Decoding media data using generative artificial intelligence

Abstract

Various embodiments provide methods, apparatuses, and computer program products. An example apparatus includes at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: encoding signaling information in or along a bitstream, wherein the signaling information enables or supports performing visual temporal extrapolation; and signaling the signaling information.

Description

Technical Field

[0001] The exemplary and non-limiting embodiments generally relate to multimedia transmission, and more specifically enable the use of generative artificial intelligence to encode, decode, or signal media data. Background Technology

[0002] Standardized formats are known to be provided for encoding, signaling, or decoding media data. Summary of the Invention

[0003] Example 1: An apparatus includes at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: encode signaling information in or along a bit stream, wherein the signaling information enables or supports the performance of visual time extrapolation; and signal the signaling information.

[0004] Example 2: An apparatus, comprising at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: receive signaling information in or along a bit stream, the signaling information enabling or supporting visual time extrapolation; decode the signaling information to obtain decoded signaling information (referred to as signaling information); and perform visual time extrapolation at least based on the decoded signaling information.

[0005] Example 3: An apparatus according to any one of Examples 1 or 2, wherein one or more neural networks (VTNN) are used to perform visual temporal extrapolation.

[0006] Example 4: The apparatus according to any one of Examples 1 to 2, wherein the signaling information is included in: a Neural Network Post-Filter Feature (NNPFC) Supplemental Enhancement Information (SEI) message; a Generative Face Video (GFV) SEI message; or a Neural Network Post-Filter Feature (NNPFC) SEI message and a GFV SEI message.

[0007] Example 5: An apparatus according to any one of Examples 1 to 4, wherein the signaling information includes one or more of the following: for the purpose of performing visual temporal extrapolation on the decoder side; or for the purpose of characterizing the temporal extrapolation characteristics (TEC) of the neural network performing the temporal extrapolation; or for the purpose of generative modeling or generative artificial intelligence (AI) on the decoder side.

[0008] Example 6: The apparatus according to any one of Examples 1 to 5, wherein the TEC information is signaled based on one or more of the following: when the signaling information indicates that the purpose of the VTNN associated with the signaling information includes visual temporal extrapolation, generative modeling, or generative artificial intelligence (AI), wherein generative modeling or generative AI refers to using the VTNN to generate data conditioned on one or more inputs; or when the purpose includes generative facial video or generative video.

[0009] Example 7: According to the apparatus of Example 6, the purpose of generating facial video includes: generating or improving at least a portion or image of a video comprising at least one or more faces, given one or more of the following: coordinates of at least one or more faces; coordinates of landmarks or key points of at least one or more faces; a transformation matrix describing the motion or temporal changes of one or more key points of at least one or more faces; facial expression features; blink indicators; text associated with the video; audio associated with the video; or one or more other images.

[0010] Example 8: An apparatus according to any one of Examples 5 to 7, wherein the TEC information includes one or more of the following: information indicating the number of images extrapolated in a single activation or inference of the VTNN; information indicating the time interval between the extrapolated images and the most recent image used as input for inference; information indicating whether the VTNN adopts auxiliary input indicating an output instance of one or more images extrapolated in time; information indicating whether the VTNN adopts auxiliary input indicating facial parameters or auxiliary input derived from facial parameters, wherein the auxiliary input is used by the VTNN to perform temporal extrapolation of one or more faces in an image or video; information indicating whether the VTNN adopts auxiliary input derived from the output of another VTNN. Information, wherein another VTNN is run or executed on the decoder side; instructing the VTNN whether to take auxiliary input information derived from the output of another VTNN, which converts or translates facial parameters into a format acceptable or required by the VTNN; instructing the VTNN whether to take auxiliary input information, which represents a tensor generated on the encoder side or derived from a tensor; instructing the VTNN whether to take auxiliary input information representing audio or speech or derived from audio or speech; when the auxiliary input represents audio or speech, information about the characteristics of the audio or speech; instructing the VTNN whether to take auxiliary input information representing text or derived from text; instructing the VTNN to perform VT under the same input conditions. Information indicating whether different inferences or executions of the VTNN produce different outputs; information indicating the type or cause of the VTNN's output variability; information indicating whether the VTNN is instructed or operated in a way that prevents it from producing output variability; information indicating whether the VTNN is allowed to have output variability; information that makes the network not have output variability; information indicating the type of extrapolated output; information indicating whether the VTNN uses auxiliary inputs, with auxiliary input indicators such as: image order count (POC) difference, timestamp difference, relative output position, or relative output difference of the output images to be generated or extrapolated; information indicating the format of the facial parameters accepted or required by the VTNN; information about the format of tensors; information indicating whether the VTNN uses auxiliary inputs. The information includes: auxiliary input, which represents text associated with human speech, with human faces extrapolated in time; information indicating whether the VTNN adopts auxiliary input representing text; information indicating whether the VTNN uses one or more of the same inputs to produce different outputs for different inferences or executions of the VTNN; information indicating whether the VTNN's input consists of background information; information indicating whether the VTNN's input consists of foreground information; information indicating whether the VTNN's input includes information driving the image; information indicating whether the VTNN's auxiliary input is derived from a lossy encoding process; and information indicating whether the VTNN accepts input that controls one or more aspects of how the final output is determined based on the VTNN's output.One or more indications used to indicate which facial parameters are present and / or which facial parameters are used by the VTNN; information indicating that the number of extrapolated images of the neural network post-filter (NNPF) varies for each inference and that the number of extrapolated images for each inference is indicated using another signaling mechanism; an indication indicating whether the decoder will generate a random seed value for the VTNN or derive a pseudo-random seed value based on other information signaled by the encoder; an indication indicating whether the pseudo-random seed value is derived based on the value of a specific syntax element included in the TEC information or otherwise signaled to the decoder; an indication indicating whether the VTNN, as specified by the TEC information, uses the seed value as input or as an auxiliary input; an indication indicating whether the VTNN, as specified by the TEC information, uses a pseudo-random seed as input or as an auxiliary input; or characteristics related to the randomness of the VTNN.

[0011] Example 9: The apparatus according to Example 8, wherein another value includes one of the following: a sequence-level quantization parameter (QP) value, wherein the pseudo-random seed value is set to be equal to the sequence-level QP value; a slice-level QP value, wherein the pseudo-random seed value is set to be equal to the slice-level QP value; a value of a syntax element in the TEC information; a value of a syntax element in a neural network post-filter feature (NNPFC) supplemental enhancement information (SEI) message specifying the VTNN; or a value of a syntax element in a generative face video (GFV) SEI message specifying the VTNN.

[0012] Example 10: The apparatus according to any one of Examples 8, wherein the facial parameters used as auxiliary input or used to derive auxiliary input are carried in one of the following: a newly defined SEI message; a generative face video (GFV) SEI message; a neural network post-filter activation (NNPFA) SEI message; or a neural network post-filter input (NNPFI) SEI message.

[0013] Example 11: The apparatus according to any one of Examples 8, wherein the tensor is the output of another neural network that is run or executed on the encoder side, or is derived from another neural network.

[0014] Example 12: The apparatus according to Example 11, wherein the tensor is signaled within an NNPFA SEI message, a GFV SEI message, or an NNPFI SEI message.

[0015] Example 13: The apparatus according to Example 8, wherein information regarding the characteristics of audio or speech includes one or more of the following: the sampling rate of the audio; the bit depth and / or dynamic range of the audio; the number of audio channels and / or channel configuration of the audio; the duration of the audio to be included in the input tensor and / or the number of audio samples; information regarding whether the audio is processed using a source separation algorithm; information regarding whether the audio is spatial audio; or the timing of the audio provided as input relative to one or more images to be extrapolated.

[0016] Example 14: The apparatus according to Example 8, wherein the type or cause of output variability includes one of the following: pseudo-randomness; deliberate variation; or a floating-point format used to represent the values ​​of one or more parameters of the VTNN and / or the values ​​of one or more signals flowing through the VTNN.

[0017] Example 15: According to the apparatus of Example 8, information indicating whether the VTNN adopts an auxiliary input indicating an output instance of one or more images extrapolated in time is carried in an NNPFC SEI message or a neural network post-filter activation (NNPFA) SEI message.

[0018] Example 16: The apparatus according to Example 8, wherein an indication VTNN is signaled to use information of an auxiliary input indicating a POC difference, timestamp difference, relative output position, or relative output difference as part of a Neural Network Post-Filter Feature (NNPFC) Supplemental Enhancement Information (SEI) message.

[0019] Example 17: The apparatus according to any one of Examples 8 or 16, wherein an auxiliary input indicating the POC difference, timestamp difference, relative output position, or relative output difference is carried in a Neural Network Post-Filter Activation (NNPFA) Supplemental Enhancement Information (SEI) message or a Neural Network Post-Filter Input (NPFI) SEI message.

[0020] Example 18: The apparatus according to any one of Examples 8 or 10, wherein the facial parameters include one or more of the following: 2D key points; 3D key points; or one or more matrices.

[0021] Example 19: The apparatus according to Example 10, wherein facial parameters obtained from NNPFA SEI messages, GFV SEI messages, or NNPFISEI messages are converted into the desired format indicated in the TEC information.

[0022] Example 20: An apparatus according to any one of Examples 8 or 13, wherein information indicating whether the VTNN uses an auxiliary input representing audio or speech or derived from audio or speech to generate or synthesize one or more features of a face.

[0023] Example 21: According to the apparatus of Example 8, when the VTNN takes auxiliary input representing text, the TEC information further includes information indicating the characteristics of the text, including one or more of the following: whether the VTNN accepts or requires the original text; whether the VTNN accepts or requires lexical units; whether the VTNN accepts or requires text embeddings; which tokenizer is used; which text embedder is used; or the timing of the text provided as input relative to one or more images to be extrapolated.

[0024] Example 22: The apparatus according to Example 8, wherein the information indicating the type of extrapolation output includes one or more categories of visual content that the VTNN is capable of extrapolating.

[0025] Example 23: The device according to Example 22, wherein one or more categories include face, hand, or body.

[0026] Example 24: The apparatus according to any one of Examples 3 to 23, wherein the signaling information further includes an indication: whether the VTNN is adapted for all images to which the VTNN is applied and is activated, or whether the VTNN is adapted at the sequence level.

[0027] Example 25: The apparatus according to Example 24, wherein the signaling information further includes: indicating the purpose of sequence-level adaptation.

[0028] Example 26: The apparatus according to Example 24 or 25, wherein a persistent auxiliary input (PAI) signal included in the signaling information is used to indicate whether the VTNN is applicable to all images to which the VTNN is applied.

[0029] Example 27: The apparatus according to Example 26, wherein the signaling information further includes: an indication of the type of PAI signal.

[0030] Example 28: The apparatus according to Example 27, wherein the type of PAI includes tensor, text, or image.

[0031] Example 29: The apparatus according to any one of Examples 3 to 23, wherein the signaling information further includes: information indicating the purpose of the neural network, which converts input facial parameters into an output representing the converted facial parameters; information indicating the purpose of the neural network, which converts input parameters into an output representing the converted parameters; or information indicating the purpose of the neural network, which converts the format of the input parameters into another format of the output parameters.

[0032] Example 30: According to the apparatus of Example 8, the Neural Network Post-Filter Input (NNPFI) SEI message is used to provide auxiliary input to a neural network post-processing filter defined by the (NNPFC) SEI message and activated by the Neural Network Post-Filter Activation (NNPFA) SEI message.

[0033] Example 31: An apparatus includes at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: infer a temporally interleaved frame packing arrangement in a bitstream; infer a constitutive frame parity, i.e., the current frame in which a temporally extrapolated filter is activated includes a temporally interleaved frame packing arrangement; select one or more input frames for the temporally extrapolated filter, the one or more input frames including the inferred constitutive frame parity; and apply the temporally extrapolated filter as input to the one or more input frames.

[0034] Example 32: A method comprising: encoding signaling information in or along a bitstream, wherein the signaling information enables or supports the performance of visual time extrapolation; and signaling the signaling information.

[0035] Example 33: A method comprising: receiving signaling information in or along a bitstream, the signaling information enabling or supporting visual temporal extrapolation; decoding the signaling information to obtain decoded signaling information (signaling information); and performing visual temporal extrapolation based at least on the decoded signaling information.

[0036] Example 34: The method according to any one of Examples 32 or 33, wherein one or more neural networks (VTNN) are used to perform visual temporal extrapolation.

[0037] Example 35: The method according to any one of Examples 32 to 33, wherein the signaling information is included in: a Neural Network Post-Filter Feature (NNPFC) Supplemental Enhancement Information (SEI) message, a Generative Face Video (GFV) SEI message, or a Neural Network Post-Filter Feature (NNPFC) SEI message and a GFV SEI message.

[0038] Example 36: The method according to any one of Examples 32 to 35, wherein the signaling information includes one or more of the following: for the purpose of performing visual temporal extrapolation on the decoder side; or temporal extrapolation characteristics (TEC) information characterizing the neural network performing the temporal extrapolation; or for the purpose of generative modeling or generative artificial intelligence (AI) on the decoder side.

[0039] Example 37: The method according to any one of Examples 32 to 36, wherein the TEC information is signaled based on one or more of the following: when the signaling information indicates that the purpose of the VTNN associated with the signaling information includes visual temporal extrapolation, generative modeling, or generative artificial intelligence (AI), wherein generative modeling or generative AI refers to using the VTNN to generate data conditioned on one or more inputs; or when the purpose includes generative facial video or generative video.

[0040] Example 38: According to the method of Example 37, the purpose of the generative facial video includes: generating or improving at least a portion or image of a video comprising at least one or more faces, given one or more of the following: coordinates of at least one or more faces; coordinates of landmarks or keypoints of at least one or more faces; a transformation matrix describing the motion or temporal changes of one or more keypoints of at least one or more faces; facial expression features; blink indicators; text associated with the video; audio associated with the video; or one or more other images.

[0041] Example 39: The method according to any one of Examples 36 to 38, wherein the TEC information includes one or more of the following: information indicating the number of images extrapolated in a single activation or inference of the VTNN; information indicating the time interval between the extrapolated images and the most recent image used as input for inference; information indicating whether the VTNN adopts auxiliary input indicating an output instance of one or more images extrapolated in time; information indicating whether the VTNN adopts auxiliary input indicating facial parameters or auxiliary input derived from facial parameters, wherein the auxiliary input is used by the VTNN to perform temporal extrapolation of one or more faces in an image or video; information indicating whether the VTNN adopts auxiliary input derived from the output of another VTNN. Information, wherein another VTNN is run or executed on the decoder side; instructing the VTNN whether to take information from auxiliary input derived from the output of another VTNN, which converts or translates facial parameters into a format accepted or required by the VTNN; instructing the VTNN whether to take information from auxiliary input, which represents a tensor generated on the encoder side or derived from a tensor; instructing the VTNN whether to take information from auxiliary input representing audio or speech or derived from audio or speech; when the auxiliary input represents audio or speech, information about the characteristics of the audio or speech; instructing the VTNN whether to take information from auxiliary input representing text or derived from text; instructing the VTNN to perform V under the same input conditions. Information indicating whether different inferences or executions of a TNN produce different outputs; information indicating the type or cause of output variability in a VTNN; information indicating whether a VTNN is instructed or operated in a way that prevents output variability; information indicating whether a VTNN is allowed to have output variability; information that makes the network unvariable in output; information indicating the type of extrapolated output; information indicating whether a VTNN uses auxiliary inputs, which indicate: the difference in picture order counts (POCs), timestamp differences, relative output positions, or relative output differences of the output images to be generated or extrapolated; information indicating the format of facial parameters accepted or required by the VTNN; information about the format of tensors; information indicating whether a VTNN uses... Information from auxiliary inputs, which represent text associated with human speech and extrapolated human faces in time; information indicating whether the VTNN adopts auxiliary input representing text; information indicating whether the VTNN uses one or more of the same inputs to produce different outputs for different inferences or executions of the VTNN; information indicating whether the VTNN's input consists of background information; information indicating whether the VTNN's input consists of foreground information; information indicating whether the VTNN's input includes information driving the image; information indicating whether the VTNN's auxiliary input is derived from a lossy encoding process; information indicating whether the VTNN accepts input that controls one or more aspects of how the final output is determined based on the VTNN's output;One or more indications used to indicate which facial parameters are present and / or which facial parameters are used by the VTNN; information indicating that the number of extrapolated images of the neural network post-filter (NNPF) varies for each inference and that the number of extrapolated images for each inference is indicated using another signaling mechanism; an indication indicating whether the decoder will generate a random seed value for the VTNN or derive a pseudo-random seed value based on other information signaled by the encoder; an indication indicating whether the pseudo-random seed value is derived based on the value of a specific syntax element included in the TEC information or otherwise signaled to the decoder; an indication indicating whether the VTNN, as specified by the TEC information, uses the seed value as input or as an auxiliary input; an indication indicating whether the VTNN, as specified by the TEC information, uses a pseudo-random seed as input or as an auxiliary input; or characteristics related to the randomness of the VTNN.

[0042] Example 40: According to the method described in Example 39, the other value includes one of the following: a sequence-level quantization parameter (QP) value, wherein the pseudo-random seed value is set to be equal to the sequence-level QP value; a slice-level QP value, wherein the pseudo-random seed value is set to be equal to the slice-level QP value; a value of a syntax element in the TEC information; a value of a syntax element in the Neural Network Post-Filter Feature (NNPFC) Supplemental Enhancement Information (SEI) message specifying the VTNN; or a value of a syntax element in the Generative Face Video (GFV) SEI message specifying the VTNN.

[0043] Example 41: According to the method of Example 39, the facial parameters used as auxiliary input or used to derive auxiliary input are carried in one of the following: a newly defined SEI message; a generative face video (GFV) SEI message; a neural network post-filter activation (NNPFA) SEI message; or a neural network post-filter input (NNPFI) SEI message.

[0044] Example 42: The method described in Example 39, wherein the tensor is the output of another neural network that is run or executed on the encoder side, or is derived from another neural network.

[0045] Example 43: The method described in Example 42, wherein the tensor is signaled within an NNPFA SEI message, a GFV SEI message, or an NNPFI SEI message.

[0046] Example 44: The method according to Example 39, wherein the information regarding the characteristics of the audio or speech includes one or more of the following: the sampling rate of the audio; the bit depth and / or dynamic range of the audio; the number of audio channels and / or channel configuration of the audio; the duration of the audio to be included in the input tensor and / or the number of audio samples; information regarding whether the audio is processed using a source separation algorithm; information regarding whether the audio is spatial audio; or the timing of the audio provided as input relative to one or more images to be extrapolated.

[0047] Example 45: The method described in Example 39, wherein the type or cause of the output variability includes one of the following: pseudo-randomness; deliberate variation; or a floating-point format used to represent the values ​​of one or more parameters of the VTNN and / or the values ​​of one or more signals flowing through the VTNN.

[0048] Example 46: According to the method of Example 39, information indicating whether the VTNN adopts auxiliary input indicating one or more output instances of images extrapolated in time is carried in an NNPFC SEI message or a neural network post-filter activation (NNPFA) SEI message.

[0049] Example 47: According to the method of Example 39, the VTNN is signaled to use information of the auxiliary input indicating the POC difference, timestamp difference, relative output position, or relative output difference as part of the neural network post-filter feature (NNPFC) supplemental enhancement information (SEI).

[0050] Example 48: The method according to any one of Examples 39 or 47, wherein an auxiliary input indicating the POC difference, timestamp difference, relative output position, or relative output difference is carried in a Neural Network Post-Filter Activation (NNPFA) Supplemental Enhancement Information (SEI) message or a Neural Network Post-Filter Input (NPFI) SEI message.

[0051] Example 49: The method of any one of Examples 39 or 41, wherein the facial parameters include one or more of the following: 2D key points; 3D key points; or one or more matrices.

[0052] Example 50: The method according to Example 41, wherein facial parameters obtained from NNPFA SEI messages, GFV SEI messages, or NNPFISEI messages are converted into the desired format indicated in the TEC information.

[0053] Example 51: The method according to any one of Examples 39 or 44, wherein information indicating whether the VTNN uses auxiliary input representing audio or speech or derived from audio or speech to generate or synthesize one or more features of the face.

[0054] Example 52: According to the method described in Example 39, when the VTNN takes auxiliary input representing text, the TEC information also includes information indicating the characteristics of the text, including one or more of the following: whether the VTNN accepts or requires the original text; whether the VTNN accepts or requires lexical units; whether the VTNN accepts or requires text embeddings; which tokenizer is used; which text embedder is used; or the timing of the text provided as input relative to one or more images to be extrapolated.

[0055] Example 53: According to the method of Example 39, the information indicating the type of extrapolation output includes one or more categories of visual content that the VTNN is able to extrapolate.

[0056] Example 54: According to the method described in Example 53, one or more categories include face, hands, or body.

[0057] Example 55: The method according to any one of Examples 34 to 54, wherein the signaling information further includes an indication: whether the VTNN is adapted for all images to which the VTNN is applied and activated, or whether the VTNN is adapted at the sequence level.

[0058] Example 56: According to the method described in Example 55, the signaling information further includes: indicating the purpose of sequence-level adaptation.

[0059] Example 57: The method described in Example 55 or 56, wherein the persistent auxiliary input (PAI) signal included in the signaling information is used to indicate whether the VTNN is applicable to all pictures to which the VTNN is applied.

[0060] Example 58: The method described in Example 57, wherein the signaling information further includes: indicating the type of PAI signal.

[0061] Example 59: The method described in Example 58, wherein the type of PAI includes tensor, text, or image.

[0062] Example 60: The method according to any one of Examples 34 to 54, wherein the signaling information further includes: information indicating the purpose of the neural network, which converts input facial parameters into an output representing the converted facial parameters; information indicating the purpose of the neural network, which converts input parameters into an output representing the converted parameters; or information indicating the purpose of the neural network, which converts the format of the input parameters into another format of the output parameters.

[0063] Example 61: According to the method described in Example 39, the Neural Network Post-Filter Input (NNPFI) SEI message is used to provide auxiliary input to a neural network post-processing filter defined by the (NNPFC) SEI message and activated by the Neural Network Post-Filter Activation (NNPFA) SEI message.

[0064] Example 62: A method comprising: inferring a time-interleaved frame packing arrangement in a bitstream; inferring a constitutive frame parity, i.e., the current frame in which a time extrapolation filter is activated includes the time-interleaved frame packing arrangement; selecting one or more input frames for the time extrapolation filter, the one or more input frames including the inferred constitutive frame parity; and applying the time extrapolation filter to the one or more input frames as input.

[0065] Example 63: An apparatus comprising components for performing the method according to any one of Examples 32 to 62.

[0066] Example 64: A computer-readable medium including program instructions that, when executed by a device, cause the device to perform the method according to any one of Examples 32 to 62.

[0067] Example 65: A computer-readable medium according to Example 64, wherein the computer-readable medium includes a non-transitory computer-readable medium. Attached Figure Description

[0068] The above embodiments and other features are explained in the following description taken in conjunction with the accompanying drawings, in which:

[0069] Figure 1 An electronic device employing an embodiment of the examples described herein is illustrated schematically.

[0070] Figure 2 A user equipment suitable for employing an embodiment of the examples described herein is illustrated schematically.

[0071] Figure 3 An electronic device employing an embodiment of the examples described herein is further illustrated schematically using wireless and wired network connections.

[0072] Figure 4 A block diagram showing the general structure of a video encoder is provided.

[0073] Figure 5 This is an example device that can be implemented in hardware and is made to implement the examples described herein.

[0074] Figure 6 A representation of an example of a non-volatile storage medium for storing instructions that implement the examples described herein is shown.

[0075] Figure 7 This is an example method for implementing the embodiments described herein, according to another embodiment.

[0076] Figure 8 This is another example method for implementing the embodiments described herein, according to another embodiment.

[0077] Figure 9 This is another example method for implementing the embodiments described herein, according to another embodiment.

[0078] Figure 10 This is a block diagram of a possible non-limiting system in which example embodiments can be practiced. Detailed Implementation

[0079] The following abbreviations and acronyms that may appear in the instruction manual and / or drawings are defined as follows: 4CC: Four-character code 5G: Fifth Generation Cellular Network Technology 5GC: 5G Core Network aka: also known as AVC: Advanced Video Coding CU: Central Unit DSP: Digital Signal Processor DU: Distributed Unit eNB (or eNodeB): Evolved Node B (e.g., LTE base station) EN-DC: E-UTRA-NR Dual Connection en-gNB or En-gNB: A node that provides NR user plane and control plane protocol termination to the UE and acts as a secondary node in the EN-DC. E-UTRA: Evolved Universal Terrestrial Radio Access, such as LTE radio access technology F1 or F1-C: Interface between CU and DU control interfaces gNB (or gNodeB): A 5G / NR base station, for example, a node that provides NR user plane and control plane protocol termination to the UE and connects to the 5GC via the NG interface. IEC: International Electrotechnical Commission IoT: Internet of Things ISO: International Organization for Standardization ISOBMFF: ISO Basic Media File Format JPEG: Joint Photo Experts Group LTE: Long Term Evolution mdat:MediaDataBox MIME: Multipurpose Internet Mail Extensions MME: Mobility Management Entity moov: MovieBox MP4: The file format of MPEG-4 Part 14 files. MPEG: Moving Pictures Experts Group MPEG-2: H.222 / H.262 as defined by the ITU. MPEG-4: The audio and video coding standard of ISO / IEC 14496 ng or NG: the new generation ng-eNB or NG-eNB: Next-generation eNB NR: New Wireless (5G Wireless) N / W or NW: Network PDCP: Packet Data Convergence Protocol PHY: Physical Layer PNG: Portable Network Graphics RAN: Radio Access Network RFC: Comment Request RLC: Radio Link Control RRC: Radio Resource Control RRH: Remote Wireless Header RU: Wireless Unit Rx: Receiver SDAP: Service Data Adaptation Protocol SGW: Service Gateway SMF: Session Management Function SPS: Sequence Parameter Set SVC: Scalable Video Codec S1: Interface between eNodeB and EPC trak: TrackBox Tx: Transmitter UE: User Equipment UICC: General Purpose Integrated Circuit Card UPF: User-Face Functionality URL: Uniform Resource Locator X2: Interconnection interface between two eNodeBs in an LTE network Xn: Interface between two NG-RAN nodes

[0080] Some embodiments will now be described more fully with reference to the accompanying drawings, which illustrate some, but not all, of the embodiments. In fact, various embodiments of the invention may be embodied in many different forms and should not be construed as limited to the embodiments described herein; rather, these embodiments are provided so that this disclosure will meet applicable legal requirements. The same reference numerals throughout refer to the same elements. As used herein, the terms 'data,' 'content,' 'information,' and similar terms are used interchangeably to refer to data that can be transmitted, received, and / or stored according to embodiments of the invention. Therefore, the use of any such terms should not be considered as limiting the spirit and scope of the embodiments.

[0081] Furthermore, as used herein, the term 'circuit system' refers to (a) a purely hardware circuit implementation (e.g., an implementation in an analog circuit and / or digital circuit system); (b) a combination of circuitry and (multiple) computer program products, including software and / or firmware instructions stored on one or more computer-readable storage media, which cooperate to cause a device to perform one or more functions described herein; and (c) a circuit, such as (multiple) microprocessors or a portion thereof, which requires software or firmware to function even if the software or firmware is not physically present. This definition of 'circuit system' applies to all uses of the term herein, including in any claim. As another example, as used herein, the term 'circuit system' also includes an implementation comprising one or more processors and / or portions thereof, along with accompanying software and / or firmware. As yet another example, for instance, the term 'circuit system' as used herein also includes, for example, baseband integrated circuits or application processor integrated circuits for mobile phones, or similar integrated circuits in servers, cellular network devices, other network devices, and / or other computing devices.

[0082] As defined herein, a 'computer-readable storage medium' (which refers to a non-transitory physical storage medium (e.g., a volatile or non-volatile storage device)) can be distinguished from a 'computer-readable transmission medium' (which refers to an electromagnetic signal).

[0083] According to an example embodiment, a method, apparatus, and computer program product are provided for enabling the encoding, decoding, or signaling of media data using generative artificial intelligence.

[0084] In one example, suitable apparatus and possible mechanisms for enabling the encoding, decoding, or signaling of media data using generative artificial intelligence are described in detail below. In this regard, reference is made first to… Figure 1 and Figure 2 ,in Figure 1An example block diagram of device 50 is shown. This device can be an Internet of Things (IoT) device configured to perform various functions, such as collecting, receiving, or transmitting information via one or more sensors, analyzing information collected or received by the device, and so on. The device may include a video encoding system that can incorporate a codec. Figure 2 The layout of the device according to an example embodiment is shown. The following will explain... Figure 1 and Figure 2 Components.

[0085] Device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag, or a low-power device. However, it should be understood that the exemplary embodiments described herein can be implemented in any electronic device or apparatus that can process data via a neural network.

[0086] Device 50 may include a housing 30 for accommodating and protecting the device. Device 50 may also include a display 32, such as a liquid crystal display, a light-emitting diode display, an organic light-emitting diode display, or the like. In other embodiments of the examples described herein, the display may be any suitable display technology suitable for displaying media or multimedia content (e.g., images or videos). Device 50 may also include a keypad 34. In other embodiments of the examples described herein, any suitable data or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or data input system as part of a touch-sensitive display.

[0087] The device may include a microphone 36 or any suitable audio input, which may be a digital or analog signal input. The device 50 may also include an audio output device, which in the example embodiments described herein may be a headset 38, a speaker, or any analog or digital audio output connection. The device 50 may also include a battery (or, in other example embodiments described herein, the device may be powered by any suitable mobile energy device, such as a solar cell, fuel cell, or clockwork generator). The device may also include a camera 42 capable of recording or capturing images and / or video. The device 50 may also include an infrared port for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may also include any suitable short-range communication technology, such as a Bluetooth wireless connection or a USB / FireWire wired connection.

[0088] The device 50 may include a controller 56, a processor, or a processor circuitry for controlling the device 50. The controller 56 may be connected to a memory 58, which, in the example embodiments described herein, may store data in the form of image, audio, and video data, and / or may also store instructions for implementation on the controller 56. The controller 56 may also be connected to a codec circuitry 54 adapted to perform encoding and / or decoding of audio, image, and / or video data, or to assist in encoding and / or decoding performed by the controller.

[0089] The device 50 may also include a card reader 48 and a smart card 46, such as a Universal Integrated Circuit Card (UICC) and a UICC reader, for providing user information and suitable for providing authentication information for authenticating and authorizing the user at the network.

[0090] Device 50 may include a wireless interface circuitry system 52 connected to a controller and adapted to generate wireless communication signals, such as communication with a cellular communication network, a wireless communication system, or a wireless local area network. Device 50 may also include an antenna 44 connected to the wireless interface circuitry system 52 for transmitting and / or receiving radio frequency signals generated at the wireless interface circuitry system 52 to and / or from other devices.

[0091] The device 50 may include a camera 42 capable of recording or detecting individual frames, which are then passed to a codec circuitry system 54 or a controller for processing. The device may receive video image data for processing from another device before transmission and / or storage. The device 50 may also receive images wirelessly or via a wired connection for encoding / decoding. The structural elements of the device 50 described above represent examples of components used to perform the corresponding functions.

[0092] about Figure 3 This document illustrates an example of a system in which embodiments of the examples described herein can be utilized. System 10 includes multiple communication devices that can communicate over one or more networks. System 10 may include any combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, CDMA, LTE, 4G, 5G networks, etc.), wireless local area networks (WLANs) such as those defined by the IEEE 802.x standard, Bluetooth personal area networks, Ethernet LANs, token ring LANs, wide area networks, and the Internet.

[0093] System 10 may include wired and wireless communication devices and / or apparatus 50 suitable for implementing embodiments of the examples described herein.

[0094] For example, Figure 3The system shown illustrates a representation of mobile phone network 11 and Internet 28. Connection to Internet 28 can include, but is not limited to, long-range wireless connections, short-range wireless connections, and various wired connections, including but not limited to telephone lines, power lines, and similar communication paths.

[0095] The example communication device shown in System 10 may include, but is not limited to: electronic device or apparatus 50, a combination of personal digital assistant (PDA) and mobile phone 14, PDA 16, integrated messaging device (IMD) 18, desktop computer 20, laptop computer 22, or head-mounted device 21 (which may be a head-mounted display (HMD), or glasses with a camera or other device for processing images and / or video). Apparatus 50 may be stationary or mobile (when carried by a moving individual). Apparatus 50 may also be located in a vehicle, including but not limited to cars, trucks, taxis, buses, trains, ships, airplanes, bicycles, motorcycles, or any similar suitable means of transport.

[0096] Implementations may also be: implemented in a set-top box (e.g., a digital television receiver) which may or may not have a display or wireless capability; implemented in a tablet or (laptop) personal computer (PC) with hardware and / or software for processing neural network data; implemented in various operating systems; and implemented in chipsets, processors, DSPs, and / or embedded systems that provide hardware / software-based coding.

[0097] Some or additional devices can send and receive calls and messages, and communicate with service providers via a wireless connection 25 to base station 24. Base station 24 can be connected to a network server 26, which allows communication between mobile phone network 11 and the Internet 28. The system may include additional communication equipment and various types of communication devices.

[0098] Communication devices can communicate using various transmission technologies, including but not limited to Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), Time Division Multiplexing (TDMA), Frequency Division Multiplexing (FDMA), Transmission Control Protocol / Internet Protocol (TCP-IP), Short Message Service (SMS), Multimedia Messaging Service (MMS), Email, Instant Messaging Service (IMS), Bluetooth, IEEE 802.11, 3GPP Narrowband IoT, and any similar wireless communication technologies. Communication devices implementing various embodiments of the examples described herein can communicate using various media, including but not limited to wireless, infrared, laser, cable connections, and any suitable connection.

[0099] In telecommunications and data networks, a channel can refer to a physical channel or a logical channel. A physical channel can refer to a physical transmission medium such as a wired connection, while a logical channel can refer to a logical connection on a multiplexed medium capable of transmitting several logical channels. A channel can be used to transmit information signals (e.g., bit streams) from one or more transmitters (or transmitters) to one or more receivers.

[0100] These embodiments can also be implemented in so-called IoT devices. For example, the Internet of Things (IoT) can be defined as the interconnection of uniquely identifiable embedded computing devices within existing internet infrastructure. The convergence of various technologies has enabled and can enable many embedded systems domains, such as wireless sensor networks, control systems, home / building automation, etc., to be incorporated into the IoT. To utilize the internet, IoT devices are provided with IP addresses as unique identifiers. IoT devices can be provided with wireless transmitters, such as WLAN or Bluetooth transmitters, or RFID tags. Alternatively, IoT devices can access IP-based networks via wired networks, such as Ethernet-based networks or power line connections (PLC).

[0101] Figure 4 A block diagram showing the general structure of a video encoder is provided. Figure 4 An encoder for two layers is presented, but it should be understood that the encoder presented can be similarly extended to encode more than two layers. Figure 4 The illustration shows a video encoder including a first encoder portion 401 for a base layer and a second encoder portion 451 for an enhancement layer. Each of the first encoder portion 401 and the second encoder portion 451 may include similar elements for encoding an incoming image. Encoder portions 401 and 451 may include pixel predictors 402 and 452, prediction error encoders 403 and 453, and prediction error decoders 404 and 454. Figure 4An embodiment of pixel predictors 402, 452 is also shown, which includes inter-frame predictors 406, 456, intra-frame predictors 408, 458, mode selectors 410, 460, filters 416, 466, and reference frame memories 418, 468. Pixel predictor 402 of the first encoder section 401 receives multiple base layer pictures / images of a video stream to be encoded at both the inter-frame predictor 406 (which determines the difference between the image and the motion-compensated reference frame) and the intra-frame predictor 408 (which determines the prediction of image patches based only on the processed portion of the current frame or picture). The outputs of both the inter-frame and intra-frame predictors are passed to mode selector 410. Intra-frame predictor 408 may have more than one intra-frame prediction mode. Thus, each mode can perform intra-frame prediction and provide a prediction signal to mode selector 410. Mode selector 410 also receives copies of multiple base layer images 400. Accordingly, the pixel predictor 452 of the second encoder section 451 receives multiple enhancement layer pictures / images 450 of the video stream to be encoded at the inter-frame predictor 456 (which determines the difference between the image and the motion-compensated reference frame) and the intra-frame predictor 458 (which determines the prediction of image patches based only on the processed portion of the current frame or picture). The outputs of both the inter-frame and intra-frame predictors are passed to the mode selector 460. The intra-frame predictor 458 may have more than one intra-frame prediction mode. Therefore, each mode can perform intra-frame prediction and provide the prediction signal to the mode selector 460. The mode selector 460 also receives a copy of the enhancement layer picture 450.

[0102] Depending on which encoding mode is selected to encode the current block, the outputs of inter-frame predictors 406 and 456, or the output of one of the optional intra-frame predictor modes, or the output of the surface encoder within the mode selector, are passed to the outputs of mode selectors 410 and 460. The outputs of mode selectors 410 and 460 are then passed to first summing devices 421 and 471. The first summing devices can subtract the outputs of pixel predictors 402 and 452 from the base layer image(s) 400 and enhancement layer image(s) 450 to generate first prediction error signals 420 and 470, which are input to prediction error encoders 403 and 453.

[0103] Pixel predictors 402 and 452 also receive a combination of the predicted representations of image blocks 412 and 462 and the outputs 438 and 488 of the prediction error decoders 404 and 454 from the preliminary reconstructors 439 and 489. The preliminary reconstructed images 414 and 464 can be passed to intra-frame predictors 408 and 458 and filters 416 and 466. Filters 416 and 466, receiving the preliminary representations, can filter the preliminary representations and output final reconstructed images 440 and 490, which can be stored in reference frame memories 418 and 468. Reference frame memory 418 can be connected to inter-frame predictor 406 to serve as a reference image for comparison with the future base layer image 400 during inter-frame prediction operations. According to some embodiments, when a base layer is selected and designated as the source for inter-layer sample prediction and / or inter-layer motion information prediction of the enhancement layer, a reference frame memory 418 may also be connected to an inter-frame predictor 456 to serve as a reference image for comparison with (multiple) future enhancement layer images 450 during inter-frame prediction operations. Furthermore, a reference frame memory 468 may be connected to an inter-frame predictor 456 to serve as a reference image for comparison with (multiple) future enhancement layer images 450 during inter-frame prediction operations.

[0104] According to some embodiments, the filtering parameters of the filter 416 from the first encoder section 401 can be provided to the second encoder section 451, provided that the base layer is selected and indicated as the source for predicting the filtering parameters of the enhancement layer.

[0105] The prediction error encoders 403 and 453 include transform units 442 and 492 and quantizers 444 and 494. Transform units 442 and 492 transform the first prediction error signals 420 and 470 to the transform domain. For example, this transform is a DCT transform. Quantizers 444 and 494 quantize the transform domain signals (e.g., DCT coefficients) to form quantized coefficients.

[0106] Prediction error decoders 404 and 454 receive outputs from prediction error encoders 403 and 453 and perform the reverse processing of the prediction error encoders 404 and 453 to generate decoded prediction error signals 438 and 488. These signals, when combined with the prediction representations of image blocks 412 and 462 at second summing devices 439 and 489, produce preliminary reconstructed images 414 and 464. The prediction error decoders can be considered to include dequantizers 446 and 496 and inverse transform units 448 and 498. The dequantizers dequantize quantized coefficient values ​​(e.g., DCT coefficients) to reconstruct the transform signal, and the inverse transform units perform an inverse transform on the reconstructed transform signal. The outputs of the inverse transform units 448 and 498 include multiple reconstructed blocks. The prediction error decoders may also include block filters that can filter the multiple reconstructed blocks based on further decoded information and filter parameters.

[0107] Entropy encoders 430 and 480 receive the outputs of prediction error encoders 403 and 453 and can perform appropriate entropy coding / variable length coding on the signal to provide a compressed signal. The outputs of entropy encoders 430 and 480 can be inserted into the bitstream, for example, via multiplexer 465.

[0108] The Advanced Video Coding Standard (which can be abbreviated as H.264, AVC, or H.264 / AVC) was developed by the Joint Video Team (JVT) of the Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard was published by its two parent standardization organizations as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC). The H.264 / AVC standard has several versions, each integrating new extensions or features into the specification. These extensions include Scalable Video Coding (SVC) and Multi-View Video Coding (MVC).

[0109] The High Efficiency Video Coding Standard (which may be abbreviated as H.265, HEVC, or H.265 / HEVC) was developed by the Joint Collaboration Team on Video Coding (JCT-VC) of VCEG and MPEG. This standard was published by two higher-level standardization organizations as the ITU-T H.265 Recommendation and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Extensions to H.265 / HEVC include Scalable, Multi-View, 3D, and Fidelity Range extensions, which may be referred to as SHVC, MV-HEVC, 3D-HEVC, and REXT, respectively. Unless otherwise stated, H.265 / HEVC, SHVC, MV-HEVC, 3D-HEVC, and REXT referenced in this specification for the purpose of understanding the definitions, structure, or concepts of these standard specifications should be understood as the latest versions of these standards available prior to the date of reference.

[0110] Universal Video Coding (which can be abbreviated as VVC, H.266, or H.266 / VVC) is a video compression standard developed as a successor to HEVC. VVC is specified in the ITU-T H.266 Recommendation and is equivalently specified in ISO / IEC 23090-3, which is also known as MPEG-I Part 3.

[0111] The Alliance for Open Media (AOM) developed the specifications for the AV1 bitstream format and decoding process. The AV1 specification was released in 2018. It is reported that AOM is currently specifying the AV2 specification.

[0112] In most cases, the basic units used as input to the encoder and output to the decoder are images. The image used as input to the encoder can also be called the source image, and the image decoded by the decoder can be called the decoded image or the reconstructed image.

[0113] The source image and the decoded image each consist of one or more sample arrays, such as one of the following sample arrays: - Luminance (Y) only (monochrome). - Brightness and two chromaticities (YCbCr or YCgCo). - Green, blue, and red (GBR, also known as RGB). - Indicates an array of other unspecified monochromatic or tristimulus color samples (e.g., YZX, also known as XYZ).

[0114] In the following text, these arrays may be referred to as luminance (or L or Y) and chrominance, where the two chrominance arrays may be referred to as Cb and Cr or Cg and Co; regardless of the actual color representation used. For example, the actual color representation used may be indicated in the encoded bitstream using Video Availability Information (VUI) syntax. A component may be defined as an array or a single sample from one of the three sample arrays (luminance and two chrominance), or an array or a single sample of an array that makes up an image in monochrome format.

[0115] An image can be defined as a frame or a field. A frame consists of a matrix of luminance samples and possible corresponding chrominance samples. A field is a set of alternating sample rows of frames and can be used as encoder input when the source signals are interleaved. The chrominance sample array may not exist (and therefore monochrome sampling can be used), or the chrominance sample array may be subsampled compared to the luminance sample array.

[0116] Some chroma formats can be summarized as follows: - In monochrome sampling, there is only one sample array, which can be nominally considered as a brightness array. In 4:2:0 sampling, each of the two chroma arrays has half the height and width of the luminance array. In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width of the luminance array. - In 4:4:4 sampling, when no separate color plane is used, each of the two chroma arrays has the same height and width as the luminance array.

[0117] Encoding formats or standards can allow sample arrays to be encoded as individual color planes into a bitstream, and the individually encoded color planes to be decoded from the bitstream accordingly. When using individual color planes, each of them is processed separately (by the encoder and / or decoder) as a monochrome sampled image.

[0118] A video codec consists of an encoder that transforms the input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a visual form. Typically, the encoder discards some information from the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate).

[0119] Typical hybrid video codecs (e.g., ITU-T H.263 and H.264) encode video information in two stages. First, pixel values ​​in a given picture region (or “block”) are predicted, for example, through motion compensation methods (finding and indicating a region in a previously encoded video frame that closely corresponds to the block being encoded) or spatial methods (using pixel values ​​around the block to be encoded in a specified manner). Second, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This is typically achieved by transforming the difference in pixel values ​​using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant thereof), quantizing the coefficients, and entropy encoding the quantization coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (picture quality) and the size of the final encoded video representation (file size or transmission bit rate).

[0120] Inter-frame prediction (also known as temporal prediction, motion compensation, or motion-compensated prediction) utilizes temporal redundancy. In inter-frame prediction, the prediction source is the previously decoded image (also known as the reference image).

[0121] In temporal inter-frame prediction, the prediction source is a previously decoded image within the same scalable layer. In intra-block copy (IBC) (also known as intra-block copy prediction), prediction can be applied similarly to temporal inter-frame prediction, but the reference image is the current image, and only previously decoded samples can be referenced during the prediction process. Inter-layer or inter-view prediction can be applied similarly to temporal inter-frame prediction, but the reference images are decoded images from a different scalable layer or a different view. In some cases, inter-frame prediction may refer only to temporal inter-frame prediction, while in others, inter-frame prediction can be collectively referred to as temporal inter-frame prediction, as well as any of intra-block copy, inter-layer, and inter-view predictions, provided these predictions are performed using the same or similar procedures as temporal prediction. Inter-frame prediction, temporal inter-frame prediction, or temporal prediction may sometimes be referred to as motion compensation or motion compensation prediction.

[0122] Intra-frame prediction leverages the fact that adjacent pixels within the same image may be correlated. Intra-frame prediction can be performed in the spatial domain or the transform domain; that is, it can predict sample values ​​or transform coefficients. Intra-frame prediction is typically used in intra-frame coding, where inter-frame prediction is not applied.

[0123] One outcome of the encoding process is a set of encoding parameters, such as motion vectors and quantization transform coefficients. These parameters can be entropy-coded more efficiently when many parameters are first predicted from spatially or temporally adjacent parameters. For example, motion vectors can be predicted based on spatially adjacent motion vectors, and only the differences relative to the predicted motion vector values ​​can be encoded. The prediction of encoding parameters and intra-frame prediction can be collectively referred to as intra-picture prediction.

[0124] The decoder reconstructs the output video by applying an encoder-like prediction method to form a predictive representation of pixel blocks (using motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (the inverse operation of prediction error encoding, used to recover the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding methods, the decoder adds the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) can also apply additional filtering methods to improve the quality of the output video before sending it for display and / or storing it as a prediction reference for upcoming frames in a video sequence.

[0125] Image and video codecs can use a set of filters to improve the visual quality of the predicted visual content. Filters can be applied in-loop, out-of-loop, or both. In-loop filters (also known as loop filters) are used to reconstruct a prediction reference, which can be used to predict the upcoming video signal. In other words, in the case of in-loop filters, a filter applied to a block in the currently encoded frame may affect the encoding of another block in the same frame and / or another frame predicted based on the current frame. In-loop filters can affect bitrate and / or visual quality. In fact, enhancing a block may result in a smaller residual (the difference between the original block and the predicted and filtered block), thus requiring fewer bits to encode. After a frame or part of a frame has been reconstructed, out-of-loop filters (also known as post-processing filters or post-filters) can be applied to it. The filtered visual content may not be used as a source for prediction, and therefore it may only affect the visual quality of the frame output by the decoder.

[0126] In a typical video codec, motion information is indicated by motion vectors associated with each motion-compensated image block. Each of these motion vectors represents the displacement of an image block in the picture to be encoded (on the encoder side) or decoded (on the decoder side) and a predicted source block in one of the previously encoded or decoded images.

[0127] To efficiently represent motion vectors, motion vectors are typically differentially encoded relative to block-specific predicted motion vectors. In typical video codecs, predicted motion vectors are created in a predefined manner, such as by calculating the median of the encoded or decoded motion vectors of adjacent blocks.

[0128] Another method for creating motion vector predictions is to generate a list of candidate predictions from neighboring and / or co-located blocks in a time reference image, and to signal the selected candidates as motion vector prediction values. In addition to predicting motion vector values, reference indices from previously encoded / decoded images can also be predicted. Reference indices are typically predicted based on neighboring and / or co-located blocks in the time reference image.

[0129] Furthermore, typical high-efficiency video codecs employ an additional motion information encoding / decoding mechanism, often referred to as a merge / fusion mode, where all motion field information (including motion vectors and corresponding reference image indices for each available list of reference images) is predicted and used without any modification / correction. Similarly, motion field information prediction is performed using motion field information from adjacent and / or co-located blocks in the temporal reference image, and the motion field information used is signaled in a motion field candidate list populated by the motion field information from available adjacent / co-located blocks.

[0130] In a typical video codec, the motion-compensated prediction residuals are first transformed using a transform kernel (such as DCT) before encoding. This is done because some correlations often remain in the residuals, and in many cases, the transform helps reduce these correlations and provides more efficient encoding.

[0131] Typical video encoders utilize a Lagrange cost function to find the optimal coding pattern, such as the optimal macroblock pattern and associated motion vectors. This cost function uses a weighting factor λ to correlate the (exact or estimated) image distortion caused by lossy coding methods with the (exact or estimated) amount of information needed to represent pixel values ​​in image regions: Where C is the Lagrangian cost to be minimized, D is the image distortion (e.g., mean squared error) considering pattern and motion vectors, and R is the number of bits required to reconstruct the image patch in the decoder (including the amount of data used to represent candidate motion vectors).

[0132] Out-of-band transmission, signaling, or storage can refer to the ability to send, signal, or store information in a manner that associates information with a video bitstream. Out-of-band transmission can use more reliable transmission mechanisms compared to protocols used to carry encoded video data, such as slices. Out-of-band transmission, signaling, or storage can be used alternatively or separately, for example, to facilitate access or session negotiation. For example, sample entries for tracks in a file conforming to the ISO Basic Media File Format may include parameter sets, while the encoded data in the bitstream is stored elsewhere in the file or in another file. Another example of out-of-band transmission, signaling, or storage includes including information such as NN and / or NN updates in a file format track separate from the track(s) containing encoded video data.

[0133] In the embodiments described in the claims, the phrase "along the bitstream" (e.g., indicating along the bitstream) or "along the encoding unit" (e.g., indicating along the encoding tile) can be used to refer to transmissions, signaling, or storage that are associated with, but not included in, the bitstream or encoding unit, respectively, as 'out-of-band' data. Decoding of the phrase "along the bitstream" or "along the encoding unit" can refer to decoding the referenced out-of-band data associated with the bitstream or encoding unit (which may be obtained from out-of-band transmission, signaling, or storage). For example, the phrase "along the bitstream" can be used when the bitstream is included in a container file, such as a file conforming to the ISO Basic Media File Format, and some file metadata is stored in the file in a manner that associates metadata with the bitstream, such as boxes in sample entries for tracks that include the bitstream, sample groups for tracks that include the bitstream, or timing metadata tracks associated with tracks that include the bitstream. In another example, the phrase "along the bitstream" can be used when the bitstream is available as a stream over a communication protocol and a media description, such as a stream list, is provided to describe the stream.

[0134] A bitstream can be defined as a sequence of bits or a sequence of syntax structures. The bitstream format can constrain the order of the syntax structures in the bitstream.

[0135] Syntax elements can be defined as data elements represented in a bitstream. Syntax structures can be defined as zero or more syntax elements that appear together in a bitstream in a specified order.

[0136] For example, syntax constructs can be specified using arithmetic, logical, relational, bitwise, and assignment operators similar to those available in many programming languages. For instance, '&' can indicate a bitwise AND operation. Furthermore, syntax constructs can be specified by referencing mathematical functions.

[0137] Syntactic structures and semantics can use the values ​​of variables derived from the values ​​of syntactic elements. Naming conventions can be defined for variables. For example, variables can be named using a mix of lowercase and uppercase letters, without any underscore characters. Variables beginning with an uppercase letter can be derived for decoding the current syntactic structure and all dependent syntactic structures. In some cases, variables beginning with an uppercase letter can be used in subsequent syntactic structures during decoding without mentioning the variable's original syntactic structure. Variables beginning with a lowercase letter can only be used within syntactic structures or functions where they have already been defined.

[0138] The basic units used for the output of a video encoder and the input of a video decoder can be Network Abstraction Layer (NAL) units. For transmission over packet-oriented networks or storage in structured files, NAL units can be encapsulated within packets or similar structures. Byte stream formats encapsulating NAL units can be used in transmission or storage environments that do not provide frame structures. Byte stream formats can separate NAL units by appending start codes to each NAL unit. To avoid erroneous detection of NAL unit boundaries, the encoder can run a byte-oriented start code anti-raceback algorithm, which adds anti-raceback bytes to the NAL unit payload if a start code would otherwise be present. To enable direct gateway operation between packet-oriented and stream-oriented systems, start code emulation prevention can be performed regardless of whether a byte stream format is used. A NAL unit can be defined as a syntax structure that includes an indication of the data type to be followed, and bytes that include that data in the form of a raw byte sequence payload, interspersed with anti-raceback bytes where necessary. A raw byte sequence payload (RBSP) can be defined as a syntax structure that includes an integer number of bytes encapsulated within the NAL unit. RBSP can be empty or in the form of a data bit string, including syntax elements, followed by RBSP stop bits, followed by zero or more subsequent bits equal to 0.

[0139] A bitstream can be defined as logically including syntactic structures, such as NAL units, when the syntactic structure is sent along the bitstream but can be included within the bitstream according to the bitstream format. A bitstream can also be defined as natively including syntactic structures when it includes them.

[0140] In some encoding formats or standards, a bitstream can be in the form of a Network Abstraction Layer (NAL) unit stream or a byte stream, which forms a representation of encoded images and associated data to form one or more encoded video sequences.

[0141] In some formats or standards, a first bitstream may be followed by a second bitstream in the same logical channel, such as within the same file or on the same connection of a communication protocol. A primary stream (in the context of video coding) can be defined as a sequence of one or more bitstreams.

[0142] In some encoding formats, such as AV1, the bitstream may include a sequence of Open Bit Stream Units (OBUs). An OBU includes a header and a payload, where the header identifies the type of OBU. Furthermore, the header may include the size of the payload in bytes.

[0143] In some coding standards, a NAL unit includes a header and a payload. The NAL unit header indicates the type of NAL unit. In some coding standards, the NAL unit header indicates a scalability layer identifier (e.g., nuh_layer_id in H.265 / HEVC and H.266 / VVC), which can be used to indicate, for example, a spatial or quality layer, a view or auxiliary layer (such as a depth map or alpha plane) in multi-view video. In some coding standards, the NAL unit header includes a temporal sublayer identifier, which can be used to indicate a temporal subset of the bitstream, such as a 30-frame-per-second subset of a 60-frame-per-second bitstream.

[0144] Bitstreams or encoded video sequences can be encoded to be temporally scalable, as shown below. Each picture can be assigned to a specific temporal sublayer. Temporal sublayers can be equivalently referred to as sub-layers, temporal sublayers, or temporal levels. Temporal sublayers can be enumerated, for example, starting from 0. The lowest temporal sublayer (sublayer 0) can be decoded independently. The picture at temporal sublayer 1 can be predicted based on the reconstructed pictures at temporal sublayers 0 and 1. The picture at temporal sublayer 2 can be predicted based on the reconstructed pictures at temporal sublayers 0, 1, and 2, and so on. In other words, the picture at temporal sublayer N does not use any pictures at temporal sublayers greater than N as a reference for inter-frame prediction. A bitstream created by excluding all pictures greater than or equal to the selected sublayer value and including the pictures is still compliant with the specification.

[0145] Each picture in a time-scalable bitstream can be assigned a time identifier (also known as a time layer identifier, time sublayer identifier, or time layer ID), for example, this identifier can be assigned to the variable TemporalId. The time identifier can be indicated in the NAL unit header or the OBU extended header, for example. A TemporalId equal to 0 corresponds to the lowest time level. A bitstream created by excluding all encoded pictures with a TemporalId greater than or equal to the selected value and including all other encoded pictures is still compliant with the specification. Therefore, a picture with a TemporalId equal to tid_value does not use any picture with a TemporalId greater than tid_value as a prediction reference.

[0146] NAL units can be divided into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are typically encoded slice NAL units.

[0147] For example, a non-VCL NAL unit can be one of the following types: Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Adaptive Parameter Set (APS), Supplemental Enhancement Information (SEI) NAL unit, Access Unit Delimiter, Sequence End NAL unit, Bitstream End NAL unit, or Padding Data NAL unit. Reconstructing a decoded image may require a parameter set, while many other non-VCL NAL units are not necessary for reconstructing decoded sample values.

[0148] Video coding standards enable the use of Supplemental Enhancement Information (SEI) messages. Some video coding standards include SEI NAL units, while others include both prefix SEI NAL units and suffix SEI NAL units, where the former can begin a picture unit (PU), and the latter can end a picture unit. An SEI NAL unit contains one or more SEI messages that are not essential for decoding the output picture but can assist in related processes such as picture output timing, post-processing of decoded pictures, rendering, error detection, error hiding, and resource reservation. Several SEI messages are specified in the H.264 / AVC, H.265 / HEVC, H.266 / VVC, and H.274 / VSEI standards, and user data SEI messages allow organizations and companies to specify SEI messages for their own use. These standards may include the syntax and semantics for specifying SEI messages, but may not define the procedures for processing the messages in the receiver. Therefore, encoders may need to follow a standard for specifying SEI messages when creating (multiple) SEI messages, while decoders may not need to process SEI messages to maintain output order consistency. One reason for including the syntax and semantics of SEI messages in the standard is to allow different system specifications to interpret supplementary information in the same way, thus enabling interoperability. System specifications can require the use of specific SEI messages at both the encoder and decoder ends, and can also specify the procedure for processing specific SEI messages at the receiver.

[0149] Some video coding standards enable the metadata OBU. The metadata OBU includes a type field that specifies the type of metadata.

[0150] In some encoding formats, a coded video sequence (CVS) can be defined as a sequence of coded images in decoding order, which is independently decodeable and followed by the end of another coded video sequence or bitstream.

[0151] In some encoding formats (such as AV1), the encoded video sequence comprises one or more time units. A time unit consists of a series of OBUs, beginning with a time delimiter, an optional sequence header, an optional metadata OBU, and a sequence of one or more frame headers (each frame header followed by zero or more tile group OBUs and optional padding OBUs). A time unit can be defined to include all OBUs associated with a specific, distinct moment. A time unit may include the time delimiter OBU and all subsequent OBUs up to but not including the next time delimiter. The time delimiter OBU can be defined as an indication that subsequent OBUs will have a different rendering / decoding timestamp than the last frame preceding the time delimiter.

[0152] A Coding Layer Video Sequence (CLVS) can be defined as a sequence of images and other associated data within the same scalable layer (e.g., having the same nuh_layer_id value in VVC) that can be decoded independently of other images in the same layer.

[0153] Some codecs use the concept of Picture Order Count (POC). A POC value is derived for each picture, and the POC value increases rather than decreases as the pictures are positioned in the output order. Therefore, the POC indicates the output order of the pictures. The POC can be used in the decoding process, such as for implicit scaling of motion vectors and initialization of a list of reference pictures. Furthermore, the POC can be used to verify the consistency of the output order. The variable that includes the POC value of a picture can be called PicOrderCntVal.

[0154] A decoded picture buffer (DPB) can be used in both the encoder and / or decoder. Buffering decoded pictures can serve two purposes: as a reference in inter-frame prediction and to reorder decoded pictures in the output order. Some encoding formats (such as HEVC) offer great flexibility for both reference picture marking and output reordering, and separate buffers for reference and output pictures could waste memory resources. Therefore, the DPB can include a unified decoded picture buffering process for both reference and output reordering. Decoded pictures can be removed from the DPB when they are no longer used as a reference and are no longer needed for output.

[0155] The output order can be defined as the order in which decoded images are output from the decoded image buffer (for decoded images that will be output from the decoded image buffer).

[0156] Output time can be defined as the time it takes for a decoded image to be output from the decoder or the decoder's DPB (for a decoded image to be output from the DPB), for example, as specified by a hypothetical reference decoder specification based on the output timing of the DPB operation.

[0157] Images with the same output order can be defined as images with the same output time.

[0158] Decoding order can be defined as the order in which syntax elements are processed during the decoding process. It may be necessary to sort the syntax elements in the bitstream according to their decoding order.

[0159] Identifiers can be defined as syntactic elements that identify syntactic structures. For example, the value of an identifier can differ in different instances of the same syntactic structure, such as a parameter set. A specific instance of a syntactic structure can be referenced by its identifier value. For example, the parameter set referenced by the (de)coding of an encoded video slice can be identified by providing the identifier value of the parameter set in the header of the encoded video slice.

[0160] An indicator (idc) can be defined as a syntax element whose value indicates a choice between more than two values ​​(for which semantics have already been assigned). The name of an indicator syntax element may have the suffix _idc.

[0161] Video coding standards may enable the use of Supplemental Enhancement Information (SEI) messages or similar messages. Several SEI messages are specified in the H.264 / AVC, H.265 / HEVC, H.266 / VVC, and H.274 / VSEI standards. User Data SEI messages allow organizations and companies to specify SEI messages for their own use. Standards may include syntax and semantics for specifying SEI messages, which encoders must follow, but decoders may not need to process SEI messages to conform to output order.

[0162] Video Usability Information (VUI) can be defined as a syntactic structure that identifies the interpretive properties of a decoded image used for display purposes, specifically including color representation information. VUI parameters may include, but are not limited to, one or more of the following: progressive scan source indicator, interlaced scan source indicator, frame packing indicator, projection content indicator, sample aspect ratio information, overscan information, primary colors, transport characteristics, color transformation matrix coefficients, sample value range indicator (e.g., indicating full range or studio range), and chroma sample position information. VUI parameters may be carried as part of a parameter set, such as a sequence parameter set or a video parameter set.

[0163] ITU-T Recommendation H.274 (equivalent to ISO / IEC 23002-7) can be referred to as "General Supplemental Enhancement Information Messages for Encoding Video Bitstreams" and is known as "General Supplemental Enhancement Information" or VSEI. The VSEI standard specifies the syntax and semantics of Video Availability Information (VUI) parameters and Supplemental Enhancement Information (SEI) messages. The VUI parameters and SEI messages defined in the VSEI standard are designed to be transmitted within the encoded video bitstream in the manner specified in the video coding specification, or in other ways determined by the specification of the system using such encoded video bitstreams. The VSEI standard is intended for use with VVC encoded video bitstreams, although it was drafted in a manner intended to be sufficiently general to allow its use with other types of encoded video bitstreams. For example, VUI parameters and SEI messages can assist in processes related to decoding, display, or other purposes. In the VSEI standard, the vui_parameters(payloadSize) syntax structure is specified for the VUI, where payloadSize is an input parameter indicating the number of bits in the VUI.

[0164] Frame packing can be defined as arranging more than one input image (which may be called (input) component frames) into an output image, or arranging input images as alternating first and second component frames with temporal interleaving.

[0165] The parity of a constituent frame can be defined as the first constituent frame or the second constituent frame, or equivalently as constituent frame 0 or constituent frame 1.

[0166] Typically, frame packing is not limited to any particular type of component frame, or the component frames do not need to have a specific relationship with each other. In many cases, frame packing is used to arrange the component frames of a stereoscopic video clip into a single sequence of images, as explained in more detail in the following paragraph. This arrangement may include placing input images in spatially non-overlapping regions within the output images. For example, in a side-by-side arrangement, two input images are placed horizontally adjacent to each other within the output images. The arrangement may also include dividing one or more input images into two or more component frame partitions and placing the component frame partitions in spatially non-overlapping regions within the output images. The output images, or the sequence of frame-packed output images, may be encoded into a bitstream, for example, by a video encoder. The bitstream may be decoded, for example, by a video decoder. The decoder or post-processing operations following decoding may extract the decoded component frames from the decoded images(s), for example, for display.

[0167] In frame-compatible stereoscopic video (also known as frame packing for stereoscopic video), stereo pairs are spatially packed into individual frames at the encoder as a preprocessing step for encoding. These packed frames are then encoded using a conventional 2D video coding scheme. The output frame produced by the decoder comprises the constituent frames of the stereo pairs.

[0168] In typical operating mode, the original frame and the packed single frame of each view have the same spatial resolution. In this case, the encoder downsamples both views of the stereoscopic video before the packing operation. Spatial packing can use, for example, side-by-side or top-bottom formats, and downsampling needs to be performed accordingly.

[0169] The encoder can indicate the use of frame packing by including one or more frame packing arrangement SEI messages in the bitstream, as defined in VSEI. Similarly, the decoder can infer the use of frame packing by decoding one or more frame packing arrangement SEI messages from the bitstream. When a frame packing arrangement SEI message is applied to CLVS, the cropped decoded picture consists of samples from multiple spatially packed components that are packed into a single frame, or, using the indicated frame packing arrangement scheme, the cropped decoded picture output in output order forms alternating first and second component frames with temporal interleaving. This information can be used by the decoder to appropriately rearrange the samples and process the component frames appropriately for display or other purposes.

[0170] In some video codecs, Video Availability Information (VUI) can be included in the Sequence Parameter Set (SPS). The VUI specified in VSEI includes the following: `vui_non_packed_constraint_flag` equal to 1 indicates that there should be no frame packing SEI messages applicable to CLVS in the bitstream. `vui_non_packed_constraint_flag` equal to 0 does not impose such a constraint.

[0171] Information about post-filtering in neural networks

[0172] Images decoded by an image or video decoder can be further processed through post-processing operations. Post-processing operations may include one or more neural networks (NNs) and / or one or more non-NN operations. A post-processing operation can take one or more input images as input and can output one or more output images. Post-processing can be used for several reasons or use cases, including but not limited to: - Enhance images based on one or more objective metrics, such as Peak Signal-to-Noise Ratio (PSNR), which can be derived from pixel-wise distortion metrics (such as PSNR) or from perceptual metrics (such as video multi-method evaluation fusion, VMAF, or feature-based metrics). Enhancements can include removing coded artifacts, denoising, etc. - Enhance the image relative to one or more subjective scores or metrics. Enhancements can include removing coded artifacts, denoising, etc. - Upsampling and super-resolution; - Given two or more input images, perform temporal interpolation on one or more output images; - Given one or more input images, perform temporal extrapolation on one or more output images; and - Coloring.

[0173] An encoder (such as a video encoder) or transmitter can signal information to a decoder (such as a video decoder) or receiver, wherein the information indicates, but is not limited to, the following: - Whether to apply post-processing operations; - Characteristics of post-processing operations, such as the format of post-processing operation inputs and outputs; and - How to apply post-processing operations.

[0174] Additional enhancements regarding Neural Network Post-Filter Properties (NNPFC) and Neural Network Post-Filter Activation (NNPFA). SEI (Search Engine Information) message information

[0175] The NNPFC SEI message and the NNPFA SEI message have been described in version 3 of the General Supplemental Enhancement Information (VSEI) standard.

[0176] The syntax structure for specifying an NNPFC SEI message can be called nn_post_filter_characteristics. The syntax structure for specifying an NNPFA SEI message can be called nn_post_filter_activation.

[0177] The NNPFC SEI message includes an `nnpfc_id` syntax element, which contains an identifier that can be used to identify post-processing filters. The basic post-processing filter is a filter included in or identified by the first NNPFC SEI message in decoding order, and has a specific `nnpfc_id` value within the Coding Layer Video Sequence (CLVS). When a second NNPFC SEI message exists whose `nnpfc_id` value is the same as the `nnpfc_id` value defining the basic post-processing filter, an update relative to the basic post-processing filter is applied to obtain the post-processing filter associated with the `nnpfc_id` value. This update can be obtained by decoding the encoded neural network bitstream in the second NNPFC SEI message. Otherwise, the post-processing filter associated with the `nnpfc_id` value is assigned the same as the basic post-processing filter.

[0178] The NNPFC SEI message includes the nnpfc_mode_idc syntax element, the semantics of which can be defined as follows:

[0179] An nnpfc_mode_idc value of 1 indicates that the base post-processing filter associated with the nnpfc_id value, or the update relative to that base post-processing filter, is a neural network identified by a Uniform Resource Identifier (URI) nnpfc_uri, the format of which is identified by the tag URI nnpfc_tag_uri.

[0180] An nnpfc_mode_idc value of 0 indicates that the SEI message contains an ISO / IEC 15938-17 bitstream that specifies a base post-processing filter with the same nnpfc_id value or an update relative to that base post-processing filter.

[0181] NNPFC SEI messages may also include: - The purpose of the post-processing filter may include, but is not limited to, one or more of the following: Improved visual quality; Chromaticity upsampling from 4:2:0 chroma format to 4:2:2 or 4:4:4 chroma format, or from 4:2:2 chroma format to 4:4:4 chroma format; Increase the width or height of the input image; Frame rate upsampling; Bit depth upsampling; or Coloring. - The format of the input tensor given as input to neural network inference - Formatting of the output tensors generated by neural network inference; and - A representation of the complexity of neural networks.

[0182] The NNPFC SEI message syntax includes nnpfc_base_flag. An nnpfc_base_flag equal to 1 specifies that the SEI message specifies the base NNPF. An nnpfc_base_flag equal to 0 specifies that the SEI message specifies an update relative to the base NNPF.

[0183] When nnpfc_base_flag equals 0, the following applies: - This SEI message defines an update relative to the previous base NNPF in decoding order, with the same nnpfc_id value. The updates are not cumulative; instead, each update is applied to the base NNPF, i.e., the NNPF specified in decoding order by the first NNPFC SEI message, which has a specific nnpfc_id value within the current CLVS. The NNPF defined by this SEI message is obtained by applying the update defined by this SEI message relative to the base NNPF with the same nnpfc_id value. - This SEI message relates to the currently decoded picture and all subsequent decoded pictures of the current layer in output order, up to the end of the current CLVS, or up to the decoded picture (excluding the current picture) that is in output order after the current decoded picture and is associated with a subsequent NNPFC SEI message in decoding order (the NNPFC SEI message has nnpfc_base_flag equal to 0 and has a specific nnpfc_id value within the current CLVS), whichever is earlier.

[0184] The NNPFC SEI message syntax includes the syntax element nnpfc_num_input_pics_minus1. nnpfc_num_input_pics_minus1 plus 1 specifies the number of images used as input to NNPF. The variable numInputPics can be set to equal nnpfc_num_input_pics_minus1 + 1.

[0185] Frame rate upsampling filters can be interchangeably called image rate upsampling filters. Such filters generate or interpolate one or more images between a pair of images that serve as filter inputs. Frame rate upsampling filters can also be used where the number of input images can be greater than two. Such frame rate upsampling filters can generate images between more than one pair of input images. Frame rate upsampling filters can include neural networks, in which case the interpolated images between a pair of input images are generated through inference from the neural network. A frame rate upsampling filter can exist that extrapolates images before or after the input images(s), as an alternative or supplement to extrapolation between input images.

[0186] When the filtering objective includes frame rate upsampling, the NNPFC SEI message includes the nnpfc_interpolated_pics[i] syntax element for a value i in the range of 0 (inclusive) to nnpfc_num_input_pics_minus1 (exclusive). nnpfc_interpolated_pics[i] specifies the number of interpolated pictures generated by NNPF between the i-th picture used as input to NNPF and the (i+1)-th picture.

[0187] The NNPFC SEI message syntax can include an indication of how images not originating from the current bitstream are expected to be replaced in the input tensor; this can be referred to as `nnpfc_absent_input_pic_zero_flag`. `nnpfc_absent_input_pic_zero_flag` equal to 1 indicates that NNPF expects input images not present in the current bitstream to be represented by an array of samples with a value of 0. `nnpfc_absent_input_pic_flag` equal to 0 indicates that NNPF expects input images not present in the current bitstream to be represented by the closest input image in the current bitstream in output order.

[0188] The NNPFC SEI message syntax can include an indication, which may be referred to as nnpfc_auxiliary_inp_idc, indicating whether auxiliary input data exists in the NNPF input tensor in addition to the sample array(s) of the input images(s). A nnpfc_auxiliary_inp_idc greater than 0 indicates that auxiliary input data exists in the NNPF input tensor. Specific semantics can be specified for specific non-zero values ​​of nnpfc_auxiliary_inp_idc. A nnpfc_auxiliary_inp_idc equal to 0 indicates that auxiliary input data does not exist in the input tensor.

[0189] The NNPFA SEI message specifies a Neural Network Post-Processing Filter (NNPF) that can be used for post-processing filtering of the current image, or for post-processing filtering of the current image and one or more other images. The NNPFA SEI message includes the nnpfa_target_id syntax element, which indicates that a neural network post-processing filter with nnpfc_id equal to nnpfa_target_id can be used for post-processing filtering of the indicated persistence. The indicated persistence can be limited to the current image only (indicated by nnpfa_persistence_flag equal to 0). Alternatively, NNPF activations can be indicated as persistent by nnpfa_persistence_flag equal to 1, in which case the persistence of the NNPF activation can persist until the end of the current CLVS, or until the next image in the current layer (excluding that image) associated with an NNPFA SEI message having the same nnpfa_target_id as the current SEI message, in the output order.

[0190] The NNPFA SEI message syntax may include syntax elements indicating whether the base post-processing filter or the latest post-processing filter is active, where the latest post-processing filter is defined by the base post-processing filter that has been updated (if any) with the latest filter. This syntax element may be referred to as `nnpfa_target_base_flag`. `nnpfa_target_base_flag` equal to 1 indicates that the target NNPF is the base NNPF, where `nnpfc_id` equals `nnpfa_target_id`. `nnpfa_target_base_flag` equal to 0 indicates that the target NNPF is the NNPF specified by the last NNPFC SEI message, where `nnpfc_id` equals `nnpfa_target_id`, which is located before the first VCL NAL unit of the current image in decoding order and is not a duplicate of the NNPFC SEI message containing the base NNPF.

[0191] The NNPFA SEI message syntax can include an indication of which images from the filtered images corresponding to the input images are output by the NNPF process. For the i-th input image filtered by NNPF, the NNPFA SEI message syntax can include the nnpfa_output_flag[i] syntax element, which specifies that the filtered image is not output by the NNPF process when it is equal to 0, and specifies that the filtered image is output by the NNPF process when it is equal to 1.

[0192] Regarding NNPFA SEI messages, two picture sets can be defined: nnpfcTargetPictures and nnpfaTargetPictures. nnpfcTargetPictures can be defined as the picture set belonging to the last NNPFC SEI message preceding the current NNPFASEI message in decoding order, where nnpfc_id equals nnpfa_target_id. nnpfaTargetPictures can be defined as the picture set of the target NNPF activated by the current NNPFA SEI message. For a compliant bitstream, it can be required that any pictures included in nnpfaTargetPictures should also be included in nnpfcTargetPictures.

[0193] The NNPF process involves performing NNPF inference on a given input image. NNPF inference can be performed block-by-block, filtering the entire image region. After NNPF inference, the NNPF generated images can be output in ascending index order, where all NNPF generated images interpolated by NNPF are output, and those NNPF generated images corresponding to any input image of NNPF are output according to the semantics specified in the NNPFA SEI message.

[0194] The general post-processing filtering procedure using NNPF can be described as follows. The input to this procedure is a bitstream BitstreamToFilter. The output of this procedure is a list of NNPF output images, ListNnpfOutputPics. First, BitstreamToFilter is decoded, and the list CroppedDecodedPictures is set as a list of cropped decoded images obtained from decoding BitstreamToFilter in output order. Second, for each cropped decoded image in CroppedDecodedPictures (where one or more NNPFs are activated), the filtering procedure for one image is repeated in output order, as described below. The order of the images in ListNnpfOutputPics is consistent with the output order. It may be necessary that there should not be more than one image associated with any particular output time instance within ListNnpfOutputPics. When multiple NNPFs are activated for any particular image in CroppedDecodedPictures and only one NNPF is allowed to be selected for application, although any NNPF can be selected, the above constraints should apply regardless of which NNPF is selected for a particular image.

[0195] The filtering process for an image using NNPF can be described as follows. The NNPF filtering process can be applied to each cropped decoded image, called the current image (i.e., the image in CroppedDecodedPictures), where one or more NNPFs are activated. When NNPF is applied to the current image, NNPF generates filtered and / or interpolated images by applying the NNPF process to the current image. When NNPF is applied to the current image, the images generated by NNPF through the application process are stored in the NNPF output tensor in the output order. When the applied NNPF is the last NNPF applied to the current image, the images generated by NNPF and output by the NNPF process will be included in ListNnpfOutputPics in the same order as when the images were stored in the NNPF output tensor.

[0196] The use of the NNPFC and NNPFA SEI messages for VVC is described in version 3 of the Universal Video Coding (VVC) standard. It should be understood that the NNPFC and NNPFA SEI messages can be used similarly in any other video coding specification.

[0197] When NNPFC and NNPFA SEI messages are used in VVC, the decoder selects input images for NNPF. Input images can be selected in reverse output order, starting with the image for which NNPF was activated via the NNPFA SEI message. Input images can be indexed, starting with index 0, which is assigned to the image for which NNPF was activated via the NNPFA SEI message. In one example, the decoder selects the input image at index i (where i is greater than 0) as the latest cropped decoded output image preceding the input image at index i-1 in output order. When the result of decoding the bitstream is that there is no cropped decoded output image preceding the input image at index i-1 in the output order, the input image at index i can be considered not present in the current bitstream (e.g., lost), and subsequent input images from indices i+1 to numInputPics-1 (if any) are also lost. Lost input images can be handled as described above regarding the nnpfc_absent_input_pic_zero_flag syntax element.

[0198] When NNPFC and NNPFA SEI messages are used in VVC, and NNPF is continuously activated for upsampling the image rate of the interpolated image between a single pair of input images until the end of the bitstream, NNPF is repeatedly applied at the end of the bitstream for different sets of input images, up to but not including the following sets of input images that would cause any interpolated image to be created after the last image of the bitstream in output order. In these sets of input images, some images may be lost and, for example, may be replaced by the last image of the bitstream in output order.

[0199] Visual time extrapolation

[0200] It should be understood that, in various embodiments, the terms visual temporal extrapolation, temporal extrapolation, and video prediction are used interchangeably. Visual temporal extrapolation can be defined as a method, algorithm, or process that, given one or more past images as input, generates one or more future images. Visual temporal extrapolation can be implemented through neural network inference, but is not necessarily based on or limited to neural network inference.

[0201] Use cases for visual time extrapolation

[0202] Use cases for visual temporal extrapolation include, but are not limited to: - Extremely low latency computer vision in fields such as robotics and autonomous driving, where extrapolated future images help make predictive decisions; - In very low-latency applications (such as cloud gaming), the rate of rendering images increases relative to the rate of decoding images; - Reduce end-to-end latency in low-latency applications by extrapolating and displaying future images before they are received; and - Generative facial videos for very low bitrate video encoding.

[0203] Information about generative artificial intelligence (AI), including temporal extrapolation and generative facial videos.

[0204] The terms generative artificial intelligence (AI), generative modeling, or generative machine learning (and other similar terms) are generally used to refer to a class of models that learn from data and are able to generate new data. State-of-the-art generative models are based on neural networks. The basic components or layers of a generative neural network are often indistinguishable from those of a non-generative neural network. Examples of such components include convolutional layers, non-linear layers, fully connected layers, normalization layers, attention layers, etc.

[0205] A typical example of a neural network architecture that allows the generation of text data is a transformer-based "decoder," where the "decoder" doesn't necessarily refer to part of a codec that compresses the input data into a small bitstream. Instead, the decoder is a neural network that takes an input word set, a portion of an input word, or a token extracted from an input word, and outputs an output word set, a portion of a word, or a token. During inference, such a NN operates in an autoregressive mode, where the generated word(s) or lexical(s) are provided as part of the input word(s) or lexical(s). To enable such a NN to generate data, it is trained to predict the next word(s) (or an estimate of the probability distribution of the next word) given an input word set. The NN can be based on a transformer architecture that includes the use of a self-attention mechanism, where an attention score is assigned to each input lexical or word based on all other input lexical or word, including previously generated words or lexical(s). During training a decoder-style transformer architecture, future data items (words or lexical(s)) are masked to avoid revealing future information. In some cases, decoder-style transformer architectures are called "unidirectional" because they use or process information from left to right, while in contrast, some encoder-style transformer architectures are called "bidirectional" (because they use or process information from left to right and from right to left).

[0206] Another example of generative modeling is visual temporal extrapolation, where an image is generated by a neural network based on one or more previously decoded or generated images and one or more other data items. The one or more previously decoded or generated images can be images decoded without involving a generative modeling process, such as conventional codecs, like VVC-compliant codecs. The one or more data items can include parameters or features describing the difference between one or more previously decoded images and the current image to be temporally extrapolated. Examples of such parameters are facial parameters (such as facial keypoints or facial landmarks and their positions or differences relative to facial landmarks in the previous image), or parameters of other objects. The one or more data items can be signaled from the encoder to the decoder.

[0207] Another example of a parameter or feature describing the difference between one or more previously decoded images and the current image to be extrapolated over time is a face matrix, which can represent the transformation between one or more previously decoded images and the current image. For example, the matrix could be one of the following (where...) (It is the multiplication symbol): - Size is 2 2 or 3 Affine translation matrix of 3; - Size is 2 2 or 3 The covariance matrix of 3; - The mouth matrix representing mouth movements; - An eye matrix representing the opening and closing state and degree of the eyes; - Size is 2 2 or 3 The head rotation parameter of 3 represents the head rotation in 2D or 3D space; - Size is 1 2 or 1 A 3x3 head translation matrix, representing head translation in 2D or 3D space; or - Size is 1 2 or 1 A 3D head position matrix, representing the head position in 2D or 3D space.

[0208] Facial parameters may include, but are not limited to, any one or both of the following: facial key points and / or one or more facial matrices.

[0209] An example of an encoder utilizing generative AI can be described as follows: An input base image is encoded into a bitstream of an encoded base image using any video coding method (such as VVC). Subsequent input images are fed into an analysis model to extract feature parameters. These feature parameters are encoded. The feature parameters can be encoded independently of the feature parameters of any other image, for example, for the first subsequent image following the base image in decoding order. Alternatively, the feature parameters can be encoded predictively with reference to earlier encoded feature parameters in decoding order, where feature residuals can be encoded relative to the predicted feature parameters. The feature residuals can be quantized and entropy-encoded as part of the feature encoding. The encoded feature parameters are included in the bitstream or along the bitstream, for example, in a generative face video SEI message. For subsequent images, the video encoder can encode either virtual or driving images, as described later. The encoded feature parameters of subsequent images can be included in or along the corresponding encoded virtual or driving images, for example, within the same image unit.

[0210] An example of a decoder utilizing generative AI can be described as follows: The decoder decodes an encoded base image from a bitstream. For subsequent encoded virtual or driving images, the decoder decodes the encoded feature parameters. The decoded feature parameters and the decoded base image, along with the optional decoded driving image, are input into the generative neural network model inference that generates the output image.

[0211] Information regarding Generative Face Video (GFV) SEI messages

[0212] In JVET, a generative face video (GFV) SEI message has been proposed for the new version of the General Supplemental Enhancement Information (VSEI) standard to support visual temporal extrapolation of faces in videos. One of the functions of the GFV SEI message is to signal face parameters that represent auxiliary inputs to the neural network performing the temporal extrapolation. The most recent document describing the GFV SEI message is JVET-AF0234, which can be obtained from [https: / / jvet-experts.org / doc_end_user / documents / 32_Hannover / wg11 / JVET-AF0234-v2.zip (last accessed December 21, 2023)]. An example summary is provided below.

[0213] The GFV SEI message defines the interface between the video decoder and the generator NN, which performs visual temporal extrapolation of faces in the video.

[0214] The following are the image types considered in GFV SEI messages: - Basic Image: The decoded output image that generative networks can use to generate new facial images. It is encoded by a traditional codec such as a VVC-compatible codec; - Virtual image: A picture unit containing only the minimum allowed resolution of the SEI message; and - (Optional) Master image (also known as driving image), which may be optionally fed into the generative neural network to improve background texture and / or facial details. This driving image may also be encoded by a conventional codec, such as a VVC-compatible codec.

[0215] The second neural network (called the converter NN) is used to convert the facial parameters (signaled in the GFV SEI message) into the following conversion parameters: - 15 3D key points; - A 3×3 matrix; and - A 1×3 matrix.

[0216] The input to a generator neural network (also known as a generative neural network) can be one or more of the following: - (Previously decoded) base image. - Transformed parameters (output of the converter NN) - (Optional) Master / Driver Image.

[0217] Various embodiments are designed to address problems such as enabling the use of generative AI on a receiver or decoder to generate video based on information signaled by a transmitter or encoder.

[0218] At least some embodiments describe methods and apparatus for supporting the performance of visual temporal extrapolation.

[0219] In addition, at least some embodiments describe methods and apparatus for enabling neural network post-filter (NNPFC) SEI messages to signal information required by a decoder or receiver to perform visual temporal extrapolation using one or more post-processing neural networks.

[0220] In one embodiment, the encoder encodes signaling information in or along the bitstream, and the decoder decodes the signaling information from or along the bitstream. This enables or supports visual temporal extrapolation on the decoder side, where performing visual temporal extrapolation may include using one or more neural networks (NNs), where, for simplicity, the one or more NNs are referred to as VTNNs. In one example, the signaling information may be included in a Neural Network Post-Filter Feature (NNPFC) Supplemental Enhancement Information (SEI) message.

[0221] In one embodiment, the signaling information may include the purpose of performing visual temporal extrapolation on the decoder side.

[0222] In one embodiment, signaling information may include generative modeling or generative artificial intelligence (AI) purposes on the decoder side.

[0223] In one embodiment, the signaling information may include information characterizing the neural network performing the time extrapolation, which may include, but is not limited to, information characterizing the inputs and / or outputs of the neural network. This information may be referred to as temporal extrapolation characteristic (TEC) information.

[0224] In one embodiment, when the purpose includes visual time extrapolation, TEC information can be signaled.

[0225] In another embodiment, when the signaling information indicates that the purpose of one or more neural networks associated with the signaling information includes one or more of time extrapolation, generative modeling, or generative artificial intelligence (AI), at least some TEC information is signaled, wherein generative modeling or generative AI may refer to using one or more neural networks to generate data conditioned on one or more inputs.

[0226] In one embodiment, when the purpose includes generating facial video or generating video, TEC information can be signaled.

[0227] In one embodiment, TEC information may include information indicating whether the VTNN can extrapolate a variable number of images or a fixed number of images.

[0228] In one embodiment, TEC information may include information indicating the maximum number of images that the VTNN can extrapolate.

[0229] In one embodiment, when the TEC information indicates that the VTNN can extrapolate a variable number of images, for each inference of the VTNN, the encoder can indicate to the decoder the number of images to be extrapolated.

[0230] In one embodiment, TEC information may include information indicating the number of images extrapolated in a single activation or inference of the VTNN.

[0231] In one embodiment, TEC information may include information indicating the time interval between images extrapolated relative to the (latest) image used as input for inference.

[0232] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary input, which indicates an output instance of time-extrapolated images(s).

[0233] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary input that indicates facial parameters or is derived from facial parameters, wherein the auxiliary input may be used by the VTNN to perform temporal extrapolation on one or more faces in an image.

[0234] In one embodiment, TEC information may include indications(s) for indicating which facial parameters are present and / or which VTNN is used. For example, TEC information may include a first flag for indicating when facial keypoints are used and a second flag for indicating when a facial matrix is ​​used.

[0235] In one embodiment, facial parameters used as auxiliary inputs or used to derive auxiliary inputs can be carried within a newly defined SEI message, for example, the SEI message could be called a Neural Network Post-Filter Input (NNPFI) SEI message.

[0236] In another embodiment, facial parameters used as auxiliary input or used to derive auxiliary input can be carried within a Generative Face Video (GFV) SEI message.

[0237] In one embodiment, TEC information may include information indicating whether auxiliary inputs to the VTNN (such as auxiliary inputs indicating facial parameters or derived from facial parameters) can be derived from a lossy encoding process. In one example, the encoder performs lossy (and optionally lossless) compression on the facial parameters; the decoder performs lossless and / or lossy decompression on the encoded facial parameters; and the decompressed facial parameters are input to the VTNN.

[0238] In one embodiment, TEC information may include information indicating whether the VTNN adopts auxiliary input derived from the output of another neural network, which is run or executed on the decoder side.

[0239] In one embodiment, TEC information may include information indicating whether the VTNN adopts auxiliary input derived from the output of another neural network that converts or translates facial parameters into a format acceptable or required by the VTNN. Facial parameters may be carried within an NNPFA SEI message, a GFV SEI message, or an NNPFI SEI message (or more generally, frame-by-frame (FW) signaling) or in any other suitable manner.

[0240] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary inputs, which may represent tensors generated on the encoder side or derived from tensors.

[0241] In one embodiment, the tensor is the output (or derived from) another neural network that is run or executed on the encoder side, and the tensor can be signaled in an NNPFA SEI message, a GFV SEI message, or an NNPFI SEI message (or more generally, FW signaling).

[0242] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary input representing or derived from audio or speech.

[0243] This information can be used by VTNN to generate or synthesize several aspects of the face, such as the mouth, lips, facial expressions, blinking, etc.

[0244] In one embodiment, when the VTNN uses auxiliary input representing audio or speech, the TEC information may include information about the characteristics of the audio or speech.

[0245] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary input representing text or derived from text.

[0246] This information can be used by VTNN to generate or synthesize several facial features, such as the mouth, lips, facial expressions, blinking, etc.

[0247] In one embodiment, TEC information may include information indicating whether the input to the VTNN can be an input that only includes background information.

[0248] In one embodiment, TEC information may include information indicating whether the input to the VTNN can be an input that only includes foreground information (such as a face).

[0249] In one embodiment, TEC information may include information indicating whether the input to the VTNN can be a driving image. In one example, the encoder indicates to the decoder that the VTNN supports two types of input images, where the first input image is a base image and the second image is a driving image.

[0250] In one embodiment, TEC information may include information indicating whether the VTNN can produce different outputs with different inferences or executions of the VTNN using the same input(s). This behavior or characteristic of the VTNN is referred to as the output variability of the VTNN.

[0251] In another embodiment, the TEC information may include information indicating the type or cause of the variability in the VTNN's output.

[0252] In one embodiment, TEC information may include information indicating whether the VTNN can be indicated or operated in a manner that does not produce output variability.

[0253] In one embodiment, TEC information may include information indicating whether the VTNN is allowed to have output variability.

[0254] In one embodiment, TEC information may include information that makes the VTNN have no output variability.

[0255] In one embodiment, TEC information may include information indicating the type of extrapolated output.

[0256] In one embodiment, signaling information may include an indication of whether the VTNN is applicable to all images to which it is applied or activated.

[0257] In one embodiment, the VTNN can be adapted to specialize certain aspects of the generated output, such as for a specific face.

[0258] In one embodiment, the VTNN can be adapted to be used specifically for a particular task or purpose.

[0259] In one embodiment, the signaling information may include an indication that the NN is sequence-level adapted.

[0260] In one embodiment, the signaling information may include: indicating the purpose of sequence-level adaptation.

[0261] In one embodiment, the signaling information may include an indication of whether the VTNN is adapted to all images to which it is applied by inputting a persistent auxiliary input (PAI) signal.

[0262] In one embodiment, the encoder encodes signaling information in or along the bitstream, and the decoder decodes the signaling information from or along the bitstream. This enables or supports visual temporal extrapolation on the decoder side, where performing visual temporal extrapolation may include using one or more neural networks. For simplicity, it is assumed that visual temporal extrapolation is performed by a neural network called a VTNN. It should be understood that embodiments can be similarly implemented by a combination of neural networks, such as an optical flow-derived neural network followed by a synthetic network that takes the optical flow and one or more images as input.

[0263] It should be understood that the embodiments are not limited to any particular type of neural network. For example, the embodiments can be implemented using a model that takes one or more seed images and one or more driving features as input. In one example, the model is a generative model, such as a normalized flow-based model. In another example, such a model is based on a diffusion model (also known as a diffusion probability model, or a score-based generative model). In yet another example, such a model is based on a convolutional neural network.

[0264] In at least some embodiments, signaling information is associated with one or more neural networks that perform at least a portion of a visual temporal extrapolation task.

[0265] In one example, signaling information can be included in the Neural Network Post-Filter Feature (NNPFC) Supplemental Enhancement Information (SEI) message. In another example, signaling information can be included in the Generative Face Video (GFV) SEI message. In yet another example, signaling information can be included in both the NNPFC SEI message and the GFV SEI message.

[0266] In some embodiments, certain features may refer to signaling information associated with one or more images of the video. This can be achieved through a frame-by-frame signaling mechanism. This type of signaling is called frame-by-frame (FW) signaling. Examples of FW signaling include Neural Network Post-Filter Activation (NNPFA) SEI messages and Generative Face Video (GFV) SEI messages.

[0267] Purpose

[0268] In one embodiment, the signaling information may include the purpose of performing visual temporal extrapolation on the decoder side.

[0269] In another embodiment, this objective may be associated with a defined neural network.

[0270] In another embodiment, the purpose can be indicated as a bit field, wherein a specific bit position is assigned to visual temporal extrapolation, and the bit value at that specific bit position indicates whether visual temporal extrapolation is one of the purposes of the defined neural network. Other bit positions can be assigned to other purposes and can be indicated together with visual temporal extrapolation.

[0271] Example implementations of signaling information include the purpose of performing visual temporal extrapolation.

[0272] Adding time extrapolation is one of the purposes of nnpfc_purpose. This addition includes the following features: - A new bit in nnpfc_purpose is specifically used for time extrapolation. - When nnpfc_purpose indicates time extrapolation, nnpfc_extrapolated_pics_minus1 plus 1 indicates the number of extrapolated pictures generated by NNPF.

[0273] Adding time extrapolation purpose

[0274] NNPFC SEI Message Syntax

[0275] NNPFC SEI Message Semantics

[0276] The table below provides an example of the nnpfc_purpose definition. It should be understood that the bitmask allocation presented is an example, and embodiments can be implemented similarly with any other bitmask allocation.

[0277] The variables ChromaUpsamplingFlag, ResolutionResamplingFlag, PictureRateUpsamplingFlag, BitDepthUpsamplingFlag, ColourizationFlag, and TemporalExtrapolationFlag (which respectively specify whether nnpfc_purpose indicates that the purpose of NNPF includes chroma upsampling, resolution resampling, picture rate upsampling, bit depth upsampling, colorization, and temporal extrapolation) are derived as follows: ChromaUpsamplingFlag = ( ( nnpfc_purpose&0x02 )>0 ) ? 1 : 0 ResolutionResamplingFlag = ( ( nnpfc_purpose&0x04 )>0 ) ? 1 : 0 PictureRateUpsamplingFlag = ( ( nnpfc_purpose&0x08 )>0 ) ? 1 : 0 BitDepthUpsamplingFlag = ( ( nnpfc_purpose&0x10 )>0 ) ? 1 : 0 ColourizationFlag = ( ( nnpfc_purpose&0x20 )>0 ) ? 1 : 0 TemporalExtrapolationFlag = ( ( nnpfc_purpose&0x40 )>0 ) ? 1 : 0

[0278] nnpfc_interpolated_pics[i] specifies the number of interpolated pictures generated by NNPF between the i-th input picture and the (i+1)-th input picture.

[0279] When the PictureRateUpsamplingFlag of the NNPF is equal to 1 and the nnpfa_persistence_flag of the NNPFA SEI message that activates the NNPF is equal to 1, the value of nnpfc_interpolated_pics[i] is greater than 0 only for a single i value in the range of 0 to numInputPics-1 (inclusive).

[0280] The increment of 1 in nnpfc_extrapolated_pics_minus1 specifies the number of extrapolated pictures generated by NNPF after all input pictures in NNPF, in the order of output.

[0281] The variables NumInpPicsInOutputTensor (which specifies the number of images with corresponding input images that exist in the NNPF output tensor), InpIdx[idx] (which specifies the idx-th image in the NNPF output tensor that has a corresponding input image, and is the input image index of the input image list in reverse output order), and numPicsInOutputTensor (which specifies the total number of images in the NNPF output tensor) are derived as follows: for( i = 0, numPicsInOutputTensor = 0; i <numInputPics; i++ ) if( nnpfc_input_pic_filtering_flag[ i ] ) { InpIdx[numPicsInOutputTensor] = i numPicsInOutputTensor++ } NumInpPicsInOutputTensor = numPicsInOutputTensor if(PictureRateUpsamplingFlag) for( i = 0; i <= numInputPics - 2; i++ ) numPicsInOutputTensor += nnpfc_interpolated_pics[ i ] if( TemporalExtrapolationFlag ) numPicsInOutputTensor += nnpfc_extrapolated_pics + 1

[0282] When TemporalExtrapolationFlag equals 1, the extrapolated images generated by NNPF are output after all the input images of NNPF in the order they appear.

[0283] In one embodiment, when TemporalExtrapolationFlag equals 1 and there are decoded output images following the current image where NNPF is activated in output order, the extrapolated image generated by NNPF precedes the decoded output image in output order.

[0284] In one embodiment, when using temporal frame packing arrangement, the decoder can select decoded images of frames with the same constituent frames (e.g., left-view only or right-view only images of a temporally frame-packed stereoscopic video) as input images for NNPF to perform temporal extrapolation. For example, when an input image with PictureRateUpsamplingFlag or TemporalExtrapolationFlag equal to 1 and index 0 is associated with a frame packing arrangement SEI message with fp_arrangement_type equal to 5, all input images are associated with a frame packing arrangement SEI message with fp_arrangement_type equal to 5 and fp_current_frame_is_frame0_flag value equal to 5.

[0285] In one embodiment, the decoder: Infer that time-interleaved frames are used to pack and arrange in the bitstream, for example, SEI messages are packed and arranged according to frames; The parity of the constituent frames is inferred, and the current frame in which the filter is activated after time extrapolation has the same constituent frame parity for the time-interleaved frame packing arrangement. Select input frames with the inferred parity of the constituent frames for the time extrapolation filter; and The time extrapolation filter is applied using the input frame as input.

[0286] Constraints to avoid output order conflicts.

[0287] In one embodiment, to avoid conflicts or obfuscated output order constraints associated with the general post-processing filtering procedure, the extrapolation of images is involved in addition to the interpolated images. Within this procedure, for any given pair of inputPicA and inputPicB consecutive in output order within CroppedDecodedPictures, when there are one or more intermediatePicSetA in ListNnpfOutputPics between inputPicA and inputPicB in output order, it may be necessary that one and only one of the following applies: - When a specific image currPicA in CroppedDecodedPictures is the current image, the images in intermediatePicSetA should be the images output by applying a specific NNPF nnpfA, where the specific NNPF nnpfA has a PictureRateUpsamplingFlag equal to 1. - When a specific image currPicA in CroppedDecodedPictures is the current image, the images in intermediatePicSetA should be the images output by applying a specific NNPF nnpfA, where the specific NNPF nnpfA has TemporalExtrapolationFlag equal to 1.

[0288] What might be needed is: the application of any other NNPF used in the filtering process of an image when currPicA is the current image, or the application of any NNPF (including nnpfA) used in the filtering process of an image when any other image in CroppedDecodedPictures, currPicB, is the current image, without outputting any images between inputPicA and inputPicB in the output order.

[0289] Add generative modeling purpose

[0290] The table below provides an example of the nnpfc_purpose definition. It should be understood that the bitmask allocation presented is an example, and embodiments can be implemented similarly with any other bitmask allocation.

[0291] The variables GeneralVisualQualityImprovementFlag and GenerativeModellingFlag (which specify whether nnpfc_purpose indicates that the purpose of NNPF includes overall visual quality improvement and generative modeling, respectively) are derived as follows: GeneralVisualQualityImprovementFlag = (nnpfc_purpose&0x01) GenerativeModellingFlag = ( ( nnpfc_purpose&0x40 )>0 ) ? 1 : 0

[0292] A GenerativeModellingFlag of 1 indicates that the output images of (multiple) NNPF do not represent the same visual content as the input images, but are generated by NNPF based on the input images and auxiliary inputs (if any).

[0293] It may be necessary that when GeneralVisualQualityImprovementFlag equals 1, GenerativeModellingFlag should be equal to 0.

[0294] Example Features

[0295] In one embodiment, the signaling information may include information characterizing the neural network performing the time extrapolation, which may include, but is not limited to, information characterizing the inputs and / or outputs of the neural network. This information may be referred to as temporal extrapolation characteristic (TEC) information.

[0296] In one embodiment, TEC information can be signaled when the purpose includes visual temporal extrapolation. In one example, TEC information is signaled when signaling information indicates that the purpose of one or more neural networks associated with the signaling information includes visual temporal extrapolation.

[0297] In another embodiment, at least some TEC information may be signaled when the purpose includes objectives other than visual temporal extrapolation. In one example, some TEC information may be signaled when the signaling information indicates that the purpose of one or more neural networks associated with the signaling information includes generative modeling or generative artificial intelligence (AI), where generative modeling or generative AI refers to using one or more neural networks to generate data conditioned on one or more inputs. An example of generative modeling or generative AI is using a neural network to generate an image based on a natural language input description. Another example of generative modeling or generative AI is using a neural network to generate additional data for a given image, such as pixels in areas not captured by the imaging process that produced the given image (e.g., a 360-degree view).

[0298] In one embodiment, TEC information can be signaled when the purpose includes generating facial video or generative video. In one example, the purpose of generating facial video includes generating or improving an image (part of a video) comprising at least one or more faces at the decoder or receiver side, given one or more of the following: - Coordinates of at least one or more faces; - Coordinates of at least one or more landmarks or key points on the face; - A transformation matrix that describes the motion or temporal changes of one or more key points on at least one or more faces; - Facial expression features; - Blinking instruction; - Text associated with the video; - Audio associated with the video; or - One or more other images, such as key images or driving images, where key images can be images encoded using techniques other than generative models, and driving images can be images used to refresh or enhance parts of an image.

[0299] It should be noted that at least some characteristics in TEC information may also be relevant to or effective for neural networks performing tasks or purposes other than visual temporal extrapolation, such as tasks or purposes performed by other generative NNs or generative AI models (such as spatial extrapolation generative NNs).

[0300] Feature: Number of extrapolated images

[0301] In one embodiment, TEC information may include information indicating the number of images extrapolated in a single activation or inference of the VTNN.

[0302] The above nnpfc_extrapolated_pics_minus1 syntax element is an example embodiment of information used to indicate the number of pictures extrapolated by the VTNN in a single activation or inference.

[0303] In one embodiment, the TEC information may include information indicating that the number of extrapolated images in the NNPF can vary for each inference iteration, and that the number of extrapolated images for each inference iteration is indicated by another signaling mechanism, such as another SEI message (e.g., a GFV SEI message or an NNPFA SEI message). For example, the TEC information may include a flag indicating when the TEC information is included in the number of images extrapolated in a single activation or inference iteration of the VTNN. When the flag indicates that the TEC information is not included in the number of images extrapolated in a single activation or inference iteration of the VTNN, it indicates that the number of extrapolated images in the NNPF can vary for each inference iteration, and that the number of extrapolated images for each inference iteration is indicated by another signaling mechanism, such as another SEI message (e.g., a GFV SEI message or an NNPFA SEI message). In another example, the TEC information may include a syntax element indicating the number of pictures extrapolated in a single activation or inference of the VTNN, and when the syntax element is equal to 0, it indicates that the number of extrapolated pictures of the NNPF may vary for each inference, and the number of extrapolated pictures for each inference is indicated by another signaling mechanism, such as another SEI message (e.g., GFV SEI message or NNPFA SEI message).

[0304] Feature: Time interval for extrapolating images

[0305] In one embodiment, TEC information may include information indicating the time interval between images extrapolated relative to the (latest) image used as input for inference.

[0306] Features: Auxiliary input indicates output position / POC / timestamp difference

[0307] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary input, which indicates an output instance of time-extrapolated images(s). The output instance may be defined in one or more of the following ways: a relative output position provided in units of a given or predefined fraction of a nominal image interval; the POC difference or timestamp difference between the time-extrapolated images(s) and the input images (e.g., relative to the most recent input image, i.e., the input image with index 0). For example, the timestamp considered may be an output timestamp or a display timestamp.

[0308] In one embodiment, information indicating whether the VTNN uses auxiliary input can be carried in an NNPFC SEI message, which indicates the output instance of time extrapolated (multiple) images, for example in a syntax element indicating the type of (multiple) auxiliary input, which may be called nnpfc_auxiliary_inp_idc.

[0309] In another embodiment, information indicating whether the VTNN uses auxiliary input can be carried in a neural network post-filter activation (NNPFA) SEI message, which indicates the output instance of time-extrapolated (multiple) images.

[0310] In one embodiment, when such auxiliary input is not available, or when the VTNN is not designed to take such auxiliary input, the VTNN may assume a fixed image rate.

[0311] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary input, which indicates: the difference in picture order count (POC) of the output image to be generated or extrapolated, the difference in timestamps, the relative output position, or the relative output difference (the interval relative to the input images).

[0312] In one example, TEC information may include information indicating whether the VTNN uses auxiliary inputs, such as POC difference, timestamp difference, relative output position, or relative output difference, which is signaled as part of the NNPFC SEI message.

[0313] In an example embodiment, this auxiliary input can be carried within a neural network post-filter activation (NNPFA) SEI message.

[0314] In another example embodiment, this auxiliary input can be carried within a newly defined SEI message, for example, which may be referred to as the Neural Network Post-Filter Input (NNPFI) SEI message.

[0315] In one example, NNPF can take two images as input and extrapolate one image as output. In the NNPFC SEI message, the syntax element `nnpfc_auxiliary_inp_idc` indicates the auxiliary input of the NN associated with that NNPFC SEI message relative to the output difference (relative to the interval between the input images). In the NNPFA SEI message, the auxiliary input parameter can be set to 0.5 to instruct the VTNN to generate the extrapolated image at half the interval between the input images. For example, when the input images have timestamps 0 and 2, setting 0.5 will cause the VTNN to generate the extrapolated image at timestamp 3. In another example, when the input images have timestamps 0 and 1, setting 0.5 will cause the VTNN to generate the extrapolated image at timestamp 1.5, for example, the image between timestamps 1 and 2.

[0316] Example implementations of auxiliary input features are described below:

[0317] To enable multiple neural network inferences for the same set of input images, the following features are included: - Specify the nnpfc_auxiliary_inp_idc value to indicate that the relative output position of the image to be extrapolated is provided as input to the temporal extrapolation NN. - A Neural Network Post-Filter Input (NNPFI) SEI message is proposed. The NNPFI SEI message carries auxiliary input data to provide for inference of the associated NNPF. The proposed NNPFI SEI message includes the following: ○ nnpfi_target_id, which associates the corresponding NNPF (with the same nnpfc_id value) with this SEI message; ○ Similar to many other SEI messages, specify nnpfi_cancel_flag and nnpfi_persistence_flag; ○ nnpfi_cnt specifies the NNPFI SEI message instance count value for the nnpfi_target_id value within the image unit; ○ nnpfi_auxiliary_inp_idc and nnpfi_extrapolated_pics_minus1, which should have the same values ​​as the associated NNPF's nnpfc_auxiliary_inp_idc and nnpfc_extrapolated_pics_minus1, respectively, and be repeated in the NNPFI SEI message to avoid resolving dependencies; and ○ When nnpfi_auxiliary_inp_idc indicates the relative output position of the image to be extrapolated, the relative output position is provided in units of a given fraction of the nominal image interval. Within the NNPFI SEI messages of the same image unit, the output positions need to be increased in ascending order of nnpfi_cnt. - For each unique value of nnpfi_cnt within an image unit, the NNPF process is executed in ascending order of nnpfi_cnt.

[0318] SEI message input to the post-filter of the neural network

[0319] In one embodiment, the Neural Network Post-Filter Input (NNPFI) SEI message is used to provide auxiliary input data to a neural network post-processing filter defined by (multiple) NNPFC SEI messages and activated by (multiple) NNPFA SEI messages.

[0320] In one embodiment, when there are multiple NNPFI SEI messages for the same target NNPF with different SEI payload contents in an image unit, each occurrence of these NNPFI SEI messages will cause inference about the target NNPF.

[0321] In one embodiment, the NNPFI SEI message includes an identifier or counter value, wherein the identifier or counter value is an NNPFI SEI message identifying the same target NNPF in the same picture unit. When the identifier or counter value is the same as a previous NNPFI SEI message for the same target NNPF in the same picture unit, the current NNPFI SEI message can be inferred to be a copy of the previous NNPFI SEI message in the same picture unit.

[0322] In one embodiment, for the same target NNPF in the same image unit, the target NNPF inference is performed for each unique identifier or counter value in the NNPFI SEI message.

[0323] In the example embodiment, the syntax of the NNPFI SEI message payload can be specified as follows:

[0324] The semantics of the syntax elements of the NNPFI SEI message can be specified as follows.

[0325] `nnpfi_target_id` indicates the `nnpfc_id` of the target NNPF, which is specified by one or more NNPFC SEI messages related to the current picture, and `nnpfc_id` is equal to `nnpfi_target_id`. It may be necessary that no NNPFC SEI message with a specific value `nnpfi_target_id` exists in the current PU, unless one or both of the following conditions are true: 1) Within the current CLVS, there is an NNPFC SEI message whose `nnpfc_id` is equal to the specific value of `nnpfi_target_id` that existed in the PU prior to the current PU in the decoding order. 2) There exists an NNPFC SEI message whose `nnpfc_id` is equal to the specific value of `nnpfi_target_id` in the current PU.

[0326] When a PU contains an NNPFA SEI message with a specific value of nnpfa_target_id and an NNPFI SEI message with an nnpfi_target_id that has a specific value equal to nnpfa_target_id, it may be necessary for the NNPFA SEI message to precede the NNPFI SEI message in the decoding order.

[0327] A `nnpfi_cancel_flag` value of 1 indicates that the persistence of NNPF auxiliary input data established by any previous NNPFI SEI message with the same `nnpfi_target_id` as the current SEI message is cancelled. That is, the NNPF auxiliary input data is no longer used unless it is activated by another NNPFI SEI message with the same `nnpfi_target_id` as the current SEI message and `nnpfi_cancel_flag` equal to 0. A `nnpfi_cancel_flag` value of 0 indicates that the syntax element specifying the NNPF auxiliary input data immediately follows.

[0328] `nnpfi_persistence_flag` specifies the persistence of the NNPF auxiliary input data for the current layer. A value of 0 for `nnpfi_persistence_flag` indicates that the NNPF auxiliary input data can only be used for post-processing filtering of the current image. A value of 1 for `nnpfi_persistence_flag` indicates that the NNPF auxiliary input data can be used for post-processing filtering of the current image and all subsequent images of the current layer in output order, until one or more of the following conditions are true: - A new CLVS begins for the current layer. - End of bitstream. - The image associated with the NNPFI SEI message that has the same nnpfi_target_id as the current SEI message in the current layer is output, and this image is output after the current image in the output order. - For subsequent images associated with an NNPFI SEI message in the current layer that has the same nnpfi_target_id as the current SEI message, NNPF auxiliary input data should not be applied.

[0329] Let nnpfiTargetPictures be a set of images of NNPF auxiliary input data that persists based on the current NNPFI SEI message. Let nnpfaTargetPictures be a set of images such that for this set, there exists an NNPF with nnpfc_id equal to nnpfi_target_id, activated by any NNPFASEI message with nnpfa_target_id equal to nnpfi_target_id. It may be necessary that any images included in nnpfiTargetPictures should also be included in nnpfaTargetPictures.

[0330] `nnpfi_cnt` specifies the NNPFI SEI message instance count within the picture unit for the given `nnpfi_target_id` value. It may be necessary that the `nnpfi_cnt` of the first NNPFI SEI message in the picture unit with a specific `nnpfi_target_id` value, in decoding order, be equal to 0. It may also be necessary that NNPFI SEI messages with the same `nnpfi_target_id` value and an `nnpfi_cnt` equal to `currNnpfiCnt-1`, in decoding order, precede the current NNPFI SEI message within the same picture unit, when the `nnpfi_cnt` assigned to `currNnpfiCnt` is greater than 0.

[0331] nnpfi_auxiliary_inp_idc has the same semantics as nnpfc_auxiliary_inp_idc and should be set to the value of nnpfc_auxiliary_inp_idc in the NNPFC SEI message, where nnpfc_id is equal to nnpfi_target_id applied to the current image.

[0332] nnpfi_extrapolated_pics_minus1 has the same semantics as nnpfc_extrapolated_pics_minus1 and should be set to the value of nnpfc_extrapolated_pics_minus1 in the NNPFC SEI message where nnpfc_id is equal to nnpfi_target_id applied to the current picture.

[0333] nnpfi_output_interval_div_minus2 is used to export units with a specified relative output position.

[0334] `nnpfi_output_interval_mul_minus1[i]+1` specifies the output position of the image, in units of 1 ÷ (nnpfi_output_interval_div_minus2+2) nominal image intervals. If the target NNPF has at least two input images, the nominal image interval is the interval between each pair of consecutive input images in output order. Otherwise, the nominal image interval is an arbitrary interval that should always be used for the `nnpfi_target_id` value in the same CLVS. The output position `OutputPos[nnpfi_cnt][i]` is set to equal to `(nnpfi_output_interval_mul_minus1[i]+1) ÷ (nnpfi_output_interval_div_minus2+2)`.

[0335] When nnpfi_cnt is greater than 0 and i equals 0, it may be necessary that OutputPos[nnpfi_cnt][0] is greater than OutputPos[nnpfi_cnt-1][nnpfi_extrapolated_pics_minus1] derived from NNPFI SEI messages with the same nnpfi_target_id in the same picture unit.

[0336] What might be needed is that when i is greater than 0, OutputPos[nnpfi_cnt][i] should be greater than OutputPos[nnpfi_cnt][i-1].

[0337] When the image exists in both nnpfaTargetPictures and nnpfiTargetPictures, the auxiliary input tensor AuxInputTensor0 is a real-valued vector equal to OutputPos[nnpfi_cnt]. Otherwise, the auxiliary input tensor AuxInputTensor0 is a real-valued vector equal to OutputPos[nnpfi_cnt], which is derived by setting nnpfi_cnt to 0, setting nnpfi_extrapolated_pics_minus1 to nnpfc_extrapolated_pics_minus1, setting nnpfi_output_interval_div_minus2 to nnpfi_extrapolated_pics_minus1, and setting nnpfi_output_interval_mul_minus1[i] to i.

[0338] Feature: Auxiliary input indicating facial parameters

[0339] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary input that indicates facial parameters or is derived from facial parameters, wherein the auxiliary input may be used by the VTNN to perform temporal extrapolation of one or more faces in an image.

[0340] In one embodiment, facial parameters used as auxiliary input or used to derive auxiliary input can be carried from the encoder to the decoder within an SEI message or other suitable signaling mechanism.

[0341] In one embodiment, information indicating whether the VTNN uses auxiliary input that indicates facial parameters or is derived from facial parameters may include an SEI message indicating the auxiliary input based on a given payload type value, and further indicating the SEI message payload type value of the generative face video (GFV) SEI message.

[0342] In one embodiment, information indicating whether the VTNN uses auxiliary input that indicates facial parameters or is derived from facial parameters may further include an identifier value that is included in or derived from one or more SEI messages for a given SEI payload type value, wherein the indication of the given SEI payload type value may also be included in TEC information. The auxiliary input will be derived based on the SEI message associated with the identifier value.

[0343] In one example, the facial parameters include one or more of the following: - 2D key points. - 3D key points. - One or more matrices.

[0344] In one embodiment, facial parameters used as auxiliary input or for deriving auxiliary input can be carried within an NNPFA SEI message.

[0345] In another embodiment, facial parameters used as auxiliary inputs or used to derive auxiliary inputs can be carried within a newly defined SEI message, for example, the SEI message can be called a Neural Network Post-Filter Input (NNPFI) SEI message.

[0346] In another embodiment, facial parameters used as auxiliary input or used to derive auxiliary input can be carried within a Generative Face Video (GFV) SEI message.

[0347] In one embodiment, TEC information may include information indicating the format of facial parameters accepted or required by the VTNN. In one example, two or more possible formats may be defined in the standard, each format being associated with a format identifier, and the TEC information will include the value of the format identifier.

[0348] Facial parameters obtained from NNPFA SEI messages, GFV SEI messages, or NNPFI SEI messages (or more generally, from FW signaling) can be converted into the desired format indicated in the TEC information.

[0349] Feature: Auxiliary input includes the output of another network running on the decoder side.

[0350] In one embodiment, TEC information may include information indicating whether the VTNN adopts auxiliary input derived from the output of another neural network, which is run or executed on the decoder side.

[0351] In one embodiment, the TEC information may include information indicating whether the VTNN adopts auxiliary input derived from the output of another neural network that converts or translates facial parameters into a format acceptable or required by the VTNN. The facial parameters may be carried within an NNPFA SEI message, a GFV SEI message, or an NNPFI SEI message (or more generally, FW signaling) or in any other suitable manner.

[0352] It should be understood that many embodiments of the feature “indicating auxiliary input of facial parameters” also apply to this feature.

[0353] In one embodiment, the auxiliary input type (nnpfc_auxiliary_inp_idc) indicated in the NNPFC SEI message identifies a specific auxiliary input or a specific SEI message used to derive the auxiliary input. For example, the auxiliary input type can be defined for a GFV SEI message that derives facial parameters and a facial matrix based on (multiple) GFV SEI messages.

[0354] In one embodiment, the auxiliary input type (nnpfc_auxiliary_inp_idc) indicated in the NNPFC SEI message specifies that the auxiliary input is derived from the SEI payload type value in (a plurality of) SEI messages, which is provided in a syntax element (e.g., called nnpfc_sei_aux_payload_type) within the NNPFC SEI message.

[0355] In one embodiment, the syntax element (nnpfc_auxiliary_inp_idc) indicated in the NNPFC SEI message for defining the auxiliary input type is treated as a bit field, where each bit position defines the use of a specific auxiliary input type. In one example, (nnpfc_auxiliary_inp_idc&2) being greater than 1 indicates that the auxiliary input is derived from (multiple) SEI messages with a given payload type value (nnpfc_sei_aux_payload_type).

[0356] In one embodiment, when nnpfc_sei_aux_payload_type exists in the NNPFC SEI message, the syntax element nnpfc_sei_aux_id also exists. nnpfc_sei_aux_id specifies an identifier that can be used to identify which(s) of the SEI messages with a payload type equal to nnpfc_sei_aux_payload_type were used to derive the NNPF auxiliary input data or auxiliary input tensors(s).

[0357] In one embodiment, when the NNPFC SEI message indicates that the auxiliary input is derived from the GFV SEI message, nnpfc_sei_aux_id exists and indicates the gfv_id value of the GFV SEI message, for which the NNPF defined in the NNPFC SEI message is used as a GenerativeNN.

[0358] In one embodiment, when the NNPFC SEI message indicates that the auxiliary input is derived from the GFV SEI message, the NNPFC SEI message includes flags, one value of which indicates that the nnpfc_id value is used as the gfv_id value, for which the NNPF defined in the NNPFC SEI message is used as the GenerativeNN, and the other value indicates that the nnpfc_sei_aux_id exists and indicates the gfv_id value of the GFV SEI message, for which the NNPF defined in the NNPFC SEI message is used as the GenerativeNN.

[0359] The following paragraphs describe an example embodiment that combines this feature with features of the previous feature (“auxiliary input indicating facial parameters”).

[0360] One possible syntax included in an NNPFC SEI message may be:

[0361] Another possible syntax included in an NNPFC SEI message may be:

[0362] When (nnpfc_auxiliary_inp_idc&2) equals 2 and nnpfc_sei_aux_payload_type equals SEI_GFV_PAYLOAD_TYPE, nnpfc_num_input_pics_minus1 should equal 1.

[0363] A value greater than 0 for nnpfc_auxiliary_inp_idc indicates that auxiliary input data exists in the input tensor of NNPF, or that (multiple) auxiliary input tensors are provided to NNPF. A value equal to 0 for nnpfc_auxiliary_inp_idc indicates that no auxiliary input data exists in the input tensor, and that no auxiliary input tensor is provided to NNPF.

[0364] (nnpfc_auxiliary_inp_idx&2) equals 2, which specifies that the auxiliary input data or (multiple) auxiliary input tensors are derived based on the given payload type value and the given identifier value of the SEI message.

[0365] nnpfc_sei_aux_payload_type specifies the SEI message payload type value used to export auxiliary input data or auxiliary input tensors(s) of NNPF.

[0366] nnpfc_sei_aux_id specifies an identifier that can be used to identify which(s) of the SEI messages of payload type equal to nnpfc_sei_aux_payload_type were used to derive the NNPF auxiliary input data or auxiliary input tensors(s).

[0367] In one example, when nnpfc_sei_aux_payload_type is equal to the payload type value of the GFV SEI message (hereinafter referred to as SEI_GFV_PAYLOAD_TYPE), it may be necessary that when there is a generative face video SEI message with gfv_id equal to nnpfc_sei_aux_id in the picture unit, the picture unit includes an NNPFA SEI message with nnpfa_target_id equal to nnpfc_id.

[0368] When (nnpfc_auxiliary_inp_idc&2) equals 2 and nnpfc_sei_aux_payload_type equals SEI_GFV_PAYLOAD_TYPE, the following applies: - inputBaseY, inputBaseCb (if applicable), inputBaseCr (if applicable), inputBaseKeyPoint, and inputBaseMatrix are derived as specified for the semantics of a generative face video SEI message, where gfv_id is equal to nnpfc_sei_aux_id and gfv_base_pic_flag is equal to 1. If multiple such generative face video SEI messages exist, the derivation is based on the last SEI message preceding the image unit containing the NNPFA SEI message (whose nnpfa_target_id is equal to the nnpfc_id used to activate NNPF) in decoding order. - The associated generative face video SEI message is a generative face video SEI message that satisfies the following: gfv_id equals nnpfc_sei_aux_id, and the generative face video SEI message exists in the image unit containing an NNPFA SEI message whose nnpfa_target_id equals nnpfc_id. Subsequently, the syntax elements of the generative face video SEI message reference the associated generative face video SEI message. - The inputDriveKeyPoint and inputDriveMatrix are exported as specified for the semantics of the associated generative facial video SEI message. - When gfv_drive_pic_fusion_flag equals 0, the sample values ​​of CroppedYPic[0], CroppedCbPic[0] (if applicable), and CroppedCrPic[0] (if applicable) are all set to 0. - The input image for NNPF, which has a sample array with a sample value of 0, is a virtual image defined in the generative facial video SEI message. - The input image at index 1 is overridden by the base image defined in the generative face video SEI message as follows: CroppedYPic[1] is set to equal inputBaseY, CroppedCbPic[1] is set to equal inputBaseCb (if applicable), and CroppedCrPic[1] is set to equal inputBaseCr.

[0369] The procedure DeriveInputTensors() is used to derive the input tensor inputTensor given the vertical sample coordinates cTop and the horizontal sample coordinates cLeft (which specify the top-left sample position of the sample block included in the input tensor), the number of auxiliary input tensors numAuxInputTensors, and the auxiliary input tensor auxInputTensor[j] (if any) (for each j value in the range from 0 to numAuxInputTensors-1, including the end values). The procedure is defined as follows: if((nnpfc_auxiliary_inp_idx&2)&& ( nnpfc_sei_aux_payload_type == SEI_GFV_PAYLOAD_TYPE ) ) numAuxInputTensors = 2 else numAuxInputTensors = 0 for( i = 0; i <numInputPics; i++ ) { if( nnpfc_inp_order_idc == 0 ) for( yP = -nnpfc_overlap; yP <inpPatchHeight + nnpfc_overlap; yP++) for( xP = -nnpfc_overlap; xP <inpPatchWidth + nnpfc_overlap; xP++ ){ inpVal = InpY( InpSampleVal( cTop + yP, cLeft + xP,CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) ) yPovlp = yP + nnpfc_overlap xPovlp = xP + nnpfc_overlap if( !nnpfc_component_last_flag )inputTensor[ 0 ][ i ][ 0 ][ yPovlp ][ xPovlp ] = inpVal else inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 0 ] = inpVal if( nnpfc_auxiliary_inp_idc = = 1 ) if( !nnpfc_component_last_flag ) inputTensor[ 0 ][ i ][ 1 ][ yPovlp ][ xPovlp ] =strengthControlScaledVal[ i ] else inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 1 ] =strengthControlScaledVal[ i ] } else if( nnpfc_inp_order_idc = = 1 ) for( yP = -nnpfc_overlap; yP<inpPatchHeight + nnpfc_overlap; yP++) for( xP = -nnpfc_overlap; xP<inpPatchWidth + nnpfc_overlap; xP++ ){ inpCbVal = InpC( InpSampleVal( cTop + yP, cLeft + xP,CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCbPic[ i ], 1 ) ) inpCrVal = InpC( InpSampleVal( cTop + yP, cLeft + xP,CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCrPic[ i ], 2 ) ) yPovlp = yP + nnpfc_overlap xPovlp = xP + nnpfc_overlap if( !nnpfc_component_last_flag ) { inputTensor[ 0 ][ i ][ 0 ][ yPovlp ][ xPovlp ] = inpCbVal inputTensor[ 0 ][ i ][ 1 ][ yPovlp ][ xPovlp ] = inpCrVal } else { inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 0 ] = inpCbVal inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 1 ] = inpCrVal } if( nnpfc_auxiliary_inp_idc = = 1 ) if( !nnpfc_component_last_flag ) inputTensor[ 0 ][ i ][ 2 ][ yPovlp ][ xPovlp ] =strengthControlScaledVal[ i ] else inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 2 ] =strengthControlScaledVal[ i ] } else if( nnpfc_inp_order_idc = = 2 ) for( yP = -nnpfc_overlap; yP<inpPatchHeight + nnpfc_overlap; yP++) for( xP = -nnpfc_overlap; xP<inpPatchWidth + nnpfc_overlap; xP++ ){ yY = cTop + yP xY = cLeft + xP yC = yY / SubHeightC xC = xY / SubWidthC inpYVal = InpY( InpSampleVal( yY, xY, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) ) inpCbVal = InpC( InpSampleVal( yC, xC, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCbPic[ i ], 1 ) ) inpCrVal = InpC( InpSampleVal( yC, xC, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCrPic[ i ], 2 ) ) yPovlp = yP + nnpfc_overlap xPovlp = xP + nnpfc_overlap if( !nnpfc_component_last_flag ) { inputTensor[ 0 ][ i ][ 0 ][ yPovlp ][ xPovlp ] = inpYVal inputTensor[ 0 ][ i ][ 1 ][ yPovlp ][ xPovlp ] = inpCbVal inputTensor[ 0 ][ i ][ 2 ][ yPovlp ][ xPovlp ] = inpCrVal } else { inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 0 ] = inpYVal inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 1 ] = inpCbVal inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 2 ] = inpCrVal } if( nnpfc_auxiliary_inp_idc = = 1 ) if( !nnpfc_component_last_flag ) inputTensor[ 0 ][ i ][ 3 ][ yPovlp ][ xPovlp ] =strengthControlScaledVal[ i ] else inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 3 ] =strengthControlScaledVal[ i ] } else if( nnpfc_inp_order_idc = = 3 ) for( yP = -nnpfc_overlap; yP<inpPatchHeight + nnpfc_overlap; yP++) for( xP = -nnpfc_overlap; xP<inpPatchWidth + nnpfc_overlap; xP++ ){ yTL = cTop + yP 2 xTL = cLeft + xP 2 yBR = yTL + 1 xBR = xTL + 1 yC = cTop / 2 + yP xC = cLeft / 2 + xP inpTLVal = InpY( InpSampleVal( yTL, xTL, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) ) inpTRVal = InpY( InpSampleVal( yTL, xBR, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) ) inpBLVal = InpY( InpSampleVal( yBR, xTL, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) ) inpBRVal = InpY( InpSampleVal( yBR, xBR, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) ) inpCbVal = InpC( InpSampleVal( yC, xC, CroppedHeight / 2, CroppedWidth / 2, CroppedCbPic[ i ], 1 ) ) inpCrVal = InpC( InpSampleVal( yC, xC, CroppedHeight / 2, CroppedWidth / 2, CroppedCrPic[ i ], 2 ) ) yPovlp = yP + nnpfc_overlap xPovlp = xP + nnpfc_overlap if( !nnpfc_component_last_flag ) { inputTensor[ 0 ][ i ][ 0 ][ yPovlp ][ xPovlp ] = inpTLVal inputTensor[ 0 ][ i ][ 1 ][ yPovlp ][ xPovlp ] = inpTRVal inputTensor[ 0 ][ i ][ 2 ][ yPovlp ][ xPovlp ] = inpBLVal inputTensor[ 0 ][ i ][ 3 ][ yPovlp ][ xPovlp ] = inpBRVal inputTensor[ 0 ][ i ][ 4 ][ yPovlp ][ xPovlp ] = inpCbVal inputTensor[ 0 ][ i ][ 5 ][ yPovlp ][ xPovlp ] = inpCrVal } else { inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 0 ] = inpTLVal inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 1 ] = inpTRVal inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 2 ] = inpBLVal inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 3 ] = inpBRVal inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 4 ] = inpCbVal inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 5 ] = inpCrVal } if( nnpfc_auxiliary_inp_idc = = 1 ) if( !nnpfc_component_last_flag ) inputTensor[ 0 ][ i ][ 6 ][ yPovlp ][ xPovlp ] =strengthControlScaledVal[ i ] else inputTensor[ 0 ][ i ][ yPovlp ][ xPovlp ][ 6 ] =strengthControlScaledVal[ i ] } if((nnpfc_auxiliary_inp_idx&2)&& ( nnpfc_sei_aux_payload_type == SEI_GFV_PAYLOAD_TYPE ) ) { if( i = = 0 ) { / / drive picture auxInputTensor0[ 0 ][ i ] = inputDriveKeyPoint auxInputTensor1[ 0 ][ i ] = inputDriveMatrix } else { / / i == 1, base picture auxInputTensor0[ 0 ][ i ] = inputBaseKeyPoint auxInputTensor1[ 0 ][ i ] = inputBaseMatrix } } }

[0370] NNPF PostProcessingFilter() is the target NNPF derived from the NNPFA SEI message semantics. NNPFPostProcessingFilter() takes an inputTensor as input, followed by a numAuxInputTensors tensor auxInputTensor. j Where j is in the range of 0 to numAuxInputTensors-1, inclusive. The following example procedure can be used with NNPF PostProcessingFilter() to generate (multiple) filtered and / or interpolated images in a block-by-block manner, containing Y, Cb, and Cr sample arrays FilteredYPic, FilteredCbPic, and FilteredCrPic, respectively, as shown in nnpfc_out_order_idc: if( nnpfc_inp_order_idc = = 0 || nnpfc_inp_order_idc = = 2 ) for( cTop = 0; cTop<CroppedHeight; cTop += inpPatchHeight ) for( cLeft = 0; cLeft<CroppedWidth; cLeft += inpPatchWidth ) { DeriveInputTensors( ) outputTensor = PostProcessingFilter( ) StoreOutputTensors( outputTensor ) } else if( nnpfc_inp_order_idc = = 1 ) for( cTop = 0; cTop<CroppedHeight / SubHeightC; cTop +=inpPatchHeight ) for( cLeft = 0; cLeft<CroppedWidth / SubWidthC; cLeft += inpPatchWidth ) { DeriveInputTensors( ) outputTensor = PostProcessingFilter( ) StoreOutputTensors( outputTensor ) } else if( nnpfc_inp_order_idc = = 3 ) for( cTop = 0; cTop<CroppedHeight; cTop += inpPatchHeight 2 ) for( cLeft = 0; cLeft<CroppedWidth; cLeft += inpPatchWidth 2 ) { DeriveInputTensors( ) outputTensor = PostProcessingFilter( ) StoreOutputTensors(outputTensor) }

[0371] In one embodiment, the signaling information may include information indicating the purpose of the neural network, which converts input facial parameters into an output representing the converted facial parameters, wherein the converted facial parameters or data derived from the converted facial parameters are included in the input of the VTNN.

[0372] In one embodiment, the signaling information may include information indicating the purpose of the neural network, which converts input parameters into an output representing the converted parameters.

[0373] In one embodiment, the signaling information may include information indicating the purpose of the neural network, which converts the format of the input parameters into another format of the output parameters.

[0374] Feature: Identifies NNPF as the generator NN and activates the generator NN.

[0375] In one embodiment, when the NNPFC SEI message indicates that the NNPF uses auxiliary input based on the SEI message payload type value indicating generative AI (such as the GFV SEI message), the encoder indicates that the NNPF is the generator NN and / or the decoder infers that the NNPF is the generator NN.

[0376] In one embodiment, when a generative AI SEI message (such as a GFV SEI message) includes an identifier for an NNPF, the encoder indicates that the NNPF is a generator NN and / or the decoder infers that the NNPF is a generator NN. In an example embodiment, when the gfv_id of the GFV SEI message is equal to the nnpfc_id of the NNPFC SEI message that defines the NNPF, the encoder indicates that the NNPF is a generator NN and / or the decoder infers that the NNPF is a generator NN. In another example embodiment, the encoder indicates that the NNPF is a generator NN and / or the decoder infers that the NNPF is a generator NN through identifier syntax elements in the GFV SEI message other than gfv_id, where the identifier syntax elements are equal to the nnpfc_id of the NNPFC SEI message that defines the NNPF. In one embodiment, the GFV SEI message may include two identifiers: a first identifier identifying a face and a second identifier identifying a generator NN. In another example embodiment, when the second identifier is equal to the nnpfc_id of the NNPFC SEI message that defines the NNPF, the encoder indicates that the NNPF is a generator NN and / or the decoder infers that the NNPF is a generator NN.

[0377] In one embodiment, the generator NN is activated on the decoder side as signaled via an NNPFA SEI message for an NNPF, for which an auxiliary input is indicated derived from a generative AI SEI message (such as a GFV SEI message). An NNPF with nnpfc_id equal to nnpfa_target_id is activated, and the auxiliary input is derived from a generative AI SEI message (such as a GFV SEI message), as described in other embodiments.

[0378] In one embodiment, the generator NN is activated on the decoder side, as signaled by a generative AI SEI message (such as a GFV SEI message). An NNPF with nnpfc_id, indicated in or for a generative AI SEI message (such as a GFV SEI message), is activated, and auxiliary inputs are derived from the generative AI SEI message (such as a GFV SEI message), as described in other embodiments.

[0379] Feature: Auxiliary inputs include tensors generated on the encoder side.

[0380] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary inputs, which may represent tensors generated on the encoder side or derived from tensors.

[0381] A tensor is an N-dimensional array. Typically, tensors will be 3D or 4D tensors, where two dimensions represent spatial dimensions (height and width), one dimension represents the channel, and one dimension (in the case of a 4D tensor) represents the batch size in the case of batch processing. However, even 1D tensors (e.g., an array of M elements) are possible when the batch size is assumed to be equal to 1, or 2D tensors (e.g., a matrix of M1 by M2 elements) are possible, where one axis or dimension represents the batch size.

[0382] In one example, a tensor is the output (or derived from) another neural network that is run or executed on the encoder side, and the tensor can be signaled in an NNPFA SEI message, a GFV SEI message, or an NNPFI SEI message (or more generally, FW signaling).

[0383] In one example, a tensor represents facial parameters or those derived from them. On the encoder side, the facial parameters can be input into a neural network (different from a VTNN with execution time extrapolation), the NN is run or executed, and the output of this NN will be a tensor. The tensor can be lossily or losslessly encoded on the encoder side and lossily or losslessly decoded on the decoder side. On the decoder side, the lossily or losslessly decoded tensor, or the signal derived from the lossily or losslessly decoded tensor, is input into the VTNN.

[0384] In one embodiment, TEC information may include information about the tensor's format, such as the number of dimensions, the size of each dimension, the numerical format of the tensor's values ​​(e.g., 32-bit floating-point or 16-bit integer), and the tensor's serialization order or format. In one example, such information may be signaled using a standard that signals the tensor (such as a tensor serialization format standard).

[0385] In one embodiment, a VTNN may employ more than one tensor.

[0386] In one embodiment, the number of auxiliary tensors to be input to the VTNN can be a characteristic of the VTNN, and information about this number can be signaled in an NNPFC SEI message. In another embodiment, the number of auxiliary tensors to be input to the VTNN can vary, and information about this number can be signaled in an NNPFA SEI message, a GFV SEI message, or a newly defined SEI message, which can be referred to as a neural network post-filter input SEI message (or more generally, FW signaling).

[0387] In one example, the VTNN accepts more than one auxiliary tensor, where different auxiliary tensors represent different facial features that will be extrapolated over time. The auxiliary tensors may include, but are not limited to, one or more of the following: a first auxiliary tensor representing facial expression, a second auxiliary tensor representing head pose, and a third auxiliary tensor representing facial details.

[0388] Features: Auxiliary input includes audio or speech.

[0389] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary input representing audio or speech or derived from audio or speech.

[0390] This information can be used by VTNN to generate or synthesize several facial features, such as the mouth, lips, facial expressions, blinking, etc.

[0391] In one embodiment, when the VTNN uses auxiliary input representing audio or speech, the TEC information may include information about the characteristics of the audio or speech, such as one or more of the following: - The expected sampling rate of the audio signal; - Bit depth and / or dynamic range; - Number of audio channels and / or channel configuration; - The duration of the audio signal to be included in the input tensor and / or the number of audio samples; - Information regarding whether the audio was processed using a source separation algorithm. For example, are the sounds isolated? (If there are multiple faces in an image) Are the different sounds separated? - Information regarding whether the audio is spatial audio; or - The timing of the audio signal provided as input relative to the (multiple) images to be extrapolated.

[0392] In one embodiment, the auxiliary input can be derived from the audio track associated with the video.

[0393] In another embodiment, the auxiliary input can be derived by processing the audio track using a neural network or other processing algorithm, wherein the neural network or other processing algorithm is defined in a standard or associated with an identifier that the encoder can indicate. In one example, the encoder can signal an identifier of the NN for processing the audio track and deriving the auxiliary input of the VTNN.

[0394] In one embodiment, the auxiliary input may be a signal carried within another signaling mechanism (or may be derived from there), such as an NNPFA SEI or GFV SEI message or a newly defined SEI message, which may be referred to as a neural network post-filter input SEI message (or more generally, FW signaling).

[0395] Feature: Auxiliary input includes text

[0396] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary input representing text or derived from text.

[0397] This information can be used by VTNN to generate or synthesize several facial features, such as the mouth, lips, facial expressions, blinking, etc.

[0398] In one embodiment, the text may include, for example, text derived from an audio track associated with a video image using speech-to-text technology.

[0399] In one embodiment, the text may include instructions for the VTNN, such as those relating to facial expressions, emotions, moods, or whether the eyes are closed. These instructions may or may not be in the form of natural language.

[0400] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary input, which represents text associated with the speech of a person whose face will be extrapolated over time.

[0401] In one embodiment, TEC information may include information indicating whether the VTNN uses auxiliary input, which represents textual instructions for high-level or abstract concepts related to the time extrapolation subject, such as emotion, mood, and facial expression in the case of the time extrapolation subject being the face.

[0402] In one embodiment, when the VTNN uses auxiliary input representing text, the TEC information may include information indicating characteristics of the text, such as one or more of the following: - Does VTNN accept or require the original text? - Whether VTNN accepts or requires lexical units (e.g., words, parts of words, or dictionary indices, where the indices are associated with the corresponding words or parts of words); - Does VTNN accept or require text embeddings, such as features extracted from the original text or lexical units? - Which word segmenter to use; - Which text embedder to use; or - The timing of the text provided as input relative to the (multiple) images to be extrapolated.

[0403] In one embodiment, the auxiliary input can be derived from a timed text track associated with the video.

[0404] In one embodiment, the auxiliary input can be derived from captions or explanatory text included in or along the video bitstream. For example, captions or explanatory text can be included in one or more SEI messages.

[0405] In one embodiment, the text can be obtained on the decoder side. For example, it can be exported based on an audio track.

[0406] In another embodiment, text or data derived from text can be signaled from the encoder to the decoder via a FW signaling mechanism (e.g., GFV SEI message of NNPFASEI or NNPFI SEI message).

[0407] Characteristics: Temperature

[0408] In one embodiment, TEC information may include information indicating whether the VTNN accepts input that controls how one or more aspects of the final output are determined based on the VTNN's output. In one example, the VTNN may accept an input parameter called "temperature," where the temperature parameter is used to control the probability of the VTNN's output, with higher values ​​of the temperature parameter allowing more randomness in the final output, while lower values ​​allow for a more predictable final output.

[0409] Feature: Output variability

[0410] In one embodiment, TEC information may include information indicating whether the VTNN can produce different outputs with different inferences or executions of the VTNN using the same input(s). This behavior or characteristic of the VTNN is referred to as the output variability of the VTNN.

[0411] In another embodiment, the TEC information may include information indicating the type or cause of the VTNN's output variability. For example, the type or cause of the output variability may be one of the following: - Pseudo-randomness; - Intentional variation, for example, the characteristics of VTNNs are output variability implemented by design; or - A floating-point format used to represent the values ​​of one or more parameters of the VTNN and / or the values ​​of one or more signals flowing through the VTNN.

[0412] In one embodiment, the TEC information may include information indicating whether the VTNN can be indicated or operated in a manner that does not produce output variability. In another embodiment, this information is included in the TEC information when the TEC information indicates that the VTNN is characterized by output variability.

[0413] In one embodiment, the TEC information may include information indicating whether the VTNN is allowed to have output variability. In other words, the VTNN associated with the TEC information may feature or support output variability, but the encoder may signal to the decoder whether the VTNN will operate or execute in a manner that does not produce output variability. In another embodiment, the TEC information is included in the TEC information when it indicates that the VTNN is characterized by output variability. In one example, the TEC information includes a flag indicating whether the VTNN is allowed to have output variability.

[0414] In one embodiment, TEC information may include information that makes the VTNN have no output variability. This information is referred to as deterministic reasoning (DI) information. In one example, the DI information includes a pseudo-random seed value.

[0415] In one embodiment, TEC information includes indications, such as flags, that indicate whether the decoder side will generate a random seed for the VTNN on its own or derive a pseudo-random seed based on other information signaled by the encoder.

[0416] A hash value can be defined as the result of a hash function. A hash value may be alternatively or additionally referred to as a checksum or hash sum. A hash function can be defined as any function that can be used to map numerical data of arbitrary size to numerical data of fixed size, where small differences in the input data can lead to large differences in the output data. A cryptographic hash function can be defined as a hash function that is practically irreversible, i.e., it creates input data based solely on the hash value. Cryptographic hash functions can include, for example, the MD5 function. An MD5 value can be a null-terminated UTF-8 character string containing a base64-encoded MD5 digest of the input data. A method for calculating a string is specified in IETF RFC 1864. It should be understood that other types of integrity checking schemes can also be used in various embodiments as alternatives to or supplements to MD5, such as different forms of Cyclic Redundancy Check (CRC), such as the CRC scheme used in ITU-T Recommendation H.271.

[0417] In one embodiment, the pseudo-random seed value can be derived on the decoder side or receiver side based on other information, which is signaled by the encoder or exists or is determined within the decoder, and this other information can also be used for purposes other than deriving the pseudo-random seed value. In one example, the other information is a sequence-level quantization parameter (QP) value, and the pseudo-random seed is set equal to that sequence-level QP value. In another example, the other information is a slice-level QP value, and the pseudo-random seed is set equal to that slice-level QP value. In yet another example, the other information is the value of a syntax element in the TEC information. In yet another example, the other information is the value of a syntax element specifying the VTNN in the NNPFC SEI message, such as nnpfc_id, and the pseudo-random seed is set equal to the value of nnpfc_id. In yet another example, the other information is the value of a syntax element specifying the VTNN in the Generative Face Video (GFV) SEI message, such as gfv_id, and the pseudo-random seed is set equal to the value of gfv_id. In another example, the hash value is derived from encoded data, such as NNPFC SEI messages, GFV SEI messages, SEI NAL units including NNPFC SEI messages or GFV SEI messages, encoded images associated with VTNN, or image units associated with VTNN. The hash value is used as a pseudo-random seed. In yet another example, the hash value is derived from decoded data (such as one or more decoded images or one or more input images of VTNN) and is used as a pseudo-random seed.

[0418] In one embodiment, the pseudo-random seed of the VTNN is derived in two stages on either the decoder or receiver side: In the first stage, as described in any of the embodiments in the preceding paragraph, a first pseudo-random seed is derived. In the second stage, the first pseudo-random seed is provided as initialization to a pseudo-random number generator, which returns a second pseudo-random seed. The second pseudo-random seed is then provided to the VTNN. The pseudo-random number generator can be predefined, for example, in a standard, or can be freely chosen by either the decoder or receiver side.

[0419] In one embodiment, the TEC information may include indications, such as flags, indicating whether a pseudo-random seed value can be derived based on the value of a specific syntax element, which is included in the TEC information or otherwise signaled to the decoder. In one example, the NNPFC SEI message includes the flag nnpfc_seed_from_nnpfc_id, which, when equal to 1, indicates that the pseudo-random seed value is equal to the value of the syntax element nnpfc_id. In another example, the NNPFC SEI message includes the flag nnpfc_seed_from_nnpfc_id, which, when equal to 1, indicates that the pseudo-random seed value should be input to the NNPF specified by the NNPFC SEI message, and that the value is equal to the value of the syntax element nnpfc_id; while, when equal to 0, it indicates that the pseudo-random seed value should not be input to the NNPF specified by the NNPFC SEI message.

[0420] In one embodiment, the TEC information may include an indication, specified by the TEC information, that the VTNN will use a seed value as input or auxiliary input. The seed value may be a random seed generated on the decoder side or a pseudo-random seed based on other information signaled by the encoder.

[0421] In one embodiment, the TEC information may include an indication, specified by the TEC information, that the VTNN will use a pseudo-random seed as input or auxiliary input.

[0422] In one embodiment, the TEC information may include an indication, specified by the TEC information, that the VTNN will use a pseudo-random seed as input or auxiliary input, wherein the pseudo-random seed may be determined or obtained in one of the following ways: the pseudo-random seed is specified in a standard specification; the pseudo-random seed is signaled as part of the TEC information; the pseudo-random seed is set to be equal to or determined based on the value of another syntax element included in the TEC information or otherwise signaled to the decoder; or the pseudo-random seed is provided to the decoder or receiver by external means, such as through an application.

[0423] In one embodiment, a specific bit of nnpfc_auxiliary_inp_idc being equal to 1 can be used as an indication, specified by the TEC information, that the VTNN uses the seed value as input or auxiliary input. For example, when nnpfc_auxiliary_inp_idc&4 is greater than 0, the auxiliary input data includes the seed value of the NNPF.

[0424] In one example, when a specific NNPFC SEI message of NNPF that specifies the execution of spatial or temporal extrapolation (or other generative task) includes a syntax element equal to 4 (nnpfc_auxiliary_inp_idc&4), NNPF will use a pseudo-random seed equal to the value of the syntax element nnpfc_id of the specific NNPFC SEI message as input or auxiliary input.

[0425] In another example, when the specific NNPFC SEI message for an NNPF that specifies the performance of spatial or temporal extrapolation (or other generative tasks) includes a syntax element (nnpfc_auxiliary_inp_idc&4) equal to 4, the NNPFC SEI message also includes a syntax element nnpfc_seed that specifies the value of the pseudo-random seed to be provided to the NNPF as input or auxiliary input. nnpfc_seed (if present) specifies the seed value used as auxiliary input to the NNPF. The following syntax included in the NNPFC SEI message can be used in this example:

[0426] In one embodiment, auxiliary inputs such as a seed value can be provided to NNPF in one of the following ways: - The channel or matrix of the input tensor includes one or more occurrences of the seed value. - The scalar seed value is provided separately from the input tensor of the NNPF procedure.

[0427] In one embodiment, the NNPF process or similar process for the VTNN includes a stage of adding pseudo-random noise to the input images(s) provided to the NNPF process. The stage of adding pseudo-random noise can be performed before the neural network inference of the VTNN. This stage may include initializing a pseudo-random number generator with a seed value that has been determined as described in other embodiments. Furthermore, this stage may include one or more of the following: - Derive pseudo-random numbers for samples of the input images. - Scale pseudo-random numbers to a range of values ​​that can depend on the range of sample values ​​(e.g., it can be a range of integer values ​​with a given bit depth or floating-point values ​​from 0 to 1). - Add scaled pseudo-random numbers to the sample values, and - Trim the obtained sample values ​​to the range of sample values.

[0428] In one embodiment, pseudo-random noise addition and neural network inference of the VTNN are performed iteratively so that the NNPF-generated images from the previous iteration are used as input for the next iteration.

[0429] In one embodiment, pseudo-random noise is first added to the input images(s) of the VTNN. Then, neural network inference of the VTNN is performed iteratively so that the images generated by the NNPF in previous iterations are used as input for the next iteration.

[0430] In one embodiment, the DI information or TEC information includes characteristics related to the randomness of the VTNN, which may include, but are not limited to, one or more of the following: - Multiple ranges of pseudo-random noise values ​​added to samples of (multiple) input images. A range of values ​​can be provided for all color components together, or separate ranges can be provided for luminance and chrominance, or separate ranges can be provided for each color component (such as Y, Cb, and Cr). - Distribution function of pseudo-random noise. - The number of iterations for neural network inference.

[0431] In one embodiment, at least a portion of the signaling information, or at least a portion of the TEC information, or at least a portion of the DI information may be included in the Neural Network Post-Filter Activation (NNPFA) SEI message.

[0432] In one example, information indicating the seed value can be included in the NNPFA SEI message, where the seed value or data derived from the seed value is used as input or auxiliary input for NNPF. In another example, information indicating whether NNPF uses the seed value as input or auxiliary input can be included in the NNPFA SEI message.

[0433] In another example, the DI information includes flags indicating the quantization version of the VTNN used.

[0434] In another example, the DI information includes one or more quantization parameters that will be used to quantize the VTNN.

[0435] In another or alternative embodiment, DI information may be carried in an NNPFA SEI message (or more generally, FW signaling).

[0436] In one example, the VTNN can be responsible for generating two or more parts or aspects of an image, such as two faces. The NNPFA SEI message or the GFV SEI message can carry two or more sets of DI information used to generate two or more parts or aspects of the image, such as two or more pseudo-random seeds.

[0437] Feature: Extrapolation output type

[0438] In one embodiment, TEC information may include information indicating the type of extrapolated output.

[0439] In another embodiment, the information indicating the type of extrapolation output may include one or more categories of visual content that the NN can extrapolate.

[0440] In one example, one or more categories include the following categories: - Face; - (multiple) hands; and / or - Body.

[0441] In one example, the VTNN performing visual temporal extrapolation only supports extrapolating the face. In another example, the VTNN performing visual temporal extrapolation supports extrapolating both the face and hands.

[0442] Sequence-level adaptation

[0443] In one embodiment, signaling information may include an indication of whether the VTNN is applicable to all images to which it is applied or activated.

[0444] Example use case 1: Specialization of generated content

[0445] In one embodiment, the VTNN can be adapted to specialize certain aspects of the generated output, such as for a specific face. In one example, the VTNN may be trained on a large dataset and may not perform well on all possible faces; the VTNN can be adapted by providing some additional information as auxiliary input, such as the unique facial features of the person whose face will be extrapolated over time. In another example, the VTNN can be adapted to generate faces under specific lighting conditions by providing additional information about lighting characteristics and ambient hue as auxiliary input.

[0446] Example Use Case 2: Task Specialization

[0447] In one embodiment, a VTNN can be adapted to be specialized for a particular task or purpose. In one example, when the purpose of the VTNN is visual temporal extrapolation, the VTNN can be specialized for extrapolating in a certain visual or artistic style. In another example, when the purpose of the VTNN is visual generative AI (e.g., a VTNN capable of generating visual content), the VTNN can be specialized for adding glasses to a face in an image. In yet another example, when the purpose of the VTNN is visual generative AI (e.g., a VTNN capable of generating visual content), the VTNN can be specialized for performing visual temporal extrapolation.

[0448] Example use case 3: Specialization regarding what to generate and what not to generate

[0449] In one embodiment, signaling information may include: an input signal instructing the VTNN to apply to all images for its application, the signal indicating that the VTNN should not modify or generate one or more aspects. In one example, the signaling instructs the VTNN not to modify background information. In another example, the signaling instructs the VTNN not to modify faces.

[0450] In one embodiment, signaling information may include: an input signal instructing the VTNN to apply to all images for its application, the signal indicating that the VTNN should only modify or generate one or more aspects. In one example, the signaling instructs the VTNN to modify only background information. In another example, the signaling instructs the VTNN to modify only faces. In yet another example, the signaling instructs the VTNN to horizontally expand frames (e.g., from a normal image to a 360-degree view).

[0451] Signaling to notify sequence-level adaptation to be executed

[0452] In one embodiment, signaling information may include an indication that the NN is sequence-level adapted.

[0453] In an example implementation, a new nnpfc_mode_idc value (e.g., 2) can be used in the NNPFC SEI message. When nnpfc_mode_idc equals 2, the NN is sequence-level adapted.

[0454] The purpose of sending signals to notify the adaptation

[0455] In one embodiment, signaling information may include an indication of the purpose of sequence-level adaptation. The standard specification may specify several purposes for sequence-level adaptation, which may be referred to as adaptation purposes, where each adaptation purpose may be associated with an identifier value; the encoder may signal the identifier value used to identify the adaptation purpose. In one example, an adaptation purpose may include specializing the NN for one or more specific faces appearing in the video. In another example, an adaptation purpose may include specializing the NN for a more specific task than indicated by the NN's general purpose (e.g., nnpfc_purpose in the NNPFC SEI message).

[0456] For example, the syntax element name could be nnpfc_adaptation_purpose.

[0457] In an example embodiment, the adaptation purpose can be indicated as a bit field, wherein a specific bit position is assigned to a specific adaptation purpose, and the bit value in the specific bit position indicates whether the corresponding specific adaptation purpose is within the scope of the defined neural network's adaptation purposes.

[0458] How is the signal notification adaptation executed?

[0459] In one embodiment, the signaling information may include an indication of whether the VTNN is adapted to all images to which it is applied by inputting a persistent auxiliary input (PAI) signal.

[0460] In one embodiment, signaling information may include an indication of the type of PAI signal, such as: - Tensor; - Text (e.g., a text prompt); or - Images (e.g., image cues).

[0461] For example, the syntax element name could be nnpfc_persistent_aux_input_type.

[0462] Example of the number of NNPFC SEI messages required to perform the adaptation

[0463] In one embodiment, sequence-level adaptation of the NN can be indicated using a single NNPFC SEI message associated with the NN. The syntax element nnpfc_base_flag is set to 1.

[0464] In another embodiment, the sequence-level adaptation of the NN can be indicated in a second NNPFC SEI message associated with the NN, wherein the second NNPFC SEI message has the same nnpfc_id as the first NNPFC SEI message present in the same CLVS. The first NNPFC SEI message has an nnpfc_base_flag equal to 1, and the second NNPFC SEI message has an nnpfc_base_flag equal to 0. In addition to the second NNPFC SEI message, other NNPFC SEI messages may indicate sequence-level adaptations of the same NN. The adapted NN is applied to all images in which the NN is activated with the nnpfc_id value and nnpfa_target_base_flag equal to 0, and the specific NNPFC SEI message where nnpfc_base_flag equal to 0 persists. Thus, more than one sequence-level adaptation or specialization of the base NN can be indicated.

[0465] In one embodiment, the signaling information may include the purpose of the base model. In one example, the base model is a machine learning model, such as a neural network, that has been trained on a large dataset for general tasks, such as next word prediction, next pixel prediction, or next frame prediction, inpainting, interpolation, etc. The base model may be specialized to perform more specific tasks, such as through one or more of the following: further training, fine-tuning, or adaptation based on input instructions or prompts.

[0466] In one embodiment, when the purpose of signaling by a first SEI message (such as a first NNPFC SEI message) includes one of the following: base model, visual temporal extrapolation, or generative AI, it may be necessary for the encoder to signal a second SEI message, such as a second NNPFC SEI message, indicating sequence-level adaptation. In one example, a neural network whose characteristics (and the weights of the final neural network) are signaled in the first NNPFC SEI message cannot be used in inference until the second NNPFC SEI message indicating sequence-level adaptation of that neural network is signaled.

[0467] Figure 5This is an example device, which can be implemented in hardware, and is enabled to encode, decode, or signal media data using generative artificial intelligence. Device 500 includes at least one processor 502 (e.g., an FPGA and / or a CPU), and one or more memories 504 including computer program code 505 having instructions for performing the methods described herein. The at least one memory 504 and the computer program code 505 are configured, together with at least one processor 502, to cause device 500 to implement circuitry, processes, components, modules, or functions (implemented by control module 506) to implement the examples described herein, including enabling the encoding, decoding, or signaling of media data using generative artificial intelligence. Optionally, an encoder 580 of control module 506 performs encoding, and optionally, a decoder 590 performs decoding. Memory 504 may be a non-transitory memory, a transient memory, a volatile memory (e.g., RAM), or a non-volatile memory (e.g., ROM).

[0468] Device 500 includes a display and / or I / O interface 508, which includes user interface (UI) circuitry and components. This display and / or I / O interface can be used to display features or states of the methods described herein (e.g., while one of these methods is being performed or at a subsequent time), or to receive input from a user, such as using a keypad, camera, touchscreen, touch area, microphone, biometrics, one or more sensors, etc. Device 500 includes one or more communication (e.g., network (N / W)) interfaces (I / F) 510. The communication I / Fs 510 can be wired and / or wireless and communicate over the Internet / (multiple) other networks via any communication technology, including via one or more links 524. The communication I / Fs 510 can include one or more transmitters or one or more receivers.

[0469] Transceiver 516 includes one or more transmitters 518 and one or more receivers 520. Transceiver 516 and / or (multiple) communication I / F 510 may include standard, well-known components such as amplifiers, filters, frequency converters, modulators (demodulators), and encoder / decoder circuitry, as well as one or more antennas, such as antenna 514 for communication via wireless link 522.

[0470] The control module 506 of device 500 includes one or both of components 506-1 and / or 506-2, which can be implemented in various ways. Control module 506 can be implemented in hardware as control module 506-1, such as being implemented as part of one or more processors 502. Control module 506-1 can also be implemented as an integrated circuit or by other hardware such as a programmable gate array. In another example, control module 506 can be implemented as control module 506-2, which is implemented as computer program code (with corresponding instructions) 505 and executed by one or more processors 502. For example, one or more memories 504 store instructions that, when executed by one or more processors 502, cause device 500 to perform one or more operations as described herein. Furthermore, one or more processors 502, one or more memories 504, and example algorithms encoded as instructions, programs, or code (e.g., as flowcharts and / or signaling diagrams) are means for performing the operations described herein.

[0471] The means 500 used to implement the functions of the control module 506 may correspond to any means described herein. Alternatively, the means 500 and its elements may not correspond to any other means described herein, as the means 500 may be part of an self-organizing / optimized network (SON) node or other node, such as a node in the cloud.

[0472] Device 500 can also be distributed throughout the network (e.g., Internet 28), including within and between device 500 and any network elements (such as base station 24 and / or device 90).

[0473] like Figure 5 As shown, interface 512 enables data communication and signaling between the various components of device 500. For example, interface 512 may be one or more buses, such as address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optic cables, or other optical communication devices. Computer program code (e.g., instructions) 505, including control 506, may include object-oriented software configured to pass data or messages between objects within computer program code 505. Device 500 does not need to include every feature described above, or may include other features. The various components of device 500 may be at least partially located in a common housing 528, or subsets of the various components of device 500 may be at least partially located in different housings, which may include housing 528.

[0474] Figure 6A schematic diagram is shown of a non-volatile storage medium 600a (e.g., a computer / optical disc (CD) or digital universal optical disc (DVD)) and 600b (e.g., a Universal Serial Bus (USB) Memory Stick) and 600c (e.g., cloud storage for downloading instructions and / or parameters 602 or receiving email instructions and / or parameters 602), which, when executed by a processor, allow the processor to perform one or more operations of the methods described herein.

[0475] Figure 7 This is an example method 700 for implementing the embodiments described herein, according to embodiments. At 702, method 700 includes: encoding signaling information in or along a bitstream. The signaling information enables or supports the performance of visual temporal extrapolation. At 704, method 700 includes: signaling the signaling information.

[0476] Method 700 can be performed using the apparatus described herein, for example, Figures 1 to 5 Any device Figure 10 Any device thereof, or any other device described herein.

[0477] Figure 8 This is another example method 800 for implementing the embodiments described herein, according to embodiments. At 802, method 800 includes receiving signaling information in or along a bitstream, the signaling information enabling or supporting visual temporal extrapolation. At 804, method 800 includes decoding the signaling information to obtain decoded signaling information. The decoded signaling information may simply be referred to as signaling information. At 806, method 800 includes performing visual temporal extrapolation based at least on the decoded signaling information.

[0478] Method 800 can be performed using the apparatus described herein, for example, Figures 1 to 5 Any device Figure 10 Any device thereof, or any other device described herein.

[0479] Figure 9 This is yet another example method 900 for implementing the embodiments described herein, according to embodiments. At 902, method 900 includes: inferring a temporally interleaved frame packing arrangement in a bitstream. At 904, method 900 includes: inferring a constitutive frame parity, i.e., the current frame in which the time extrapolation filter is activated includes a temporally interleaved frame packing arrangement. At 906, method 900 includes: selecting one or more input frames for the time extrapolation filter, the one or more input frames including the inferred constitutive frame parity. At 908, method 900 includes: applying the time extrapolation filter as input to the one or more input frames.

[0480] Method 900 can be performed using the apparatus described herein, for example, Figures 1 to 5 Any device Figure 10 Any device thereof, or any other device described herein.

[0481] refer to Figure 10 This diagram illustrates a possible and non-limiting example in which practical examples can be exemplified. It shows a user equipment (UE) 110, a radio access network (RAN) node 170, and (multiple) network elements 190. Figure 1 In the example, User Equipment (UE) 110 wirelessly communicates with Wireless Network 100. The UE is a wireless device that can access Wireless Network 100. UE 110 includes one or more processors 120, one or more memories 125, and one or more transceivers 130 interconnected via one or more buses 127. Each of the one or more transceivers 130 includes a receiver Rx 132 and a transmitter Tx 133. The one or more buses 127 may be address, data, or control buses and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, optical fiber, or other optical communication devices. The one or more transceivers 130 are connected to one or more antennas 128. The one or more memories 125 include computer program code 123. UE 110 includes a module 140, which includes one or both of portions 140-1 and / or 140-2, and this module may be implemented in various ways. Module 140 may be implemented in hardware as module 140-1, such as being implemented as part of one or more processors 120. Module 140-1 can also be implemented as an integrated circuit or via other hardware such as a programmable gate array. In another example, module 140 can be implemented as module 140-2, which is implemented as computer program code 123 and executed by one or more processors 120. For example, one or more memories 125 and computer program code 123 can be configured, together with one or more processors 120, to cause user equipment 110 to perform one or more of the operations described herein. UE 110 communicates with radio access network (RAN) node 170 via radio link 111.

[0482] In this example, RAN node 170 is a base station that provides access to wireless network 100 for wireless devices such as UE 110. RAN node 170 can be, for example, a base station for fifth-generation cellular network technology (5G), also known as New Radio (NR). In 5G, RAN node 170 can be an NG-RAN node, which is defined as a gNB (e.g., a base station for 5G / NR, such as a node providing NR user plane and control plane protocol termination to the UE, and connected to the 5GC via an NG interface) or an ng (next-generation)-eNB. A gNB is a node providing NR user plane and control plane protocol termination to the UE and connected to the 5G core network (5GC) (e.g., (multiple) network elements 190) via an NG interface. An ng-eNB is a node providing evolved Universal Terrestrial Radio Access (E-UTRA), such as LTE radio access technology, user plane and control plane protocol termination, and connected to the 5GC via an NG interface. An NG-RAN node may include multiple gNBs, which may also include a central unit (CU) (gNB-CU) 196 and (multiple) distributed units (DUs) (gNB-DUs), of which DU 195 is shown. Note that a DU may include or be coupled to and control a radio unit (RU). A gNB-CU is a logical node that carries the radio resource control (RRC), serving data adaptation protocol (SDAP), and PDCP protocols of a gNB, or carries the RRC and packet data convergence protocol (PDCP) protocols of an en-gNB (e.g., a node that provides NR user plane and control plane protocol termination to the UE and acts as an auxiliary node in E-UTRA-NR dual connectivity (EN-DC). The gNB-CU controls the operation of one or more gNB-DUs. The gNB-CU terminates the interface between the CU and the DU control interface (F1 or F1-C) connected to the gNB-DU. The F1 interface is shown as reference numeral 198, although reference numeral 198 also shows links between remote elements of RAN node 170 and centralized elements of RAN node 170, such as the link between gNB-CU 196 and gNB-DU 195. A gNB-DU is a logical node carrying the Radio Link Control (RLC), MAC, and Physical Layer (PHY) layers of a gNB or en-gNB, and its operation is partially controlled by the gNB-CU. One gNB-CU supports one or more cells. A cell is supported by only one gNB-DU. The gNB-DU terminates at the F1 interface 198 connected to the gNB-CU. Note that DU 195 is considered to include transceiver 160, for example, as part of an RU; however, some examples in this regard could allow transceiver 160 to be part of a separate RU, for example, under the control of and connected to DU 195.RAN node 170 can also be an eNB (evolved NodeB) base station for, for example, Long Term Evolution (LTE), or any other suitable base station or node.

[0483] RAN node 170 includes one or more processors 152, one or more memories 155, one or more network interfaces (N / WI / F) 161, and one or more transceivers 160 interconnected via one or more buses 157. Each of the one or more transceivers 160 includes a receiver Rx 162 and a transmitter Tx 163. The one or more transceivers 160 are connected to one or more antennas 158. The one or more memories 155 include computer program code 153. CU 196 may include processor(s) 152, memories 155, and network interfaces 161. Note that DU 195 may also include its own memory(s) and processor(s), and / or other hardware, but these are not shown.

[0484] RAN node 170 includes module 150, which comprises one or both of portions 150-1 and / or 150-2, and can be implemented in various ways. Module 150 can be implemented in hardware as module 150-1, such as being implemented as part of one or more processors 152. Module 150-1 can also be implemented as an integrated circuit or via other hardware such as a programmable gate array. In another example, module 150 can be implemented as module 150-2, which is implemented as computer program code 153 and executed by one or more processors 152. For example, one or more memories 155 and computer program code 153 are configured, together with one or more processors 152, to cause RAN node 170 to perform one or more operations as described herein. Note that the functionality of module 150 can be distributed, such as distributed between DU 195 and CU 196, or implemented solely in DU 195.

[0485] One or more network interfaces 161 communicate over a network, such as via links 176 and 131. Two or more gNBs 170 may communicate using, for example, link 176. Link 176 may be wired, wireless, or both, and may implement, for example, an Xn interface for 5G, an X2 interface for LTE, or other suitable interfaces for other standards.

[0486] One or more buses 157 may be address, data, or control buses and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, optical fiber or other optical communication equipment, wireless channels, etc. For example, one or more transceivers 160 may be implemented as a Remote Radio Header (RRH) 195 for LTE or a Distributed Unit (DU) 195 for a gNB implementation of 5G, wherein other elements of the RAN node 170 may be physically located in a different location from the RRH / DU 195, and one or more buses 157 may be partially implemented as, for example, fiber optic cables or other suitable network connections for connecting other elements of the RAN node 170 (e.g., Central Unit (CU), gNB-CU) to the RRH / DU 195. Reference numeral 198 also indicates these suitable network links(s).

[0487] Note that the description in this document indicates that a 'cell' performs a function; however, it should be clear that the equipment forming the cell can perform this function. A cell constitutes part of a base station. That is, each base station can have multiple cells. For example, a single carrier frequency and associated bandwidth can have three cells, each covering one-third of a 360-degree area, thus the coverage area of ​​a single base station is approximately elliptical or circular. Furthermore, each cell can correspond to a single carrier, and a base station can use multiple carriers. So if each carrier has three 120-degree cells and there are two carriers, the base station has a total of six cells.

[0488] Wireless network 100 may include one or more network elements 190, which may include core network functions and provide connectivity to additional networks such as telephone networks and / or data communication networks (e.g., the Internet) via one or more links 181. Such core network functions for 5G may include Access and Mobility Management Functions (AMF) and / or User Plane Functions (UPF) and / or Session Management Functions (SMF). Such core network functions for LTE may include MME (Mobility Management Entity) / SGW (Serving Gateway) functions. These are merely illustrative functions that can be supported by network elements(s) 190, and it should be noted that both 5G and LTE functions can be supported. RAN node 170 is coupled to network element 190 via link 131. Link 131 may be implemented as, for example, an NG interface for 5G, or an S1 interface for LTE, or other suitable interfaces for other standards. Network element 190 includes one or more processors 175, one or more memories 171, and one or more network interfaces (N / WI / F) 180 interconnected via one or more buses 185. One or more memories 171 include computer program code 173. The one or more memories 171 and computer program code 173 are configured, together with one or more processors 175, to cause network element 190 to perform one or more operations.

[0489] Wireless network 100 can implement network virtualization, a process that combines hardware and software network resources and network functions into a single software-based managed entity (i.e., a virtual network). Network virtualization involves platform virtualization, which is often used in conjunction with resource virtualization. Network virtualization is divided into external network virtualization and internal network virtualization. External network virtualization combines many networks or network components into virtual units, while internal network virtualization provides network-like functionality to software containers on a single system. Note that the virtualized entities created by network virtualization are still implemented to some extent using hardware such as processors 152 or 175 and memories 155 and 171, and such virtualized entities also produce technical effects.

[0490] Computer-readable storage devices 125, 155, and 171 can be of any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, magnetic storage devices and systems, optical storage devices and systems, fixed storage, and removable storage. Computer-readable storage devices 125, 155, and 171 can be components for performing storage functions. Processors 120, 152, and 175 can be of any type suitable for the local technical environment and, by way of non-limiting example, can include one or more of general-purpose computers, special-purpose computers, microprocessors, digital signal processors (DSPs), and processors based on multi-core processor architectures. Processors 120, 152, and 175 can be components for performing functions such as control of UE 110, RAN node 170, network element(s) 190, and other functions described herein.

[0491] Typically, various embodiments of user equipment 110 may include, but are not limited to, cellular phones with wireless communication capabilities (such as smartphones, tablets, personal digital assistants (PDAs)), portable computers with wireless communication capabilities, image capture devices with wireless communication capabilities (such as digital cameras), gaming devices with wireless communication capabilities, music storage and playback devices with wireless communication capabilities, internet devices that allow wireless internet access and browsing, tablets with wireless communication capabilities, and portable units or terminals that combine such functions.

[0492] Modules 140-1, 140-2, 150-1, and 150-2 can be enabled to encode, decode, or signal media data using generative artificial intelligence. Computer program code 173 can also be enabled to encode, decode, or signal media data using generative artificial intelligence.

[0493] As mentioned above, Figures 7 to 9 Flowcharts include apparatus (e.g., 50, 500, or any other apparatus described herein), methods, and computer program products according to certain example embodiments. It should be understood that each block of the flowchart, and combinations of blocks in the flowchart, can be implemented in various ways, such as hardware, firmware, processors, circuit systems, and / or other devices associated with the execution of software including one or more computer program instructions. For example, one or more processes described above can be embodied by computer program instructions. In this regard, computer program instructions embodying the processes described above can be stored in the memory (e.g., 58, 125, or 504) of an apparatus employing embodiments of the invention and executed by the processing circuitry of that apparatus (e.g., 56, 120, or 502). It should be understood that any such computer program instructions can be loaded onto a computer or other programmable device (e.g., hardware) to produce a machine that causes the resulting computer or other programmable device to perform the functions specified in the flowchart blocks. These computer program instructions can also be stored in a computer-readable storage medium that can instruct a computer or other programmable device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of art whose execution performs the functions specified in the flowchart blocks. Computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operations to be performed on the computer or other programmable device, thereby producing a computer-implemented process, such that the instructions, which execute in the computer or other programmable device, provide operations for implementing the function specified in the flowchart box.

[0494] Therefore, when computer program instructions (such as computer-readable program code portions) are stored by at least one non-transitory computer-readable storage medium, a computer program product is defined, wherein the computer program instructions (such as computer-readable program code portions) are configured, upon execution, to perform the aforementioned functions, such as combining... Figures 7 to 9 The flowchart(s) are as follows. In other embodiments, computer program instructions (such as computer-readable program code portions) do not need to be stored or otherwise embodied by a non-transitory computer-readable storage medium, but may be embodied by a transient medium, wherein the computer program instructions (such as computer-readable program code portions) are still configured to perform the above-described functions when executed.

[0495] Therefore, the boxes in a flowchart support combinations of components used to perform a specified function and combinations of operations used to perform a specific function. It should also be understood that one or more boxes in a flowchart, as well as combinations of boxes in a flowchart, can be implemented by a dedicated hardware-based computer system or a combination of dedicated hardware and computer instructions that performs the specified function.

[0496] In some embodiments, certain operations described above may be modified or further extended. Furthermore, in some embodiments, additional optional operations may be included. Modifications, additions, or extensions to the operations described above may be made in any order and in any combination.

[0497] Some embodiments related to performing visual temporal extrapolation have been described. It should be understood that the embodiments can be implemented using any generative modeling neural network.

[0498] Some embodiments have been described in conjunction with specific SEI messages, such as GFV SEI messages, NNPFC SEI messages, and / or NNPFA SEI messages. It should be understood that the embodiments are not limited to these specific SEI messages, but can be implemented with any similar SEI message.

[0499] Some embodiments have been described in conjunction with SEI messages. It should be understood that the embodiments are not limited to SEI messages, but can be implemented using any similar syntax structure, such as metadata OBU.

[0500] Some embodiments have been described for certain terms (such as picture units) applicable to some video coding formats (such as VVC). It should be understood that the embodiments are not limited to the video coding formats to which the terms apply, but can be implemented with any video coding format having its respective terms.

[0501] In the preceding text, some example embodiments have been described using the syntax of bitstreams. However, it should be understood that corresponding structures and / or computer programs may reside at the encoder used to generate the bitstream and / or the decoder used to decode the bitstream.

[0502] In the foregoing description of an example embodiment with reference to the encoder, it should be understood that the resulting bitstream and decoder have corresponding elements. Similarly, in the description of an example embodiment with reference to the decoder, it should be understood that the encoder has a structure and / or computer program for generating the bitstream to be decoded by the decoder.

[0503] Benefiting from the teachings given in the foregoing description and the accompanying drawings, those skilled in the art will be able to conceive of many modifications and other embodiments of the invention described herein. Therefore, it should be understood that the invention is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Furthermore, although the foregoing description and accompanying drawings have described exemplary embodiments in the context of certain example combinations of elements and / or functions, it should be understood that alternative embodiments may provide different combinations of elements and / or functions without departing from the scope of the appended claims. In this regard, for example, as described in some of the appended claims, combinations of elements and / or functions different from those explicitly described above may also be contemplated. Therefore, this specification is intended to cover all such alternatives, modifications, and variations that fall within the scope of the appended claims. Although specific terms are used herein, they are used only in a general and descriptive sense and not for limiting purposes.

[0504] It should be understood that the above description is illustrative only. Those skilled in the art can devise various alternatives and modifications. For example, the features recited in the various dependent claims can be combined with each other in any suitable combination. Furthermore, features of the different embodiments described above can be selectively combined to form new embodiments. Therefore, this specification is intended to cover all such alternatives, modifications, and variations falling within the scope of the appended claims.

[0505] References to 'computer', 'processor', etc., should be understood to include not only computers with different architectures, such as single / multiprocessor architectures and sequential (von Neumann) / parallel architectures, but also special-purpose circuits, such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), signal processing devices, and other processing circuit systems. References to computer programs, instructions, code, etc., should be understood to include the software or firmware of programmable processors, such as the programmable content of hardware devices, such as processor instructions, or the configuration settings of fixed-function devices, gate arrays, or programmable logic devices.

[0506] As used herein, the term 'circuit system' can refer to any of the following: (a) a hardware circuit implementation, such as an implementation in an analog and / or digital circuit system; and (b) a combination of circuitry and software (and / or firmware), such as (if applicable): (i) a combination of (multiple) processors, or (ii) portions of (multiple) processors / software (including (multiple) digital signal processors), software, and (multiple) memories, which work together to enable a device to perform various functions; and (c) circuitry that requires software or firmware to operate (even if the software or firmware does not physically exist), such as (multiple) microprocessors or portions of (multiple) microprocessors. This description of 'circuit system' applies to the use of the term in this application. As another example, as used herein, the term 'circuit system' also covers an implementation of only one processor (or multiple processors) or a portion of a processor and its accompanying software and / or firmware. For example, if applicable to a particular element, the term 'circuit system' also covers a baseband integrated circuit or application processor integrated circuit for a mobile phone, or a similar integrated circuit in a server, cellular network device, or another network device.

[0507] Circuit system or circuit: As used in this application, the terms "circuit system" or "circuit" may refer to one or more or all of the following: (a) Hardware circuit implementation only (such as implementation only in analog and / or digital circuit systems); and (b) A combination of hardware circuitry and software, such as (if applicable): (i) A combination of (multiple) analog and / or digital hardware circuits with software / firmware; and (ii) Any part of a hardware processor (including digital signal processors), software, and memory (including multiple memory) that works together to enable a device such as a mobile phone or server to perform various functions; and (c) (Multiple) hardware circuits and / or (multiple) processors, such as (multiple) microprocessors or a portion thereof, which require software (e.g., firmware) to function, but may be absent when operation does not require the software.

[0508] This definition of circuit system applies to all uses of the term in this application, including in any claim. As another example, as used in this application, the term circuit system also covers only hardware circuitry or a processor (or multiple processors) or a portion of hardware circuitry or a processor and its accompanying software and / or firmware. For example, if applicable to a particular claim element, the term circuit also covers baseband integrated circuits or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

Claims

1. An apparatus comprising at least one processor; and at least one memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Encoding signaling information in or along a bitstream, wherein the signaling information enables or supports visual temporal extrapolation; and The signaling information is sent to notify the sender.

2. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Receive signaling information in or along the bitstream, the signaling information enabling or supporting the execution of visual time extrapolation; Decode the signaling information to obtain decoded signaling information (signaling information); and The visual time extrapolation is performed based at least on the decoded signaling information.

3. The apparatus according to any one of claims 1 or 2, wherein one or more neural networks (VTNNs) are used to perform the visual temporal extrapolation.

4. The apparatus according to any one of claims 1 to 2, wherein the signaling information is included in: a Neural Network Post-Filter Feature (NNPFC) Supplemental Enhancement Information (SEI) message; a Generative Face Video (GFV) SEI message; or a Neural Network Post-Filter Feature (NNPFC) SEI message and a GFV SEI message.

5. The apparatus according to any one of claims 1 to 4, wherein the signaling information includes one or more of the following: The purpose of performing the visual temporal extrapolation is on the decoder side; Temporal extrapolation (TEC) information of neural networks that characterize execution time extrapolation; or The purpose of generative modeling or generative artificial intelligence (AI) on the decoder side.

6. The apparatus according to any one of claims 1 to 5, wherein the TEC information is signaled based on one or more of the following: When the signaling information indicates that the purpose of the VTNN associated with the signaling information includes visual temporal extrapolation, generative modeling, or generative artificial intelligence (AI), wherein the generative modeling or the generative AI refers to using the VTNN to generate data conditioned on one or more inputs; or When the stated purpose includes generative facial videos or generative videos.

7. The apparatus of claim 6, wherein the purpose of the generative facial video includes: Given one or more of the following, generate or improve at least a portion of a video or image that includes at least one or more faces: The coordinates of at least one or more faces; The coordinates of at least one or more facial landmarks or key points; A transformation matrix describing the motion or temporal changes of one or more key points of the at least one or more faces; Facial expression features; Blinking signal; Text associated with the video; The audio associated with the video; or One or more other images.

8. The apparatus according to any one of claims 5 to 7, wherein the TEC information includes one or more of the following: Information indicating the number of images extrapolated during a single activation or inference of the VTNN; Information indicating the time interval between the extrapolated image and the latest image used as input for the inference; Information indicating whether the VTNN uses auxiliary input that indicates an output instance of one or more images extrapolated in time; The VTNN is indicated by information indicating facial parameters or auxiliary input derived from facial parameters, wherein the auxiliary input is used by the VTNN to perform temporal extrapolation of one or more faces in an image or video; Indicates whether the VTNN adopts information from the output of another VTNN, which is run or executed on the decoder side; Indicates whether the VTNN adopts auxiliary input information derived from the output of the other VTNN, which converts or translates facial parameters into a format accepted or required by the VTNN; Information indicating whether the VTNN uses auxiliary input, which represents a tensor generated on the encoder side or derived from the tensor; Indicates whether the VTNN uses information representing audio or speech or auxiliary input derived from the audio or speech; When the auxiliary input represents the audio or the speech, information about the characteristics of the audio or the speech; Indicates whether the VTNN uses information representing text or auxiliary input derived from the text; Information indicating whether different inferences or executions of the VTNN under the same input produce different outputs; Information indicating the type or cause of the variability in the VTNN's output; Information indicating whether the VTNN is indicated or operated in a manner that does not produce output variability; Information indicating whether the VTNN is allowed to have output variability; This prevents the network from having information with output variability; Information indicating the type of extrapolated output; Information indicating whether the VTNN uses auxiliary input, wherein the auxiliary input indicates: the difference in image order count (POC) of the output image to be generated or extrapolated, the difference in timestamp, the relative output position, or the relative output difference; Information indicating the format of the facial parameters accepted or required by the VTNN; Information regarding the format of the tensor; Information indicating whether the VTNN uses auxiliary input, which represents text associated with a person's speech, and the person's face will be extrapolated in time; Indicates whether the VTNN uses auxiliary input information representing text; Information indicating whether the VTNN produces different outputs for different inferences or executions of the VTNN using one or more of the same inputs; Information indicating whether the input of the VTNN consists of background information; Information indicating whether the input of the VTNN consists of foreground information; Indicates whether the input of the VTNN includes information about the driving image; Information indicating whether the auxiliary input of the VTNN is derived from the lossy encoding process; Indicates whether the VTNN accepts input information, and the input controls one or more aspects of how the final output is determined based on the output of the VTNN; One or more indicators used to indicate which facial parameters are present and / or which facial parameters are used by the VTNN; The number of extrapolated images for the neural network post-filter (NNPF) varies for each inference, and the number of extrapolated images for each inference is indicated using another signaling mechanism. An indication of whether the decoder side will generate a random seed value for the VTNN or derive a pseudo-random seed value based on other information signaled by the encoder; An indication of whether the pseudo-random seed value is derived based on the value of a specific syntax element, which is included in the TEC information or otherwise signaled to the decoder; An indication of whether the VTNN, as specified by the TEC information, will use the seed value as an input or an auxiliary input; An indication of whether the VTNN, as specified by the TEC information, will use the pseudo-random seed as the input or the auxiliary input; or Regarding the randomness characteristic of the VTNN.

9. The apparatus of claim 8, wherein the other value comprises one of the following: The sequence-level quantization parameter (QP) value, wherein the pseudo-random seed value is set to be equal to the sequence-level QP value; Slice-level QP value, wherein the pseudo-random seed value is set to be equal to the slice-level QP value; The values ​​of the syntax elements in the TEC information; Specify the values ​​of syntax elements in the neural network post-filter feature (NNPFC) supplemental enhancement information (SEI) message of the VTNN; or Specify the value of the syntax element in the Generative Face Video (GFV) SEI message of the VTNN.

10. The apparatus of claim 8, wherein the facial parameters used as auxiliary input or used to derive the auxiliary input are carried in one of the following: Newly defined SEI message; Generative Face Video (GFV) SEI message; Neural Network Post-Filter Activation (NNPFA) SEI message; or Neural Network Post-Filter Input (NNPFI) SEI message.

11. The apparatus of claim 8, wherein the tensor is the output of another neural network that is run or executed on the encoder side, or is derived from the other neural network.

12. The apparatus of claim 11, wherein the tensor is signaled within the NNPFA SEI message, the GFV SEI message, or the NNPFI SEI message.

13. The apparatus of claim 8, wherein the information regarding the characteristics of the audio or speech includes one or more of the following: The sampling rate of the audio; The bit depth and / or dynamic range of the audio; The number of audio channels and / or channel configuration of the audio; The duration of the audio and / or the number of audio samples will be included in the input tensor; Information regarding whether the audio was processed using a source separation algorithm; Information regarding whether the audio is spatial audio; or The timing of the audio provided as input relative to one or more images to be extrapolated.

14. The apparatus of claim 8, wherein the type or cause of the output variability includes one of the following: Pseudo-randomness; Intentional alteration; or A floating-point format used to represent the values ​​of one or more parameters of the VTNN and / or the values ​​of one or more signals flowing through the VTNN.

15. The apparatus of claim 8, wherein the information indicating whether the VTNN adopts an auxiliary input indicating an output instance of one or more images extrapolated in time is carried in the NNPFC SEI message or the Neural Network Post-Filter Activation (NNPFA) SEI message.

16. The apparatus of claim 8, wherein the VTNN is signaled to use the information of the auxiliary input indicating the POC difference, timestamp difference, relative output position, or relative output difference as part of a Neural Network Post-Filter Feature (NNPFC) Supplemental Enhancement Information (SEI) message.

17. The apparatus of any one of claims 8 or 16, wherein the auxiliary input indicating the POC difference, timestamp difference, relative output position, or relative output difference is carried in a Neural Network Post-Filter Activation (NNPFA) Supplemental Enhancement Information (SEI) message or a Neural Network Post-Filter Input (NPFI) SEI message.

18. The apparatus of claim 8 or 10, wherein the facial parameters include one or more of the following: 2D key points; 3D key points; or One or more matrices.

19. The apparatus of claim 10, wherein the facial parameters obtained from the NNPFA SEI message, the GFV SEI message, or the NNPFI SEI message are converted into the desired format indicated in the TEC information.

20. The apparatus of any one of claims 8 or 13, wherein information indicating whether the VTNN uses the auxiliary input representing the audio or the speech or derived from the audio or the speech to generate or synthesize one or more facial features.

21. The apparatus of claim 8, wherein when the VTNN employs the auxiliary input representing the text, the TEC information further includes information indicating characteristics of the text, the characteristics including one or more of the following: Does the VTNN accept or require the raw text? Does the VTNN accept or require lexical units? Does the VTNN accept or require text embedding? Which word segmenter to use; Which text embedder to use; or The timing of the text provided as input relative to one or more images to be extrapolated.

22. The apparatus of claim 8, wherein the information indicating the extrapolation output type includes one or more categories of visual content that the VTNN can extrapolate.

23. The device of claim 22, wherein one or more categories include face, hand, or body.

24. The apparatus according to any one of claims 3 to 23, wherein the signaling information further includes an indication of whether the VTNN is adapted for all images to which the VTNN is applied and is activated, or whether the VTNN is sequence-level adapted.

25. The apparatus of claim 24, wherein the signaling information further comprises: Indicates the purpose of the sequence-level adaptation.

26. The apparatus of claim 24 or 25, wherein the persistent auxiliary input (PAI) signal included in the signaling information is used to indicate whether the VTNN is applicable to all images to which the VTNN is applied.

27. The apparatus of claim 26, wherein the signaling information further comprises: Indicates the type of the PAI signal.

28. The apparatus of claim 27, wherein the type of the PAI includes tensor, text, or image.

29. The apparatus according to any one of claims 3 to 23, wherein the signaling information further comprises: Information indicating the purpose of the neural network, which converts input facial parameters into an output representing the converted facial parameters; Information indicating the purpose of a neural network that converts input parameters into an output representing the converted parameters; or information indicating the purpose of a neural network that converts the format of input parameters into another format of output parameters.

30. The apparatus of claim 8, wherein the Neural Network Post-Filter Input (NNPFI) SEI message is used to provide the auxiliary input to a neural network post-processing filter defined by the (NNPFC) SEI message and activated by the Neural Network Post-Filter Activation (NNPFA) SEI message.

31. An apparatus comprising at least one processor; and at least one memory storing instructions, said instructions, when executed by said at least one processor, causing the apparatus to perform at least: Infer that time-interleaved frames are used in the bitstream for packing and arrangement; The inference constitutes the frame parity, that is, the current frame in which the filter is activated after time extrapolation includes the time-interleaved frame packing arrangement; Select one or more input frames for the time extrapolation filter, the one or more input frames including the inferred constituent frame parity; and The time extrapolation filter is applied using the one or more input frames as input.

32. A method comprising: Encoding signaling information in or along a bitstream, wherein the signaling information enables or supports the performance of visual temporal extrapolation; as well as The signaling information is sent to notify the sender.

33. A method comprising: Receive signaling information in or along the bitstream, the signaling information enabling or supporting the execution of visual time extrapolation; The signaling information is decoded to obtain the decoded signaling information (signaling information); as well as The visual time extrapolation is performed based at least on the decoded signaling information.

34. The method of claim 31 or 32, wherein one or more neural networks (VTNNs) are used to perform the visual temporal extrapolation.

35. The method according to any one of claims 31 to 32, wherein the signaling information is included in: a Neural Network Post-Filter Feature (NNPFC) Supplemental Enhancement Information (SEI) message; a Generative Face Video (GFV) SEI message; or a Neural Network Post-Filter Feature (NNPFC) SEI message and a GFV SEI message.

36. The method according to any one of claims 31 to 34, wherein the signaling information includes one or more of the following: The purpose of performing the visual temporal extrapolation on the decoder side; or Temporal extrapolation (TEC) information of neural networks that characterize execution time extrapolation; or The purpose of generative modeling or generative artificial intelligence (AI) on the decoder side.

37. The method according to any one of claims 31 to 35, wherein the TEC information is signaled based on one or more of the following: When the signaling information indicates that the purpose of the VTNN associated with the signaling information includes visual temporal extrapolation, generative modeling, or generative artificial intelligence (AI), wherein the generative modeling or the generative AI refers to using the VTNN to generate data conditioned on one or more inputs; or When the stated purpose includes generative facial videos or generative videos.

38. The method of claim 36, wherein the purpose of the generative facial video includes: Given one or more of the following, generate or improve at least a portion of a video or image that includes at least one or more faces: The coordinates of at least one or more faces; The coordinates of at least one or more facial landmarks or key points; A transformation matrix describing the motion or temporal changes of one or more key points of the at least one or more faces; Facial expression features; Blinking signal; Text associated with the video; The audio associated with the video; or One or more other images.

39. The method according to any one of claims 35 to 37, wherein the TEC information includes one or more of the following: Information indicating the number of images extrapolated during a single activation or inference of the VTNN; Information indicating the time interval between the extrapolated image and the latest image used as input for the inference; Information indicating whether the VTNN uses auxiliary input that indicates an output instance of one or more images extrapolated in time; The VTNN is indicated by information indicating facial parameters or auxiliary input derived from facial parameters, wherein the auxiliary input is used by the VTNN to perform temporal extrapolation of one or more faces in an image or video; Indicates whether the VTNN adopts information from the output of another VTNN, which is run or executed on the decoder side; Indicates whether the VTNN adopts auxiliary input information derived from the output of the other VTNN, which converts or translates facial parameters into a format accepted or required by the VTNN; Information indicating whether the VTNN uses auxiliary input, which represents a tensor generated on the encoder side or derived from the tensor; Indicates whether the VTNN uses information representing audio or speech or auxiliary input derived from the audio or speech; When the auxiliary input represents the audio or the speech, information about the characteristics of the audio or the speech; Indicates whether the VTNN uses information representing text or auxiliary input derived from the text; Information indicating whether different inferences or executions of the VTNN under the same input produce different outputs; Information indicating the type or cause of the variability in the VTNN's output; Information indicating whether the VTNN is indicated or operated in a manner that does not produce output variability; Information indicating whether the VTNN is allowed to have output variability; This prevents the network from having information with output variability; Information indicating the type of extrapolated output; Information indicating whether the VTNN uses auxiliary input, wherein the auxiliary input indicates: the difference in image order count (POC) of the output image to be generated or extrapolated, the difference in timestamp, the relative output position, or the relative output difference; Information indicating the format of the facial parameters accepted or required by the VTNN; Information regarding the format of the tensor; Information indicating whether the VTNN uses auxiliary input, which represents text associated with a person's speech, and the person's face will be extrapolated in time; Indicates whether the VTNN uses auxiliary input information representing text; Information indicating whether the VTNN produces different outputs for different inferences or executions of the VTNN using one or more of the same inputs; Information indicating whether the input of the VTNN consists of background information; Information indicating whether the input of the VTNN consists of foreground information; Indicates whether the input of the VTNN includes information about the driving image; Information indicating whether the auxiliary input of the VTNN is derived from the lossy encoding process; Indicates whether the VTNN accepts input information, and the input controls one or more aspects of how the final output is determined based on the output of the VTNN; One or more indicators used to indicate which facial parameters are present and / or which facial parameters are used by the VTNN; The number of extrapolated images for the neural network post-filter (NNPF) varies for each inference, and the number of extrapolated images for each inference is indicated using another signaling mechanism. An indication of whether the decoder side will generate a random seed value for the VTNN or derive a pseudo-random seed value based on other information signaled by the encoder; An indication of whether the pseudo-random seed value is derived based on the value of a specific syntax element, which is included in the TEC information or otherwise signaled to the decoder; An indication of whether the VTNN, as specified by the TEC information, will use the seed value as an input or an auxiliary input; An indication of whether the VTNN, as specified by the TEC information, will use the pseudo-random seed as the input or the auxiliary input; or Regarding the randomness characteristic of the VTNN.

40. The method of claim 39, wherein the other value comprises one of the following: The sequence-level quantization parameter (QP) value, wherein the pseudo-random seed value is set to be equal to the sequence-level QP value; Slice-level QP value, wherein the pseudo-random seed value is set to be equal to the slice-level QP value; The values ​​of the syntax elements in the TEC information; Specify the values ​​of syntax elements in the neural network post-filter feature (NNPFC) supplemental enhancement information (SEI) message of the VTNN; or Specify the value of the syntax element in the Generative Face Video (GFV) SEI message of the VTNN.

41. The method of claim 39, wherein the facial parameters used as auxiliary input or used to derive the auxiliary input are carried in one of the following: Newly defined SEI message; Generative Face Video (GFV) SEI message; Neural Network Post-Filter Activation (NNPFA) SEI message; or Neural Network Post-Filter Input (NNPFI) SEI message.

42. The method of claim 39, wherein the tensor is the output of another neural network that is run or executed on the encoder side, or is derived from the other neural network.

43. The method of claim 42, wherein the tensor is signaled within the NNPFA SEI message, the GFV SEI message, or the NNPFI SEI message.

44. The method of claim 39, wherein the information regarding the characteristics of the audio or speech includes one or more of the following: The sampling rate of the audio; The bit depth and / or dynamic range of the audio; The number of audio channels and / or channel configuration of the audio; The duration of the audio and / or the number of audio samples will be included in the input tensor; Information regarding whether the audio was processed using a source separation algorithm; Information regarding whether the audio is spatial audio; or The timing of the audio provided as input relative to one or more images to be extrapolated.

45. The method of claim 39, wherein the type or cause of the output variability includes one of the following: Pseudo-randomness; Intentional alteration; or A floating-point format used to represent the values ​​of one or more parameters of the VTNN and / or the values ​​of one or more signals flowing through the VTNN.

46. ​​The method of claim 39, wherein the information indicating whether the VTNN adopts an auxiliary input indicating an output instance of one or more images extrapolated in time is carried in the NNPFC SEI message or the Neural Network Post-Filter Activation (NNPFA) SEI message.

47. The method of claim 39, wherein the VTNN is signaled to indicate whether the information of the auxiliary input indicating the POC difference, timestamp difference, relative output position, or relative output difference is used as part of the neural network post-filter feature (NNPFC) supplementary enhancement information (SEI).

48. The method of any one of claims 39 or 47, wherein the auxiliary input indicating the POC difference, timestamp difference, relative output position, or relative output difference is carried in a Neural Network Post-Filter Activation (NNPFA) Supplemental Enhancement Information (SEI) message or a Neural Network Post-Filter Input (NPFI) SEI message.

49. The method of claim 39 or 41, wherein the facial parameters include one or more of the following: 2D key points; 3D key points; or One or more matrices.

50. The method of claim 41, wherein the facial parameters obtained from the NNPFA SEI message, the GFV SEI message, or the NNPFI SEI message are converted into the desired format indicated in the TEC information.

51. The method of any one of claims 39 or 44, wherein information indicating whether the VTNN uses the auxiliary input representing the audio or the speech or derived from the audio or the speech to generate or synthesize one or more facial features.

52. The method of claim 39, wherein when the VTNN uses the auxiliary input representing the text, the TEC information further includes information indicating characteristics of the text, the characteristics including one or more of the following: Does the VTNN accept or require the raw text? Does the VTNN accept or require lexical units? Does the VTNN accept or require text embedding? Which word segmenter to use; Which text embedder to use; or The timing of the text provided as input relative to one or more images to be extrapolated.

53. The method of claim 39, wherein the information indicating the extrapolation output type includes one or more categories of visual content that the VTNN can extrapolate.

54. The method of claim 53, wherein the one or more categories include face, hands, or body.

55. The method according to any one of claims 34 to 54, wherein the signaling information further includes an indication of whether the VTNN is adapted for all images to which the VTNN is applied and is activated, or whether the VTNN is sequence-level adapted.

56. The method of claim 55, wherein the signaling information further comprises: Indicates the purpose of the sequence-level adaptation.

57. The method of claim 55 or 56, wherein the persistent auxiliary input (PAI) signal included in the signaling information is used to indicate whether the VTNN is applicable to all images to which the VTNN is applied.

58. The method of claim 57, wherein the signaling information further includes an indication of the type of the PAI signal.

59. The method of claim 58, wherein the type of the PAI includes tensor, text, or image.

60. The method according to any one of claims 34 to 54, wherein the signaling information further comprises: Information indicating the purpose of the neural network, which converts input facial parameters into an output representing the converted facial parameters; Information indicating the purpose of a neural network that converts input parameters into an output representing the converted parameters; or information indicating the purpose of a neural network that converts the format of input parameters into another format of output parameters.

61. The method of claim 39, wherein the Neural Network Post-Filter Input (NNPFI) SEI message is used to provide the auxiliary input to a neural network post-processing filter defined by the (NNPFC) SEI message and activated by the Neural Network Post-Filter Activation (NNPFA) SEI message.

62. A method comprising: Infer that time-interleaved frames are used in the bitstream for packing and arrangement; The inference constitutes the frame parity, that is, the current frame in which the filter is activated after time extrapolation includes the time-interleaved frame packing arrangement; Select one or more input frames for the time extrapolation filter, the one or more input frames including the inferred constituent frame parity; and The time extrapolation filter is applied using the one or more input frames as input.

63. An apparatus comprising components for performing the method according to any one of claims 32 to 62.

64. A computer-readable medium comprising program instructions that, when executed by a device, cause the device to perform the method according to any one of claims 32 to 62.

65. The computer-readable medium of claim 64, wherein the computer-readable medium includes a non-transitory computer-readable medium.

Images