Method and apparatus for encoding / decoding

Abstract

In various implementations, methods and apparatus are disclosed that allow for generating more than one extrapolated picture instance of a decoded picture by applying multiple neural network post filters to the decoded picture. Various embodiments of encoded video data, one or more syntax structures for temporal extrapolation neural network post filtering NNPFC, one or more syntax structures for neural network post filter activation NNPFA associated with the one or more NNPFC, and their use in generating more than one extrapolated picture instance of a decoded picture are disclosed.

Description

Cross-references to related applications

[0001] This application claims the benefit of European Patent Application No. 24305445.9, filed on 25 March 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This embodiment generally relates to a method and apparatus for generating multiple extrapolated image instances of a decoded image by applying multiple neural network post-filters to the decoded image. Background Technology

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transform to utilize spatial and temporal redundancy in video content. Generally, intra-frame or inter-frame prediction is used to fully leverage intra-frame or inter-frame image correlations. Then, the difference between the original block and the predicted block (often called the prediction error or prediction residual) is transformed, quantized, and entropy-coded. To reconstruct the video, the compressed data is decoded through the inverse processes corresponding to entropy coding, quantization, transform, and prediction. Summary of the Invention

[0004] Among various implementations, methods and apparatus are disclosed that provide a syntax structure adapted to generate more than one extrapolated frame instance by applying multiple neural network post-filters, based on the timestamp, frame number, or frame sequence count of the decoded frames of a video.

[0005] According to a first aspect, a video encoding method is disclosed, comprising: encoding and decoding at least one frame of a video; deriving multiple neural network post-filter (NNPF) models for temporal extrapolation, such that more than one extrapolated frame instance of the decoded frame is generated by applying the multiple NNPF models to a decoded frame of the video; encoding more than one syntactic structure (NNPFC) for temporal extrapolation neural network post-filter associated with the multiple NNPF models; and encoding more than one syntactic structure (NNPFA) for neural network post-filter activation associated with the more than one NNPFC.

[0006] According to a second aspect, a video decoding method is disclosed, comprising: decoding at least one frame of a video; obtaining more than one syntax structure for a neural network post-filter (NNPFC); decoding multiple neural network post-filter (NNPF) models from the more than one NNPFC; obtaining more than one syntax structure for activation of the neural network post-filter associated with the more than one NNPFC; identifying multiple activated NNPF models; and generating more than one extrapolated frame instance of the decoded frame by applying the multiple activated NNPF models to the decoded frame of the video.

[0007] According to a third aspect, a video coding method is disclosed, comprising: encoding and decoding at least one frame of a video; deriving multiple neural network post-filter (NNPF) models for temporal extrapolation, such that more than one extrapolated frame instance of the decoded frame is generated by applying the multiple NNPF models to the decoded frame of the video; encoding a syntax structure (NNPFC) for a neural network post-filter for temporal extrapolation associated with the multiple NNPF models, the NNPFC enabling the implementation of multiple instances of the neural network post-filter; and encoding more than one syntax structure (NNPFA) for activation of the neural network post-filter associated with the NNPFC.

[0008] According to a fourth aspect, a video decoding method is disclosed, comprising: decoding at least one frame of a video; obtaining a syntax structure (NNPFC) for temporal extrapolation of a neural network post-filter associated with a plurality of NNPF models, which enables the implementation of a plurality of instances of the neural network post-filter; decoding the plurality of neural network post-filter NNPF models from the NNPFC; obtaining more than one syntax structure (NNPFA) for activation of the neural network post-filter associated with the NNPFC; identifying a plurality of activated NNPF models; and generating more than one extrapolated frame instance of the decoded frame by applying the plurality of activated NNPF models to the decoded frame of the video.

[0009] According to a fifth aspect, a video coding method is disclosed, comprising: encoding and decoding at least one frame of a video; deriving a neural network post-filter (NNPF) model for temporal extrapolation, such that more than one extrapolated frame instance of the decoded frame is generated by applying the NNPF model to a decoded frame of the video and a plurality of control parameter sets; deriving a plurality of control parameter sets, which enable the generation of a plurality of extrapolated frame instances of the decoded frame from the neural network post-filter; encoding a syntax structure (NNPFC) for temporal extrapolation of the neural network post-filter associated with the NNPF model; signaling the plurality of control parameter sets for generating a plurality of extrapolated frame instances of the decoded frame from the neural network post-filter; and encoding one or more syntax structures (NNPFA) for neural network post-filter activation associated with the NNPFC.

[0010] According to a sixth aspect, a video decoding method is disclosed, comprising: decoding at least one frame of a video; obtaining a syntax structure (NNPFC) for a temporal extrapolation neural network post-filter associated with an NNPF model, such that more than one extrapolated frame instance of the decoded frame is generated by applying the NNPF model to the decoded frame of the video and a plurality of control parameter sets; obtaining a plurality of control parameter sets, which enable the generation of a plurality of extrapolated frame instances of the decoded frame from the neural network post-filter; decoding the neural network post-filter NNPF model from the NNPFC; obtaining one or more syntax structures (NNPFA) for neural network post-filter activation associated with the NNPFC; identifying the plurality of control parameter sets; and generating more than one extrapolated frame instance of the decoded frame by applying the NNPF model to the decoded frame of the video, wherein the NNPF model further uses each of the activated control parameter sets as input.

[0011] One or more embodiments also provide methods and / or apparatus for signaling one or more variations of the syntax structure of a time-domain extrapolation neural network post-filter according to any embodiment described herein.

[0012] One or more embodiments also provide an apparatus for encoding / decoding video, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to perform an encoding method according to any embodiment described herein.

[0013] One or more embodiments also provide a computer program including instructions that, when executed by one or more processors, cause the one or more processors to perform an encoding method according to any of the embodiments described herein. One or more embodiments of the invention also provide a computer-readable storage medium storing instructions thereon for encoding or decoding video according to the methods described herein.

[0014] One or more embodiments also provide a computer-readable storage medium storing video data generated according to the method described above. One or more embodiments also provide a method and apparatus for transmitting or receiving video data generated according to the method described herein. Attached Figure Description

[0015] Figure 1 A block diagram of a system that can implement aspects of embodiments of the present invention is shown.

[0016] Figure 2 A block diagram of an embodiment of a video encoder is shown.

[0017] Figure 3 A block diagram of an embodiment of a video decoder is shown.

[0018] Figure 4 An example of the usage of the NNPF procedure according to existing technology is shown.

[0019] Figure 5 Another example of the use of the NNPF procedure based on existing technology is shown.

[0020] Figure 6 An example of the use of the NNPF procedure according to the first embodiment is shown.

[0021] Figure 7 An example of the use of the NNPF procedure according to the second embodiment is shown.

[0022] Figure 8 An example of the usage of a variant of the NNPF procedure according to the second embodiment is shown.

[0023] Figure 9 A method for video decoding according to an embodiment is shown.

[0024] Figure 10 A method for video encoding according to an embodiment is shown.

[0025] Figure 11 A method for video decoding according to an embodiment is shown.

[0026] Figure 12 A method for video encoding according to an embodiment is shown. Detailed Implementation

[0027] Figure 1 A block diagram illustrating a system example that can implement various aspects and embodiments is shown. System 100 can be implemented as a device including the various components described below and configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, networked home appliances, and servers. The elements of system 100 can be implemented individually or in combination in an integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more aspects described in this application.

[0028] System 100 includes at least one processor 110 configured to execute instructions loaded thereon to implement various aspects, such as those described in this application. Processor 110 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 140 may include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0029] System 100 includes an encoder / decoder module 130 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a module that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both encoding and decoding modules. Furthermore, the encoder / decoder module 130 may be implemented as a separate element of system 100, or may be incorporated into processor 110 as a combination of hardware and software, as is known to those skilled in the art.

[0030] Program code to be loaded onto processor 110 or encoder / decoder 130 to execute the various aspects described in this application may be stored in storage device 140 and subsequently loaded into memory 120 for execution by processor 110. According to various embodiments, one or more of processor 110, memory 120, storage device 140, and encoder / decoder module 130 may store one or more items during the execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0031] In several embodiments, memory within processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be processor 110 or encoder / decoder module 130) is used for one or more of these functions. External memory may be memory 120 and / or storage device 140, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external volatile memory (e.g., RAM) is used as working memory for video encoding and decoding operations, such as for MPEG-2, HEVC, or VVC.

[0032] Inputs to the components of system 100 can be provided by a variety of input devices as indicated in box 105. Such input devices include, but are not limited to: (i) an RF section that receives, for example, RF signals transmitted over the air by a broadcaster; (ii) a composite input terminal; (iii) a USB input terminal; and / or (iv) an HDMI input terminal.

[0033] In various embodiments, the input device of block 105 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or band-limiting a signal to a frequency band); (ii) down-converting the selected signal; (iii) band-limiting it again to a narrower frequency band to select, for example, a signal band, which in some embodiments may be referred to as a channel; (iv) demodulating the down-converted and band-limited signal; (v) performing error correction; and (vi) demultiplexing to select a desired data packet stream. The RF section in various embodiments includes one or more elements (e.g., frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers) to perform these functions. The RF section may include a tuner that performs multiple of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In many embodiments, the RF section includes an antenna.

[0034] Furthermore, the USB and / or HDMI terminals may include corresponding interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) can be implemented on demand, for example, within a separate input processing IC or processor 110. Similarly, aspects of USB or HDMI interface processing can be implemented on demand within a separate interface IC or processor 110. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110 and encoder / decoder 130, which operate in conjunction with memory and storage elements to process the data stream on demand for presentation on the output device.

[0035] Various components of system 100 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data between them using suitable connection means 115 (e.g., internal buses known in the art, including I2C buses, wiring, and printed circuit boards).

[0036] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to send and receive data on the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 190 may be implemented, for example, in a wired and / or wireless medium.

[0037] In various embodiments, data is streamed to system 100 using a Wi-Fi network (e.g., IEEE 802.11). In these embodiments, the Wi-Fi signal is received via a communication channel 190 and a communication interface 150 adapted for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router, providing access to external networks, including the Internet, to allow streaming applications and other over-the-top services to communicate. Other embodiments use a set-top box to provide streaming data to system 100, which transmits data via an HDMI connection to input block 105. Still other embodiments use an RF connection to input block 105 to provide streaming data to system 100.

[0038] System 100 can provide output signals to a variety of output devices, including a display 165, a speaker 175, and other peripheral devices 185. Other peripheral devices 185 include, in various embodiment examples, one or more of a standalone DVR, an optical disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 100. In various embodiments, control signals communicate between system 100 and the display 165, speaker 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control (with or without user intervention). Output devices can be communicatively coupled to system 100 via dedicated connections through corresponding interfaces 160, 170, and 180. Alternatively, output devices can be connected to system 100 via communication interface 150 using communication channel 190. The display 165 and speaker 175 can be integrated with other components of system 100 into a single unit of an electronic device, such as a television set. In various embodiments, display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0039] Display 165 and speaker 175 may alternatively be separated from one or more other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments where display 165 and speaker 175 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0040] Figure 2 An example video encoder 200 is shown, such as a VVC (Variety Video Coding) encoder. Figure 2 Encoders that improve upon the VVC standard or employ VVC-like technologies can also be shown.

[0041] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "encoded" and "coded," and the terms "image," "picture," and "frame." Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0042] Before encoding, the video sequence may undergo pre-coding (201), such as applying color transformations to the input color picture (e.g., converting from RGB 4:4:4 to YcbCr 4:2:0), or performing remapping of the input picture components to obtain a signal distribution that is more robust to compression. Metadata may be associated with pre-processing and attached to the bitstream (e.g., in the form of Supplemental Enhancement Information (SEI) messages for AVC, HEVC, and VVC standards).

[0043] In encoder 200, the frame is encoded by encoder elements as described below. The frame to be encoded is divided (202) and processed in units, for example, CUs. Each unit is encoded using, for example, an intra-frame or inter-frame mode. When a unit is encoded in intra-frame mode, it performs intra-frame prediction (260). In inter-frame mode, motion estimation (275) and compensation (270) are performed. The encoder determines (205) which of the intra-frame and inter-frame modes to use to encode the unit and indicates the intra-frame / inter-frame decision by, for example, a prediction mode flag. After prediction, prediction enhancement (285) is applied to the prediction block. For example, the prediction residual is calculated by subtracting (210) the prediction block from the original image block.

[0044] The predicted residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with the motion vector and other syntax elements, are entropy-encoded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, meaning the residual is encoded directly without applying either the transform or quantization process.

[0045] The encoder decodes the encoded block to provide a reference for further prediction. The quantization transform coefficients are dequantized (240) and inversely transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (255) to reconstruct the image block. A loop filter (265) is applied to the reconstructed image to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference image buffer (280).

[0046] Figure 3 A block diagram of an example video decoder 300 is shown. In decoder 300, the bitstream is decoded by decoder elements as described below. Video decoder 300 typically performs operations similar to... Figure 2 The encoding process is the reverse of the decoding process. Encoder 200 typically also performs video decoding as part of the video data encoding.

[0047] Specifically, the decoder's input includes a video bitstream, which can be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other encoded information. Frame segmentation information indicates how the frame is segmented. Therefore, the decoder can segment (335) the frame based on the decoded frame segmentation information. The transform coefficients are dequantized (340) and inversely transformed (350) to decode the prediction residual. The decoded prediction residual and prediction block are combined (355) to reconstruct the image block. The prediction block can be obtained from intra-frame prediction (360) or motion-compensated prediction (i.e., inter-frame prediction) (375) (370). After prediction, prediction enhancement (390) is applied to the prediction block. A loop filter (365) is applied to reconstruct the image. The filtered image is stored in a reference frame buffer (380).

[0048] The decoded image can also undergo post-decoding processing (385), such as inverse color transformation (e.g., conversion from YcbCr 4:2:0 to RGB 4:4:4) or inverse remapping as the inverse of the remapping process performed in the pre-encoding process (201). Post-decoding processing can use metadata derived in the pre-encoding process and signaled in the bitstream. As described below, post-decoding processing may include post-processing filtering for temporal extrapolation.

[0049] Recent updates to video compression technology include versions of various industry standards, reference software, and / or documentation, such as the Enhanced Compression Model (ECM) developed by the JVET (Joint Video Exploration Team) group. The aim is to further improve upon existing VVC (Multi-Functional Video Coding) standards and VSEI specifications.

[0050] Recent advancements in video compression techniques include the Neural Network Post-Filter (NNPF), designed for various post-processing filtering purposes such as general visual quality improvement, chroma upsampling, resolution resampling, and frame rate upsampling. However, recent additions to post-processing filtering include temporal extrapolation neural networks (NNs), also known as video prediction neural networks (NNs), which generate one or more future frames given one or more past frames as input. Typical use cases for temporal extrapolation include extremely low-latency computer vision in fields such as robotics and autonomous driving, where extrapolated future frames facilitate predictive decision-making; or improving the rendering frame rate relative to the decoding frame rate in extremely low-latency applications such as cloud gaming. Various implementations of the NNPF utilize parameterized NN models.

[0051] NNPFC SEI The Neural Network Post-Filter (NNPF) feature SEI message is defined in the revision of the VSEI specification. It allows the transmission of information related to the NN model (as external or internal parameters, depending on the syntax element nnpfc_mode_idc), along with information related to the purpose of the NN model (the "nnpfc_purpose" syntax element). This information is used to indicate the intended purpose of the NNPF signal.

[0052] In the first prior art method, the NNPFC SEI syntax is defined as shown in the table below.

[0053]

[0054] In an improvement to this first prior art method, the NNPC SEI syntax was subsequently extended to address signal notification of information related to a temporal extrapolation neural network (NN). The following is an excerpt of the syntax and semantics related to the nnpfc_purpose, with the addition of a new purpose: temporal extrapolation of frames. The added syntax is highlighted in bold italics and underlined in the table below.

[0055]

[0056] The semantic definition of the “nnpfc_purpose” syntax element is as follows: `nnpfc_purpose` indicates the NNPF purpose as specified in Table 20, where `(nnpfc_purpose&bitMask)` not equal to 0 indicates that the NNPF has the purpose associated with the `bitMask` value in Table 20. When `nnpfc_purpose` is greater than 0 and `(nnpfc_purpose&bitMask)` equals 0, the purpose associated with the `bitMask` value is not applicable to the NNPF. When `nnpfc_purpose` equals 0, the NNPF may be used as determined by the application and as specified by `nnpfc_application_purpose_tag_uri`. In bitstreams conforming to this version of this document, the value of `nnpfc_purpose` should be in the range of 0 to 63 (inclusive). `nnpfc_purpose` values ​​from 64 to 65535 (inclusive) are reserved for future use by ITU-T|ISO / IEC and should not exist in bitstreams conforming to this version of this document. Decoders conforming to this version of this document should ignore NNPC SEI messages with `nnpfc_purpose` in the range of 64 to 65535 (inclusive).

[0057] Table 20 - Definition of nnpfc_purpose

[0058] The variables ChromaUpsamplingFlag, ResolutionResamplingFlag, PictureRateUpsamplingFlag, BitDepthUpsamplingFlag, ColourizationFlag, and TemporalExtrapolationFlag specify whether nnpfc_purpose indicates the purpose of NNPF, including chroma upsampling, resolution resampling, frame rate upsampling, bit depth upsampling, colorization, and temporal extrapolation, and are derived as follows: ChromaUpsamplingFlag = ( ( nnpfc_purpose&0x02 )>0 ) ? 1 : 0 ResolutionResamplingFlag = ( ( nnpfc_purpose&0x04 )>0 ) ? 1 : 0 PictureRateUpsamplingFlag = ( ( nnpfc_purpose&0x08 )>0 ) ? 1 : 0(75) BitDepthUpsamplingFlag = ( ( nnpfc_purpose&0x10 )>0 ) ? 1 : 0 ColourizationFlag = ( ( nnpfc_purpose&0x20 )>0 ) ? 1 : 0 TemporalExtrapolationFlag = ( ( nnpfc_purpose&0x40 )>0 ) ? 1 : 0 Note 2 - When the reserved nnpfc_purpose value is adopted by ITU-T|ISO / IEC in the future, the syntax of this SEI message can be extended with syntax elements whose existence is conditional on nnpfc_purpose being equal to that value or any of the set of values ​​that include that value.

[0059] The update mechanism in the current NNPFC SEI specification In the NNPFC feature SEI of the first prior art method, the concept of sending a base NNPF in one NNPFC SEI and updating it in another NNPFC SEI is defined. An NNPF with a given identifier ID specified in nnpfc_id can be signaled as a base model, which can be used as a reference, and can also be updated by sending / receiving a new NNPFC SEI with the same nnpfc_id. The syntax element nnpfc_base_flag is defined as indicating whether the NN model is an update of the base model or relative to the base NNPF. Several basic constraints are specified for the update mechanism: - The first NNPF that appears in the bitstream with a given ID (notified by a signal in the NNPFC SEI) must be the basic NNPF.

[0060] - All subsequent NNPFCs with the same ID and that are base NNPFCs must have the same content as the first base NNPFC with that ID.

[0061] - Updates are not cumulative; instead, each update is applied to the underlying NNPF.

[0062] The semantics of nnpfc_base_flag are as follows:

[0063] An nnpfc_base_flag value of 1 indicates that this SEI message specifies the underlying NNPF. An nnpfc_base_flag value of 0 indicates that this SEI message specifies an update relative to the underlying NNPF.

[0064] The following constraints apply to the value of nnpfc_base_flag: - When the NNPFC SEI message is the first NNPFC SEI message in the current CLVS with a specific nnpfc_id value in the decoding order, the value of nnpfc_base_flag should be equal to 1.

[0065] - All NNPFC SEI messages in CLVS with a specific nnpfc_id value and nnpfc_base_flag equal to 1 should have the same SEI payload content.

[0066] When nnpfc_base_flag equals 0, the following applies: - This SEI message defines an update relative to the previous base NNPF in decoding order with the same nnpfc_id value. Updates are not cumulative; instead, each update is applied to the base NNPF, which is the NNPF specified by the first NNPFC SEI message in decoding order within the current CLVS with a specific nnpfc_id value. The NNPF defined by this SEI message is obtained by applying the update defined by this SEI message to the base NNPF with the same nnpfc_id value.

[0067] - This SEI message pertains to the current decoded screen and all subsequent decoded screens in the current layer in output order, until the end of the current CLVS, or until, but not including, the decoded screens in the current CLVS that follow the current decoded screen in output order and are associated with subsequent NNPFC SEI messages in decoding order (which have an nnpfc_base_flag equal to 0 and that particular nnpfc_id value in the current CLVS), whichever is earlier.

[0068] Neural Network Post-Filter Activation SEI Message (NNPFA) SEI In addition to the NNPFC SEI message, the first prior art method also specifies a Neural Network Post-Filter Activation SEI message (NNPFA). The NNPFA SEI is used to activate or deactivate a Neural Network Post-Processing Filter (NNPF) identified by the syntax elements nnpfa_target_id and nnpfa_target_base_flag for possible post-processing filtering of a set of frames. If nnpfa_target_base_flag equals 1, the target NNPF is the base NNPF with an nnpfc_id equal to nnpfa_target_id. Otherwise, the target NNPF is the NNPF specified by the last NNPFC SEI message with an nnpfc_id equal to nnpfa_target_id, which is the base NNPF.

[0069] It should be noted that there can be several NNPFA SEI messages for the same image, for example, when NNPF is used for different purposes or for filtering different color components.

[0070] The syntax and semantics of NNPFA are excerpted below.

[0071]

[0072] An nnpfa_target_base_flag value of 1 indicates that the target NNPF is the base NNPF with an nnpfc_id equal to nnpfa_target_id. An nnpfa_target_base_flag value of 0 indicates that the target NNPF is the NNPF specified by the last NNPFC SEI message with an nnpfc_id equal to nnpfa_target_id, which is decoded before the first VCL NAL unit of the current frame and is not a duplicate of the NNPFC SEI message containing the base NNPF.

[0073] Note 3 - When the update of the underlying NNPF with a specific nnpfc_id value is active, the NNPFA message can activate the underlying NNPF, which will switch the target NNPF from the updated NNPF to the underlying NNPF.

[0074] When the nnpfa_target_base_flag in the NNPFA SEI message is equal to 0, there should be at least one NNPFC SEI message with an nnpfc_id equal to nnpfa_target_id and an nnpfc_base_flag equal to 0, which is decoded before the NNPFA SEI message.

[0075] Illustrative example Figure 4 An example of NNPF procedure usage is shown. This example demonstrates inserting NNPFC SEIs into a stream, some as bases and some as updates. It also shows the insertion of NNPFA SEIs to activate NNPF. The NNPF version is applied to the decoded frame based on the positions of the NNPFC SEIs and NNPFA SEIs inserted in the stream. The first received NNPFA (NNPFA0) activates the base NNPF (NNPFC0) for subsequent decoded frames. Then, an update to the base NNPF (NNPFC1) is received, and this update is activated by the second received NNPFA SEI (NNPFA1) for subsequent decoded frames. Another update to the base NNPF (NNPFC2) is received and applied to subsequent decoded frames. Another base NNPFC (NNPFC3) is received, which is reactivated by a new NNPFA (NNPFA2), which activates the base NNPF (NNPF0) for subsequent decoded frames. In this example, the output of the NNPF procedure is a post-filtered version of the decoded frame.

[0076] In this figure and the following figures, to simplify the labeling, nnpfc_purpose is replaced with "Purpose", nnpfc_id is replaced with "Id", and nnpfc_base_flag is replaced with "Base_flag".

[0077] Figure 5 Another usage example of the NNPF procedure for the frame temporal extrapolation use case is shown. In this case, the output from NNPF consists of a series of output frames that follow the last decoded frame in time. Typical use cases for this video prediction feature are robotics, autonomous driving, and motion prediction. For example, for autonomous vehicles, predicting the trajectory of pedestrians in traffic is very useful for predicting future events. First, a base NNPFC0 is received and activated by NNPFA0, which activates the application of NNPF0 to generate (1 + nnpfc_extrapolated_pics_minus1) temporal extrapolated output frames following the latest decoded frame. The input to NNPF0 is the last decoded frame, but may also include more than one previous decoded frame. Then, for subsequent decoded frames, the application of NNPF0 is canceled by NNPFA1 (with the CANCEL flag equal to 1). Then, an updated NNPFC1 is received and activated by NNPFA2, which activates the application of NNPF0 updated by NNPF1 to generate temporal extrapolated output frames following the latest decoded frame. Then, for subsequent decoded screens, the application of NNPF0 updated by NNPF1 is canceled by NNPFA2 (CANCEL flag equal to 1).

[0078] In the current design of NNPF SEI messages, post-filtering using the NN model defined in NNPFC can only generate one instance per frame, regardless of the purpose of NNPF (as indicated by the syntax element nnpfc_purpose). NNPF can generate several frames at different timestamps, frame numbers, or frame order counts (POCs) from decoded frames, for example, in the case of interpolation based on two surrounding non-contiguous decoded frames, or in the case of extrapolation based on previously decoded frames. However, for a given timestamp, frame number, or POC, only a single processed frame can be generated from the application of NNPF. For example, when the purpose is frame interpolation, only one instance is generated for each output interpolated frame; when the purpose is frame extrapolation, only one instance is generated for each output extrapolated frame. Therefore, signaling that allows the generation of multiple instances of the same output frame is desirable. Such signaling is particularly advantageous for the use case of frame temporal extrapolation described herein. However, this principle will also be advantageous and easily implemented for other purposes, such as frame interpolation, frame upsampling, and frame filtering.

[0079] Therefore, this paper proposes allowing more than one output screen instance for the same timestamp, screen number, or POC to be generated for a specific purpose (e.g., temporal extrapolation), either by applying several different NNPF models or by applying one NNPF model configured with different inputs (or control parameter sets). This paper proposes: signaling several NN models (including one NN model with configurable inputs) for the purpose of temporal extrapolation; activating these NN models by inserting several associated NNPF activation SEIs before the decoded screen, wherein these NN models will be used to perform several temporal extrapolation instances from the decoded screen; and optionally, depending on the variant considered, signaling information (one or more syntax elements) in the bitstream indicating that multiple instances of the same temporal extrapolation screen can be generated by the NNPF process for the purpose of temporal extrapolation.

[0080] In the primary variant, at least one of the signaled NN models serves as a reference for the other NN models, and signaling to the other NN models corresponds to an update to the reference NN model. In another variant, the signaled NN model is a reference NN model and is therefore signaled independently (i.e., not as an update to a previously signaled model).

[0081] In the first embodiment, it is proposed to signal more than one NNPFC SEI with the same time-domain extrapolation purpose and the same nnpfc_id, wherein some of these NNPFC SEIs can generate different instances (versions) of the same output time-domain extrapolation screen, and wherein a syntax element is added to indicate the id of the instance associated with the NNPFC SEI.

[0082] In the second embodiment, it is proposed to signal more than one (N) NNPFC SEIs with the same time-domain extrapolation purpose and different nnpfc_ids, activate the N NNPFC SEIs through N NNPFA SEIs, each NNPFA SEI refers to one of the N NNPFC SEIs, and enable the use of N NNPFs to apply N time-domain extrapolation NNPF processes to the same decoded picture in a non-cascaded manner to generate multiple instances of each time-domain extrapolated picture.

[0083] In the third embodiment, it is proposed to signal different NN models within the same NNPFC SEI, with each model corresponding to one instance of each extrapolation frame. Information related to the number of instances is signaled within this SEI.

[0084] In the fourth embodiment, a signal is used to notify the NN model, which also uses a set of control parameters as input, each set of control parameters for one instance of each extrapolated frame. The control parameter sets can be notified by signal or derived from video data characteristics.

[0085] Although various embodiments for signaling have been proposed, they share the following concept: with the goal of temporal extrapolation, signaling several NN models (where a NN model with configurable inputs can be regarded as several configured NN models) that generate more than one output screen instance for the same timestamp, screen number, or POC.

[0086] The following describes various embodiments of the grammatical structures, which can be used in combination or independently.

[0087] Figure 6 An example of the use of the NNPF procedure according to the first embodiment is shown. According to the first embodiment, more than one NNPFC having the same nnpfc_id is signaled, where some of these NNPFCs can generate different instances (versions) of the same output temporal extrapolated pics. Furthermore, when nnpfc_purpose is temporal extrapolation, a new syntax element is signaled to indicate the identifier Id (denoted as nnpfc_extrapolated_pics_id) of the output temporal extrapolated pics instance.

[0088] Additionally, syntax elements can be added to indicate the maximum number of instances (denoted as nnpfc_extrapolated_pics_max_id). In the syntax below, these new syntax elements are added to the NNPFC SEI (highlighted by underscores, bold, and italics). Alternatively, these parameters can be signaled in another SEI attached to the NNPFC SEI, such as in an NNPFA SEI with an nnpfa_target_id equal to the nnpfc_id of the NNPFC SEI.

[0089]

[0090] The semantics are as follows.

[0091] nnpfc_extrapolated_pics_max_id indicates the maximum possible value of nnpfc_extrapolated_pics_id.

[0092] nnpfc_extrapolated_pics_id indicates the ID of each extrapolated picture instance. nnpfc_extrapolated_pics_id should be in the range of 0 to nnpfc_extrapolated_pics_max_id.

[0093] All active NNPFC SEIs with the same nnpfc_id are required to be consistent, which means that several common parameters are constrained to have the same value, such as nnpfc_extrapolated_pics_minus1 or nnpfc_extrapolated_pics_max_id.

[0094] In one variant, the NNPFCSEI associated with the first instance (i.e., nnpfc_extrapolated_pics_id = 0) must be the underlying NNPFC.

[0095] In another variant, the NNPFC SEI that is not related to the first instance (i.e., nnpfc_extrapolated_pics_id>0) must be the updated NNPFC relative to the latest base NNPFC.

[0096] As mentioned earlier, in order to... Figure 6 In this context, the term "nnpfc_extrapolated_pics_id" is replaced with "Instance". Therefore, the base NNPFC0 (Instance = 0) is received first, followed by NNPFC1 (Instance = 1), which is the updated SEI of NNPFC0. Then, NNPFA0 is received, which activates the application of NNPFC0 to generate instance 0 of the temporal extrapolated output frame following the latest decoded frame. Then, NNPFA1 is received, which activates the application of NNPFC0 updated by NNPFC1 to generate instance 1 of the temporal extrapolated output frame following the latest decoded frame. NNPFA2 cancels the temporal extrapolation application for subsequent decoded frames.

[0097] Figure 7 An example of the usage of the NNPF procedure according to the second embodiment is shown. According to the second embodiment, more than one (N) NNPFC SEIs with different nnpfc_ids are signaled. To activate N NNPFC SEIs via N NNPFA SEIs, each NNPFA SEI references one of the N NNPFC SEIs. Advantageously, this arrangement allows the use of N NNPFs to non-cascade N temporal extrapolation NNPF procedures on the same decoded frame, generating more than one instance of each temporal extrapolation frame. Figure 7 As shown, NNPFC0 and NNPFC1 have different IDs, and NNPFA0 and NNPFA1 also have different reference IDs. This is because... Figure 6The implementation differs. The process is as follows: First, a base NNPFC0 (Id = k) is received, followed by a base NNPFC1 (Id = m). Then, NNPFA0 is received, which activates the application of NNPFC0 to generate a first instance of a time-domain extrapolated output frame following the latest decoded frame. Then, NNPFA1 is received, which activates the application of NNPFC1 to generate a second instance of a time-domain extrapolated output frame following the latest decoded frame. NNPFA2 and NNPFA3 respectively cancel the time-domain extrapolation application from NNPFC0 and NNPFC1 to subsequent decoded frames.

[0098] Based on additional features, a new flag (hereinafter denoted as nnpfc_new_instance_flag) can be inserted into the NNPFC syntax to indicate whether the temporal extrapolation NNPFC is allowed to generate a new instance of each temporal extrapolation frame, independent of other simultaneous instances that may be generated by other NNPFCs. Example syntax is shown in the table below. When nnpfc_new_instance_flag is true, an NNPF model from the NNPFC SEI can be applied to a given decoded frame, even if another NNPF model has been or will be applied to the same decoded frame. Each NNPF model with this attribute can generate a different instance of the output frame. In other words, all activated NNPF models with a true nnpfc_new_instance_flag can be applied to the same decoded frame and can each generate their own version of the extrapolation frame.

[0099]

[0100] In one variant, the NNPFC SEI can be an updated NNPFC that references a base NNPFC with a different nnpfc_id (but with the same time-domain extrapolation purpose). When the NNPFC SEI has an nnbpc_base_flag equal to 0, a new syntax element nnpfc_base_id is added to indicate the base NNPFC SEI used as the reference. The syntax is shown in the table below.

[0101]

[0102] Figure 8 A usage example of a variant of the NNPFC procedure according to the second embodiment is shown. Figure 7The main difference is that the new syntax element nnpfc_base_id (denoted as Base_id in the diagram) exists in NNPFC1. It is equal to k, which is the Id of NNPFC0, meaning that NNPFC1 is an update of NNPFC0. The process is as follows: First, the base NNPFC0 (Id = k) is received, then NNPFC1 (Id = m) is received as an update to NNPFC0 (referenced by Base_id = k). Then, NNPFA0 is received, which activates the application of NNPFC0 to generate the first instance of the temporal extrapolation output frame following the latest decoded frame. Then, NNPFA1 is received, which activates the application of NNPFC1 to generate the second instance of the temporal extrapolation output frame following the latest decoded frame. NNPFA2 and NNPFA3 respectively cancel the temporal extrapolation application from NNPFC0 and NNPFC1 to subsequent decoded frames.

[0103] In one variant, removing the base NNPFC will remove all (updating) NNPFCs that reference that base NNPFC. In this case, refer to... Figure 8 NNPFA3 may not need to send a signal notification because NNPFA2 cancels the base NNPFC0 and therefore cancels the update of NNPFC1.

[0104] According to the third embodiment, different NN models are notified by a signal within the same NNPFC SEI, with each model targeting one instance of each extrapolated picture. In the example syntax table below, the maximum instance ID is indicated (the parameter nnpfc_extrapolated_pics_max_id gives the maximum ID number, so the number of instances is derived as nnpfc_extrapolated_pics_max_id + 1).

[0105] Then, when nnpfc_mode_idc equals 1, in addition to the Uniform Resource Identifier (URI) of instance 0 indicated in the existing syntax element nnpfc_uri, the URI (nnpfc_uri_instance[i]) of each instance at index i (where i is in [1, nnpfc_extrapolated_pics_max_id]) is signaled. Another syntax element, nnpfc_tag_uri_instance[i], can also be signaled, but in the preferred implementation, this information is inferred from nnpfc_tag_uri.

[0106] When nnpfc_mode_idc equals 0, in addition to the payload byte of instance 0 indicated in the existing syntax element nnpfc_payload_byte[i], the payload byte (nnpfc_payload_byte[j][i]) of each instance at index j (j is in [1, nnpfc_extrapolated_pics_max_id]) is also signaled.

[0107]

[0108] In one variant, when TemporalExtrapolationFlag is true, nnpfc_base_flag is forced to be equal to 1.

[0109] In another variant, NNPF instances other than instance 0 (notified by signal via nnpfc_uri_instance[i] or nnpfc_payload_byte[j][i]) are forced to update the NNPF of instance 0 (notified by signal via nnpfc_uri or nnpfc_payload_byte[i]).

[0110] According to the fourth embodiment, a neural network (NN) model is signaled in the NNPFC SEI, and a set of control parameters is also signaled, either in the NNPFC SEI or in another syntactic structure referencing the NNPFC SEI (e.g., in an accompanying new SEI message). Each set of control parameters serves as additional input to the NNPF and is specific to an instance of the extrapolated frame. The set of control parameters can be derived or defined by default through a pre-coding analysis step. For example, the pre-analysis step can derive parameters for modeling the estimated global motion on the decoded frame (e.g., translation, affine, quadratic, or homography model parameters of the object of interest in a scene captured in the video). The set of control parameters can be derived from these estimated global motion parameters. For example, one set corresponds to keeping the estimated global motion parameters unchanged. Another set corresponds to modifying the estimated global motion parameters to amplify the motion of the object of interest toward the right side of the frame. For example, in the case of a translation model, the set of control parameters consists of two parameters, dx and dy, corresponding to the horizontal and vertical components of the translational motion, respectively. If the estimated motion is (dx0, dy0), then the first set of control parameters can correspond to (dx0, dy0), and the second set of control parameters can correspond to ((1+a)). dx0,dy0), the third set of control parameters can correspond to ((1-a) dx0, dy0), where a is a slightly varying parameter, for example, equal to 0.2. The fourth set of control parameters can be set to (0,0). In other examples, the control parameter set describes modifications to dy, either alone or together with dx, corresponding to the vertical component of the translational motion. Another set corresponds to modifying the estimated global motion parameters to amplify the motion of the object of interest toward the left side of the frame. Another set corresponds to empty global motion. Other examples of control parameters are qualitative motion indications (e.g., indicating "maintain" for the next frame, "accelerate" for the next frame, and "decelerate" for the next frame). This qualitative information can be input by the user using external means via text prompts, such as by an end-user video renderer interfaced with a decoder, or by an end-user video analyzer interfaced with a decoder (e.g., when the machine performs video analysis and makes decisions from it). Another example of control parameters is a quantitative value indicating a hypothesis number (value k corresponds to hypothesis k). The NNPF model can be trained to generate different versions of extrapolated frames based on the hypothesis value. Those skilled in the art will conceive of other examples of control parameter sets; therefore, the above examples of control parameter sets are illustrative, and the aspects are not limited to these specific examples.

[0111] The following is a syntax example, where the number of control parameter sets is represented as nnpfc_extrapolated_pics_nb_control_params_sets, and the number of control parameters for each set is represented as nnpfc_extrapolated_pics_size_control_params. Each control parameter `control_parameter[i][j]` is then signaled for each set (index i). Index `j` indicates the index of the control parameter in the control parameter set. In a variant, the number of control parameter sets is inferred from `nnpfc_extrapolated_pics_max_id` (e.g., set to be equal to it) without signaling.

[0112]

[0113] Alternatively, the control parameter set can be determined by external means on the client or decoder side, such as by the end user or the end user's machine. In this case, the control parameter set may not need to be notified by signals.

[0114] In an embodiment, an NNPFA can be used to activate an NNPF and a set of control parameters to generate an instance of an extrapolated picture from a decoded picture using the NNPF model. If k sets of control parameters are used, k NNPFAs can be signaled to activate the NNPF with those sets of control parameters. An NNPFA can contain information indicating which set of control parameters signaled in an NNPFC or another syntax structure is selected and activated. For example, this can be achieved by adding a syntax element to the NNPFA corresponding to the index of the set of control parameters to be used as input to the NNPF. In another variation, an NNPFA is signaled to activate all sets of control parameters signaled in an NNPFC or another syntax structure linked to the NNPFC (or NNPF). In yet another variation, an NNPFA can contain an array of ids indicating which sets of control parameters signaled in an NNPFC or another syntax structure are activated. More generally, an NNPFA can contain information about the sets of control parameters used as input to the NNPF.

[0115] Decoding and post-filtering processes Figure 9 A method for decoding and post-filtering video according to an embodiment is illustrated. A bitstream is obtained in step 900. The bitstream may include encoded video data, one or more syntax structures (NNPFCs) for a temporal extrapolation neural network post-filter according to any of the foregoing illustrative examples. Furthermore, the bitstream includes one or more syntax structures (NNPFAs) associated with the one or more NNPFCs for activating the neural network post-filter.

[0116] Step 901 performs screen decoding to generate a decoded screen. Step 902 obtains several NNPFC SEIs with a purpose corresponding to temporal extrapolation. In step 903, several NNPF models are decoded from the several NNPFC SEIs. In step 904, the NNPFA SEIs associated with the several NNPFC SEIs are obtained and decoded. In step 905, the identification of NNPF models (or equivalent NNPFC SEIs) activated for generating temporal extrapolation screens is performed. This identification can be based on the decoding of the syntax elements defined above, for example: • As described in the first embodiment, nnpfc_extrapolated_pics_max_id and nnpfc_extrapolated_pics_id • As described in the second embodiment, nnpfc_base_flag or nnpfc_base_id • As described in conjunction with the third embodiment, nnpfc_tag_uri_instance[i], nnpfc_uri_instance[i], and nnpfc_payload_byte[j][i].

[0117] In step 906, temporal extrapolation is performed using several identified NNPF models to generate several extrapolated picture instances of the latest decoded picture. Advantageously, the activated NNPF model is applied to generate (1 + nnpfc_extrapolated_pics_minus1) temporal extrapolated output pictures following the latest decoded picture, which has a given timestamp, picture number, or point of view (POC). However, the input to NNPF is not limited to the last decoded picture and can include more pictures. For example, NNPF can also include more than one picture decoded before the latest decoded picture with a given timestamp, picture number, or POC. For example, these pictures can allow NNPF to estimate motion information on the previously decoded picture and generate multiple temporal extrapolated pictures of the latest decoded picture. For clarity, this single latest decoded picture is used as a reference picture for a specific timestamp, picture number, or POC used for extrapolation.

[0118] Encoding and bitstream generation process Figure 10 A method for encoding video and generating a bitstream according to an embodiment is illustrated. The generated bitstream may include encoded video data and one or more NNPFC syntax structures for temporal extrapolation neural network post-filters according to any of the foregoing illustrative examples. Furthermore, the generated bitstream may include one or more NNPFA syntax structures for neural network post-filter activation associated with the one or more NNPFCs. The input to the encoding method is the input video obtained in step 1000. Step 1001 performs encoding and decoding (i.e., frame reconstruction). Step 1002 derives several temporal extrapolation NNPF models with the aim of generating more than one extrapolated frame instance of the decoded video frame using these multiple NNPF models. In step 1003, one or more associated NNPFC SEIs associated with the multiple NNPF models are encoded, wherein information is inserted into these one or more NNPFC SEIs according to any of the foregoing illustrative examples to allow the generation of more than one instance of the extrapolated frame. This information may use syntax elements defined above, such as: • As described in the first embodiment, nnpfc_extrapolated_pics_max_id and nnpfc_extrapolated_pics_id • As described in the second embodiment, nnpfc_base_flag or nnpfc_base_id • As described in conjunction with the third embodiment, nnpfc_tag_uri_instance[i], nnpfc_uri_instance[i], and nnpfc_payload_byte[j][i].

[0119] The NNPFA SEI, which is signaled to activate the NNPFC SEI, is also encoded before the decoding screen, where time-domain extrapolation should be applied to the decoding screen (step 1004). The data is then concatenated to generate the output bitstream (step 1005).

[0120] Other illustrative examples of the decoding and post-filtering process Figure 11 A method for decoding and post-filtering video according to another embodiment is shown. For example, Figure 11The embodiments described herein are applicable to the proposed fourth embodiment of signaling employing control parameters. In step 1100, a bitstream is obtained. The bitstream may include encoded video data, one or more syntax structures (NNPFCs) for a neural network post-filter according to any example. For example, those skilled in the art will understand that a first variant embodiment, which can signal different NN models to generate multiple instances, is compatible with the fourth variant embodiment because a NN model in the first variant embodiment may further use different control parameters as in the fourth variant embodiment. Furthermore, the bitstream includes one or more syntax structures (NNPFAs) for activating the neural network post-filter associated with the one or more NNPFCs and the control parameter set. Step 1101 performs screen decoding to generate a decoded screen. Step 1102 obtains an NNPFC SEI with a purpose corresponding to temporal extrapolation. In step 1103, an NNPF model is decoded from the NNPFC SEI. In step 1104, a control parameter set is obtained from the syntax structure of the bitstream or from an external means. In step 1105, one or more NNPFA SEIs associated with the NNPFC SEI and the associated control parameter set are obtained and decoded. In step 1106, the identification of control parameter sets activated for generating temporal extrapolation images is performed. This identification can be based on the decoding of syntax elements defined above, such as nnpfc_extrapolated_pics_nb_control_params_sets, nnpfc_extrapolated_pics_size_control_params, and control_parameter[i][j]. Additional syntax elements (e.g., those inserted into the NNPFA SEI) can also be signaled to indicate which control parameter sets are activated. For example, an NNPFA SEI or another syntax structure can contain an array of id values ​​indicating the ids of the activated control parameter sets obtained in step 1104.

[0121] In step 1107, temporal extrapolation is performed using several sets of activated control parameters used as input to the NNPF model, thereby generating several extrapolated screen instances. As mentioned above, the NNPF model can also use more than one previously decoded screen as input.

[0122] Encoding and bitstream generation process Figure 12A method for encoding video and generating a bitstream according to an embodiment is illustrated. The generated bitstream may include encoded video data, one or more NNPFC syntax structures for a neural network post-filter according to any of the foregoing illustrative examples. Furthermore, the generated bitstream may include one or more NNPFA syntax structures for activating the neural network post-filter associated with the one or more NNPFCs. The input to the encoding method is the input video obtained in step 1200. Step 1201 performs encoding and decoding (i.e., scene reconstruction). Step 1202 derives a temporal extrapolation NNPF model with the aim of generating more than one extrapolated scene instance of the decoded video using multiple sets of control parameters. In step 1203, the associated NNPFC SEIs associated with the NNPF model are encoded. According to any of the foregoing illustrative examples, in step 1204, a set of control parameters is derived that enables the generation of more than one extrapolated scene instance for the decoded scene using the NNPF model, and information associated with the derived set of control parameters is encoded. This information can be obtained using the syntax elements defined above, such as nnpfc_extrapolated_pics_nb_control_params_sets, nnpfc_extrapolated_pics_size_control_params, and control_parameter[i][j].

[0123] The NNPFA SEI, which is signaled to activate the NNPFC SEI, is also encoded before the decoding screen, where time-domain extrapolation should be applied to the decoding screen (step 1205). The data is then concatenated to generate the output bitstream (step 1206).

[0124] This document describes various methods, and each method includes one or more steps or actions for implementing the method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions can be modified or combined. Furthermore, terms such as "first" and "second" can be used in various embodiments to modify elements, components, steps, operations, etc., e.g., "first decoding" and "second decoding." The use of such terms does not imply an ordering of the modified operations unless specifically required. Therefore, in this example, the first decoding does not need to be performed before the second decoding and can occur, for example, before, during, or within an overlapping period of time with the second decoding.

[0125] The various methods and other aspects described in this application can be used to modify the module, for example... Figure 2The motion compensation module (270) of the video encoder 200 shown is illustrated. Furthermore, this aspect is not limited to ECM and VVC, and can be applied to, for example, other standards and recommendations, as well as any extensions of such standards and recommendations. Unless otherwise indicated or technically excluded, the aspects described in this application may be used individually or in combination.

[0126] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0127] Various implementations involve decoding. As used herein, "decoding" can encompass all or part of the process performed, for example, on a received encoded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to refer to a broader decoding process will be clear from the specific context of the description and is thought to be well understood by those skilled in the art.

[0128] Various implementations involve encoding. In a manner similar to the discussion of "decoding" above, as used in this application, "encoding" can encompass, for example, all or part of the process performed on an input video sequence to produce an encoded bitstream.

[0129] It should be noted that the grammatical elements used in this article are descriptive terms. Therefore, the use of other grammatical element names is not excluded.

[0130] The implementations and aspects described herein can be implemented as, for example, methods or processes, apparatus, software programs, data streams, or signals. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), implementations of the discussed features can be implemented in other forms (e.g., apparatus or program). Apparatus can be implemented as, for example, suitable hardware, software, and firmware. Methods can be implemented in, for example, apparatus, such as a processor, which generally refers to processing devices, including, for example, computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0131] References to “an embodiment” or “an embodiment” or “an implementation” or “implementation”, and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases “in an embodiment” or “in an embodiment” or “in an implementation” or “in an implementation”, and any other variations appearing in multiple places throughout this application, do not necessarily refer to the same embodiment.

[0132] Furthermore, this application may refer to "determining" various types of information. Determining information may include, for example, one or more of the following: estimation information, calculation information, prediction information, or information retrieved from memory.

[0133] Furthermore, this application may refer to "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0134] Furthermore, this application may refer to "receiving" various types of information. "Receiving," like "accessing," is a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) from one or more sources. Moreover, "receiving" generally refers in one or more ways to operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0135] It should be understood that the use of any of the following “ / ”, “and / or”, and “…at least one of…”, such as in the cases of “A / B”, “A and / or B”, and “at least one of A and B”, is intended to cover selecting only the first listed option (A), selecting only the second listed option (B), or selecting both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, this wording is intended to cover selecting only the first listed option (A), selecting only the second listed option (B), selecting only the third listed option (C), selecting only the first and second listed options (A and B), selecting only the first and third listed options (A and C), selecting only the second and third listed options (B and C), or selecting all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as possible listed.

[0136] Furthermore, as used herein, the term "for signal notification" specifically refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals the quantization matrix used for dequantization. Thus, in this embodiment, the same parameters are used on both the encoder and decoder sides. Therefore, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has the specific parameters as well as other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select the specific parameters. Bit savings are achieved in many embodiments by avoiding the transmission of any actual functionality. It should be understood that signal notification can be done in a variety of ways. For example, in many embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the foregoing refers to the verb form of the term "for signal notification," the term "signal" can also be used as a noun herein.

[0137] As will be apparent to those skilled in the art, implementations can generate various signals formatted to carry information, which can, for example, be stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of a spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted via a variety of different wired or wireless links known in the art. The signal may be stored on a processor-readable medium.

[0138] We have described numerous embodiments. The features of these embodiments may be provided individually or in any combination across a variety of claim classes and types. Furthermore, embodiments may include one or more of the following features, devices, or aspects individually or in any combination across a variety of claim classes and types.

Claims

1. A method comprising: Decode at least one frame of the video (1101). Obtain the syntax structure NNPFC (1102) for the post-filter of the time-domain extrapolation neural network; Decode the (1103) NNPF model from the NNPFC; Obtain (1104) the set of control parameters for the NNPF model; Obtain (1105) one or more syntax structures NNPFA for neural network post-filter activation associated with the NNPFC; Identify (1106) multiple activated control parameter sets; and (1107) More than one extrapolated frame instance of the decoded frame is generated by applying the NNPF model to the decoded frame of the video, wherein the NNPF model also uses each of the activated control parameter set as input.

2. A method comprising: Encode and decode at least one frame of the video (1201). Derive (1202) a neural network post-filter NNPF model for temporal extrapolation, such that by applying the NNPF model to the decoded frame of the video and more than one set of control parameters, more than one extrapolated frame instance of the decoded frame is generated; The syntax structure NNPFC used for the post-filter of the time-domain extrapolation neural network is encoded (1203). Deriving (1204) more than one set of control parameters, generating extrapolated image instances of the decoded image by applying the NNPF model to the decoded image of the video and one of the more than one set of control parameters; Encode one or more syntax structures NNPFA for neural network post-filter activation associated with the NNPFC (1205).

3. The method of any one of claims 1 and 2, wherein the one or more syntax structures NNPFA for post-filter activation of the neural network associated with the NNPFC further include information about the set of control parameters.

4. The method of any one of claims 1 to 3, wherein the set of control parameters includes qualitative information.

5. The method of any one of claims 1 to 3, wherein the set of control parameters is obtained from an external means.

6. A method comprising: Decode at least one frame of the video (901); Obtain (902) more than one syntax structure NNPFC for the post-filter of the temporal extrapolation neural network; Decode multiple neural network post-filter NNPF models from more than one NNPFC (903); Obtain (904) more than one syntax structure NNPFA for neural network post-filter activation associated with more than one of the NNPFCs; Identify (905) multiple activated NNPF models; and More than one extrapolated image instance of the decoded image of the video (906) is generated by applying the multiple activated NNPF models.

7. A method comprising: Encode and decode at least one frame of the video (1001). Derive (1002) multiple neural network post-filter NNPF models for temporal extrapolation, such that more than one extrapolated image instance of the decoded image is generated by applying multiple of the NNPF models to the decoded image of the video; Encoding (1003) of more than one syntax structure NNPFC for a neural network post-filter associated with multiple of the aforementioned NNPF models; and Encode more than one syntax structure NNPFA for neural network post-filter activation associated with more than one of the NNPFCs (1004).

8. The method of any one of claims 6 and 7, wherein the syntax structure NNPFC for the post-filter of the temporal extrapolation neural network comprises: Indication of the purpose associated with time-domain extrapolation; Neural network post-filter model identifier; Indications for either the base model or the base model update used in neural network post-filters; and Indication of instances of post-filters in neural networks.

9. The method of claim 8, wherein the indication of one of the base model and the base model update for the neural network post-filter indicates the base model for a first instance of the neural network post-filter, and the indication of one of the base model and the base model update for the neural network post-filter indicates the update model for a second instance of the neural network post-filter.

10. The method of claim 8, wherein obtaining (904) more than one syntax structure NNPFA for neural network post-filter activation associated with more than one of the NNPFCs comprises: Obtain (904) the first syntax structure NNPFA associated with the first instance of the neural network post-filter for activation of the neural network post-filter; as well as Obtain (904) a second syntax structure NNPFA for activation of the neural network post-filter associated with a second instance of the same neural network post-filter.

11. The method of any one of claims 6 and 7, wherein obtaining (902) more than one syntax structure NNPFC for a post-filter of a temporal extrapolation neural network comprises: A first syntax structure NNPFC for a neural network post-filter used for temporal extrapolation is obtained, comprising: an indication of the purpose associated with temporal extrapolation, a first neural network post-filter model identifier, a first indication of either the base model for the neural network post-filter or an update of the base model; and A second syntax structure NNPFC for a neural network post-filter is obtained, comprising: an indication of the purpose associated with the temporal extrapolation, a second neural network post-filter model identifier different from the first neural network post-filter model identifier, and a second indication of either the base model for the neural network post-filter or the base model update.

12. The method of claim 11, wherein obtaining (904) more than one syntax structure NNPFA for neural network post-filter activation associated with more than one of the NNPFCs comprises: Obtain the first syntax structure NNPFA associated with the first neural network post-filter model identifier for neural network post-filter activation; as well as Obtain the second syntax structure NNPFA associated with the second neural network post-filter model identifier for neural network post-filter activation.

13. The method of any one of claims 11 and 12, wherein the first indication of one of the base model and the base model update for the neural network post-filter indicates the base model for the first neural network post-filter model identifier, and the second indication of one of the base model and the base model update for the neural network post-filter indicates the base model for the second neural network post-filter model identifier.

14. The method of any one of claims 11 and 12, wherein the first indication of one of the base model and the base model update for the neural network post-filter indicates the base model for the first neural network post-filter model identifier, the second indication of one of the base model and the base model update for the neural network post-filter indicates the updated model for the second neural network post-filter model identifier, and wherein the second syntax structure NNPFC for the temporal extrapolation neural network post-filter further includes the first neural network post-filter model identifier as an indication of the updated model.

15. The method of any one of claims 6 to 14, wherein the more than one syntax structure NNPFC for time-domain extrapolation neural network post-filter and the more than one syntax structure NNPFA associated with the more than one NNPFC for neural network post-filter activation are obtained from or encoded in the supplemental enhancement information (SEI) message.

16. A method comprising: Decode at least one frame of the video (901); Obtain (902) a syntax structure NNPFC for a neural network post-filter for temporal extrapolation, the syntax structure NNPFC for a neural network post-filter for temporal extrapolation includes: an indication of the purpose associated with temporal extrapolation, a model identifier for a first instance, an indication of either the base model for the first instance or the base model update, an indication of a number of neural network post-filter instances greater than one, and an indication of the neural network post-filter NNPF model for each instance different from the first instance. Decode multiple NNPF models from the NNPFC (903); Obtain (904) more than one syntax structure NNPFA for post-filter activation of the neural network associated with the NNPFC; Identify (905) multiple activated NNPF models; and More than one extrapolated frame instance of the decoded frame is generated by applying the multiple activated NNPF models to the decoded frame of the video (906).

17. A method comprising: Encode and decode at least one frame of the video (1001). Derive (1002) multiple neural network post-filter NNPF models for temporal extrapolation, such that more than one extrapolated image instance of the decoded image is generated by applying multiple of the NNPF models to the decoded image of the video; The syntax structure NNPFC for the temporal extrapolation neural network post-filter is encoded (1003), the NNPFC including: an indication of the purpose associated with temporal extrapolation, a model identifier for a first instance, an indication of either the base model for the first instance or an update of the base model, an indication of a number of neural network post-filter instances greater than one, and an indication of the neural network post-filter NNPF model for each instance different from the first instance; and Encode more than one syntax structure NNPFA for neural network post-filter activation associated with the NNPFC (1004).

18. An apparatus comprising one or more processors, wherein the one or more processors are configured to implement the method as claimed in any one of claims 1 to 17.

Images