Method and apparatus for generating a synthetic reference picture for image decoding

Abstract

A decoding method (1500), implemented by a decoder for predicting a currently decoded picture in dependence on a synthetic reference picture, the method comprising: parsing (1501) multiple bitstream parts, the multiple bitstream parts comprising an additional bitstream part (921) containing information relevant to the formation of the synthetic reference picture; from the information contained in the additional bitstream part (921), determining (1502) a synthetic reference picture type for the synthetic reference picture; and generating (1503) the synthetic reference picture in dependence on the determined synthetic reference picture type; wherein the synthetic reference picture differs from pictures previously decoded from the bitstream. Generating the synthetic reference picture from information contained in the additional bitstream part may allow for content adaptation and the handling of Intra areas (such as occlusions or independently coded areas).

Description

[0001] METHOD AND APPARATUS FOR GENERATING A SYNTHETIC REFERENCE PICTURE FOR IMAGE

[0002] DECODING

[0003] TECHNICAL FIELD

[0004] Embodiments of the present disclosure generally relate to the field of encoding and decoding data. In particular, some embodiments relate to methods and apparatuses for such encoding and decoding images and / or videos from a bitstream.

[0005] BACKGROUND

[0006] Hybrid image and video codecs have been used to compress image and video data. In such codecs, signal is typically encoded block- wisely by predicting a block and by further coding only the difference between the original bock and its prediction. In particular, such coding may include transformation, quantization and generating the bitstream, usually including some entropy coding. Typically, the three components of hybrid coding methods - transformation, quantization, and entropy coding - are separately optimized. Modem video compression standards like High-Efficiency Video Coding (HEVC), Versatile Video Coding (VVC) and Essential Video Coding (EVC) also use transformed representation to code residual signal after prediction.

[0007] Recently, neural network architectures have been applied to image and / or video coding. In general, these neural network (NN) based approaches can be applied in various different ways to the image and video coding. For example, some end- to-end optimized image or video coding frameworks have been discussed. Moreover, deep learning has been used to determine or optimize some parts of the end-to-end coding framework such as selection or compression of prediction parameters or the like. Besides, some neural network based approached have also been discussed for usage in hybrid image and video coding frameworks, e.g. for implementation as a trained deep learning model for intra or inter prediction in image or video coding.

[0008] The end-to-end optimized image or video coding applications discussed above have in common that they produce some feature map data, which is to be conveyed between encoder and decoder.

[0009] Neural networks are machine learning models that employ one or more layers of nonlinear units based on which they can predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. A corresponding feature map may be provided as an output of each hidden layer. Such corresponding feature map of each hidden layer may be used as an input to a subsequent layer in the network, i.e., a subsequent hidden layer or the output layer.

[0010] Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. In a neural network that is split between devices, e.g. between encoder and decoder, a device and a cloud or between different devices, a feature map at the output of the place of splitting (e.g. a first device) is compressed and transmitted to the remaining layers of the neural network (e.g. to a second device).

[0011] In image and video codecs, a picture (pic) or frame is set of pixels (triplet of color components) to be coded (referred to as the original picture) or reconstructed after decoding (referred to as the reconstructed picture). A Picture Order Count (POC) is an index specifying picture number in display order. The coding order can be different to the display order.

[0012] The reconstructed picture can be formed in dependence on a reference picture (RefPic). In traditional video coding, the reference picture is one of the pictures in a decoded picture buffer (BPB): that is, a previously decoded picture. A reference picture list can include one or more reference pictures. Each of the reference pictures can be identified by a reference picture index (Refldx). There could be more than one reference picture list (RefPicLX, X=0,l,..). Typically, there are two reference picture lists.

[0013] If a picture is coded independently from other picture in video, and so without access to previously decoded pictures (i.e. the reference picture list is empty) then the picture type is I-picture (or I-frame). I-pictures are inserted periodically (for example) in order to enable random access (RA) functionality.

[0014] If there is only one reference picture list, then the picture is coded without a combination of different predictions. This picture type is P-picture.

[0015] In applications which require ultra-low delay on the encoder side, usually the first picture is of I-picture type and all others are P-pictures.

[0016] For each block type of prediction (Intra, Uni, Bi or SKIP) is defined by syntax elements signalled in the bitstream. I- picture has only Intra predicted blocks.

[0017] During a motion compensation (MC) process, a prediction for the coded block is generated from the parts of reference pictures. If more than two reference picture lists are supported by video coding, then a prediction can be uni- or bidirectional. For P-pictures only uni-directional prediction is allowed. A uni-directional prediction can be generated from the part of reference picture in a single RefPicLO. Which picture to predict from a be defined by Refldx. Which part of the reference picture to use can be specified by a motion vector.

[0018] Figures la- Id illustrates the video frames in a video coder with motion compensation. Figure la shows multiple video frames as displayed. The POC defines the display order. The coding / decoding order can be different to the display order. The frame shown in Figure lb is at POC=0. In this example, this frame is coded as an I-picture without access to previously decoded frames, but with access to previously decoded pixels. The frame shown in Figure 1c is at POC=2. In this example, this frame is coded as a B-picture with access to one previously decoded picture The frame shown in Figure Id is at POC=4. In this example, this frame is coded as a P-picture with access to only one previously decoded picture.

[0019] Figures 2a and 2b schematically illustrate motion vector field construction for a reference picture.

[0020] For the purposes of temporal motion vector prediction (TMVP), a reference picture contains not only reconstructed pixel values, but also an aggregated motion vector field, as schematically illustrated in Figure 2a. The granularity of the aggregated motion vector field is different from partitioning used during reference picture reconstruction.

[0021] The process of reference picture creation generally includes assigning a motion vector field, POC, a map of quality parameters (QP) and other parameters. This is schematically illustrated in Figure 2b for a reference picture RefPic. Motion vector prediction may use motion vector fields form several previously decoded frames. Figure 3 schematically illustrates motion vector prediction which computes prediction for a current motion vector from previous motion vectors along a motion trajectory, using POC for extrapolation along the motion trajectory.

[0022] In single layer video coding, a reference picture list may comprise pictures from the DPB, where the reference pictures are all identical to output pictures. For example, all pictures in the reference picture list may have the same resolution as the currently predicted picture.

[0023] Figure 4 shows a video coder process diagram for single layer video coding. The video coding system 400 comprises a processor 401 and a memory 402. At memory 402 is a DPB 451 and a stored reference picture list 452.

[0024] The input image is shown at 403. The input image 403 is input to a coding tree unit (CTU) partition 404. CTU partitioning is the process of diving a picture into coding units (CUs) and then further dividing those CUs into prediction units (PUs). A block partition map 405 is produced and applied to the input image at 406.

[0025] The partitioned image 406 is then put through transform 407 and quantization 408 operations, before entropy coding 409 to form a bitstream 410. The transformed and quantized image can undergo inverse quantization 411 and inverse transform 412 operations to code the prediction residual. One or more loop filters 416 are applied in the inter-picture prediction loop. The filtered image is stored in the DPB 451 as a reference for inter-picture prediction. The image can be displayed on a display 417.

[0026] As mentioned above, images in the DPB 451 are used in the reference picture list 452 for motion estimation 413 and inter-picture prediction 414. Intra-picture prediction is also performed at 415.

[0027] Multi-layer video coding is used for example for simultaneous coding of the same video in different resolutions (layers). For example, the base layer is coded in smaller resolution, independently from other layers. An enhancement layer is coded in higher spatial resolution (which may be in dependence on a scaling factor, S) and refers to an already decoded base layer.

[0028] Figure 5 schematically illustrates an example of a scalable multi-layer coder. The operations for the base layer are shown generally at 501 and for the enhancement layer at 502. The generated motion vector (MV) are stored for temporal motion vector prediction (TMVP).

[0029] In multi-layer video coding, the DPB stores a respective decoded picture for the base and enhancement layers. For example, the reference picture list of the enhancement layer may be composed of a previously decoded picture of the enhancement layer and an up-sampled picture of the base layer, as illustrated in Figure 5. Therefore, in this example the reference picture list of the enhancement layer additionally includes previously decoded pictures of enhancement layer generated reference picture, which is up-sampled (with scaled factors) from the decoded base layer picture. In this prior implementation, the generated reference picture is synthesized using only previously decoded pictures. Multi-layer coding is a functionality of version 1 of the standard. It is also based on up-sampling decoded picture of base layer and insertion of this picture to the reference picture list.

[0030] In scalable codec, the generated reference picture has the same POC (corresponding to the same time moment) as the predicted picture. Even if it is possible to derive the motion vector field of the reference picture by scaling the motion vector field of the base layer, for complexity reasons, the motion vector of the up-scaled reference picture is generally assigned to be 0.

[0031] Figure 6 schematically illustrates an example of a video coder process with a generated reference picture in a video coding system 600. Components common with Figure 4 have been labelled with the same reference numerals. The decoded picture buffer is shown at 651. Here, the reference pictures are generated using only the decoded pictures in the DPB, as illustrated at 652. The generated reference pictures are added to the reference picture list 653.

[0032] Generated reference pictures can also be generated using neural network-based Inter (NN-Inter) coding. Different variants of neural networks (such as deep neural networks and convolutional neural networks) can be used for reference picture synthesis. In these techniques, reference picture synthesis uses only previously decoded pictures as input. Furthermore, these methods generally use two previously reconstructed pictures as input, to compute optical flow and to synthesize the reference picture.

[0033] The reference picture generation process should preferably be bit-exact, as this results in the same pixel values on all devices. Otherwise, mismatch between the encoder and decoder reference picture generation process can cause an error during decoding. This error can drift (propagate), producing a corrupt video.

[0034] In prior methods where a neural network is used for reference generation, neural network parameters are often float points. Float point arithmetic is not a deterministic process, and behaves differently depending on platform. Therefore, the float point process in not bit-exact. If the neural network is quantized, then model parameters and each layer output may be rounded to (for example) 16 bits, but without consideration of register overflow. If register overflow occurs, then it is handled differently on different platforms and, again, the process is not bit-exact.

[0035] An overflow aware quantizer is used in the entropy part of the JPEG Al standard. Figure 7 schematically illustrates an example of an overflow aware quantized entropy neural network of the JPEG Al standard. The process uses quantized convolutions, which in addition to the convolution itself includes de-scaling shift and clipping operations. Clipping and de-scaling values in quantized convolutions are trainable in a way that no overflow can be guaranteed.

[0036] Figure 8 schematically illustrates an example of bitstream partitioning in the JPEG Al standard. In the JPEG Al standard, the residual is partitioned, and partitioning is aligned with part of picture.

[0037] An end-to-end (E2E) artificial intelligent (Al) codec converts a picture to a tensor representation. Each tile of the latent tensor corresponds to an area in the picture. Several E2E Al based video codecs are capable of encoding video with uniprediction.

[0038] Examples of video coding with uni-prediction can be seen in “Neural Video Compression with Feature Modulation”, J. Li, B. Li and Y. Yu, Computer Vision and Pattern Recognition (CVPR) 2024 (https: / / arxiv.org / abs / 2402.17414) and “AlphaVC: High-Performance and Efficient Learned Video Compression”, Y. Shi, Y. Ge, J. Wang and J. Mao, European Conference on Computer Vision (ECCV) 2022. In such examples, the next picture can be decoded using data from the bitstream and previously decoded picture in signal or latent representation.

[0039] Another example of an E2E Al based image codec can be seen in Draft International Standard ISO / IEC DIS 6048-1, Information technology - JPEG Al learning-based image coding system. In this example, the image codec converts an image to a latent tensor representation, which is encoded in a predictive manner. The picture is reconstructed from the bitstream.

[0040] Several attempts to combine Al-based intra coding with classical coding previously resulted in a new Intra coding mode, called at block level. Since the rest of the coding is performed on the CPU, it is not efficient to perform Al-based Intra mode coding on a graphics processing unit (GPU) or neural processing unit (NPU), as data transfer has high costs in terms of time and diminishes the benefits.

[0041] Areas with occlusions are coded as Intra blocks. Generated reference pictures in prior methods are generally used only for Inter-coded blocks. Independently coded pictures cannot be handled. Only previously reconstructed pictures are used for reference picture generation, so there is no adaptation of reference picture generation per content, there is no encoder control over the reference picture generation mechanism.

[0042] Furthermore, if a reference picture is generated using a neural network algorithm, bit-exact behaviour of the reference picture generation is not guaranteed.

[0043] It is desirable to develop an approach that can overcome at least some of the above issues.

[0044] SUMMARY

[0045] The present disclosure provides methods and apparatuses to improve the encoding and decoding of picture data.

[0046] The foregoing and other objects are achieved by the subject matter of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures.

[0047] Particular embodiments are outlined in the attached independent claims, with other embodiments in the dependent claims.

[0048] According to a first aspect, there is provided a decoding method, implemented by a decoder for predicting a currently decoded picture in dependence on a synthetic reference picture, the method comprising: parsing multiple bitstream parts, the multiple bitstream parts comprising an additional bitstream part containing information relevant to the formation of the synthetic reference picture; from the information contained in the additional bitstream part, determining a synthetic reference picture type for the synthetic reference picture; and generating the synthetic reference picture in dependence on the determined synthetic reference picture type; wherein the synthetic reference picture differs from pictures previously decoded from the bitstream.

[0049] Generating the synthetic reference picture from information contained in the additional bitstream part may allow for content adaptation and the handling of Intra areas (such as occlusions or independently coded areas).

[0050] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination.

[0051] The method may comprise generating the synthetic reference picture in dependence on the determined synthetic reference picture type and one or more pictures previously decoded from the bitstream. The one or more pictures previously decoded from the bitstream may be decoded from other parts of the bitstream to the additional part. The multiple parts of the bitstream may comprise substreams of the bitstream. For example, the additional part may be an additional part to one or more residual substreams of the bitstream.

[0052] The method may comprise generating the synthetic reference picture from the information contained in the additional bitstream part. The method may comprise generating the synthetic reference picture from syntax elements signalled in the additional bitstream part. This may allow the information relevant to the formation of the synthetic reference picture and the type of synthetic reference picture to be decoded from the additional bitstream part and used for reference picture generation.

[0053] The method may comprise generating the synthetic reference picture from the information contained in the additional bitstream part using an overflow aware quantized neural network. This may ensure bit-exact behavior.

[0054] The synthetic reference picture may not be displayed by a display device in communication with the decoder. This may result in improved efficiency.

[0055] The synthetic reference picture type may indicate one of a plurality of reference picture generation methods. This may allow the synthetic reference picture to be generated at the decoder using an appropriate method for its type.

[0056] The method may comprise assigning a motion vector field of the synthetic reference picture to be used for further motion vector prediction in dependence on the synthetic reference picture type and / or optical flow information parsed from one or more of the multiple bitstream parts and / or optical flow information derived from previously decoded picture analysis. Motion vector field construction dependency on the type of reference picture generation may allow more accurate motion vector prediction.

[0057] The method may comprise assigning quality parameter information to the synthetic reference picture based on the synthetic reference picture type and / or quality information parsed from the additional bitstream part. This may allow synthetic reference picture information to be decoded from the additional bitstream part.

[0058] The synthetic reference picture may not be equivalent to any previously reconstructed pictures stored in a memory of the decoder. The synthetic reference picture may instead be generated from information contained in an additional bitstream part. This may allow the decoding and reference picture generation to be split across different processors in a computing system.

[0059] The method may comprise adding the generated synthetic reference picture to a reference picture list. This may allow the decoder to use the synthetic reference picture in the decoding of a current picture.

[0060] The method may further comprise, after the decoding of the currently decoded picture using the synthetic reference picture, removing the synthetic reference picture from a memory of the decoder. This may be more memory-efficient for the video coding system.

[0061] Reference picture information of the synthetic reference picture for motion vector prediction may be dependent on the type of the synthetic reference picture. This may allow information such as motion vector field and quality parameters to be generated by the encoder using an appropriate method. The reference picture information may comprise a motion vector field and a quality parameter map for the synthetic reference picture. This may allow the generated synthetic reference picture and associated information to be used for processes such as motion vector prediction.

[0062] The reference picture information for the synthetic reference picture may be artificial intelligent-based coded. This may allow for compatibility with artificial-intelligence-based codecs, such as neural network-based codecs.

[0063] The method may further comprise generating the synthetic reference picture at a first processor; inserting the synthetic reference picture into a hybrid codec; and executing the hybrid codec at a second processor, wherein data exchange between the first processor and the second processor occurs through a shared memory of the decoder. This may allow video coding systems to make use of multiple processors of a computing system, such as graphics processing units and neural processing units in addition to the central processing unit. This may result in a faster computational speed.

[0064] The synthetic reference picture may be generated from an end-to-end artificial intelligent codec stream and a previously decoded picture of the previously decoded pictures. This may allow for end-to-end formation of the synthetic reference picture and the currently decoded picture using an artificial intelligent dedicated processor.

[0065] The end-to-end artificial intelligent codec stream may be a JPEG Al codec stream or an analogue end-to-end learnable codec. This may allow for compatibility with existing image and video codec streams.

[0066] According to another aspect, there is provided a decoding apparatus comprising processing circuitry configured to execute steps of the decoding method having any of the features above.

[0067] According to a further aspect, there is provided a decoder comprising one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors, wherein the storage medium stores programming for execution by the one or more processors, wherein the programming, when executed by the one or more processors, configures the decoder to carry out the decoding method having any of the features described above.

[0068] Such apparatus for decoding may refer to the same advantageous effect as the method for decoding according to the first aspect. Details are not described herein again. The decoding apparatus provides technical means for implementing an action in the method defined according to the first aspect. The function may be implemented by hardware, or may be implemented by hardware executing corresponding software.

[0069] According to another aspect, there is provided an encoding method, implemented by an encoder, the method comprising: receiving an input signal; and encoding the input signal into a bitstream comprising multiple bitstream parts, the multiple bitstream parts comprising an additional bitstream part containing information relevant to the formation of a synthetic reference picture, wherein the information contained in the additional bitstream part comprises an indication of a synthetic reference picture type for the synthetic reference picture.

[0070] Encoding an additional bitstream part comprising information relevant to the formation of the synthetic reference picture may allow for content adaptation and the handling of Intra areas (such as occlusions or independently coded areas) during decoding.

[0071] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. The synthetic reference picture type may indicate one of a plurality of reference picture generation types. This may allow the reference picture generation method for the synthetic reference picture to be included in the bitstream so that the decoder can generate the synthetic reference picture.

[0072] The additional bitstream part may be coded using partial region-based coding. This may allow the approach to be used in cases where only part of the reference picture and its associated information is coded by the encoder.

[0073] According to a further aspect, there is provided an encoder comprising one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors, wherein the storage medium stores programming for execution by the one or more processors, wherein the programming, when executed by the one or more processors, configures the decoder to carry out the encoding method having an of the features described above.

[0074] According to another aspect, there is provided an encoding apparatus comprising processing circuitry configured to execute steps of the encoding method having any of the features described above.

[0075] Such apparatus for encoding may refer to the same advantageous effect as the method for encoding described above. Details are not described herein again. The encoding apparatus provides technical means for implementing an action in the encoding method described above. The function may be implemented by hardware, or may be implemented by hardware executing corresponding software.

[0076] According to another aspect, there is provided a non-transitory storage medium comprising a bitstream encoded by the encoding method having any of the features described above.

[0077] According to another aspect, there is provided a computer program stored on a non-transitory medium and including code instructions, which, when executed on one or more processor, causes the one or more processor to execute the decoding method or the encoding method having any of the features described above.

[0078] According to another aspect, there is provided a system for delivering a bitstream, the system including at least one storage medium configured to store at least one bitstream generated by the encoding method having any of the features described above.

[0079] According to a further aspect, there is provided a video stream encoding apparatus, including a processor and a memory. The memory stores instructions that cause the processor to perform the encoding method having an of the features described above.

[0080] According to a further aspect, a computer-readable storage medium having stored thereon instructions that when executed cause one or more processors to encode video data is proposed. The instructions cause the one or more processors to perform the method according to the first or second aspect or any possible embodiment of the first or second aspect.

[0081] According to a further aspect, the present disclosure relates to a computer program product including program code for performing the method according to the first or second aspect or any possible embodiment of the first or second aspect when executed on a computer.

[0082] Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE FIGURES

[0083] Figures la to Id show an example of a video coder with motion compensation.

[0084] Figures 2a and 2b schematically illustrate the construction of a motion vector field of a reference picture.

[0085] Figure 3 schematically illustrates motion vector prediction.

[0086] Figure 4 schematically illustrates a video coder process diagram.

[0087] Figure 5 schematically illustrates an example of a scalable multi-layer coder.

[0088] Figure 6 schematically illustrates a video coder process with a generated synthetic reference picture.

[0089] Figure 7 schematically illustrates an overflow aware quantized entropy neural network of the JPEG Al standard.

[0090] Figure 8 schematically illustrates bitstream partitioning in the JPEG Al standard.

[0091] Figure 9 schematically illustrates an implementation where a synthetic reference picture is generated from an additional bitstream part (and optionally previously decoded pictures).

[0092] Figure 10 schematically illustrates an implementation where the synthetic reference picture generation method is selected based on the type of the synthetic reference picture.

[0093] Figure 11 schematically illustrates another implementation where the synthetic reference picture generation method is selected based on the type of the synthetic reference picture.

[0094] Figure 12 schematically illustrates an implementation where the generated synthetic reference picture and reference picture information are generated using an overflow aware quantized neural network.

[0095] Figure 13 schematically illustrates an implementation where the generated synthetic reference picture and reference picture information generation is independently from the hybrid video coder.

[0096] Figure 14 schematically illustrates an implementation where the generated synthetic reference picture and reference picture information are generated from partial bitstream.

[0097] Figure 15 shows the steps of an exemplary decoding method.

[0098] Figure 16 shows the steps of an exemplary encoding method.

[0099] Figure 17 schematically illustrates an implementation where the generated synthetic reference picture is coded as an end- to-end artificial intelligent image codec.

[0100] Figure 18 schematically illustrates an implementation of partial synthesis of the synthetic reference picture with end-to- end artificial intelligent image codec. Figure 19 schematically illustrates an implementation where partial bitstream signaling is used for end-to-end artificial intelligent image codec.

[0101] Figure 20 schematically illustrates video coding with motion compensation, where the synthetic reference picture is generated from the end-to-end artificial intelligent codec stream and a previously decoded picture.

[0102] Figure 21 schematically illustrates video coding with motion compensation and frame reordering, where the synthetic reference picture is generated from the end-to-end artificial intelligent codec stream and multiple previously decoded pictures.

[0103] Figure 22 shows the steps of another exemplary decoding method.

[0104] Figure 23 shows the steps of another exemplary encoding method.

[0105] Figure 24 shows a device for decoding.

[0106] Figure 25 shows a device for encoding.

[0107] Figure 26 is a block diagram showing an example of a video coding system configured to implement embodiments of the present disclosure.

[0108] Figure 27 is a block diagram showing another example of a video coding system configured to implement embodiments of the present disclosure.

[0109] Figure 28 is a block diagram illustrating an example of an encoding apparatus or a decoding apparatus.

[0110] Figure 29 is a block diagram illustrating another example of an encoding apparatus or a decoding apparatus.

[0111] Figure 30 is a block diagram illustrating another example of an encoding apparatus or a decoding apparatus.

[0112] DETAILED DESCRIPTION

[0113] Embodiments of the present invention concern video coding devices where the decoder can predict a currently decoded picture from a bitstream in dependence on a synthetic reference picture.

[0114] In embodiments of the present invention, the bitstream comprises multiple parts. The multiple parts comprise an additional bitstream containing information relevant to the formation of the synthetic reference picture. The additional bitstream part may not include information used to reconstruct a currently decoded picture (or frame) itself, such as an encoded residual component of a currently decoded picture. Each bitstream part may be a substream of the bitstream. One or more pictures previously decoded from the bitstream may be decoded from other parts of the bitstream to the additional part. For example, the additional part may be an additional part to one or more residual substreams of the bitstream. From the information contained in the additional bitstream part, the decoder determines a synthetic reference picture type for the synthetic reference picture. The decoder generates the synthetic reference picture in dependence on the determined synthetic reference picture type and one or more previously decoded pictures.

[0115] The synthetic reference picture differs from the previously decoded pictures, such as those stored in memory at the DPB of the video coding system.

[0116] Figure 9 schematically illustrates an example where an external synthetic reference picture (gRefPic) is generated from an additional bitstream part (and optionally previously decoded pictures).

[0117] Figure 9 shows a video coder process diagram for a video coding system 900. The video coding system 900 comprises a first processor 901, a second processor 902 and a memory 903. At memory 903 is a decoded picture buffer (DPB) 951 and a stored reference picture list 952.

[0118] The input image is shown at 904. The input image 904 is input to a coding tree unit (CTU) partition 905. CTU partitioning is the process of diving a picture into coding units (CUs) and then further dividing those CUs into prediction units (PUs). A block partition map 906 is produced and applied to the input image at 907.

[0119] The partitioned image 907 is then put through transform 908 and quantization 909 operations, before entropy coding 910 to form a bitstream 911.

[0120] The transformed and quantized image can undergo inverse quantization 912 and inverse transform 913 operations to code the prediction residual.

[0121] One or more loop filters 914 are applied in the inter-picture prediction loop. The filtered image is stored in the DPB 951 as a reference for inter-picture prediction. The image can be displayed on a display 915.

[0122] In this implementation, the second processor 902 performs the reference picture synthesis. However, in some implementations, the coder and the reference picture synthesis may be performed by the same processor.

[0123] The first processor 901 may be a central processing unit (CPU). The second processor may be a graphics processing unit (GPU) or neural processing unit (NPU). One or more of the processors (for example, the NPU) may perform NN-based picture synthesis.

[0124] The decoder parses multiple bitstream parts, the multiple bitstream parts comprising the additional bitstream part containing information relevant to the formation of the synthetic reference picture.

[0125] The processor 902 parses the additional bitstream part for reference picture synthesis, indicated at 921. The synthesised reference picture is generated from the additional bitstream part, as indicated at 922. The generated synthesised reference picture can be stored in the reference picture list 952 at memory 903.

[0126] Images in the reference picture list 952 are used for motion estimation 916 and inter-picture prediction 917. Intra-picture prediction is also performed at 918. The synthetic reference picture is different to previously reconstructed pictures. The synthetic reference picture may not be equivalent to any previously reconstructed pictures stored in a memory of the decoder. In this example, the synthetic reference picture is different to pictures stored in the DPB 951. The synthetic reference picture can be added to the reference picture list (RefPicList). The generated synthetic reference picture is not necessarily displayed at display 915 in communication with the decoder.

[0127] Generally, the synthetic reference picture is generated using information signalled in the additional bitstream part, for example from syntax elements signalled in the additional bitstream part.

[0128] After the decoding of the currently decoded picture using the synthetic reference picture, the synthetic reference picture can be removed from memory 903. For example, the generated synthetic reference picture can be removed from the DPB and no longer storied in shared memory.

[0129] Further details regarding the formation of the synthetic reference picture ad reference picture information from the information contained in the additional bitstream part will now be described.

[0130] The information contained in the additional bitstream part comprises an indication of a synthetic reference picture type for the synthetic reference picture. The synthetic reference picture type may indicate one of a plurality of reference picture generation methods.

[0131] Figure 10 schematically illustrates an example of a video coding system 1000 where the generation method for the synthetic reference picture is selected based on the type of the synthetic reference picture.

[0132] The operation of the video coder at the first processor 901 is the same as described previously with reference to Figure 9.

[0133] As mentioned previously, the type of synthetic reference picture is signalled in the bitstream. The type of synthetic reference picture is signalled in the additional bitstream part. The type of the synthetic reference picture can be included in the information signalled in the additional bitstream part.

[0134] At 1001, the additional bitstream part is parsed and at 1002 the type of synthetic reference picture is determined from the information contained in the additional bitstream part.

[0135] In dependence on the type of the synthetic reference picture signalled in the additional bitstream part, the synthetic reference picture is generated by one of multiple synthetic reference picture generators. In the example of Figure 10, there are three synthetic reference picture generators: synthetic reference picture generator #1 1003, synthetic reference picture generator #2 1004, synthetic reference picture generator #3 1005.

[0136] The type of the synthetic reference picture may indicate one of multiple different types of reference picture generation method. The synthetic reference picture generator that generators the synthetic reference picture used to decode a currently decoded picture may be selected to correspond to the reference picture generation method indicted by the type of the synthetic reference picture.

[0137] The decoder generates the synthetic reference picture at one of the synthetic reference picture generators 1003, 1004, 1005 in dependence on the determined synthetic reference picture type and one or more previously decoded pictures. The synthetic reference picture differs from the previously decoded pictures, for example the pictures stored in the DPB 951.

[0138] The decoder then uses the synthetic reference picture to decode the currently decoded picture as described previously.

[0139] Figure 11 schematically illustrates another example where the generation method for the synthetic reference picture is selected based on the type of the synthetic reference picture. This example illustrates the formation of motion vector (MV) field for the synthetic reference picture.

[0140] The decoder may assign a motion vector field of the synthetic reference picture to be used for further motion vector prediction in dependence on the synthetic reference picture type and / or optical flow information parsed from one or more of the multiple bitstream parts and / or optical flow information derived from previously decoded picture analysis.

[0141] The decoder may further assign quality parameter information (not pictured) to the synthetic reference picture based on the synthetic reference picture type and / or quality information parsed from the additional bitstream part.

[0142] Reference picture information of the synthetic reference picture for motion vector prediction may be dependent on the type of the synthetic reference picture. The reference picture information may comprise a motion vector field and a quality parameter map (not pictured) for the synthetic reference picture.

[0143] For each of the decoded pictures in the DPB, indicated at 1101, the MV field for MVP is equal to the aggregated MV from decoded. The MV field from decoding are indicated at 1102.

[0144] In the implementation shown in F igure 11 , for motion vector prediction (MVP), reference picture information (including the MV field) of the synthetic reference picture depends on the type of the synthetic reference picture.

[0145] At 1103, the additional bitstream part is parsed and at 1104 and 1105 the type of synthetic reference picture is determined from the information contained in the additional bitstream part.

[0146] In dependence on the type of the synthetic reference picture signalled in the additional bitstream part, the MV field for the synthetic reference picture is generated by one of multiple MV field generators. In the example of Figure 11, there are three MV field generators: MV field generator #1 1106, MV field generator #2 1107, MV field generator #3 1108.

[0147] The type of the synthetic reference picture may indicate one of multiple different types of MV field generation method. The MV field generator that generators the MV field used for MVP for a currently decoded picture may be selected to correspond to the MV field generation method indicted by the type of the synthetic reference picture.

[0148] In dependence on the type of the synthetic reference picture signalled in the additional bitstream part, the synthetic reference picture is generated by one of multiple synthetic reference picture generators. In the example of Figure 11, there are three synthetic reference picture generators: synthetic reference picture generator #1 1109, synthetic reference picture generator #2 1110, synthetic reference picture generator #3 1111.

[0149] The type of the synthetic reference picture may indicate one of multiple different types of reference picture generation method. The synthetic reference picture generator that generators the synthetic reference picture used to decode a currently decoded picture may be selected to correspond to the reference picture generator method indicted by the type of the synthetic reference picture.

[0150] The decoder generates the synthetic reference picture at one of the synthetic reference picture generators 1109, 1110, 1111 in dependence on the determined synthetic reference picture type and one or more previously decoded pictures. The synthetic reference picture and the corresponding MV fields differ from the previously decoded pictures and corresponding MV fields, for example the pictures and MV fields stored in the DPB shown at 1101, 1102. The synthetic reference picture can be stored in a synthetic reference picture list, as indicated at 1112, along with the corresponding MV fields, as indicated at 1113.

[0151] Figure 12 schematically illustrates an implementation in system 1200 where the generated synthetic reference picture and reference picture information are generated using an overflow aware quantized neural network (NN). This may be implemented for cases where the synthetic reference picture is Al-based coded.

[0152] The synthetic reference picture and the corresponding synthetic reference picture information (such as the MV field and in some implementations a QP map) are generated from the information contained in the additional bitstream part using the overflow aware quantized neural network. The overflow aware quantized neural network (with quantized convolutions) can be used to ensure bit-exact behaviour.

[0153] The parsing of the additional bitstream part and information at 1103 and the determination of the type of the synthetic reference picture at 1104 and 1105 are the same as described previously with reference to Figure 11.

[0154] The overflow aware quantised NN is indicated at 1201. Multiple MV field generators and multiple synthetic reference picture generators can be implemented by the NN 1201.

[0155] In this example, in dependence on the type of the synthetic reference picture signalled in the additional bitstream part, the MV field for the synthetic reference picture is generated by one of multiple MV field generators. In the example of Figure 12, there are three MV field generators: MV field generator #1 1202, MV field generator #2 1203, MV field generator #3 1204.

[0156] The type of the synthetic reference picture may indicate one of multiple different types of MV field generation method. The MV field generator that generators the MV field used for MVP for a currently decoded picture may be selected to correspond to the MV field generation method indicted by the type of the synthetic reference picture.

[0157] In dependence on the type of the synthetic reference picture signalled in the additional bitstream part, the synthetic reference picture is generated by one of multiple synthetic reference picture generators. In the example of Figure 12, there are three synthetic reference picture generators: synthetic reference picture generator #1 1205, synthetic reference picture generator #2 1206, synthetic reference picture generator #3 1207.

[0158] The synthetic reference picture and the corresponding MV fields differ from the previously decoded pictures and corresponding MV fields, for example the pictures and MV fields stored in the DPB shown at 1101 , 1102. The synthetic reference picture can be stored in a synthetic reference picture list, as indicated at 1208, along with the corresponding MV fields, as indicated at 1209. Figure 13 schematically illustrates an implementation in a system 1300 where the generated synthetic reference picture and reference picture information (MV fields, etc) are independent from the hybrid video coder.

[0159] In this example, a first processor 1301 implements the video coder. A second processor 1302 generates the synthetic reference picture and reference picture information (MV field, etc). The synthetic reference picture is generated using processor 1302, independently of processor 1301. The synthetic reference picture can be inserted into the hybrid codec, which is executed in processor 1301. Data exchange between processor 1301 and processor 1302 can occur through a shared memory 1303.

[0160] In this case, the decoding method comprises generating the synthetic reference picture at processor 1302; inserting the synthetic reference picture into a hybrid codec; and executing the hybrid codec at processor 1301, with data exchange between the first processor and the second processor occurring through shared memory 1303 of the video coding system 1300.

[0161] The parsing of the additional bitstream part and information at 1306 and the determination of the type of the synthetic reference picture at 1307 and 1308 are the same as described previously with reference to Figure 11.

[0162] In this example, in dependence on the type of the synthetic reference picture signalled in the additional bitstream part, the MV field for the synthetic reference picture is generated by one of multiple MV field generators. In the example of Figure 13, there are three MV field generators: MV field generator #1 1309, MV field generator #2 1310, MV field generator #3 1311.

[0163] The type of the synthetic reference picture may indicate one of multiple different types of MV field generation method. The MV field generator that generators the MV field used for MVP for a currently decoded picture may be selected to correspond to the MV field generation method indicted by the type of the synthetic reference picture.

[0164] In dependence on the type of the synthetic reference picture signalled in the additional bitstream part, the synthetic reference picture is generated by one of multiple synthetic reference picture generators. In the example of Figure 13 there are three synthetic reference picture generators: synthetic reference picture generator #1 1312, synthetic reference picture generator #2 1313, synthetic reference picture generator #3 1314.

[0165] The synthetic reference picture and the corresponding MV fields differ from the previously decoded pictures and corresponding MV fields, for example the pictures and MV fields stored in the DPB shown at 1304 and 1305 respectively. The synthetic reference picture can be stored in a synthetic reference picture list, as indicated at 1315, along with the corresponding MV fields, as indicated at 1316.

[0166] Figure 14 schematically illustrates an implementation in a system 1400 where the generated synthetic reference picture and reference picture information are generated from a partial bitstream. The video coder and the reference picture generation are shown in this example as being performed by two different processors 1401 and 1402 respectively. However, in some implementations they may be performed by the same processor.

[0167] In this implementation, the additional bitstream part may be coded by the encoder using partial region-based coding.

[0168] The partial additional bitstream part and information is parsed at 1405 and the type of the synthetic reference picture is determined at 1406 and 1407. In this example, in dependence on the type of the synthetic reference picture signalled in the additional bitstream part, the MV field for the synthetic reference picture is generated by one of multiple MV field generators. In the example of Figure 14, there are three MV field generators: MV field generator #1 1408, MV field generator #2 1409, MV field generator #3 1410.

[0169] The type of the synthetic reference picture may indicate one of multiple different types of MV field generation method. The MV field generator that generators the MV field used for MVP for a currently decoded picture may be selected to correspond to the MV field generation method indicted by the type of the synthetic reference picture.

[0170] In dependence on the type of the synthetic reference picture signalled in the additional bitstream part, the synthetic reference picture is generated by one of multiple synthetic reference picture generators. In the example of Figure 14, there are three synthetic reference picture generators: synthetic reference picture generator #1 1411, synthetic reference picture generator #2 1412, synthetic reference picture generator #3 1413.

[0171] If the synthetic reference picture was partially coded, then the reference picture index (Refldx) and MV field does not refer to non-encoded areas of the synthetic reference picture. This combination of Refldx and MV field for the nonencoded areas does not appear in the bitstream.

[0172] The partial synthetic reference pictures and the corresponding MV fields differ from the previously decoded pictures and corresponding MV fields, for example the pictures and MV fields stored in the DPB shown at 1403, 1404. The partial synthetic reference pictures can be stored in a synthetic reference picture list, as indicated at 1414, along with the corresponding partial MV fields, as indicated at 1415.

[0173] After decoding of the current picture, the synthetic reference picture can be removed from the reference picture list and no longer stored in memory.

[0174] In the examples described herein, the reference picture information for the synthetic reference picture may be artificial intelligent-based coded. For example, the synthetic reference picture may be generated from an end-to-end artificial intelligent codec stream and a previously decoded picture of the previously decoded pictures. The end-to-end artificial intelligent codec stream may be a JPEG Al codec stream or an analogue end-to-end learnable codec.

[0175] Figure 15 shows an example of the steps of a decoding method 1500, implemented by a decoder for predicting a currently decoded picture in dependence on a synthetic reference picture. At step 1501, the method comprises parsing multiple bitstream parts, the multiple bitstream parts comprising an additional bitstream part containing information relevant to the formation of the synthetic reference picture. At step 1502, the method comprises, from the information contained in the additional bitstream part, determining a synthetic reference picture type for the synthetic reference picture. At step 1503, the method comprises generating the synthetic reference picture in dependence on the determined synthetic reference picture type (and optionally one or more previously decoded pictures), wherein the synthetic reference picture differs from pictures previously decoded from the bitstream.

[0176] The decoder performs reconstruction of part(s) of or the whole reference picture according to the method defined by the synthetic reference picture type from the bitstream (for example, from the additional bitstream part) or / and previously decoded pictures. The decoder can assign a motion vector field of the synthetic reference picture to be used for further motion vector prediction depending on the type of the synthetic reference picture or / and optical flow information parsed from the bitstream. The decoder can assign quality parameter (QP) information of the synthetic reference picture based on the type of the synthetic reference picture or / and quality information parsed from the bitstream (for example, from the additional bitstream part).

[0177] Figure 16 shows an example of the steps of an encoding method 1600, implemented by an encoder. At step 1601, the method comprises receiving an input signal. At step 1602, the method comprises encoding the input signal into a bitstream comprising multiple bitstream parts, the multiple bitstream parts comprising an additional bitstream part containing information relevant to the formation of a synthetic reference picture, wherein the information contained in the additional bitstream part comprises an indication of a synthetic reference picture type for the synthetic reference picture.

[0178] In another aspect, the synthetic reference picture may be generated from an end-to-end (E2E) artificial intelligent code stream.

[0179] In such implementations, the decoder is configured to parse one or more bitstream parts comprising an end-to-end artificial intelligent image codec stream and generate the synthetic reference picture from information contained in the end-to-end artificial intelligent image codec stream.

[0180] The method may comprise generating the synthetic reference picture by means of the codec of the end-to-end artificial intelligent image codec stream.

[0181] This aspect may enable the use of a second processor dedicated to artificial intelligent operations. The synthetic reference picture may be generated at the second processor. Such operations are highly parallelizable on dedicated processors, and so the generation of the artificial reference frame can be carried out at a second processor independently from the main coding process, which can be implemented at a first processor. This may reduce the overall latency.

[0182] In such implementations, the encoder receives an input signal and encodes the input signal into a bitstream. The bitstream comprises one or more bitstream parts comprising an end-to-end artificial intelligent image codec stream. Information contained in the end-to-end artificial intelligent image codec stream comprises an indication of a synthetic reference picture type for a synthetic reference picture to be generated from the information contained in the end-to-end artificial intelligent image codec stream. As described above, the synthetic reference picture type may indicate one of a plurality of reference picture generation types.

[0183] At the encoder end, the end-to-end artificial intelligent codec stream can convert the synthetic reference picture to a latent tensor representation. Each tile of the latent tensor representation can correspond to an area in the synthetic reference picture.

[0184] The decoder may determine a synthetic reference picture type for the synthetic reference picture, wherein the determined synthetic reference picture type indicates that the synthetic reference picture is to be generated from the end-to-end artificial intelligent image codec stream.

[0185] The end-to-end artificial intelligent codec stream may be, for example, a JPEG Al codec stream or an analogue end-to- end learnable codec. Figure 17 schematically illustrates an example of video coding with motion compensation (MC) and synthetic reference picture insertion coded as an end-to-end Al image codec. An E2E Al image codec bitstream 1701 is decoded by an E2E Al image decoder 1702.

[0186] For each frame (at POC t-2, t-1, t, etc), MV are stored for MVP, as indicated at 1703 and 1704 for time frames t-2 and t- 1 respectively.

[0187] In this example, the type of the synthetic reference picture is determined from the bitstream to be an Al-Intra coded synthetic reference picture. The synthetic reference picture may be a JPEG Al-coded synthetic reference picture. In other implementations, the synthetic reference picture may be coded by an analogue end-to-end learnable codec. The synthetic reference picture type may be signalled as part of the end-to-end Al code stream.

[0188] In this example, for the synthetic reference pictured decoded from the E2E Al image codec bitstream, the MV corresponding to the synthetic reference picture may be set to zero, as illustrated at 1705. In this example, the decoding method comprises assigning a motion vector field of the synthetic reference picture for motion vector prediction to be all zeros.

[0189] If during motion compensation for a currently decoded block a reference picture index (Refldx) indicates that the synthetic reference picture is to be generated from the end-to-end artificial intelligent image codec stream, the method may comprise forcing motion vector information for the currently decoded block to be zero.

[0190] The decoding method may comprise generating only parts of the synthetic reference picture needed for referencing during motion compensation. The method may comprise parsing an additional bitstream that includes only tiles corresponding to the areas of the synthetic reference picture needed for referencing during prediction.

[0191] The method may comprise deriving a quality parameter (QP) map for luma and chroma of the synthetic reference picture from quality parameters of the end-to-end artificial intelligent codec stream. The QPs of the end-to-end artificial intelligent codec stream may comprise beta_y and beta_uv. Parameters such as chroma_QP_offset may be derived from these beta values.

[0192] The motion vector information for the currently decoded block may not be coded. This may be implemented as conditional skipping of motion vector information signaling and inferring this value to be zero for the blocks referencing to the synthetic reference picture generated from the end-to-end artificial intelligent image codec stream. In this way, bit(s) for motion vector information will be saved. This can reduce the complexity of the motion compensation process, since motion compensation with zero motion vector just involves copying without interpolation.

[0193] The method may comprise assigning a motion vector field of the synthetic reference picture for motion vector prediction to be all zeros. This would correspond to a static object or occlusions for which no information about previous motion trajectory is known.

[0194] If during motion compensation for a currently decoded block a reference picture index indicates that the synthetic reference picture is to be generated from the end-to-end artificial intelligent image codec stream, the method may comprise forcing motion vector information for the currently decoded block to be zero. This may be implemented as an encoder constraint disallowing signaling values other than zero for the blocks referencing to the synthetic reference picture generated from the end-to-end artificial intelligent image codec stream. This may reduce the complexity of the motion compensation process, since, as mentioned previously, motion compensation with zero motion vector just involves copying without interpolation.

[0195] If during motion compensation for a currently decoded block a reference picture index indicates that the synthetic reference picture is to be generated from the end-to-end artificial intelligent image codec stream, the method may comprise coding motion vector information for the currently decoded block using a mode without residual coding or / and without motion information coding. This may be implemented as conditional skipping of motion vector information and residual signaling and inferring this value to be zero for the blocks referencing to the synthetic reference picture generated from the end-to-end artificial intelligent image codec stream. This way, bit(s) for motion vector information and residual may be saved. This may reduce the complexity of the block reconstruction process, since motion compensation with zero motion vector is just copy without interpolation and zero residual does not require a transform operation.

[0196] A block type for the currently decoded block may not be coded. This may be implemented as conditional skipping of block type information signaling for the blocks referencing to the synthetic reference picture generated from the end-to- end artificial intelligent image codec stream. This can save bits for block type signaling.

[0197] In this implementation, the reference picture list to which the synthetic picture is added after decoding can be used in frame types where otherwise no reference pictures are used. For example, the image frame type may be an I-frame.

[0198] If the synthetic reference picture (gRefPic) is generated from the E2E Al image codec stream, then the type of the synthetic reference picture (gRefPicType) can be assigned to be ailntraRef. If gRefPicType = ailntraRef then motion vector field for MVP of gRefPic is assigned to be all 0. If gRefPicType= ailntraRef then a QP map for luma and chroma can be derived from quality parameters of the E2E Al codec (for example, beta_y and beta_uv in the case of JPEG Al) .

[0199] If during motion compensation in a currently coded block the Refldx indicates that RefPic is ailntraRef, motion vector information for this block may be forced to be zero.

[0200] If during motion compensation in a currently coded block the Refldx indicates that RefPic is ailntraRef, motion vector information for this block may be forced to be zero and not coded.

[0201] If during motion compensation in a currently coded block the Refldx indicates that RefPic is ailntraRef, this block can be coded using skip mode.

[0202] Skip mode is a tool that can be used in the codec to tell the entropy coder which residual elements should be decoded from the bitstream and which should not. Some functions relevant to skip mode will now be described. Skip mode has a lower decoder complexity compared to other coding modes. Greater usage of skip mode can improve the entropy coder thoughput.

[0203] Masking generation is a function which is used in skip mode. The input of the mask generation core function is a tensor with sigma (variance) samples cr (o and (Juvin primary and secondary components coding pipelines) of size [C, h4, w4], which is the output of the hyper decoder (107) and used by AE (105 and 106) for estimation of symbols’ probabilities. The size relates to the size of the input image [3, / i,w] in the following way: h4= roun . The output is a mask mask[C, h4,w4] to be used by the skip mode coding tool when enabled. To generate the mask, four syntax elements are used, namely Threshold, GreaterFlag, Mode and BlockSize. These syntax elements can be included in the picture header. Mask generation is generally a three-step process.

[0204] A first step is pooling. If the BlockSize is greater than 1, a pooling operation is applied to the input sigma samples tensor first. Based on the value of the Mode syntax element, the pooling operation is either max pooling, average pooling or min pooling, with a kernel size equal to BlockSize in horizontal and vertical dimensions.

[0205] Afterwards, each one of the pooled sigma samples are compared with the Threshold, and the comparison is stored in a pooled mask tensor maskp. The pooled mask samples can be obtained according to the following:

[0206] After the pooled mask samples tensor, maskp, is obtained, and if the BlockSize is greater than 1 , an up-sampling operation is applied to maskp to obtain the final mask samples tensor. The up-sampling operation can be based on nearest neighbors. If the BlockSize is equal to 1 , the up-sampling operation is skipped. If the BlockSize is greater than 1 , a cropping operation is applied after up- sampling resulting in an output mask tensor with size [C, h4,w4] mask[c, i,j] = maskp[c, i / BlockSize ,j / BlockSize ], c = 0. . C — 1, i = 0. . h4— 1 and j = 0. . w4— 1.

[0207] The residual skip process can use one set of control parameters, which means numSkipparams is one. Four sets of parameters Threshold[numSkipparams] , GreaterFlag[numSkipparams] , Mo de [numSkipparams] and BlockSize[numSkipparams] are used to define the mask, as described above.

[0208] - Tensors r and maskAggregate are initialized to be equal to all zeros and all ones respectively

[0209] - The counter k = 0 - For c = 0.. C — 1, i = 0.. h4— 1 , j = 0.. w4— 1 maskAggregate[c, i,j] = maskAggregate[c, i,j] • mask[idx][c, i,j]

[0210] - For c = 0.. C — 1, i = 0.. h4— 1 , j = 0.. w r[c, i,j] = ma

[0211] The output of this process is the residual tensor r.

[0212] If during motion compensation in a currently coded block the Refldx indicates that the synthetic reference picture has type ailntraRef, then this block is coded using skip mode and block type (SKIP flag in HEVC / VVC) is not coded.

[0213] Figure 18 schematically illustrates partial synthesis with an E2E Al image codec. Only parts of the synthetic reference picture are generated. In this example, parts 1801, 1802 of the complete synthetic reference picture 1800 are generated.

[0214] If the type of the synthetic reference picture is determined to be ailntraRef, only parts of the synthetic reference picture needed for referencing during the MC process may be synthesized using partial synthesis. Parts of the synthetic reference picture needed for referencing during the MC process’ may include the reference picture index (Refldx) and MV points to this / these part(s) of the synthetic reference picture.

[0215] In such implementations, the additional bitstream may be coded using partial region-based coding.

[0216] Figure 19 schematically illustrates partial bitstream signaling for an end-to-ed Al image codec for a JPEG Al codestream 1900. Some residual tiles are excluded from the bistream and some residual tiles are present in the bitstream.

[0217] If the type of the synthetic reference picture is determined to be ailntraRef, the additional part of the bitstream signalled may include only residual tiles corresponding to the areas needed for referencing during MC process (partial coding), as described above.

[0218] If the synthetic reference picture (gRefPic) is generated from the E2E Al codec stream and a previously decoded picture (for example, a decoded picture in the DPB) then the synthetic reference picture type (gRefPicType) can assigned to be aiPRef. Synthetic reference pictures with type aiPRef are thus generated from the E2E Al codec stream and a previously decoded picture.

[0219] Figure 20 schematically illustrates video coding with MC, where the synthetic reference picture type is aiPRef.

[0220] An E2E Al image codec bitstream 2001 is decoded by an E2E Al image decoder 2002. The E2E video decoder may be, for example, a known E2E Al decider such as Alpha VC, as described in “Alpha VC: High-Performance and Efficient Learned Video Compression”, Y. Shi, Y. Ge, J. Wang and J. Mao, European Conference on Computer Vision 2022 (https: / / arxiv.org / pdf / 2207.14678) or DCVC-FM, as described in “Neural Video Compression with Feature Modulation”, J. Li, B. Li, Y. Lu, The IEEE / CVF Conference on Computer Vision and Pattern Recognition 2024 (https: / / arxiv.org / pdf / 2402.17414).

[0221] For each frame (at POC t-2, t-1, t, etc), MV are stored for TMVP, as indicated at 2003 and 2004 for time frames t-2 and t-1 respectively. In this example, the type of the synthetic reference picture is determined from the bitstream to be an aiPRef. The synthetic reference picture may be a JPEG Al-coded synthetic reference picture. In other implementations, the synthetic reference picture may be coded by an analogue end-to-end learnable codec. The synthetic reference picture type may be signalled as part of the end-to-end Al code stream.

[0222] In this example, for the synthetic reference pictured decoded from the E2E Al image codec bitstream, the MV corresponding to the synthetic reference picture is derived from optical flow (OF), as illustrated at 2005.

[0223] In this example, the synthetic reference picture type is signalled to be aiPRef. Part of the stream is signalled. The MV for the synthetic reference picture is derived from signals in the E2E Al video codec MV field, specifically from OF. QP such as chroma_QP_offset are derived from ‘beta’. The motion vector field (derived from OF) is signalled to the decoder and used for deformable convolution.

[0224] If the synthetic reference picture (gRefPic) is generated from E2E Al codec stream and more than one previously decoded picture (for example, multiple decoded pictures in the DPB) then gRefPicType is assigned to be aiBRef. Synthetic reference pictures with type aiBRef are thus generated from the E2E Al codec stream and multiple previously decoded pictures.

[0225] Figure 21 schematically illustrates video coding with motion compensation (MC) and frame reordering, where the synthetic reference picture type is aiBRef.

[0226] An E2E Al image codec bitstream is decoded by an E2E Al image decoder. In this example, a mask is signalled in the bitstream, as indicated at 2101. The signalled mask at 2101 may be synthesized using, for example spare iFRNet, as described in “IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation”, L. Kong, B. Jiang, D. Luo, W. Chu, X. Huang, Y. Tai, C. Wand and J. Yang, The IEEE / CVF Conference on Computer Vision and Pattern Recognition 2022.

[0227] For each frame (at POC t-2, t-1, t, etc), MV are stored for TMVP, as indicated at 2102 and 2103 for time frames t+1 and t-1 respectively.

[0228] In this example, the type of the synthetic reference picture is determined from the bitstream to be an aiBRef. The synthetic reference picture may be a JPEG Al-coded synthetic reference picture. In other implementations, the synthetic reference picture may be coded by an analogue end-to-end learnable codec. The synthetic reference picture type may be signalled as part of the end-to-end Al code stream.

[0229] In this example, for the synthetic reference pictured decoded from the E2E Al image codec bitstream, the MV corresponding to the synthetic reference picture is derived from optical flow (OF), as illustrated at 2104.

[0230] In this example, the synthetic reference picture type is signalled to be aiBRef. Part of the stream is signalled. The MV for the synthetic reference picture is derived from signals in the E2E Al video codec MV field, specifically from OF. QP such as chroma_QP_offset are derived from ‘beta’ . The motion vector field (derived from OF) is signalled to the decoder and used for deformable convolution.

[0231] In this example, the synthetic reference picture type is signalled to be aiBRef. Part of the picture is synthesized. For MVP, the MV field is derived from sigals in the E2E Al video codec MV field, specifically from OF. QP such as chroma_QP_offset are derived from ‘beta’ . Motion vector field (OF) is signalled to the decoder and used for deformable convolution. TMVP / HMVP MV = 0 in the prior art.

[0232] Figure 22 shows an example of a decoding method, implemented by a decoder for predicting a currently decoded picture in dependence on a synthetic reference picture. At step 2201, the method comprises parsing one or more bitstream parts comprising an end-to-end artificial intelligent image codec stream. At step 2202, the method comprises generating the synthetic reference picture from information contained in the end-to-end artificial intelligent image codec stream.

[0233] Figure 23 shows an example of an encoding method, implemented by an encoder. At step 2301, the method comprises receiving an input signal. At step 2302, the method comprises encoding the input signal into a bitstream, the bitstream comprising one or more bitstream parts comprising an end-to-end artificial intelligent image codec stream; wherein information contained in the end-to-end artificial intelligent image codec stream comprises an indication of a synthetic reference picture type for a synthetic reference picture to be generated from the information contained in the end-to-end artificial intelligent image codec stream.

[0234] The encoder and decoder described herein may be implemented by an encoding apparatus and a decoding apparatus comprising processing circuitry configured to execute steps described herein. The encoding and / or decoder may comprise one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors. The storage medium can store programming for execution by the one or more processors, wherein the programming, when executed by the one or more processors, configures the decoder to carry out the decoding method having any of the features described above.

[0235] In embodiments of the present invention, content adaptation and the handling of Intra areas (occlusions or independently coded areas) of the process of reference picture synthesis is achieved by the signaling of an additional bitstream part. Furthermore, motion field construction dependency on the type of reference picture generation allows more accurate motion vector prediction. In prior approaches, only a single method of reference picture generation may be used (for example, up-scaling in scalable coding, or deep neural network with optical flow computation for NN-Inter). Additionally, bit-exact behaviour of reference picture generation may be guaranteed by the user of an overflow aware neural network quantizer.

[0236] The approaches described herein may be used, for example, in applications such as TVs, smartphones, tablets, CCTV (surveillance cameras), cloud gaming and gaming streaming, teleconferencing systems and automotive. Besides video on demand segment of video market which can afford off-line (not limited in time) encoding, all other video applications generally require real time encoding. For some applications, such as automotive and cloud gaming, the allowed encoder delay is very small (for example, milliseconds). The present approach can be advantageously used for such applications.

[0237] Modem devices are generally equipped with more than one processor, including a central processing unit (CPU), a graphics processing unit (GPU) and a neural processing unit (NPU). Classical hybrid codecs are generally deployed on the CPU (on the decoder side) and there are no benefits provided by the NPU and GPU. Embodiments of the present invention can enable second processor usage for NN-based picture synthesis with a dedicated processor (such as GPU or NPU) that is faster, for example thousands of times faster, than the hybrid encoder whilst producing the same quality of picture. Embodiments of the present invention can allow a wide range of adaptivity and balancing between encoder speed and compression efficiency quality. In an ultra-low encoder latency comer case, a second processor (for example, GPU or NPU) can generate the synthetic reference picture (very fast) as an entire picture. A first processor implementing the video coder can select only SKIP mode (with no motion vector, no transform). Essentially, only the generated reference picture is used as reconstruction (which goes to the in-loop filters process). In this case, the encoder can be super-fast, with equal speed to the decoder.

[0238] On the encoder and decoder side, a second processor may be loaded more than a first processor. In a low latency case, the second processor may generate only parts of the picture as of generated reference picture (very fast) and the first processor implementing the video coder can select SKIP mode (no motion vector, no transform) with higher priority (early termination of encoder search). In this case, on the encoder and decoder side, the two processors are loaded approximately equally.

[0239] In a high encoder latency case, where desirable, the encoder can select not to use the generated reference picture and the whole encoding-decoding may be performed using one processor (such as the CPU).

[0240] While operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0241] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0242] Figure 24 shows a device 2400 for decoding for processing by a neural network based unit. The device comprises a receiving unit 24a configured to receive a bitstream including encoded data of an input signal. The device comprises a parsing unit 24b configured to parse the bitstream. The device 2400 may perform a decoding method having any of the features described herein.

[0243] Figure 25 shows a device 2500 for encoding for processing by a neural network based unit. The device comprises a receiving unit 25a configured to receive an input signal. The device comprises an encoding unit 25b configured to encode the input signal into a bitstream. The device 2500 may perform an encoding method having any of the features described herein.

[0244] The arithmetic coding of the present disclosure may be readily applied to encoding of feature maps of a neural network or in classic picture (still or video) encoding and decoding. The neural networks may be used for any purpose, in particular for encoding and decoding or pictures (still or moving), or encoding and decoding of picture-related data such as motion flow or motion vectors or other parameters. The neural network may also be used for computer vision applications such as classification of images, depth detection, segmentation map determination, object recognition of identification or the like. The entropy decoding may be performed in parallel, for example by a multi-core decoder. In addition, only parts of the entropy decoding may be performed in parallel.

[0245] The input data channels may refer to channels obtained by processing some data by a neural network. For example, the input data may be feature channels such as output channels or latent representation channels of a neural network. In an exemplary implementation, the neural network is a deep neural network and / or a convolutional neural network or the like. The neural network may be trained to process pictures (still or moving). The processing may be for picture encoding and reconstruction or for computer vision such as object recognition, classification, segmentation, or the like. In general, the present disclosure is not limited to any particular kind of tasks or neural networks. Rather, the present disclosure is applicable for encoding any kind of data coming from a plurality of channels, which are to be generally understood as any sources of data. Moreover, the channels may be provided by a pre-processing of source data.

[0246] Some exemplary implementations in hardware and software

[0247] The corresponding system which may deploy the above-mentioned encoder-decoder processing chain is illustrated in Figure 26. Figure 26 is a schematic block diagram illustrating an example coding system, e.g. a video, image, audio, and / or other coding system (or short coding system) that may utilize techniques of this present application. Video encoder

[0248] 20 (or short encoder 20) and video decoder 30 (or short decoder 30) of video coding system 10 represent examples of devices that may be configured to perform techniques in accordance with various examples described in the present application. For example, the video coding and decoding may employ neural network such which may be distributed and which may apply the above-mentioned bitstream parsing and / or bitstream generation to convey feature maps between the distributed computation nodes (two or more).

[0249] As shown in Figure 26, the coding system 10 comprises a source device 12 configured to provide encoded picture data

[0250] 21 e.g. to a destination device 14 for decoding the encoded picture data 13.

[0251] The source device 12 comprises an encoder 20, and may additionally, i.e. optionally, comprise a picture source 16, a preprocessor (or pre-processing unit) 18, e.g. a picture pre-processor 18, and a communication interface or communication unit 22.

[0252] The picture source 16 may comprise or be any kind of picture capturing device, for example a camera for capturing a real-world picture, and / or any kind of a picture generating device, for example a computer-graphics processor for generating a computer animated picture, or any kind of other device for obtaining and / or providing a real- world picture, a computer generated picture (e.g. a screen content, a virtual reality (VR) picture) and / or any combination thereof (e.g. an augmented reality (AR) picture). The picture source may be any kind of memory or storage storing any of the aforementioned pictures.

[0253] In distinction to the pre-processor 18 and the processing performed by the pre-processing unit 18, the picture or picture data 17 may also be referred to as raw picture or raw picture data 17.

[0254] Pre-processor 18 is configured to receive the (raw) picture data 17 and to perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. Pre-processing performed by the pre-processor 18 may, e.g., comprise trimming, color format conversion (e.g. from RGB to YCbCr), color correction, or de-noising. It can be understood that the pre-processing unit 18 may be optional component. It is noted that the pre-processing may also employ a neural network which uses the presence indicator signaling. The video encoder 20 is configured to receive the pre-processed picture data 19 and provide encoded picture data 21.

[0255] Communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and to transmit the encoded picture data 21 (or any further processed version thereof) over communication channel 13 to another device, e.g. the destination device 14 or any other device, for storage or direct reconstruction.

[0256] The destination device 14 comprises a decoder 30 (e.g. a video decoder 30), and may additionally, i.e. optionally, comprise a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32) and a display device 34.

[0257] The communication interface 28 of the destination device 14 is configured receive the encoded picture data 21 (or any further processed version thereof), e.g. directly from the source device 12 or from any other source, e.g. a storage device, e.g. an encoded picture data storage device, and provide the encoded picture data 21 to the decoder 30.

[0258] The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoded picture data 21 or encoded data 13 via a direct communication link between the source device 12 and the destination device 14, e.g. a direct wired or wireless connection, or via any kind of network, e.g. a wired or wireless network or any combination thereof, or any kind of private and public network, or any kind of combination thereof.

[0259] The communication interface 22 may be, e.g., configured to package the encoded picture data 21 into an appropriate format, e.g. packets, and / or process the encoded picture data using any kind of transmission encoding or processing for transmission over a communication link or communication network.

[0260] The communication interface 28, forming the counterpart of the communication interface 22, may be, e.g., configured to receive the transmitted data and process the transmission data using any kind of corresponding transmission decoding or processing and / or de-packaging to obtain the encoded picture data 21.

[0261] Both, communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces as indicated by the arrow for the communication channel 13 in Figure 26 pointing from the source device 12 to the destination device 14, or bi-directional communication interfaces, and may be configured, e.g. to send and receive messages, e.g. to set up a connection, to acknowledge and exchange any other information related to the communication link and / or data transmission, e.g. encoded picture data transmission. The decoder 30 is configured to receive the encoded picture data 21 and provide decoded picture data 31 or a decoded picture 31.

[0262] The post-processor 32 of destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), e.g. the decoded picture 31, to obtain post-processed picture data 33, e.g. a post-processed picture 33. The post-processing performed by the post-processing unit 32 may comprise, e.g. color format conversion (e.g. from YCbCr to RGB), color correction, trimming, or re-sampling, or any other processing, e.g. for preparing the decoded picture data 31 for display, e.g. by display device 34. The display device 34 of the destination device 14 is configured to receive the post-processed picture data 33 for displaying the picture, e.g. to a user or viewer. The display device 34 may be or comprise any kind of display for representing the reconstructed picture, e.g. an integrated or external display or monitor. The displays may, e.g. comprise liquid crystal displays (LCD), organic light emitting diodes (OLED) displays, plasma displays, projectors, micro LED displays, liquid crystal on silicon (LCoS), digital light processor (DLP) or any kind of other display. Although Figure 26 depicts the source device 12 and the destination device 14 as separate devices, embodiments of devices may also comprise both or both functionalities, the source device 12 or corresponding functionality and the destination device 14 or corresponding functionality. In such embodiments the source device 12 or corresponding functionality and the destination device 14 or corresponding functionality may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0263] As will be apparent for the skilled person based on the description, the existence and (exact) split of functionalities of the different units or functionalities within the source device 12 and / or destination device 14 as shown in Figure 26 may vary depending on the actual device and application.

[0264] The encoder 20 (e.g. a video encoder 20) or the decoder 30 (e.g. a video decoder 30) or both encoder 20 and decoder 30 may be implemented via processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application- specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video coding dedicated or any combinations thereof. The encoder 20 may be implemented via processing circuitry 46 to embody the various modules including the neural network or its parts. The decoder 30 may be implemented via processing circuitry 46 to embody any coding system or subsystem described herein. The processing circuitry may be configured to perform the various operations as discussed later. If the techniques are implemented partially in software, a device may store instructions for the software in a suitable, non-transitory computer-readable storage medium and may execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Either of video encoder 20 and video decoder 30 may be integrated as part of a combined encoder / decoder (CODEC) in a single device, for example, as shown in Figure 27 for video coding system 40. Video coding device 40 comprises imaging device(s) 41, antenna 42, processor 43, memory store(s) 44 and display device 45.

[0265] Source device 12 and destination device 14 may comprise any of a wide range of devices, including any kind of handheld or stationary devices, e.g. notebook or laptop computers, mobile phones, smart phones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video gaming consoles, video streaming devices(such as content services servers or content delivery servers), broadcast receiver device, broadcast transmitter device, or the like and may use no or any kind of operating system. In some cases, the source device 12 and the destination device 14 may be equipped for wireless communication. Thus, the source device 12 and the destination device 14 may be wireless communication devices.

[0266] In some cases, video coding system 10 illustrated in Figure 26 is merely an example and the techniques of the present application may apply to video coding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding and decoding devices. In other examples, data is retrieved from a local memory, streamed over a network, or the like. A video encoding device may encode and store data to memory, and / or a video decoding device may retrieve and decode data from memory. In some examples, the encoding and decoding is performed by devices that do not communicate with one another, but simply encode data to memory and / or retrieve and decode data from memory.

[0267] Figure 28 is a schematic diagram of a video coding device 8000 according to an embodiment of the disclosure. The video coding device 8000 is suitable for implementing the disclosed embodiments as described herein. In an embodiment, the video coding device 8000 may be a decoder such as video decoder 30 of Figure 26 or an encoder such as video encoder 20 of Figure 26. The video coding device 8000 comprises ingress ports 8010 (or input ports 8010) and receiver units (Rx) 8020 for receiving data; a processor, logic unit, or central processing unit (CPU) 8030 to process the data; transmitter units (Tx) 8040 and egress ports 8050 (or output ports 8050) for transmitting the data; and a memory 8060 for storing the data. The video coding device 8000 may also comprise optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress ports 8010, the receiver units 8020, the transmitter units 8040, and the egress ports 8050 for egress or ingress of optical or electrical signals. The processor 8030 is implemented by hardware and software. The processor 8030 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGAs, ASICs, and DSPs. The processor 8030 is in communication with the ingress ports 8010, receiver units 8020, transmitter units 8040, egress ports 8050, and memory 8060. The processor 8030 comprises a neural network based codec 8070. The neural network based codec 8070 implements the disclosed embodiments described above. For instance, the neural network based codec 8070 implements, processes, prepares, or provides the various coding operations. The inclusion of the neural network based codec 8070 therefore provides a substantial improvement to the functionality of the video coding device 8000 and effects a transformation of the video coding device 8000 to a different state. Alternatively, the neural network based codec 8070 is implemented as instructions stored in the memory 8060 and executed by the processor 8030.

[0268] The memory 8060 may comprise one or more disks, tape drives, and solid-state drives and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory 8060 may be, for example, volatile and / or non-volatile and may be a read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0269] Figure 29 is a simplified block diagram of an apparatus that may be used as either or both of the source device 12 and the destination device 14 from Figure 26 according to an exemplary embodiment.

[0270] A processor 9002 in the apparatus 9000 can be a central processing unit. Alternatively, the processor 9002 can be any other type of device, or multiple devices, capable of manipulating or processing information now-existing or hereafter developed. Although the disclosed implementations can be practiced with a single processor as shown, e.g., the processor 9002, advantages in speed and efficiency can be achieved using more than one processor. A memory 9004 in the apparatus 9000 can be a read only memory (ROM) device or a random access memory (RAM) device in an implementation. Any other suitable type of storage device can be used as the memory 9004. The memory 9004 can include code and data 9006 that is accessed by the processor 9002 using a bus 9012. The memory 9004 can further include an operating system 9008 and application programs 9010, the application programs 9010 including at least one program that permits the processor 9002 to perform the methods described here. For example, the application programs 9010 can include applications 1 through N, which further include a video coding application that performs the methods described here.

[0271] The apparatus 9000 can also include one or more output devices, such as a display 9018. The display 9018 may be, in one example, a touch sensitive display that combines a display with a touch sensitive element that is operable to sense touch inputs. The display 9018 can be coupled to the processor 9002 via the bus 9012.

[0272] Although depicted here as a single bus, the bus 9012 of the apparatus 9000 can be composed of multiple buses. Further, a secondary storage can be directly coupled to the other components of the apparatus 9000 or can be accessed via a network and can comprise a single integrated unit such as a memory card or multiple units such as multiple memory cards. The apparatus 9000 can thus be implemented in a wide variety of configurations. Figure 30 is a block diagram of a video coding system 10000 according to an embodiment of the disclosure. A platform 10002 in the system 10000 can be could sever or local sever. Alternatively, the platform 10002 can be any other type of device, or multiple devices, capable of calculation, storing, transcoding, encryption, rendering, decoding or encoding. Although the disclosed implementations can be practiced with a single platform as shown, e.g., the platform 10002, advantages in speed and efficiency can be achieved using more than one platform. A content delivery network (CDN) 10004 in the system 10000 can be a group of geographically distributed servers. Alternatively, the CDN 10004 can be any other type of device, or multiple devices, capable of data buffering, scheduling, dissemination or speed up the delivery of web content by bringing it closer to where users are. Although the disclosed implementations can be practiced with a single CDN as shown, e.g., the CDN 10004, advantages in speed and efficiency can be achieved using more than one CDN. A terminal 10006 in the apparatus 10000 can be a mobile phone, computer, television, laptop, camera. Alternatively, the terminal 10006 can be any other type of device, or multiple devices, capable of displaying video or image.

[0273] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description, it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1. A decoding method (1500), implemented by a decoder for predicting a currently decoded picture in dependence on a synthetic reference picture, the method comprising: parsing (1501) multiple bitstream parts of a bitstream, the multiple bitstream parts comprising an additional bitstream part (921) containing information relevant to the formation of the synthetic reference picture; from the information contained in the additional bitstream part (921), determining (1502) a synthetic reference picture type for the synthetic reference picture; and generating (1503) the synthetic reference picture in dependence on the determined synthetic reference picture type; wherein the synthetic reference picture differs from pictures previously decoded from the bitstream.

2. The decoding method as claimed in claim 1, wherein the method comprises generating the synthetic reference picture in dependence on the determined synthetic reference picture type and one or more pictures previously decoded from the bitstream.

3. The decoding method as claimed in claim 1 or claim 2, wherein the method comprises generating the synthetic reference picture from the information contained in the additional bitstream part.

4. The decoding method as claimed in claim 3, wherein the method comprises generating the synthetic reference picture from syntax elements signalled in the additional bitstream part.

5. The decoding method as claimed in claim 3 or claim 4, wherein the method comprises generating the synthetic reference picture from the information contained in the additional bitstream part using an overflow aware quantized neural network.

6. The decoding method as claimed in any preceding claim, wherein the synthetic reference picture is not displayed by a display device in communication with the decoder.

7. The decoding method as claimed in any preceding claim, wherein the synthetic reference picture type indicates one of a plurality of reference picture generation methods.

8. The decoding method as claimed in any preceding claim, wherein the method comprises assigning a motion vector field of the synthetic reference picture to be used for further motion vector prediction in dependence on the synthetic reference picture type and / or optical flow information parsed from one or more of the multiple bitstream parts and / or optical flow information derived from previously decoded picture analysis.

9. The decoding method as claimed in any preceding claim, wherein the method comprises assigning quality parameter information to the synthetic reference picture based on the synthetic reference picture type and / or quality information parsed from the additional bitstream part.

10. The decoding method as claimed in any preceding claim, wherein the synthetic reference picture is not equivalent to any previously reconstructed pictures stored in a memory (903) of the decoder.

11. The decoding method as claimed in any preceding claim, wherein the method comprises adding the generated synthetic reference picture to a reference picture list.

12. The decoding method as claimed in any preceding claim, wherein the method further comprises, after the decoding of the currently decoded picture using the synthetic reference picture, removing the synthetic reference picture from a memory (903) of the decoder.

13. The decoding method as claimed in any preceding claim, wherein reference picture information of the synthetic reference picture for motion vector prediction is dependent on the type of the synthetic reference picture.

14. The decoding method as claimed in claim 13, wherein the reference picture information comprises a motion vector field and a quality parameter map for the synthetic reference picture.

15. The decoding method as claimed in claim 13 or claim 14, wherein the reference picture information for the synthetic reference picture is artificial intelligent-based coded.

16. The decoding method as claimed in any preceding claim, wherein the method further comprises: generating the synthetic reference picture at a first processor (902); inserting the synthetic reference picture into a hybrid codec; and executing the hybrid codec at a second processor (901), wherein data exchange between the first processor and the second processor occurs through a shared memory (903) of the decoder.

17. The decoding method as claimed in any preceding claim, wherein the synthetic reference picture is generated from an end-to-end artificial intelligent codec stream and a previously decoded picture of the previously decoded pictures.

18. The decoding method as claimed in claim 17, wherein the end-to-end artificial intelligent codec stream is a JPEG Al codec stream or an analogue end-to-end learnable codec.

19. A decoding apparatus (2400) comprising processing circuitry configured to execute steps of the method (1500) according to any of claims 1 to 18.

20. A decoder (2400) comprising one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors, wherein the storage medium stores programming for execution by the one or more processors, wherein the programming, when executed by the one or more processors, configures the decoder to carry out the method (1500) according to any of claims 1 to 18.

21. An encoding method (1600), implemented by an encoder, the method comprising: receiving (1601) an input signal encoding (1602) the input signal into a bitstream comprising multiple bitstream parts, the multiple bitstream parts comprising an additional bitstream part containing information relevant to the formation of a synthetic reference picture, wherein the information contained in the additional bitstream part comprises an indication of a synthetic reference picture type for the synthetic reference picture.

22. The encoding method as claimed in claim 21, wherein the synthetic reference picture type indicates one of a plurality of reference picture generation types.

23. The encoding method as claimed in claim 21 or claim 22, wherein the additional bitstream part is coded using partial region-based coding.

24. An encoder (2500) comprising one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors, wherein the storage medium stores programming for execution by the one or more processors, wherein the programming, when executed by the one or more processors, configures the decoder to carry out the method (1600) according to any of claims 21 to 23.

25. An encoding apparatus (2500) comprising processing circuitry configured to execute steps of the method (1600) according to any of claims 21 to 23.

26. A non-transitory storage medium comprising a bitstream encoded by the method of any of claims 21 to 23.

27. A computer program stored on a non-transitory medium and including code instructions, which, when executed on one or more processor, causes the one or more processor to execute the method according to any of claims 1 to 18 or 21 to 23.

28. A system for delivering a bitstream, the system including at least one storage medium configured to store at least one bitstream generated by the encoding method of any of claims 21 to 23.

Images