A method, apparatus, and medium for decoding an image

Abstract

Embodiments of the present disclosure relate to a method, apparatus and medium for decoding an image. The method proposed herein comprises: receiving a bitstream; determining, by decoding the bitstream, a first element indicating a type of a current frame; in response to the first element indicating that the current frame corresponds to a first type, determining, by decoding the bitstream, a second element indicating a generation mode; determining, based on the second element, a set of reference frames for generating the current frame; providing the set of reference frames to a generative model to generate image data for the current frame; and reconstructing the current frame based on the image data and residual information determined by decoding the bitstream. In this way, embodiments of the present disclosure can effectively improve the reconstruction quality of the current frame.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure generally relate to video processing technology, and more particularly, to a method, apparatus and medium for decoding an image. BACKGROUND

[0002] Nowadays, digital video capability is being applied to various aspects of people's life. Traditional video coding techniques (e.g., H.264 / AV1 / H.266) mainly rely on Inter Prediction and I-frame transmission. In order to ensure video quality, the encoder usually needs to frequently send I-frames or high-quality forward prediction frames or bidirectional prediction frames, which will cause the problems of low flexibility, large bandwidth occupation and low video quality. SUMMARY

[0003] In a first aspect, a method for decoding an image is provided. The method comprises: receiving a bitstream; determining, by decoding the bitstream, a first element indicating a type of a current frame; in response to the first element indicating that the current frame corresponds to a first type, determining, by decoding the bitstream, a second element indicating a generation mode; determining, based on the second element, a set of reference frames for generating the current frame; providing the set of reference frames to a generative model to generate image data for the current frame; and reconstructing the current frame based on the image data and residual information determined by decoding the bitstream.

[0004] In a second aspect, an apparatus for video processing is provided. The apparatus comprises a processor and a non-transitory memory having instructions thereon. The instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0005] In a third aspect, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores instructions that cause a processor to perform the method according to the first aspect of the present disclosure.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0007] The above and other objects, features and advantages of the example embodiments of the present disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout. In the example embodiments of the present disclosure, the same reference numbers in different drawings identify the same or similar components.

[0008] Figure 1 A block diagram showing an example video coding system is shown in accordance with some embodiments of the present disclosure;

[0009] Figure 2 A block diagram illustrating an example video encoder is shown in accordance with some embodiments of the present disclosure;

[0010] Figure 3 A block diagram illustrating an example video decoder is shown in accordance with some embodiments of the present disclosure;

[0011] Figure 4 An example flow diagram of a method of decoding an image is shown in accordance with some embodiments of the present disclosure;

[0012] Figure 5 A block diagram of a computing device in which various embodiments of the present disclosure can be implemented is shown.

[0013] Throughout the drawings, identical or similar reference numerals can designate identical or similar elements throughout the several views. DETAILED DESCRIPTION

[0014] The principles of the present disclosure will now be described with reference to some embodiments. It should be understood that the embodiments are described for illustrative purposes only and help the person skilled in the art to understand and implement the present disclosure, and do not imply any limitation on the scope of the present disclosure. The disclosure described herein can be implemented in various ways in addition to the ways described below.

[0015] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0016] References in the present disclosure to "one embodiment", "an embodiment", "example embodiment", etc. indicate that the embodiment described can include a particular feature, structure, or characteristic, but every embodiment can not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an example embodiment, it is submitted that the feature, structure, or characteristic is within the knowledge of one of ordinary skill in the art, whether or not explicitly described, to affect such a feature, structure, or characteristic in connection with other embodiments.

[0017] It should be understood that although the terms "first" and "second" etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element without departing from the scope of the example embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated terms.

[0018] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes" and / or "including," when used herein, specify the presence of stated features, elements and / or components, but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0019] As mentioned above, nowadays, digital video capability is being applied to various aspects of people's life. Conventional video coding techniques (e.g., H.264 / AV1 / H.266) mainly rely on Inter Prediction and I-frame transmission. In order to ensure video quality, the encoder usually needs to frequently send I-frames or high-quality forward prediction frames or bidirectional prediction frames, which will cause the problems of low flexibility, large bandwidth occupation and low video quality.

[0020] Embodiments of the present disclosure provide a scheme for decoding an image. The scheme includes: receiving a bitstream; determining, by decoding the bitstream, a first element indicating a type of a current frame; in response to the first element indicating that the current frame corresponds to a first type, determining, by decoding the bitstream, a second element indicating a generation mode; determining, based on the second element, a set of reference frames for generating the current frame; providing the set of reference frames to a generative model to generate image data for the current frame; and reconstructing the current frame based on the image data and residual information determined by decoding the bitstream.

[0021] In this way, embodiments of the present disclosure can generate image data for the current frame by the generative model, and reconstruct the current frame based on the image data and the residual information, so that the reconstruction quality of the current frame can be effectively improved.

[0022] Various example implementations of the scheme are described in further detail below in conjunction with the accompanying drawings.

[0023] Example Environment

[0024] Figure 1is a block diagram illustrating an example video coding system 100 that can utilize the techniques of this disclosure. As shown, video coding system 100 can include an encoding device (e.g., source device 110) and a decoding device (e.g., destination device 120). Source device 110 can also be referred to as a video encoding device, and destination device 120 can also be referred to as a video decoding device. In operation, source device 110 can be configured to generate encoded video data, and destination device 120 can be configured to decode the encoded video data generated by source device 110. Source device 110 can include video source 112, video encoder 114, and first I / O interface 116.

[0025] Video source 112 can include a source such as a video capture device. Examples of video capture devices include, but are not limited to, an interface to receive video data from a video content provider, a computer graphics system to generate video data, and / or a combination thereof.

[0026] Video data can include one or more pictures. Video encoder 114 encodes video data from video source 112 to generate a bitstream. The bitstream can include a sequence of bits that form an encoded representation of the video data. The bitstream can include encoded pictures and associated data. An encoded picture is an encoded representation of a picture. The associated data can include sequence parameter sets, picture parameter sets, and other syntax structures. First I / O interface 116 can include a modulator / demodulator and / or a transmitter. The encoded video data can be transmitted directly to destination device 120 via first I / O interface 116, over network 130A. The encoded video data can also be stored onto a storage medium / server 130B for access by destination device 120.

[0027] Destination device 120 can include second I / O interface 126, video decoder 124, and display device 122. Second I / O interface 126 can include a receiver and / or a modem. Second I / O interface 126 can obtain encoded video data from source device 110 or storage medium / server 130B. Video decoder 124 can decode the encoded video data. Display device 122 can display the decoded video data to a user. Display device 122 can be integrated with destination device 120, or can be external to destination device 120 which is configured to interface with an external display device.

[0028] Video encoder 114 and video decoder 124 can operate according to a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, Versatile Video Coding (VVC) standard, and other existing and / or future standards.

[0029] Figure 2is a block diagram illustrating an example of a video encoder 114, in accordance with some embodiments of the present disclosure.

[0030] The video encoder 114 can be configured to implement any or all of the techniques of the present disclosure. In Figure 2 some examples, the video encoder 114 includes a number of functional components. The techniques described in this disclosure can be shared amongst the various components of the video encoder 114. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0031] In some embodiments, the video encoder 114 can include a partitioning unit 201, a prediction unit 202, which can include a mode selection unit 203, a motion estimation unit 204, a first motion compensation unit 205, and a first intra-prediction unit 206, a residual generation unit 207, a transform unit 208, a quantization unit 209, a first dequantization unit 210, a first inverse transform unit 211, a first reconstruction unit 212, a first buffer 213, and an entropy encoding unit 214.

[0032] In other examples, the video encoder 114 can include more, less, or different functional components. In one example, the prediction unit 202 can include an intra block copy (IBC) unit. The IBC unit can perform prediction in an IBC mode in which at least one reference picture is the picture in which the current video block is located.

[0033] Furthermore, although some components, such as the motion estimation unit 204 and the first motion compensation unit 205, can be integrated, these components are shown separately in Figure 2 the example for the sake of explanation.

[0034] The partitioning unit 201 can partition a picture into one or more video blocks. The video encoder 114 and the video decoder 124 can support various video block sizes.

[0035] The mode selection unit 203 can select one of a plurality of encoding modes (intra- or inter- encoding) based on, for example, error results, and provide the resulting intra- or inter- encoded block to the residual generation unit 207 to generate residual block data and to the first reconstruction unit 212 to reconstruct the encoded block for use as a reference picture. In some examples, the mode selection unit 203 can select a combined intra-inter prediction (CIIP) mode in which prediction is based on both an inter prediction signal and an intra prediction signal. In the case of inter prediction, the mode selection unit 203 can also select a resolution for a motion vector (e.g., sub-pixel accuracy or integer pixel accuracy) for the block.

[0036] To perform inter prediction on the current video block, motion estimation unit 204 can generate motion information for the current video block by comparing one or more reference frames from first buffer 213 to the current video block. First motion compensation unit 205 can determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from first buffer 213 other than the picture associated with the current video block.

[0037] Motion estimation unit 204 and first motion compensation unit 205 can perform different operations on the current video block, e.g., depending on whether the current video block is in an I slice, a P slice, or a B slice. As used herein, an "I slice" can refer to a portion of a picture composed of macroblocks, all of which are based on macroblocks within the same picture. Further, as used herein, a "P slice" and a "B slice" can refer to portions of a picture composed of macroblocks that are independent of macroblocks in the same picture, in some aspects.

[0038] In some examples, motion estimation unit 204 can perform uni-prediction on the current video block, and motion estimation unit 204 can search reference pictures of list 0 or list 1 for a reference video block for the current video block. Motion estimation unit 204 can then generate a reference index indicating the reference picture of list 0 or list 1 containing the reference video block, and a motion vector indicating a spatial displacement between the current video block and the reference video block. Motion estimation unit 204 can output the reference index, the prediction direction indicator, and the motion vector as motion information for the current video block. First motion compensation unit 205 can generate a predicted video block for the current video block based on the reference video block indicated by the motion information for the current video block.

[0039] Alternatively, in other examples, motion estimation unit 204 can perform bi-prediction on the current video block. Motion estimation unit 204 can search reference pictures of list 0 for one reference video block for the current video block, and can also search reference pictures of list 1 for another reference video block for the current video block. Motion estimation unit 204 can then generate multiple reference indices indicating multiple reference pictures of list 0 and list 1 containing the multiple reference video blocks, and multiple motion vectors indicating multiple spatial displacements between the multiple reference video blocks and the current video block. Motion estimation unit 204 can output the multiple reference indices and the multiple motion vectors for the current video block as motion information for the current video block. First motion compensation unit 205 can generate a predicted video block for the current video block based on the multiple reference video blocks indicated by the motion information for the current video block.

[0040] In some examples, the motion estimation unit 204 can output a full set of motion information for use in the decoding process by the decoder. Alternatively, in some embodiments, the motion estimation unit 204 can signal the motion information for the current video block with reference to the motion information of another video block. For example, the motion estimation unit 204 can determine that the motion information for the current video block is sufficiently similar to the motion information of a neighboring video block.

[0041] In one example, the motion estimation unit 204 can indicate a value in a syntax structure associated with the current video block that indicates to the video decoder 124 that the current video block has the same motion information as another video block.

[0042] In another example, the motion estimation unit 204 can identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates a difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 124 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0043] As discussed above, the video encoder 114 can signal motion vectors in a predictive manner. Two examples of predictive signaling techniques that can be implemented by the video encoder 114 include advanced motion vector prediction (AMVP) and Merge mode signaling.

[0044] The first intra prediction unit 206 can perform intra prediction on the current video block. When the first intra prediction unit 206 performs intra prediction on the current video block, the first intra prediction unit 206 can generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block can include a predicted video block and various syntax elements.

[0045] The residual generation unit 207 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) the prediction video block(s) for the current video block from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components of the samples in the current video block.

[0046] In other examples, such as in skip mode, there can be no residual data for the current video block for the current video block, and the residual generation unit 207 can not perform the subtraction operation.

[0047] The transform unit 208 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video blocks associated with the current video block.

[0048] After the transform unit 208 generates a transform coefficient video block associated with the current video block, the quantization unit 209 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.

[0049] The first dequantization unit 210 and the first inverse transform unit 211 can apply dequantization and inverse transform, respectively, to the transform coefficient video block to reconstruct a residual video block from the transform coefficient video block. The first reconstruction unit 212 can add the reconstructed residual video block to corresponding samples from one or more prediction video blocks generated by the prediction unit 202 to produce a reconstructed video block associated with the current video block for storage in the first buffer 213.

[0050] After the first reconstruction unit 212 reconstructs the video block, in-loop filtering operations can be performed to reduce video block effect artifacts in the video block.

[0051] The entropy encoding unit 214 can receive data from other functional components of the video encoder 114. When the entropy encoding unit 214 receives data, the entropy encoding unit 214 can perform one or more entropy encoding operations to generate entropy encoded data and output a bitstream that includes the entropy encoded data.

[0052] Figure 3 FIG. 3 is a block diagram illustrating an example of a video decoder 124, in accordance with some embodiments of the present disclosure.

[0053] The video decoder 124 can be configured to perform any or all of the techniques of the present disclosure. In Figure 3 In examples in which the video decoder 124 includes multiple functional components, the techniques described in the present disclosure can be shared among the various components of the video decoder 124. In some examples, a processor can be configured to perform any or all of the techniques described in the present disclosure.

[0054] In Figure 3 In examples in which the video decoder 124 includes multiple functional components, the techniques described in the present disclosure can be shared among the various components of the video decoder 124. In some examples, a processor can be configured to perform any or all of the techniques described in the present disclosure.

[0055] Entropy decoding unit 301 can retrieve an encoded bitstream. The encoded bitstream can include entropy encoded video data (e.g., encoded blocks of video data). Entropy decoding unit 301 can decode the entropy encoded video data, and second motion compensation unit 302 can determine motion information from the entropy decoded video data, including motion vectors, motion vector precision, reference picture list index, and other motion information. Second motion compensation unit 302 can determine such information, for example, by performing AMVP and Merge modes. AMVP is used, including deriving a number of most probable candidates based on data from neighboring PBs and reference pictures. The motion information typically includes a horizontal motion vector displacement value and a vertical motion vector displacement value, one or two reference picture indices, and in the case of a prediction region in a B slice, an identification of which reference picture list is associated with each index. As used herein, in some aspects, "Merge mode" can refer to deriving motion information from a spatially or temporally neighboring block.

[0056] Second motion compensation unit 302 can produce a motion compensated block, possibly performing interpolation based on an interpolation filter. An identifier for the interpolation filter used at sub-pixel precision can be included in the syntax elements.

[0057] Second motion compensation unit 302 can use the interpolation filter used by video encoder 114 during encoding of the video block to calculate interpolated values for sub-integer pixels of the reference block. Second motion compensation unit 302 can determine the interpolation filter used by video encoder 114 from received syntax information, and second motion compensation unit 302 can use the interpolation filter to produce the prediction block.

[0058] Second motion compensation unit 302 can use at least some of the syntax information to determine the size of the blocks used to encode frames and / or slices of the encoded video sequence, partition information describing how each macroblock of a picture of the encoded video sequence is partitioned, modes indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-coded block, and other information used to decode the encoded video sequence. As used herein, in some aspects, a "slice" can refer to a data structure that can be decoded independently of other slices of the same picture in terms of entropy coding, signal prediction, and residual signal reconstruction. A slice can be an entire picture, or can also be a region of a picture.

[0059] The second intra prediction unit 303 can use, for example, intra prediction modes received in the bitstream to form a prediction block from spatial neighboring blocks. The second dequantization unit 304 dequantizes quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 301 (i.e., dequantizes). The second inverse transform unit 305 applies an inverse transform.

[0060] The second reconstruction unit 306 can obtain a decoded block, for example, by adding the residual block to the corresponding prediction block generated by the second motion compensation unit 302 or the second intra prediction unit 303. If needed, a deblocking filter can also be applied to filter the decoded block in order to remove blocking artifacts. The decoded video block is then stored in the second buffer 307, which provides reference blocks for subsequent motion compensation / intra prediction, and the second buffer 307 also produces decoded video for presentation on a display device.

[0061] Some example embodiments of the present disclosure will be described in detail below. It should be noted that chapter headings are used in this document for ease of understanding only and do not limit the embodiments disclosed in a chapter to only that chapter. Also, although some embodiments are described with reference to a multi-functional video codec or other specific video codec, the disclosed techniques are applicable to other video codec technologies as well. Furthermore, although some embodiments describe video encoding steps in detail, it should be understood that corresponding decoding steps to un-encode will be implemented by a decoder. Also, the term video processing includes video encoding or compression, video decoding or decompression, and video transcoding in which video pixels are represented from one compressed format to another compressed format or at a different compressed bit rate.

[0062] Example process

[0063] Figure 4 A flowchart of an example process 400 of decoding an image according to some embodiments of the present disclosure is shown. The process 400 can be implemented at the destination device 120. The process 400 is described below with reference to the video codec system 100 shown. Figure 1 The video codec system 100 can include an encoding device (e.g., the source device 110) and a decoding device (e.g., the destination device 120).

[0064] In some embodiments, the encoding device (e.g., the source device 110) can provide a set of reference frames to the second generative model to generate a predicted frame. The source device 110 can determine a first difference between the current frame and the predicted frame. Further, the source device 110 can encode, in a bitstream, residual information between the current frame and the predicted frame in response to the first difference being less than a first threshold.

[0065] As an example, a source device 110 can encode a target video before a decoding device (e.g., the destination device 120) receives the bitstream. The source device 110 can determine a plurality of image frames of the target video in succession. The plurality of image frames herein includes a set of reference frames used to generate a predicted frame. The set of reference frames herein includes at least one preceding frame adjacent to a current frame or at least one preceding frame and at least one frame following the current frame. After obtaining the set of reference frames, the source device 110 can input the set of reference frames to a second generative model to generate a predicted frame corresponding to the current frame.

[0066] In some scenarios, the data used to generate the predicted frame can further include multi-modal information associated with the target video, such as optical flow information, depth images, timestamps, etc., to provide temporal context information for generating the predicted frame. The second generative model herein can include, for example, a diffusion model, a generative adversarial network (GAN), a variational autoencoder (VAE), or a transformer-based model, etc. The selection of the specific generative model can be determined by those skilled in the art according to the needs, and the present disclosure does not limit this.

[0067] In generating the predicted frame, the source device 110 can employ different generation modes. In some scenarios, the source device 110 can generate the predicted frame corresponding to the current frame based on at least one preceding frame using the second generative model. Generating the predicted frame based on only the at least one preceding frame can effectively improve the generation speed of the predicted frame, ensure the real-time generation of the predicted frame, and ensure that the encoding delay is at a low level.

[0068] In other scenarios, the source device 110 can further generate the predicted frame based on at least one preceding frame and at least one frame following the current frame using the second generative model. Generating the predicted frame based on the at least one preceding frame and the at least one frame following the current frame can enable the second generative model to effectively utilize the context information of the preceding and following frames, thereby generating a predicted frame that is more accurate and has a smaller residual amplitude.

[0069] After the generated of the prediction frame is completed, the source device 110 can calculate a first difference between the prediction frame and the current frame using an appropriate algorithm. The first difference can also be referred to as error information. The error information can include, for example, a Peak Signal to Noise Ratio (PSNR), a Mean Square Error (MSE), a Learned Perceptual Image Patch Similarity (LPIPS), a Video Multimethod Assessment Fusion (VMAF), and the like.

[0070] After the error information is determined, the source device 110 can determine whether the error information is less than a first threshold. When the error information is less than the first threshold, the source device 110 can mark the prediction frame as a generated prediction frame. Further, the source device 110 can determine residual information associated with the prediction frame and encode the residual information, at least one previous frame, and at least one frame after the current frame to obtain a bitstream. After the bitstream is obtained, the source device 110 can transmit the bitstream to the destination device 120 to enable the destination device 120 to obtain the bitstream. When the error information reaches the first threshold, the source device 110 can encode a set of reference frames and the current frame to obtain a bitstream. Hereinafter, the case in which the bitstream is obtained by encoding the residual information and the set of reference frames will be described.

[0071] In this way, the source device 110 only needs to transmit the residual information associated with the current frame when transmitting the current frame, thereby effectively reducing the bandwidth required to transmit the bitstream and improving the transmission efficiency of the bitstream.

[0072] Additionally or alternatively, the source device 110 may also set multiple elements in the obtained bitstream so that the decoding end can select the corresponding reconstruction mode during decoding. These multiple elements may, for example, indicate the types of multiple image frames in the bitstream, the generation mode of the predicted frames, and the frame index used to locate the predicted frames. The types of multiple image frames may, for example, include a first type, a second type, a third type, and a fourth type. The first type may, for example, indicate that the identified image frame is a predicted frame. The second type may, for example, indicate that the identified image frame is an intra-coded frame (I-Frame). The third type may, for example, indicate that the identified image frame is a forward-predicted frame (P-Frame). The fourth type may, for example, indicate that the identified image frame is a bi-directional predicted frame (B-Frame). The generation mode includes a first generation mode (e.g., forward prediction) and a second generation mode (e.g., bi-directional prediction).

[0073] like Figure 4 As shown, in box 410, destination device 120 receives the bitstream. As an example, source device 110 can generate the bitstream based on at least a set of reference frames and residual information. After the bitstream is generated, source device 110 can send the bitstream to a decoding device (e.g., destination device 120). Destination device 120 can receive the generated bitstream.

[0074] In box 420, destination device 120 determines a first element indicating the type of the current frame by decoding the bitstream. As an example, destination device 120 can decode the bitstream after receiving it to determine the first element. This first element is set in the bitstream by source device 110. The first element is used to indicate the type of the current frame. For example, the first element could indicate that the current frame is a predicted frame.

[0075] In block 430, in response to a first element indicating that the current frame corresponds to a first type, the destination device 120 determines a second element for indicating a generation mode by decoding the bitstream. As an example, after determining that the first element indicates that the current frame corresponds to a first type (e.g., the current frame is a prediction frame), the destination device 120 can determine the second element for indicating a generation mode from the received bitstream to determine the generation mode used to generate image data for the current frame. This second element may, for example, indicate a first generation mode (e.g., a forward prediction mode) or a second generation mode (e.g., a bidirectional prediction mode).

[0076] In box 440, the destination device 120 determines a set of reference frames for generating the current frame based on the second element. As an example, after determining the second element indicating the generation mode by decoding the bitstream, the destination device 120 can determine the generation mode for generating image data for the current frame based on the second element. After determining the generation mode, the destination device 120 can determine a set of reference frames for generating the current frame based on the generation mode.

[0077] In some embodiments, the destination device 120 may, in response to a second element indicating a first generation mode, determine at least one preceding frame as a set of reference frames. For example, when the second element indicates a first generation mode (e.g., a forward prediction mode), the destination device 120 may determine that the predicted frame generated in the source device 110 is generated based on at least one preceding frame. Further, to ensure that the subsequently generated image data for the current frame differs little from the predicted frame, the destination device 120 may use at least one preceding frame as a set of reference frames.

[0078] In some embodiments, the destination device 120 may also, in response to a second element indicating a second generation mode, determine at least one frame preceding the current frame and at least one frame following the current frame as a set of reference frames. For example, when the second element indicates a second generation mode (e.g., a bidirectional prediction mode), the destination device 120 may determine that the predicted frame generated in the source device 110 is generated based on at least one preceding frame preceding the current frame and at least one frame following the current frame. Further, to ensure that the subsequently generated image data for the current frame differs little from the predicted frame, the destination device 120 may use at least one preceding frame preceding the current frame and at least one frame following the current frame as a set of reference frames.

[0079] In some embodiments, the destination device 120 may determine a third element by decoding the bitstream in response to a second element indicating a preset generative model. The third element indicates a set of frame indices. Further, the destination device 120 may determine a set of reference frames corresponding to the set of frame indices.

[0080] As an example, after determining that the second element indicates a preset generation mode (e.g., a first generation mode or a second generation mode), the destination device 120 can determine a third element associated with the bitstream by decoding the bitstream. The third element indicates a set of frame indices. This set of frame indices can, for example, indicate the position of reference frames used by the source device 110 when generating the predicted frame. For example, if the preset generation mode is the first generation mode, then the source device 110 generates the frame based on at least one preceding frame before the current frame. The set of frame indices here can then indicate the position of at least one preceding frame before the current frame in the bitstream. As another example, if the preset generation mode is the second generation mode, then the source device 110 generates the frame based on at least one preceding frame before the current frame and at least one frame after the current frame. The set of frame indices here can then indicate the position of at least one preceding frame before the current frame and at least one frame after the current frame in the bitstream. After obtaining the set of frame indices, the destination device 120 can determine, based on the set of frame indices, a set of reference frames from the bitstream for generating image data for the current frame.

[0081] In this way, embodiments of the present disclosure can determine a set of reference frames based on a set of frame indices associated with the bitstream, thereby effectively improving the efficiency of determining a set of reference frames and thus effectively improving the efficiency of decoding images.

[0082] In box 450, the destination device 120 provides a set of reference frames to the generative model to generate image data for the current frame. Here, the generative model is a first generative model deployed on the decoding device. The first generative model deployed on the decoding device (e.g., destination device 120) and the second generative model deployed on the encoding device (e.g., source device 110) correspond to the same model attributes. These model attributes include at least one of the following: model structure information, model weight information, and a random seed for the model. The image data is also generated based on at least one of depth information associated with the current frame and a timestamp corresponding to the current frame.

[0083] As an example, after determining a set of reference frames, the destination device 120 can input the set of reference frames into the first generative model to generate image data for the current frame. To ensure the accuracy of the generated image data, the first generative model deployed in the destination device 120 corresponds to the same attributes as the second generative model deployed in the source device 110. As an example, before generating image data, the destination device 120 can configure the attributes of the first generative model based on the attributes of the second generative model. For example, the destination device 120 can obtain the structural information, weight information, and configuration information such as the random seed used to generate the prediction frame from the second generative model. After obtaining the configuration information, the destination device 120 can configure the first generative model based on the configuration information. The model structural information here may include, for example, model type, network topology, input / output specifications, etc. The weight information here may indicate, for example, the weight data and weight tensors used to generate the prediction frame. By ensuring that the attributes of the first generative model and the second generative model are the same, the image data generated by the first generative model can correspond to the image data of the prediction frame, thereby enabling better reconstruction of the current frame. Furthermore, generating image data through generative models can effectively restore details of complex scenes and reduce block artifacts.

[0084] In some scenarios, before generating image data, the target device 120 can also provide the first generative model with depth information and a timestamp associated with the current frame, so that the first generative model can refer to the depth information and timestamp when generating image data. In this way, embodiments of this disclosure can generate image data by providing multimodal information to the generative model, thereby effectively improving the quality of the generated image data.

[0085] In some embodiments, the destination device 120 can decode a fourth element from the bitstream. The fourth element indicates a random seed used for the first generative model. As an example, after the bitstream is generated, the source device 110 can obtain a random seed for the second generative model. Further, the source device 110 can encode the random seed as a fourth element and add the fourth element to the bitstream so that the bitstream includes the random seed. After receiving the bitstream, the destination device 120 can decode the bitstream to determine the fourth element. After obtaining the fourth element, the destination device 120 can configure the first generative model based on the random seed indicated by the fourth element so that the random seed of the first generative model is consistent with the random seed of the second generative model.

[0086] In some embodiments, the target device 120 may also determine temporal motion information associated with the current frame. Further, the target device 120 may utilize a generative model to generate image data for the current frame based on a set of reference frames and the temporal motion information. The temporal motion data here includes optical flow information associated with the current frame.

[0087] As an example, source device 110 can encode the temporal motion information of multiple image frames into a bitstream. After receiving the bitstream, destination device 120 can not only obtain a set of reference frames by decoding the bitstream, but also obtain the corresponding temporal motion information. This temporal motion information may include, but is not limited to, optical flow information. For example, destination device 120 can decode the bitstream to obtain optical flow information. After obtaining the optical flow information, destination device 120 can provide the optical flow information and image data to a first generative model to generate image data for the current frame.

[0088] In this way, embodiments of this disclosure can also refer to optical flow information to generate image data for the current frame, thereby effectively improving the quality of the generated image data.

[0089] In box 460, the destination device 120 reconstructs the current frame based on image data and residual information determined by the decoded bitstream. This residual information is determined based on the difference between the image information of the current frame and the predicted frame generated by a second generative model deployed on the encoding device.

[0090] As an example, after the source device 110 generates a prediction frame, it can calculate the difference between the prediction frame and the current frame among multiple image frames of the target video. This difference can also be referred to as error information. When the error information is less than a first threshold, the source device 110 can generate a bitstream by encoding residual information and a set of reference frames. The destination device 120 can determine the residual information by decoding the bitstream. After the residual information is determined, the destination device 120 can reconstruct the current frame by fusing the generated image data and the residual information. For example, after obtaining the residual information, the destination device 120 can determine a residual correction amount based on the residual data. Further, the destination device 120 can perform pixel-by-pixel additive fusion of the generated image data and the residual correction amount to reconstruct the current frame.

[0091] In some embodiments, the target device 120 may reconstruct a first candidate frame based on image data and residual information. Further, the target device 120 may utilize a generative model to generate a second candidate frame based on optical flow information of at least one reference frame in the set of reference frames, where the at least one reference frame is determined based on a second element. Finally, in response to a second difference between the first and second candidate frames reaching a second threshold, the target device 120 may fuse the first and second candidate frames to determine the current frame.

[0092] As an example, the target device 120 may also deploy a cross-frame consistency optimization module. The target device 120 can further improve the temporal consistency and visual coherence of consecutively generated frames through a cross-frame consistency optimization model. For example, the target device 120 can utilize an optical flow-guided temporal smoothing method to further improve the temporal consistency and visual coherence of consecutively generated frames. Specifically, the target device 120 can reconstruct a first candidate frame based on image data and residual information. The target device 120 can also determine at least one reference frame from a set of reference frames for generating a second candidate frame based on a second element. For example, when the second element indicates a first generation mode, the target device 120 can determine that the previous frame of the current frame is at least one reference frame for generating the second candidate frame. When the second element indicates a second generation mode, the target device 120 can determine that the previous and next frames of the current frame are at least one reference frame for generating the second candidate frame. Further, the target device 120 can provide at least one reference frame and optical flow information to a generative model to generate the second candidate frame.

[0093] After obtaining the first candidate frame and the second candidate frame, the target device 120 can determine a second difference between the first candidate frame and the second candidate frame. This second difference may include, for example, Peak Signal-to-Noise Ratio (PSNR) or Structural Similarity Index (SSIM). When the second difference reaches a second threshold (e.g., greater than or equal to the second threshold), the target device 120 can fuse the first candidate frame and the second candidate frame to determine the current frame. For example, the target device 120 can fuse the first candidate frame and the second candidate frame using a weighted fusion method. The target device 120 can assign different weights to the first candidate frame and the second candidate frame based on factors such as scene type, motion complexity, and prediction confidence. After assigning weights, the target device 120 can add the first candidate frame and the second candidate frame to determine the current frame. When fusing the first candidate frame and the second candidate frame, it is necessary to ensure that the consistency loss between the preceding and following frames meets a preset condition. This preset condition can be set by those skilled in the art according to their needs, and this disclosure does not impose any limitations on it.

[0094] In this way, embodiments of the present disclosure can generate image data for the current frame through a generative model, and reconstruct the current frame based on the image data and residual information, thereby effectively improving the reconstruction quality of the current frame.

[0095] Example devices and equipment

[0096] Figure 5A block diagram of a computing device 500 in which various embodiments of the present disclosure may be implemented is shown. The computing device 500 may be implemented as a source device 110 (or video encoder 114) or a destination device 120 (or video decoder 124), or may be included in a source device 110 (or video encoder 114) or a destination device 120 (or video decoder 124).

[0097] It should be understood that, Figure 5 The computing device 500 shown is for illustrative purposes only and is not intended to imply any limitation on the functionality and scope of the embodiments of this disclosure.

[0098] like Figure 5 As shown, computing device 500 includes general-purpose computing device 500. Computing device 500 may include at least one or more processors or processing units 510, memory 520, storage unit 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560.

[0099] In some embodiments, the computing device 500 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server, a large computing device, etc., provided by a service provider. The user terminal can be, for example, any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, stations, units, devices, multimedia computers, multimedia tablet computers, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 500 can support any type of interface to the user (such as "wearable" circuitry devices, etc.).

[0100] Processing unit 510 can be a physical processor or a virtual processor, and can perform various processes based on programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 500. Processing unit 510 may also be referred to as a central processing unit (CPU), microprocessor, controller, or microcontroller.

[0101] Computing device 500 typically includes various computer storage media. Such media can be any media accessible by computing device 500, including but not limited to volatile and non-volatile media, or removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or flash memory) or any combination thereof. Storage cell 530 can be any removable or non-removable media and may include machine-readable media, such as memory, flash drives, disks, or other media that can be used to store information and / or data and can be accessed within computing device 500.

[0102] The computing device 500 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Although in Figure 5 Not shown, but a disk drive for reading from and / or writing to a removable non-volatile disk, and an optical disc drive for reading from and / or writing to a removable non-volatile optical disc may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0103] Communication unit 540 communicates with another computing device via a communication medium. Furthermore, the functionality of the components in computing device 500 can be implemented by a single computing cluster or multiple computing machines that can communicate via communication connections. Therefore, computing device 500 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0104] Input device 550 can be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 560 can be one or more of various output devices, such as a monitor, speaker, printer, etc. With the aid of communication unit 540, computing device 500 can also communicate with one or more external devices (not shown), such as storage devices and display devices. Computing device 500 can also communicate with one or more devices that enable a user to interact with computing device 500, or, if needed, with any device that enables computing device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via an input / output (I / O) interface (not shown).

[0105] In some embodiments, some or all of the components of computing device 500 may be deployed in a cloud computing architecture, rather than being integrated into a single device. In a cloud computing architecture, components may be remotely provided and work together to achieve the functionality described herein. In some embodiments, cloud computing provides computing, software, data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services via a wide area network (WAN), such as the Internet, using suitable protocols. For example, a cloud computing provider provides applications via a WAN that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated or distributed across remote data center locations. Cloud computing infrastructure may provide services through shared data centers, although to users they appear as a single access point. Therefore, a cloud computing architecture can be used to provide the components and functionality described herein from service providers at remote locations. Alternatively, the components and functionality described herein may be provided by conventional servers or installed directly or otherwise on client devices.

[0106] In embodiments of this disclosure, computing device 500 may be used to implement video encoding / decoding. Memory 520 may include one or more video codec modules 525 having one or more program instructions. These modules are accessible and executable by processing unit 510 to perform the functions of the various embodiments described herein.

[0107] In an example embodiment of performing video encoding, input device 550 may receive video data as input 570 to be encoded. The video data may be processed, for example, by video codec module 525 to generate an encoded bitstream. The encoded bitstream may be provided as output 580 via output device 560.

[0108] In an example embodiment of performing video decoding, input device 550 may receive an encoded bitstream as input 570. The encoded bitstream may be processed, for example, by video codec module 525 to generate decoded video data. The decoded video data may be provided as output 580 via output device 560.

[0109] While this disclosure has been specifically shown and described with reference to preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this application as defined by the appended claims. These variations are intended to be covered by the scope of this application. Therefore, the foregoing description of embodiments of this application is not intended to be limiting.

Claims

1. A method for decoding an image, characterized in that, The method includes: Receive the bitstream; By decoding the bitstream, a first element indicating the type of the current frame is determined; In response to the first element indicating that the current frame corresponds to a first type, a second element for indicating the generation mode is determined by decoding the bitstream; Based on the second element, a set of reference frames is determined for generating the current frame; The set of reference frames is provided to the generative model to generate image data for the current frame; Based on the image data and the residual information determined by decoding the bitstream, the first candidate frame is reconstructed; Using the generative model, a second candidate frame is generated based on the set of reference frames and the optical flow information associated with the current frame; and In response to a second difference between the first candidate frame and the second candidate frame reaching a second threshold, the first candidate frame and the second candidate frame are fused to determine the current frame.

2. The method according to claim 1, characterized in that, The step of determining a set of reference frames for generating the current frame based on the second element includes: In response to the second element indicating a first generation mode, at least one preceding frame prior to the current frame is determined as the set of reference frames; or In response to the second element indicating a second generation mode, at least one frame preceding the current frame and at least one frame following the current frame are determined as the set of reference frames.

3. The method according to claim 1, characterized in that, The step of determining a set of reference frames for generating the current frame based on the second element includes: In response to the second element indicating a preset generation mode, a third element is determined by decoding the bitstream, the third element indicating a set of frame indices; and Determine the set of reference frames corresponding to the set of frame indices.

4. The method according to claim 1, characterized in that, The generative model is a first generative model deployed on the decoding device. The residual information is determined based on the difference between the image information of the current frame and the predicted frame generated by the second generative model deployed on the encoding device. The first generative model and the second generative model correspond to the same model attributes.

5. The method according to claim 4, characterized in that, The model attributes include at least one of the following: Structural information of the model; The model's weight information; The random seed for the model.

6. The method according to claim 5, characterized in that, The method further includes: Decode a fourth element from the bitstream, the fourth element indicating the random seed used for the first generative model.

7. The method according to claim 4, characterized in that, The encoding device is also configured to: The set of reference frames is provided to the second generative model to generate the predicted frames; Determine the first difference between the current frame and the predicted frame; and In response to the first difference being less than a first threshold, the residual information between the current frame and the predicted frame is encoded in the bitstream.

8. The method according to claim 1, characterized in that, Providing the set of reference frames to the generative model to generate image data for the current frame includes: Determine the temporal motion information associated with the current frame; and Using the generative model, image data for the current frame is generated based on the set of reference frames and the temporal motion information.

9. The method according to claim 8, characterized in that, The temporal motion information includes optical flow information associated with the current frame.

10. The method according to claim 1, characterized in that, The image data is also generated based on at least one of the following: Depth information associated with the current frame; The timestamp corresponding to the current frame.

11. An apparatus for decoding images, characterized in that, The apparatus includes a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 10.

12. A non-transitory computer-readable storage medium for storing instructions, characterized in that, The instructions cause the processor to execute the instructions of the method according to any one of claims 1 to 10.

Images