Frame coding and relative difference metric for generative facial video compression and two-stage training

Abstract

Generative Face Video Compression (“GFVC”) technology is provided to improve the performance of face video compression. The computational system is configured to: compute a relative difference metric describing the feature differences between frames; and, based on the relative difference metric, determine whether the current frame can be synthesized without entropy coding or should be re-encoded. The computational system is configured to perform two-stage training to stabilize the training of the generative adversarial network (“GAN”) in GFVC.

Description

[0001] Cross-references to related applications This disclosure claims priority to U.S. Provisional Application No. 63 / 619,305, filed January 9, 2024, and U.S. Patent Application No. 19 / 008,505, filed January 2, 2025. All of the above applications are incorporated herein by reference in their entirety. Background Technology

[0002] Machine learning tools are being incorporated into intra-frame coding used in video coding standards to further improve compression efficiency, building upon previous standards such as H.264 / Advanced Video Coding (AVC) and H.265 / High-Efficiency Video Coding (HEVC), and more recently, Versatile Video Coding (VVC). Furthermore, learning-based coding is also likely to become part of future video coding standards following VVC.

[0003] Current image coding technologies are primarily based on lossy compression, using frameworks including transform coding, quantization, and entropy coding. For many years, lossy compression has achieved compression ratios suitable for limited-scale image acquisition and storage. However, computer systems are increasingly being configured to acquire and store images at much larger scales for applications such as surveillance, streaming media, data mining, and computer vision. Therefore, it is expected that future image coding standards will achieve smaller image sizes without significantly sacrificing image quality.

[0004] Machine learning has never been part of previous image coding standards, whether in still image compression or intra-frame coding used in video compression. In the VVC standardization process from 2018 to 2020, working groups of ISO / IEC and ITU-T reviewed learning-based coding proposals but did not adopt them. In October 2023, the 32nd meeting of the Joint Video Experts Team (JVET) convened an ad hoc group on Generative Face Video Compression (GFVC), whose work included software implementation, testing conditions, coordinated experiments, and interoperability studies.

[0005] According to GFVC, further improvements to facial video compression technology are still needed. Attached Figure Description

[0006] Detailed descriptions of the embodiments will be given with reference to the accompanying drawings. In the drawings, one or more numbers on the far left of the reference numerals indicate the figure in which the reference numeral first appears. The same reference numerals are used in different figures to indicate similar or identical items or features.

[0007] Figure 1A A block diagram illustrating the image compression process based on various image coding techniques is shown.

[0008] Figure 1B A flowchart of a first-order motion model for video generative compression based on a deep learning model is shown.

[0009] Figure 1C A flowchart of a deep learning-based video generative compression model based on compact feature representation is shown.

[0010] Figure 2 A flowchart is shown for the following operations: calculating a relative difference metric that describes the feature differences between frames; and based on the relative difference metric, determining whether to recode the current frame or synthesize the current frame using a generative neural network without entropy encoding.

[0011] Figure 3 A flowchart is shown of an encoding-end sequential frame processing method for generative facial video compression (“GFVC”) according to an exemplary embodiment of the present disclosure, and a calculation of a relative difference metric based on the frame processing method.

[0012] Figure 4 A flowchart is shown of a decoding end sequential frame processing method based on a relative difference metric according to an exemplary embodiment of the present disclosure, in accordance with the present disclosure.

[0013] Figure 5 A flowchart of the two-stage training process for training a Generative Adversarial Network (GAN) in stabilizing GFVC is shown.

[0014] Figure 6 An exemplary system for implementing the processes and methods described herein is shown, the processes and methods being used to: calculate a relative difference metric describing the feature differences between frames; and, based on the relative difference metric, determine whether the current frame is synthesized by a generative neural network without entropy coding or should be recoded. Detailed Implementation

[0015] Exemplary embodiments of this disclosure provide for computing a relative difference metric that describes the feature difference between frames, and for determining, based on the relative difference metric, whether the current frame can be synthesized by a generative neural network without entropy encoding or should be re-encoded; furthermore, a two-stage training method is provided to stabilize the training process of a generative adversarial network (“GAN”) in GFVC.

[0016] Figure 1 illustrates a block diagram of an image compression process 100 based on various image coding techniques—such as those implemented by various intra-frame coding techniques (e.g., those implemented by AVC, HEVC, and VVC). The image compression process 100 may include lossless and lossy steps.

[0017] It should be understood that while the image compression process 100 conforms to the aforementioned standards (and other image coding standards or technologies based on image compression, but not limited to these), it does not describe all the contents of the aforementioned standards (or all the contents of other image coding standards or technologies). Furthermore, the elements of the image compression process 100 can be implemented in different ways according to the aforementioned standards (and according to other image coding standards or technologies), but are not limited thereto.

[0018] According to the image compression process 100, the computing system is configured with one or more computer-executable instruction sets to perform several operations on the input image 102. First, the computing system performs a transformation operation 104 on the input image 102. Here, one or more processors of the computing system transform the image data from a spatial domain representation (i.e., image pixel data) to a frequency domain representation by calculating a Fourier transform, for example, a discrete cosine transform (DCT). In the frequency domain representation, the transformed image data is represented by transform coefficients 106.

[0019] According to the image compression process 100, the computing system then performs a quantization operation 108 on the transform coefficients 106. Here, one or more processors of the computing system generate a quantization index 110, which stores a finite subset of the color information stored in the image data.

[0020] The computing system then performs an entropy encoding operation 112 on the quantization index 110. Here, one or more processors of the computing system perform an encoding operation, such as arithmetic encoding, in which symbols are encoded as bit sequences according to their occurrence probabilities. The entropy encoding operation 112 generates a compressed image 114.

[0021] One or more processors of the computing system are also configured by one or more computer-executable instruction sets to perform several operations on the compressed image 114 to output the compressed image.

[0022] For example, according to some image coding standards, the computing system performs entropy decoding operation 116, dequantization operation 118, and inverse transform operation 120 on the compressed image 114 to output a reconstructed image 122. For instance, if transform operation 104 is a DCT calculation, the inverse transform operation 120 may be an inverse discrete cosine transform (IDCT) calculation, which restores the frequency domain representation of the image data to the spatial domain representation.

[0023] However, the decoded image can be used for other calculations without undergoing the inverse transform operation 120. One or more processors of the computing system can be configured to output the compressed image 114 in a format other than the reconstructed image. Before performing the inverse transform operation 120, or instead of performing the inverse transform operation 120, one or more processors of the computing system can be configured to perform image processing operation 124 on the decoded image 126 generated by the entropy decoding operation 116.

[0024] For example, one or more processors of the computing system can resize, rotate, reshape, crop, rescale in any or all of its color channels, translate by several pixels in any direction, change brightness or contrast, flip in any orientation, inject noise, reweight frequency channels, apply frequency jitter, etc., on the decoded image.

[0025] Before or instead of performing the inverse transform operation 120, one or more processors of the computing system may be configured to input the decoded image 126 into the learning model 128. The one or more processors of the computing system may input the decoded image 126 into any layer of the learning model 128, the layer further configuring the one or more processors to perform training or inference computations based on the decoded image 126.

[0026] The computing system may perform any, some, or all of the following operations: outputting a reconstructed image 122; performing image processing operations 124 on the decoded image 126; and inputting the decoded image 126 into a learning model 128, but is not limited thereto.

[0027] For the image compression process 100 described above using various image coding techniques, learning-based coding can be incorporated into the image compression process 100. Learned image compression (LIC) architectures are generally classified into two categories: hybrid coding and end-to-end learning-based coding.

[0028] End-to-end learning-based coding typically refers to modifying one or more steps of the entire image compression process 100 so that one or more learning models can learn parameters. Separately from the image compression process 100, on another computing system, a dataset can be input into the learning model to train the model to learn parameters, thereby improving the computation and output required to perform various computational tasks.

[0029] For example, the LIC is implemented using a variational autoencoder (VAE) architecture, which also includes an encoder. decoder Quantizer . x It is the input image. It is a latent representation. This refers to the quantized and encoded bitstream used for storage and transmission (e.g., via lossless arithmetic encoding). Since deterministic quantization is non-differentiable with respect to network parameters φ and θ, additive uniform noise is typically used to optimize the approximately differentiable rate distortion ("rate distortion, RD") loss, as described in Equation 1 below: in, It is the probability density function of all natural images. It is the distortion loss between the original input and the reconstruction (e.g., mean-square error (MSE) or mean absolute error (MAE)). It is the rate loss estimated from the bitrate of the encoded bitstream, and These are hyperparameters used to control network parameter optimization, balancing reconstruction quality and compression bit rate. Typically, for... For each target value, a model parameter set needs to be trained. and In order to optimize Formula 1 accordingly.

[0030] A learning model can include one or more computer-readable instructions executable by one or more processors of a computing system to perform tasks including processing the model's inputs and various parameters, as well as its outputs. For example, a learning model can be a hierarchical model, such as a deep neural network, which can have a fully connected structure, a feedforward structure like a convolutional neural network (CNN), a backpropagation structure like a recurrent neural network (RNN), or other architectures suitable for specific task computations. Generally, any hierarchical model with multiple layers between the input and output layers is a deep neural network (DNN).

[0031] For example, tasks can include classification, clustering, matching, regression, semantic segmentation, etc. Tasks can provide outputs that support computer vision or machine vision functions, such as identifying objects and / or boundaries in images and / or videos; tracking the movement of objects in videos in real time; matching objects identified in images and / or videos with other images and / or videos; providing annotations or transcriptions of images, videos, and / or audio in real time; and so on.

[0032] Deep generative models, including VAEs and generative adversarial networks (“GANs”), have been applied to improve the performance of facial video compression. In 2018, Wiles designed X2Face to control facial generation through image, audio, and pose codes. Furthermore, Zakharov et al. proposed a realistic neural speaking head model through few-shot adversarial learning.

[0033] For video-to-video compositing tasks, the NVIDIA research team first proposed Face-vidtovid in 2019. Subsequently, in 2020, they proposed a novel approach that utilizes compact 3D keypoint representations to drive a generative model to render the target frame. Furthermore, the Facebook research team designed a mobile-enabled video chat system based on a first-order motion model (FOMM).

[0034] Feng et al. proposed VSBNet, which reconstructs the original frame from markers using adversarial learning. Additionally, Chen et al. proposed an end-to-end speaking head video compression framework based on "compact feature learning" (CFTE) for efficient speaking facial video compression in ultra-low bandwidth scenarios. CFTE utilizes the compact feature representation to compensate for temporal evolution and reconstructs target facial video frames in an end-to-end manner, and can be incorporated into the video coding framework under rate-distortion target supervision. Furthermore, Chen et al. used a 3D morphable model (3DMM) template, utilized facial semantics to represent facial videos, and implemented facial manipulation for facial video coding.

[0035] Table 1 below further summarizes the facial representation of the generative facial video compression algorithm. Facial images exhibit strong statistical regularity, which can be concisely represented by 2D landmarks, 2D keypoints, region matrices, 3D keypoints, compact feature matrices, and facial semantics. This facial description strategy can reduce the coding bit rate and improve coding efficiency, thus making it suitable for video conferencing and real-time entertainment.

[0036] Figure 1B A flowchart of a video generative compression FOMM based on a deep learning model is shown. The FOMM proposed by Siarohin et al. deforms a reference source frame to follow the motion of the driving video and is specifically applied to facial videos. Figure 1B The FOMM shown implements an encoder-decoder architecture with motion transfer components.

[0037] The encoder encodes the source frame using a block-based image or video compression method, such as HEVC / VVC or JPEG / BPG. Figure 1B As shown in the reference above Figure 1A The block-based encoder 152 described herein configures one or more processors of a computing system to compress the source frame according to VVC.

[0038] One or more processors of the computing system are configured to learn a keypoint extractor using equivariant loss without explicit labels. Keypoints (x, y) collectively represent the most important information in the feature map. Source keypoint extractor 154 and driving keypoint extractor 156 are configured to configure one or more processors of the computing system to compute two sets of ten learned keypoints for the source and driving frames, respectively. Gaussian mapping operation 158 configures one or more processors of the computing system to map the learned keypoints from a size of... The transformation is performed on the feature map. Therefore, each corresponding key point can represent the feature information of different channels.

[0039] Dense Motion Network 160 configures one or more processors of the computing system to output dense motion fields and occlusion maps based on the markers and the source frames.

[0040] Decoder 162 configures one or more processors of the computing system to generate an image based on the deformed graph.

[0041] Figure 1C A flowchart of a deep learning-based video generative compression model based on compact feature representation is shown, namely CFTE proposed by Chen et al. Figure 1C The model described implements an encoder-decoder architecture, which processes a sequence of frames including a keyframe and multiple subsequent inter-frame frames.

[0042] The encoding end includes: a block-based encoder 172 for compressing the keyframe, a feature extractor 174 for extracting compact human features from the other inter-frame frames, and a feature encoding module 176 for compressing the inter-frame prediction residuals of the compact human features.

[0043] The feature extractor should be understood as a learning model that is trained to extract human features from image data input to the learning model.

[0044] The block-based encoder 172 configures one or more processors of the computing system to compress keyframes representing human body textures according to the VVC standard.

[0045] The compact feature extractor 174 configures one or more processors of the computing system to utilize a size of A compact feature matrix is ​​used to characterize each subsequent inter-frame. The size of the compact feature matrix is ​​not fixed, and the number of feature parameters can be increased or decreased depending on the specific bit consumption requirements.

[0046] These extracted features are used for inter-frame prediction and quantization, as referenced above. Figure 1A The feature encoding module 176 configures one or more processors of the computing system to entropy encode the residuals and transmit the encoded residuals in a bitstream.

[0047] The decoding end includes: a block-based decoder 178 for reconstructing the keyframes, a feature decoding module 180 for reconstructing the compact features through entropy decoding and compensation, and a depth generative model 182 for outputting video for display based on the reconstructed features and the decoded keyframes.

[0048] The block-based decoder 178 configures one or more processors of the computing system to output decoded keyframes from the transmitted bitstream according to the VVC standard.

[0049] The feature decoding module 180 configures one or more processors of the computing system to perform compact feature extraction on the decoded keyframes to output features.

[0050] Subsequently, based on the features from the keyframes and inter-frames, the relevant sparse motion field is calculated, thereby facilitating the generation of pixel-level dense motion maps and occlusion maps.

[0051] The deep generative model 182 configures one or more processors of the computing system to output video for display based on the decoded keyframes, pixel-by-pixel dense motion maps, and occlusion maps with implicit motion field representations, thereby generating appearance, pose, and expression.

[0052] In October 2023, the 32nd meeting of the Joint Video Experts Group (“JVET”) established an ad hoc group on Generative Face Video Compression (“GFVC”), covering software implementation, testing conditions, harmonized experiments, and interoperability studies. A unified software package was proposed to adapt to different GFVC methods using various facial video representations and to support encoding based on the VVC Main 10 profile. Results showed that GFVC can achieve significantly better reconstruction quality than existing VVC standards at ultra-low bitrates.

[0053] This software package only supports entropy encoding of the first frame and synthesizes subsequent frames based on neural networks (e.g., FOMM, Face_vid2vid, and CFTE) without entropy encoding. However, when video frames experience severe jitter, intermediate frames need to be encoded to ensure the continuity of the video sequence. Wang et al. proposed calculating the mean squared error (“MSE”) of the features generated by the neural network, where if the MSE is greater than a preset MSE threshold, the current frame is considered to have changed significantly and needs to be re-encoded instead of synthesized by the neural network. However, setting the MSE threshold is a challenging problem because different neural networks generate features with different ranges.

[0054] Therefore, exemplary embodiments of this disclosure provide for calculating a relative difference metric that describes the feature difference between frames; and based on the relative difference metric, determining whether the current frame is synthesized by a generative neural network without entropy coding or should be recoded.

[0055] The training phase of GFVC utilizes GANs to generate realistic images; however, the training of GANs is very unstable, especially in GFVC, which uses only a few key points to generate the entire video frame.

[0056] Therefore, exemplary embodiments of this disclosure provide two-stage training to stabilize GAN training in GFVC.

[0057] Figure 2 A flowchart is shown for the following operations: calculating a relative difference metric that describes the feature differences between frames; and determining, based on the relative difference metric, whether to recode the current frame or synthesize the current frame using a generative neural network.

[0058] First, one or more processors of the computing system are configured to input the current frame 202 and the reference frame 204 as forward inputs to a learning model to output their respective features. The learning model can be, but is not limited to, any learning model as described above. Next, at step 206, one or more processors of the computing system are configured to calculate a relative difference metric between the features of the current frame 202 and the features of the reference frame 204. Next, at step 208, the magnitude of the relative difference metric is compared to a relative difference threshold. At step 210, if the relative difference metric exceeds the relative difference threshold, one or more processors of the computing system are configured to re-encode the current frame 202 using entropy coding. At step 212, if the relative difference metric is less than the relative difference threshold, one or more processors of the computing system are configured at the decoding end to synthesize the current frame 202 at the decoding end using a generative neural network without entropy coding, as subsequently referenced. Figure 4 As stated above.

[0059] Then refer to Figure 3 The calculation of the relative difference measure according to step 206 is described in more detail.

[0060] Figure 3 A flowchart is shown of an encoding end sequential frame processing method according to an exemplary embodiment of the present disclosure, and a calculation of a relative difference metric based on the frame processing method.

[0061] The encoder configures one or more processors of the computing system to encode a first frame 302 of the video sequence, and the one or more processors of the computing system are configured to input the first frame 302 into a learning model 304 to output keypoints 306 of the first frame 302. The learning model 304 can be any learning model not limited to those described above, and "keypoint" should be understood as describing the coordinates of an object present in the frame. The first frame is subsequently referred to as a "reference frame," and the keypoints of the first frame are subsequently referred to as "reference keypoints." The one or more processors of the computing system are configured to generate a reference frame list 308 and a reference keypoint list 310, and store the reference frames and reference keypoints in the reference frame list 308 and the reference keypoint list 310, respectively.

[0062] One or more processors of the computing system are configured to sequentially input each subsequent current frame 312 into the learning model 304 to output the current keypoint 314 of the current frame 312. For each current frame 312, one or more processors of the computing system are configured to calculate the absolute difference between the current keypoint 314 and a reference keypoint (such as those stored in the reference keypoint list 310) using a distance function. Distance functions, such as mean absolute error (MAE), mean squared error (MSE), Kullback-Leibler (“KL”) divergence, or cross-entropy, measure the distance between two data points. The absolute difference between the current keypoint and the reference keypoint can be described by the following Equation 2: Here, This is any of the aforementioned distance functions. The previous absolute difference between the current keypoint and the reference keypoint is stored in a moving window of any length or fixed size using an additional function. For the current frame 312, the mean of the moving window is calculated by one or more processors of the computing system according to the following formula 3. The absolute difference between the current keypoint and the reference keypoint is divided by the mean of the moving window by one or more processors of the computing system, as shown in the following formula 4, to generate a relative difference between the current keypoint and the reference keypoint, which is the relative difference metric.

[0063] The calculation of the above relative difference measure should be understood. Figure 2 An example of step 206.

[0064] For each current frame 312, a relative difference metric measures the relative difference between the current frame and a reference frame globally. One or more processors of the computing system are configured at step 208 to compare the relative difference metric with a relative difference threshold: if the relative difference metric exceeds the relative difference threshold, the current frame 312 is marked as another reference frame, and the current keypoint 314 is marked as another reference keypoint; after the encoder configures one or more processors of the computing system to re-encode the current frame at step 210, one or more processors of the computing system are configured at step 320 to append the newly marked reference frames and newly marked reference keypoints to the reference frame list 308 and the reference keypoint list 310, respectively.

[0065] If the relative difference metric is less than the relative difference threshold, one or more processors of the computing system are configured to encode the current key point at step 322 using entropy coding (e.g., arithmetic coding and context-based adaptive binary arithmetic coding (CABAC)).

[0066] Each subsequent current frame 312 remaining in the video sequence is processed in the manner described above.

[0067] Figure 4 A flowchart is shown of a decoding end sequential frame processing method based on a relative difference metric according to an exemplary embodiment of the present disclosure, in accordance with the present disclosure.

[0068] At step 402, the decoder configures one or more processors of the computing system to decode reference frames stored in reference frame list 308 from the bit stream.

[0069] At step 404, for each non-reference frame, the decoder configures one or more processors of the computing system to decode reference key points stored in the reference key point list 310 from the bit stream.

[0070] At step 406, the generative neural network configures one or more processors of the computing system to input the reference keypoints and a reference frame corresponding to the current frame into the generative neural network, and outputs the current frame, which is synthesized by the generative neural network without entropy encoding.

[0071] Repeat steps 404 and 406 for each remaining subsequent frame in the video sequence.

[0072] Furthermore, exemplary embodiments of this disclosure provide two-stage training to stabilize GAN training in GFVC. Figure 5 Two-stage training according to an exemplary embodiment of this disclosure is shown.

[0073] At step 502, one or more processors of the computing system train a generative neural network without a discriminator, such that the gradient of the generative neural network is not disturbed by the discriminator during the training phase.

[0074] At step 504, one or more processors of the computing system jointly train the generative neural network using a discriminator, enabling the generative neural network to synthesize high-quality images.

[0075] Those skilled in the art will understand that all the foregoing aspects of this disclosure may be implemented simultaneously in any combination thereof, and all aspects of this disclosure may be combined as yet another embodiment of this disclosure.

[0076] Figure 6 An exemplary system 600 is shown for implementing the above-described processes and methods, which are used to: calculate a relative difference metric describing the feature differences between frames; and, based on the relative difference metric, determine whether the current frame can be synthesized by a generative neural network without entropy coding, or should be recoded.

[0077] The techniques and mechanisms described herein can be implemented by multiple instances of System 600 as well as any other computing devices, systems, and / or environments. Figure 6 The system 600 shown is merely an example and does not impose any limitation on the scope or functionality of any computing device used to perform the above processes and / or procedures. Other known computing devices, systems, environments, and / or configurations suitable for use with the embodiments include, but are not limited to: personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, game consoles, programmable consumer electronics devices, network PCs, microcomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and implementations using field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs).

[0078] System 600 may include one or more processors 602 and system memory 604 communicatively coupled to the one or more processors 602. The one or more processors 602 may execute one or more modules and / or processes to enable the one or more processors 602 to perform various functions. In some embodiments, the one or more processors 602 may include a central processing unit (“CPU”), a graphics processing unit (“GPU”), both a CPU and a GPU, or other processing units or components known in the art. Additionally, each of the one or more processors 602 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0079] Depending on the precise configuration and type of the system 600, the system memory 604 may be volatile, such as RAM, or non-volatile, such as ROM, flash memory, micro hard disk drive, memory card, or a combination thereof. The system memory 604 may include one or more computer-executable modules 606 executable by the one or more processors 602.

[0080] The module 606 may include, but is not limited to, an encoder 608, a decoder 610, a learning model 612, a reference frame list 308, a reference keypoint list 310, a relative difference metric calculation module 618, a generative neural network training module 620, and a generative neural network 622, as described above. Figure 3 , 4 And 5, as well as Table 1.

[0081] The encoder 608 configures the one or more processors 602 to perform image encoding using any of the techniques and processes described above (e.g., the image compression process 100 of FIG1).

[0082] The decoder 610 configures the one or more processors 602 to perform image encoding using any of the techniques and processes described above (e.g., the image compression process 100 of FIG1).

[0083] The learning model 612 configures the one or more processors 602 as key points of the output frame, as referenced above. Figure 3 As stated above.

[0084] The reference frame list 308 and the reference key point list 310 are as described above. Figure 3 As stated above.

[0085] The relative difference metric calculation module 618 configures the one or more processors to calculate the relative difference metric, as referenced above. Figure 3 As stated above.

[0086] The generative neural network training module 620 configures the one or more processors to perform two-stage training, as described above. Figure 5 As stated above.

[0087] The system 600 may further include an input / output (I / O) interface 640 for receiving input image data and bitstream data, and for outputting decoded images to a display, image processor, learning model, etc. The system 600 may also include a communication module 650, enabling the system 600 to communicate with other devices (not shown) via a network (not shown). The network may include the Internet, wired media such as a wired network or direct wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0088] Some or all of the operations described above can be performed by executing computer-readable instructions stored on the computer-readable storage medium 630, as defined below. The term "computer-readable instructions" as used in the specification and claims includes routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including single-processor or multi-processor systems, microcomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based systems, programmable consumer electronic devices, and combinations thereof.

[0089] Computer-readable storage media may include volatile memory (such as random-access memory, RAM) and / or non-volatile memory (such as read-only memory, ROM), flash memory, etc. Computer-readable storage media may also include additional removable and / or non-removable memory, including but not limited to flash memory, magnetic memory, optical memory, and / or magnetic tape memory, which can provide non-volatile storage of computer-readable instructions, data structures, program modules, etc.

[0090] Non-transient computer-readable storage media are examples of computer-readable media. Computer-readable media include at least two types: computer-readable storage media and communication media. Computer-readable storage media include volatile and non-volatile, removable and non-removable media, which are used in any process or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, cassette tape, magnetic tape, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media may embody computer-readable instructions, data structures, program modules, or other data in a carrier or modulated data signal, such as a carrier or other transmission mechanism. The computer-readable storage medium used in this document should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (such as optical pulses through fiber optic cables), or electrical signals propagating through wires.

[0091] Computer-readable instructions stored on one or more non-transitory computer-readable storage media, when executed by one or more processors, can perform the operations described above with reference to Figures 1-5. Typically, computer-readable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific abstract data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the operations can be combined in any order and / or in parallel to perform the process.

[0092] Although the subject matter has been described using terms specific to structural features and / or methodological operations, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described. Rather, the specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A method comprising: A relative difference metric is calculated by one or more processors of the computing system, the relative difference metric describing the feature difference between the current frame and the reference frame; as well as The one or more processors determine, based on the relative difference metric, whether to synthesize the current frame using a generative neural network without entropy encoding.

2. The method according to claim 1, wherein, Calculating the relative difference measure further includes: The current frame and the reference frame are input into the learning model by the one or more processors; and The one or more processors output the current key point of the current frame and the reference key point of the reference frame.

3. The method according to claim 2, wherein, Calculating the relative difference measure further includes: The one or more processors calculate the absolute difference between the current key point and the reference key point.

4. The method according to claim 3, wherein, Calculating the relative difference measure further includes: The absolute difference is divided by the mean of a moving window by the one or more processors, wherein the moving window includes the previous absolute difference.

5. The method according to claim 3, wherein, Calculating the absolute difference further includes: applying a distance function to the current keypoint and the reference keypoint by the one or more processors.

6. The method according to claim 4, wherein, The distance function does not include the mean square error.

7. The method according to claim 1, wherein, Based on the relative difference metric, determining whether to synthesize the current frame further includes: The relative difference metric is compared with a relative difference threshold by the one or more processors.

8. The method according to claim 1, further comprising: The one or more processors decode reference frames stored in the reference frame list from the bit stream; The one or more processors decode the reference key points stored in the reference key point list from the bit stream; The one or more processors input the reference key points and the reference frame corresponding to the current frame into the generative neural network; as well as The one or more processors output the current frame, which is synthesized by the generative neural network without entropy encoding.

9. A computing system, comprising: One or more processors, and A computer-readable storage medium communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors, the computer-readable instructions performing related operations when executed by the one or more processors, the operations including: Calculate a relative difference metric that describes the feature difference between the current frame and the reference frame.

10. The computing system according to claim 9, wherein, Calculating the relative difference measure further includes: The current frame and the reference frame are input into the learning model; and Output the current key point of the current frame and the reference key point of the reference frame.

11. The computing system according to claim 10, wherein, Calculating the relative difference measure further includes: Calculate the absolute difference between the current key point and the reference key point.

12. The computing system according to claim 11, wherein, Calculating the relative difference measure further includes: Divide the absolute difference by the mean of the moving window, wherein the moving window includes the previous absolute difference.

13. The computing system according to claim 11, wherein, Calculating the absolute difference also includes applying a distance function to the current keypoint and the reference keypoint.

14. The computing system according to claim 13, wherein, The distance function does not include the mean square error.

15. The computing system according to claim 9, wherein, Based on the relative difference metric, determining whether to synthesize the current frame further includes: The relative difference metric is compared with the relative difference threshold.

16. The computing system according to claim 9, wherein, The operation also includes: Decode reference frames stored in the reference frame list from the bitstream; Decode reference key points stored in the reference key point list from the bitstream; The reference key points and the reference frame corresponding to the current frame are input into the generative neural network; and Output the current frame, which is synthesized by the generative neural network without entropy encoding.

17. A method comprising: Generative neural networks are trained by one or more processors of a computing system without a discriminator. as well as The generative neural network is trained by one or more processors of the computing system using a discriminator.

Images