Generating supersamples

Abstract

Frame generation super-sampling may include generating an image frame embedding from a first image frame in the image stream, and predicting a synthesized second image frame in the image stream using the image frame embedding of the first image in the image stream. The synthesized second image is a frame displayed after the first image frame in the image stream.

Description

Technical Field

[0001] Various aspects of this disclosure relate to supersampling, and more specifically, various aspects of this disclosure relate to frame generation supersampling. Background Technology

[0002] Computer graphics systems can generate images from geometric data. They are commonly used in graphics-intensive applications such as video games. In recent years, artificial intelligence (AI) has been applied to real-time rendering of graphics to build sharper, higher-resolution images. For example, Nvidia's Deep Learning Super Sampling (DLSS) 3 implementation inserts a composite frame generated by a machine learning model between the most recently rendered frame and the frame preceding it before display. This introduces significant latency because the device must render both the most recent and the previous frame, resulting in a variable frame rate even though it produces a higher frame rate. These variable frame rate changes are very noticeable to the user and are sometimes referred to as "stuttering." These changes in variable frame rates degrade the user experience, as users may experience frame rate drops during intensive application processing and higher frame rates during less demanding tasks.

[0003] It is against this backdrop that the various aspects of this disclosure are presented. Attached Figure Description

[0004] The teachings of this disclosure can be readily understood by considering the following specific embodiments in conjunction with the accompanying drawings, in which: Figure 1A It is a simplified node diagram of a recurrent neural network for improved frame generation supersampling according to various aspects of this disclosure.

[0005] Figure 1B It is a simplified node diagram of a recurrent neural network for an improved frame generation supersampling based on various aspects of this disclosure.

[0006] Figure 1C This is a simplified diagram of a convolutional neural network for improved frame generation supersampling according to various aspects of this disclosure.

[0007] Figure 2 This is a flowchart illustrating a method for training an autoencoder network to generate synthetic image frames for improved frame generation supersampling, according to various aspects of this disclosure.

[0008] Figure 3 A block diagram depicts training encoder-decoder pairs via unsupervised training according to various aspects of this disclosure.

[0009] Figure 4The improved frame generation supersampling method using an autoencoder trained with a machine learning algorithm, according to various aspects of this disclosure, is illustrated graphically.

[0010] Figure 5 The flowchart illustrates a method for improved frame generation supersampling according to various aspects of this disclosure.

[0011] Figure 6 A system for improved frame generation supersampling is described according to various aspects of this disclosure. Detailed Implementation

[0012] While the following detailed description contains many specific details for illustrative purposes, those skilled in the art will understand that many variations and modifications of these details are within the scope of this disclosure. Therefore, the examples of embodiments of this disclosure described below are set forth without loss of generality and without implying any limitation on the claimed disclosure.

[0013] Computer graphics systems typically render frames at the fastest possible speed allowed by computing power and software. At variable frame rates, the monitor and computer can lose synchronization, resulting in a visual artifact known as "tearing." When a monitor with vertical synchronization or similar capabilities is connected to the computer, the monitor can tell the computer when it will render a frame, or vice versa. Alternatively, the computer can render two real frames and, for example, manually render the intermediate frames between them using motion vectors. This is called variable rendering rate (VRR) rendering. VRR still has application latency issues because the rendering rate is limited by the rate at which the most recent real frame is created. VRR does avoid screen tearing, but the results are not as good as rendering at a fixed frame rate, and screen tearing is still quite noticeable to the user during changes in the rendering rate. To achieve a fixed frame rate, the system is configured to reduce the overall rendering quality until the target fixed rate is achieved for both processing more and less dense frames.

[0014] According to various aspects of this disclosure, a computer graphics system can operate at a real, fixed frame rate and generate one or more composite frames after each real frame or after every two or more real frames, depending on the composite frame insertion interval. Composite frames can be generated from the first two real frames using algorithmic frame generation or a neural network model trained with a machine learning algorithm for predicting composite frames. Subsequent frames can be generated after the composite frames and then displayed. In some alternative implementations, composite frames can be created using the most recent frame and motion vectors generated from an image in an image stream. This improved frame generation is not constrained by the generation rate of the most recent real frame because the composite frames are generated after the most recent real frame and before the creation of subsequent real frames.

[0015] General Neural Network Training According to various aspects of this disclosure, improved frame generation for supersampling can be implemented using an autoencoder-type neural network layout having an encoder network (which participates in dimensionality reduction to output image frame embeddings) and a decoder (which uses the image frame embeddings to predict synthesized subsequent frames), collectively referred to as an autoencoder neural network. The autoencoder neural network outputs feature-length image embeddings, and the decoder includes a neural network for identification, which uses those feature-length image embeddings to generate one or more subsequent synthesized frames. The autoencoder can also be configured to employ motion vectors and / or user input information. The motion vectors and / or user input information can be used in conjunction with frame data to generate image embeddings.

[0016] Each neural network used in an autoencoder can be of any type known in the art, but preferably, the neural network is a convolutional neural network (CNN). In alternative embodiments, the CNN is any type of convolutional recurrent neural network (CRNN).

[0017] Figure 1A The basic form of a CNN with one layer of nodes 120 is described, where each node is characterized by an activation function S, an input weight U, and an output transition weight V. It should be noted that the activation function S can be any nonlinear function known in the art and is not limited to the hyperbolic tangent (tanh) function. The input weight U and output transition weight V can be randomized before training and can be changed during training based on a chosen machine learning algorithm. For example, the activation function S can be a sigmoid or ReLU function. The CNN is characterized by adding hidden layers 121 to the output of the first node layer 120, thereby creating a hierarchical structure. The CNN can have any number of hidden layers, and the hidden layers can have activation functions H and output transition weights Z different from those of the initial node layer 120. The inputs 1, 2, 3, 4, and 5 of the initial node layer can be monochrome pixel values ​​from input image frames. Each color channel of the image can have one input layer (e.g., for a red-green-blue mixture, there can be three input layers, one for each color channel), as discussed below. Other inputs (such as user input from an input device) can be concatenated to the input space of one or more color channels in the color channels, or a separate input layer can be created for that other input type. Furthermore, the initial node layer 120 in the CNN is not fully connected across all inputs, but rather operates on a window of the input, such as... Figure 1CAs discussed in [the previous section], operating only on a window of input allows CNNs to preserve spatial information better than traditional fully connected neural networks. Subsequent hidden layers can combine the outputs of all nodes from the window of previous layers; this type of layer is called a pooling layer. Max pooling layers are another specialized hidden layer similar to pooling layers, except that they only take values ​​from the input with the highest weights from the previous layers within the window. The design of a CNN is highly dependent on its intended use, and the number of nodes, layers, layer types, and activation functions are all variable and require experimental optimization.

[0018] like Figure 1B As shown, a recurrent neural network layer adds a recurrent node transition W, which returns the value from the previous iteration to the current node. Therefore, a recurrent layer can be viewed as a series of nodes 120 with the same activation function as time T and T+1 progress. Thus, the RNN maintains historical information by feeding the result from the previous time T to the current time T+1. CNN can implement recurrent node layers. Another type of RNN that can be used is the Long Short-Term Memory (LSTM) neural network, which adds memory blocks to RNN nodes with input gate activation functions, output gate activation functions, and forget gate activation functions, thereby producing a gated memory that allows the network to retain some information for a longer period of time, as described in the following literature: Hochreiter and Schmidhuber, “Long Short-Term Memory”, Neural Computation 9(8):1735-1780 (1997).

[0019] Figure 1C An example layout of a convolutional neural network for image recognition is depicted. In this depiction, a convolutional neural network is generated for image 132, which has a size of 4 units in height and 4 units in width, giving a total area of ​​16 units. The depicted convolutional neural network has a filter size 133 with a height of 2 units and a width of 2 units, where the jump value is 1, and the channel size 136 is 9. (For clarity of depiction, only the connections 134 between the first column of channels and their filter windows are depicted.) Convolutional neural networks according to various aspects of this disclosure can have any number of additional neural network node layers 131 and can include layer types such as additional convolutional layers of any size, fully connected layers, pooling layers, max pooling layers, local contrast normalization layers, etc.

[0020] Autoencoder training Figure 2 A method for training an autoencoder network to generate synthetic image frames is shown.

[0021] Training a neural network (NN) starts with initializing the weights of a 201NN (in... Figure 1A The initial weights (shown as U and V) depend on the type of activation function and the number of inputs to the nodes. The initial weights of a neural network cannot be zero, as this will lead to asymmetric computation in the hidden layers. Generally, the initial weights should be randomly distributed. For example, a neural network with a tanh activation function should have weights distributed as follows: and Random values ​​between, where n It represents the number of inputs to the node.

[0022] After initialization, image frames from the image stream are provided to the encoder neural network 202. Exemplary image stream types are not limited to: video streams, time-lapse photography, slow-motion photography, etc. In some implementations, motion vectors may be provided along with the image frames. The motion vector data may be concatenated to the end of the image frames or provided to a separate network that generates motion embeddings. The motion embeddings may be concatenated to the image frame embeddings. Alternatively, two or more images may be provided to the encoder neural network. These images may be concatenated together and used as a single array in the encoder's input space. The two or more images may include the most recent image frame generated by the renderer and one or more image frames generated by the device preceding that most recent image frame.

[0023] In some alternative implementations, user input is also provided to the encoder neural network 208. The user input can be concatenated to image frames in the encoder input space. Alternatively, a separate user input network can generate user input embeddings, and these embeddings can be concatenated with image frame embeddings and used as input to the decoder.

[0024] An autoencoder consists of a neural network trained using a method called unsupervised learning. In unsupervised learning, a corresponding encoder NN is provided to the decoder NN, and the encoder and decoder are trained together as a single unit. The basic function of an autoencoder is to obtain an element R. d The input x is mapped to an element R. d The '' represents h, and this mapped representation can also be called an image vector. It uses the type h = ƒ θ = σ ( W χ + b And the parameter is θ = { W , b A deterministic function} is used to create image vectors. Then, a decoder NN reconstructs the input from the representative image vectors using the inverse operation of ƒ: y = ƒ θ’ (h) = σ ( W'h + b’ ), where θ' = { W’ , b’These two parameter sets can be constrained as follows: The form is that the same weights are used to encode the input and decode the representation. Each training input χ i Image vector mapped to it h i and its reconstruction y i These parameters are trained by minimizing an appropriate cost function on the training set. A convolutional autoencoder works similarly to a basic autoencoder, except that weights are shared across all locations of the input. Therefore, for a single-channel input (such as a black and white image)... x The representation of the k-th feature map is given below: h k = σ(x*W k + b k The bias is broadcast across the entire graph. The variable σ represents the activation function, b represents the single bias used for each potential graph, W represents the weights shared across the graph, and * is the 2D convolution operator. To reconstruct the input, the formula is:

[0025] Each input channel has a bias C. h Identify feature map groups, and This indicates an operation that flips both the dimensions and weights. Further information on the training and weighting of convolutional autoencoders can be found in "Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction" by Masci et al. ICANN Pages 52-59, 2011.

[0026] As discussed above, the automatic encoder will input x i Mapped to its corresponding representative image embedding h i 203, those image embeddings are then provided to the decoder 204. According to aspects of this disclosure, the encoder and decoder are configured to predict one or more subsequent frames in the input image stream 205, rather than reconstructing the original input. The output of the decoder according to aspects of this disclosure differs from that of a conventional autoencoder by making the network output have more channels than the input. By way of example and not limitation, if the input is a 100×100 RGB 3-channel image, the output will be b × 3 × 100 × 100, where b is the number of channels corresponding to all timestamps within the interval {t+w}. That is, given an input image F at time t, the encoder and decoder are trained to predict interval F.{t+w} The image within the range, where w is the prediction interval. In other words, ideally, training would involve making the synthesized subsequent images F'... t+w With image stream {F1, F2, F3…F T The actual subsequent image F in} t+w Minimizing the mean square error between them yields the following equation: Equation 1 Therefore, according to various aspects of this disclosure, the encoder NN ε generates a k-dimensional vector Φ such that Φ = ε (F t ), and decoder D Generate the predicted subsequent image F from the k-th dimension vector Φ. t+w ,get: {Ft +w} = D (Φ) = D (ε (F t Equation 2 The decoder is a convolutional network with upsampling layers that transforms a k-dimensional vector Φ into a sequence of output images. The training and optimization of this decoder and encoder neural network system leverages the fact that the training set is a stream of images, and therefore previous and subsequent images are readily available. The decoder's output (i.e., the predicted subsequent image) is compared with the corresponding actual subsequent image from the image stream. In other words, the predicted image F'... t+w The predicted image is compared to the actual image at time t+w. The difference between the actual and predicted images is applied to a loss function, such as the mean squared error function or the cross-entropy loss function. The result of the loss function is then used to optimize and train the NN using known neural network training methods, such as backpropagation with stochastic gradient descent, as indicated at 207.

[0027] After multiple rounds of training, the encoder and decoder neural network outputs correctly predict subsequent images in the image stream, and the loss function has stabilized.

[0028] In some implementations, prediction is not limited to the next image immediately following the input image in the image stream. The decoder system can be trained to predict any number of images before or after the input image. In some alternative implementations, the input to the encoder and decoder neural networks can be optimized by selecting certain image points for the neural network to process, or by encoding the video using conventional video coding methods (such as MPEG-2 or H.264 encoding). According to an additional aspect of this disclosure, additional tuning of the encoder and decoder systems can be performed by manually manipulating the number of nodes in a layer or by changing the activation function. Tuning of a convolutional neural network can also be performed by changing the weights, size, and number of channels of a mask. In a fully connected network, the number of hidden units can be changed to tune the network.

[0029] Figure 3 A block diagram depicts an encoder 301 trained together with a decoder 303 via unsupervised training. The encoder is provided with the most recent image 306 as input, and in this implementation, with a previous image 308 from an image stream 309. The image stream 309 contains the previous image 308 and / or subsequent images T+1307 and T+2311. Additionally, in this implementation, user input 312 from an input device is appended to the encoder input. Here, the user input can occur simultaneously with the generation of the most recent image (e.g., within 15 ms), or slightly earlier than the generation of the most recent image frame (e.g., displayed more than 15 ms earlier but less than two seconds earlier). The idea is to bind user input to the generation of image frames; for example, when a player in a game presses the jump button, the next few frames will be jump animation frames, which the encoder can learn. User input can be one or more button presses or movement information from an input device, such as a game controller, keyboard, mouse, trackball, joystick, inertial measurement unit (IMU), etc. User input can be a single input or a combination of inputs. Input combinations can be determined by a threshold time interval between inputs; if the user made another input before the threshold time interval has elapsed, that input can be included as an element of the input combination. In some alternative implementations, instead of including previous images, the motion vector of the most recent image can be provided as input.

[0030] As discussed above, encoder 301 generates a representative image embedding 302, which is provided to decoder 303. Decoder 303 then constructs the synthesized next image F from the representative image embedding 302. t+1 305, and in some implementations, the decoder can also predict the next synthesized image F. t+2304. The predictions for the synthesized next image 305 and (optionally) the synthesized next-next image 304 are checked by comparing 310 with the actual next image 307 and optionally the original next-next image 309, respectively. As shown, the predicted next image 305 differs from the actual next image 307 because the face is sticking out its tongue in the actual image, but not in the predicted image, thus requiring more training to achieve the correct result. The results of the comparison are then used to train and optimize the encoder-decoder system, as described above. The encoder-decoder system is considered sufficiently trained when the loss function does not change significantly with variations in the parameters.

[0031] Once trained, an encoder-decoder pair can be used to output a synthesized image frame that can be displayed after the most recent image frame generated by the system in the image stream. For example, and not limited to, the system can generate a previous real image frame and a first real image frame, and then implement the encoder-decoder pair to generate a synthesized second image and (optionally) a synthesized third image. The synthesized second image can be displayed after the first real image and before the system generates the second real image. Here, generating a real image involves using one or more stages in a conventional graphics processing pipeline, such as primitive generation, input assembly, vertex shading, shell shading, subdivision, domain shading, geometry shading, rasterization, and pixel shading. In aspects of this disclosure, however, the synthesized image is generated by machine learning or by an algorithm without using stages in a conventional graphics processing pipeline.

[0032] Improved frame generation supersampling based on autoencoder Figure 4 A modified frame generation supersampling method using an autoencoder trained with a machine learning algorithm, according to various aspects of this disclosure, is illustrated graphically. In the illustrated implementation, a first real image frame 402 from an image stream generated by the system using one or more stages of a conventional graphics pipeline is provided to the trained encoder neural network 405. Additionally, motion vectors 403 generated from the first real image frame and the image stream are provided to the encoder neural network. Each image frame consists of an array of pixels, which may be grouped together into, for example, but not limited to, 4-pixel by 4-pixel (16 pixels in total) blocks, and further grouped into 16-pixel by 16-pixel macroblocks (4 × 4 square blocks). Furthermore, sub-macroblock grouping may be, for example, but not limited to, 2-block by 2-block groups. Motion vectors can be created by searching for images of matching blocks or macroblocks in the image stream and generating vector values ​​based on the shift of the matching blocks in the images.

[0033] Finally, the most recent user input from input device 404 can (optionally) be provided to encoder NN 405. As discussed above, the most recent user input from the input device can be performed simultaneously with the most recent image used to generate the input, or slightly earlier than the generation of the most recent image frame. (See also: Regarding...) Figure 2 and Figure 3 The discussion utilizes machine learning algorithms to train the encoder NN405 to generate image embeddings. The decoder NN406 is closely connected to the encoder NN405 and obtains the image embeddings, predicting a synthesized second image 407. This synthesized second image is either an image following the first image in the image stream, or an image located between the real first image and the real second image 410, but before the graphics pipeline 409 has generated the real second image. In some implementations, the decoder NN can generate one or more additional synthesized images from the image embeddings. These additional images can be displayed before the real next image. After displaying one or more synthesized next images, the real next image 410 can be displayed. The real next image 410 can be generated through one or more stages of the graphics pipeline 409 while one or more synthesized next images are being generated, or after one or more synthesized next images have been generated. Typically, generating the real next image will take longer than generating synthesized images using the autoencoder NN. In some (optional) implementations, the display of the real next image may be delayed 411 until one or more synthesized next images have been displayed. This prevents the injection of the real next image between synthesized images.

[0034] Figure 5This is a flowchart illustrating a method for improved frame generation supersampling according to various aspects of this disclosure. In this implementation, a real previous image frame and a real most recent image frame generated through one or more stages of the graphics pipeline can be provided to an encoder NN 501. Optionally, one or more recent user inputs from an input device can also be provided to the encoder NN 507. The encoder NN then generates an image frame embedding 502 from the real previous image frame, the real most recent image frame, and optionally one or more recent user inputs. The image frame embedding is then provided to a trained decoder NN 503. The trained decoder NN predicts one or more synthesized next image frames from the image frame embedding 504. The synthesized next image is then provided to a frame buffer 505, which can be provided to a display device, and the synthesized next image frame can be displayed on the screen of the display device 506. After displaying one or more synthesized next image frames, a real next image frame generated through one or more stages of the graphics pipeline can be displayed 508. In some alternative implementations, the pixel values ​​of the synthesized next image frame can be checked against the pixel values ​​of the real next image frame, and if they match, or if the difference between the pixel values ​​is less than a threshold, the real next image frame may not be displayed. The real next image frame can also be provided as input 510 to the encoder NN as the real most recent image 501 to continue the synthesized frame generation loop.

[0035] Algorithm Frame Generation Algorithmic frame generation uses algorithms, rather than machine learning and neural networks, to generate synthetic frames. According to various aspects of this disclosure, algorithmic frame generation can use motion vectors or real previous frames compared to the real most recent frame. Using real previous frames, the system can identify unchanged blocks or macroblocks within the image between the real previous frame and the real most recent frame. These unchanged macroblocks can be passed to the synthetic image without alteration.

[0036] Next, the system can identify motion between the real previous frame and the real most recent frame by searching for blocks, macroblocks, or sub-macroblocks in the real most recent frame that match the real previous frame; this is sometimes called motion search. Then, based on the assumption that the movement will generally continue along the direction and magnitude of the motion vector over a short period between frames, the position of the matching block, macroblock, or sub-macroblock in the synthesized next image frame can be predicted using the magnitude and direction of the shift in the matching block, macroblock, or sub-macroblock. In some implementations, the motion vectors may already be computed, and in this case, additional motion search is not required.

[0037] Once the location of a matching block, macroblock, or sub-macroblock has been determined, the pixel value at that location can be overwritten with the value of the matching block, macroblock, or sub-macroblock. There are blank areas within the matching block, macroblock, or sub-macroblock; if the matching block, macroblock, or sub-macroblock is not in these areas, the value can be copied from the previous, actual image, or it can be copied from the nearest neighbor pixel value.

[0038] Systems for improving frame generation supersampling Figure 6 A system according to various aspects of this disclosure is described. The system may include a computing device 600 coupled to a user input device 602. The user input device 602 may be a controller, a touchscreen, a microphone, a keyboard, a mouse, a joystick, a motion control device (IMU), or other device that allows the user to input voice data into the system.

[0039] The computing device 600 may include one or more processor units 603, which may be configured according to well-known architectures (e.g., single-core, dual-core, quad-core, multi-core, processor-coprocessor, unit processor, etc.). The computing device may also include one or more memory units 604 (e.g., random access memory (RAM), dynamic random access memory (DRAM), read-only memory (ROM), etc.).

[0040] Processor unit 603 can execute one or more programs, portions of which can be stored in memory 604, and processor 603 can be operatively coupled to memory (e.g., accessed via data bus 605). The programs can be configured to implement training of encoder 608 and decoder 622. Memory 604 may also contain software modules such as encoder module 608, decoder module 622, and / or algorithm frame generation module 609. The algorithm frame generation module can generate frames using motion vectors generated from image stream 621, as discussed above. The overall structure and possibilities of the NN can also be stored as data 618 in mass storage area 615. Processor unit 603 is further configured to execute one or more programs 617 stored in mass storage area 615 or memory 604, which cause the processor to execute method 200 for training encoder 608 and decoder 622 from image stream 621 and / or training image recognition NN from image embedding 610. The system can generate neural networks as part of the NN training process. These neural networks can be stored in memory 604 as part of encoder module 608 and decoder module 622. The complete NN can be stored in memory 604 or as data 618 in mass storage area 615. Program 617 (or parts thereof) can also be configured, for example, by appropriate programming to encode unencoded video or manipulate one or more images in an image stream stored in a buffer in memory 604.

[0041] The computing device 600 may also include well-known support circuitry, such as input / output (I / O) 607, circuitry, power supply (P / S) 611, clock (CLK) 612, and cache 613, which may communicate with other components of the system, for example, via a digital bus 605. The computing device may include a network interface 614. The processor unit 603 and network interface 614 may be configured to implement a local area network (LAN), or a PAN via a suitable network protocol for a personal area network (PAN), such as Bluetooth. The computing device may optionally include a mass storage device 615 (such as a disk drive, CD-ROM drive, tape drive, flash memory, etc.), and the mass storage device may store programs and / or data. The computing device may also include a user interface 616 for facilitating interaction between the system and a user. The user interface may include a display device, such as a monitor, flat panel screen, or other audiovisual device.

[0042] Network interface 614 facilitates communication via electronic communication network 620. Network interface 614 can be configured to enable wired or wireless communication via local area networks (LANs) and wide area networks (WANs) such as the Internet. Device 600 can send and receive data and / or requests via network 620 via one or more message packets. Message packets sent via network 620 can be temporarily stored in a buffer in memory 604.

[0043] While the foregoing is a complete description of the preferred embodiments of the invention, various alternatives, modifications, and equivalents may be used. Therefore, the scope of the invention should not be determined by reference to the above description, but rather by reference to the appended claims and their equivalents throughout. Any feature described herein (whether preferred or not) may be combined with any other feature described herein (whether preferred or not). In the appended claims, The indefinite article "a (A)" or "a (An)" "Amount" refers to one or more items following the article, unless otherwise expressly stated therein. The appended claims should not be construed as including means plus functional limitations, unless such limitations are explicitly stated in a given claim using the phrase "means for...".

Claims

1. A method for frame generation supersampling, comprising: Generate image frame embeddings from the first image frame in the image stream; The image frame embedding of the first image in the image stream is used to predict the synthesized second image frame in the image stream; The synthesized second image frame is displayed after the first image frame in the image stream.

2. The method of claim 1, wherein generating the image frame embedding further comprises: Image frame embeddings are generated from the first image frame in the image stream using user input information.

3. The method of claim 2, wherein the user input information includes one or more button presses on the input device.

4. The method of claim 2, wherein the user input information includes one or more movements of the input device.

5. The method of claim 4, wherein the input device is a mouse, trackball, or joystick.

6. The method of claim 4, wherein the input device is an inertial measurement unit.

7. The method of claim 1, wherein generating the image frame embedding further comprises: An image frame embedding is generated from the first image frame in the image stream using motion vectors from the image stream.

8. The method of claim 1, wherein generating the image frame embedding from the first image in the image stream further comprises: An image frame embedding is generated from the first image frame in the image stream by combining previous image frames in the image stream.

9. The method of claim 1, wherein generating the image frame embedding from the first image in the image stream comprises: The image frame embedding is generated using a neural network trained with machine learning algorithms.

10. The method of claim 1, wherein predicting the synthesized second image frame in the image stream using the image frame embedding of the first image in the image stream comprises: The synthesized second image frame is generated using a neural network trained with machine learning algorithms.

11. The method of claim 1, further comprising: The real second image frame is displayed after the synthesized second image frame.

12. The method of claim 1, wherein using information from the first image frame to predict a synthesized second image frame in the image stream further comprises: The image frame embedding is used to predict the synthesized third image frame, and the synthesized third image frame is displayed after the synthesized second image frame in the image stream.

13. The method of claim 12, further comprising: The real second image frame is displayed after the synthesized third image frame is displayed.

14. A system for frame generation supersampling, comprising: processor; A memory operatively coupled to the processor; Non-transitory processor-executable instructions, embodied in the memory, which, when executed by the processor, cause the processor to perform a method for frame generation supersampling, the method comprising: Generate image frame embeddings from the first image frame in the image stream; The image frame embedding of the first image in the image stream is used to predict the synthesized second image frame in the image stream; The synthesized second image frame is displayed after the first image frame in the image stream.

15. The system of claim 14, further comprising an input device, and wherein generating the image frame embedding from the first image in the image stream further comprises: An image frame embedding is generated from the first image frame in the image stream by combining user input information.

16. The method of claim 15, wherein the user input information includes one or more button presses on the input device.

17. The method of claim 16, wherein the user input information includes one or more movements of the input device.

18. The method of claim 17, wherein the input device is a mouse, a trackball, or a joystick.

19. The method of claim 17, wherein the input device is an inertial measurement unit.

20. A non-transitory computer-readable medium having embedded computer-executable instructions, which, when executed by a computer, cause the computer to implement a method for frame generation supersampling, the method comprising: Generate image frame embeddings from the first image frame in the image stream; The image frame embedding of the first image in the image stream is used to predict the synthesized second image frame in the image stream; The synthesized second image frame is displayed after the first image frame in the image stream.

Images