Predicting next frame of video using integration

Abstract

A computer-implemented method for predicting a next frame of a video is provided. Aspects include obtaining a video having a sequence of frames, obtaining a plurality of machine learning models (MLMs) trained to perform next frame prediction, and inputting a subset of the sequence of frames into each of the plurality of MLMs. Aspects also include calculating a score for each of the plurality of MLMs for a subset of the sequence of frames, and generating a next frame using one of the plurality of MLMs having the highest score, where the next frame is a frame immediately following the subset of the sequence of frames.

Description

BACKGROUND

[0001] The present invention relates generally to video processing, and more specifically, to predicting a next frame of a video using ensembling.

[0002] The use and distribution of videos for entertainment, communication, and educational purposes has greatly increased. As a result, the need to accurately generate and predict a next video frame for a given sequence of video frames has also increased. Generally, next frame prediction builds on the understanding that information in historical images can be used to predict the content of future images. A next frame prediction model receives an input of an unlabeled video frame and utilizes a trained network to generate one or more subsequent frames.

[0003] Several next frame prediction models have recently been developed. Generally, each available next frame prediction model is best suited for predicting a next frame of certain types of videos. For example, one type of next frame prediction model can be constructed to accurately predict a next frame of a sequence of frames that includes a person in motion, and another type of next frame prediction model can be constructed to accurately predict a next frame of a sequence of frames that includes the motion of a machine or animal in nature. SUMMARY

[0004] Embodiments of the present invention relate to a computer-implemented method for predicting a next frame of a video using ensembling. The computer-implemented method includes obtaining a video having a sequence of frames, obtaining a plurality of machine learning models (MLMs) trained to perform next frame prediction, and inputting a subset of the sequence of frames into each of the plurality of MLMs. The method further includes calculating a score for each of the plurality of MLMs for the subset of the sequence of frames, and generating a next frame using one of the plurality of MLMs having a highest score, wherein the next frame is a frame immediately following the subset of the sequence of frames.

[0005] Embodiments of the present invention relate to a computer program product for predicting a next frame of a video using ensembling. The computer program product includes one or more computer-readable storage media having computer-readable program code stored thereon. The computer-readable program code is executed by a processor of a computer system to cause the computer system to perform operations. The operations include obtaining a video having a sequence of frames, obtaining a plurality of machine learning models (MLMs) trained to perform next frame prediction, and inputting a subset of the sequence of frames into each of the plurality of MLMs. The operations further include calculating a score for each of the plurality of MLMs for the subset of the sequence of frames, and generating a next frame using one of the plurality of MLMs having a highest score, wherein the next frame is a frame immediately following the subset of the sequence of frames.

[0006] Various embodiments of the present invention relate to a computing system including a processor, a memory coupled to the processor, and one or more computer-readable storage media coupled to the processor. The one or more computer-readable storage media collectively contain instructions executed by the processor via the memory to cause the processor to perform operations. The operations include obtaining a video having a sequence of frames, obtaining multiple machine learning models (MLMs) trained to perform next frame prediction, and inputting a subset of the frame sequence into each of the multiple MLMs. The operations also include calculating a score for each of the multiple MLMs for the subset of the frame sequence, and using the MLM with the highest score among the multiple MLMs to generate the next frame, wherein the next frame is the frame immediately following the subset of the frame sequence.

[0007] Additional technical features and benefits are achieved through the technology of this invention. Embodiments and aspects of the invention are described in detail herein and are considered part of the claimed subject matter. For a better understanding, refer to the specific embodiments and accompanying drawings. Attached Figure Description

[0008] The details of the proprietary rights described herein are specifically pointed out and explicitly claimed in the appended claims. The foregoing and other features and advantages of embodiments of the invention will be apparent from the following detailed description taken in conjunction with the accompanying drawings, wherein:

[0009] Figure 1 This is a schematic diagram of a computing environment according to one or more embodiments of the present invention;

[0010] Figure 2 This is a block diagram of components of a machine learning training and inference system according to one or more embodiments of the present invention;

[0011] Figure 3 This is a schematic diagram illustrating the training of multiple machine learning models for performing next frame prediction according to one or more embodiments of the present invention;

[0012] Figure 4 This is a block diagram illustrating a system for predicting the next frame of a video using integration, according to one or more embodiments of the present invention.

[0013] Figure 5 This is a flowchart illustrating a computer-implemented method for predicting the next frame of a video using integration, according to one or more embodiments of the present invention; and

[0014] Figure 6 This is a flowchart illustrating a computer-implemented method for calculating a score of a machine learning model for predicting the next frame of a video, according to one or more embodiments of the present invention.

[0015] The figures depicted herein are illustrative. Many variations may be made to the figures or operations described herein without departing from the scope of the invention. For example, actions may be performed in a different order, or actions may be added, deleted, or modified. Furthermore, the term "coupled" and its variations describe a communication path between two elements and do not imply a direct connection between the elements without any intermediate elements / connections between them. All such variations are considered part of the specification. Detailed Implementation

[0016] Embodiments of the present invention relate to a computer-implemented method for predicting the next frame of a video using ensemble methods. The computer-implemented method includes obtaining a video having a sequence of frames, obtaining multiple machine learning models (MLMs) trained to perform next frame prediction, and inputting a subset of the frame sequence into each of the multiple MLMs. The method also includes calculating a score for each of the multiple MLMs for the subset of the frame sequence, and using the MLM with the highest score among the multiple MLMs to generate the next frame, wherein the next frame is the frame immediately following the subset of the frame sequence. One technical benefit of selectively using the identified MLMs to generate the next frame is that it optimizes the accuracy of prediction for each frame of the video.

[0017] Alternatively or concurrently, in embodiments of the invention, each of the plurality of MLMs has an accuracy. Therefore, when selecting an MLM for the generation of the next frame, the overall accuracy of each MLM and the accuracy of each MLM in predicting each frame are taken into consideration.

[0018] Alternatively, in embodiments of the invention, calculating the score for each of the plurality of MLMs includes calculating the frame loss value for each of the subsets of the frame sequence by comparing ground truth frames from a subset of the frame sequence with predicted frames generated by the corresponding MLMs among the plurality of MLMs. A technical advantage of calculating the frame loss value for each subset of the frame sequence is the ability to identify the performance trend of each MLM over time, which can be used to select the optimal MLM for generating the next frame.

[0019] Alternatively, in embodiments of the invention, calculating the score for each of the plurality of MLMs may further include calculating the total loss value for each of the plurality of MLMs based on the frame loss value of each of the subsets of frame sequences corresponding to the MLMs among the plurality of MLMs. A technical advantage of calculating the total loss value for each of the plurality of MLMs is that it simplifies the process of comparing the performance of various MLMs, i.e., it is not necessary to compare the frame loss value of each MLM individually.

[0020] Alternatively, in embodiments of the invention, calculating the score for each of the plurality of MLMs may further include calculating the total loss value for each of the plurality of MLMs by summing the frame loss values ​​of each of the subsets of frame sequences corresponding to the MLMs among the plurality of MLMs. A technical advantage of calculating the total loss value for each of the plurality of MLMs is that it simplifies the process of comparing the performance of various MLMs, i.e., it is not necessary to compare the frame loss values ​​of each MLM individually.

[0021] Alternatively, in embodiments of the invention, calculating the score for each of the plurality of MLMs may also include multiplying the total loss value corresponding to the MLM among the plurality of MLMs by the accuracy corresponding to the MLM among the plurality of MLMs. This approach considers the technical advantage of using the accuracy and total loss value of each MLM, as well as the accuracy of each MLM in predicting each frame, when selecting the MLM for generation of the next frame.

[0022] Additionally or alternatively, in embodiments of the invention, the multiple MLMs include Long Short-Term Memory (LSTM) models, Convolutional Neural Network (CNN) models, and Gated Recurrent Unit (GRU) models. One technical benefit of utilizing various MLM types is improved accuracy in predicting the next frame, as each model type will have different performance characteristics.

[0023] Alternatively, in embodiments of the invention, the video's metadata is updated to include an indication of the MLM among a plurality of MLMs that has the highest score for predicting the next frame. One technical advantage of storing the indication of the MLM among a plurality of MLMs that has the highest score for predicting the next frame in the metadata is that the metadata can be used to identify frames that can be discarded, and the MLMs that should be used to generate those discarded frames, thereby allowing video compression while minimizing video degradation during playback.

[0024] Embodiments of the present invention relate to a computer program product for predicting the next frame of a video using integration. The computer program product includes one or more computer-readable storage media having computer-readable program code commonly stored on the one or more computer-readable storage media. The computer-readable program code is executed by a processor of a computer system to cause the computer system to perform operations. These operations include obtaining a video having a sequence of frames, obtaining multiple machine learning models (MLMs) trained to perform next frame prediction, and inputting a subset of the frame sequence into each of the multiple MLMs. The operations also include calculating a score for each of the multiple MLMs for the subset of the frame sequence, and using the MLM with the highest score among the multiple MLMs to generate the next frame, wherein the next frame is the frame immediately following the subset of the frame sequence.

[0025] Various embodiments of the present invention relate to a computing system including a processor, a memory coupled to the processor, and one or more computer-readable storage media coupled to the processor. The one or more computer-readable storage media collectively contain instructions executed by the processor via the memory to cause the processor to perform operations. The operations include obtaining a video having a sequence of frames, obtaining multiple machine learning models (MLMs) trained to perform next frame prediction, and inputting a subset of the frame sequence into each of the multiple MLMs. The operations also include calculating a score for each of the multiple MLMs for the subset of the frame sequence, and using the MLM with the highest score among the multiple MLMs to generate the next frame, wherein the next frame is the frame immediately following the subset of the frame sequence.

[0026] As mentioned above, several next-frame prediction models have recently been developed, each designed to generate predicted frames for a specific type of video. Consequently, the accuracy of the predicted next frame can vary depending on the video type and the next-frame prediction model used. In practice, users may not know the optimal next-frame prediction model for a particular video. Furthermore, for videos containing multiple content types, different parts of the video may be best suited for different next-frame prediction models.

[0027] In exemplary embodiments, systems, methods, and computer program products are provided for using integration to predict the next frame of a video. In exemplary embodiments, multiple next-frame prediction models for an input video are continuously evaluated to sequentially identify the model best suited from the multiple models for performing next-frame prediction on a given sequence of video frames. In exemplary embodiments, identifying a model from the multiple models includes calculating a score for the frame sequence based on a comparison of frames generated by each model with ground truth frames (i.e., actual frames in the frame sequence). Once the best model for the frame sequence is identified, the identified model is used to generate the predicted next frame for the frame sequence. In exemplary embodiments, the video's metadata file may be updated to include an identifier of the identified machine learning model best suited for generating each frame in the video.

[0028] Various aspects of this disclosure are described by way of text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of a computer program product (CPP). With respect to any flowchart, depending on the technology involved, operations may be performed in a different order than that shown in a given flowchart. For example, again depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that at least partially overlaps in time.

[0029] Computer Program Product Embodiment (“CPP Embodiment” or “CPP”) is a term used in this disclosure to describe any set of one or more storage media (also referred to as “media”) collectively included in a set of one or more storage devices that collectively include machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A “storage device” is any tangible device capable of holding and storing instructions used by a computer processor. Without limitation, a computer-readable storage medium can be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include: magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD), memory sticks, floppy disks, mechanical encoding devices (e.g., punched cards or pits / platforms formed in the main surface of the disk), or any suitable combination of the foregoing. As used herein, the term computer-readable storage medium should not be construed as storage in the form of transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses through fiber optic cables, electrical signals transmitted through wires, and / or other transmission media. As those skilled in the art will understand, data typically moves at certain incidental points in time during the normal operation of the storage device (such as during access, defragmentation, or garbage collection), but this does not render the storage device transient, as the data is not transient at the time of storage.

[0030] Computing environment 100 includes examples of environments for executing at least some computer code related to performing the methods of the present invention, such as using integration to predict the next frame of a video (box 150). In addition to block 150, computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end-user equipment (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, computer 101 includes a processor set 110 (including processing circuitry 120 and cache 121), communication infrastructure 111, volatile memory 112, persistent storage device 113 (including an operating system 122 and block 150, as described above), a peripheral device set 114 (including a user interface (UI), a device set 123, storage device 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. Remote server 104 includes a remote database 132. The public cloud 105 includes a gateway 130, a cloud orchestration module 131, a host physical machine set 142, a virtual machine set 143, and a container set 144.

[0031] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device now known or to be developed in the future capable of running programs, accessing networks, or querying databases (such as remote database 132). As is well understood in the field of computer technology, and depending on the technology, the execution of the computer-implemented method may be distributed among multiple computers and / or multiple locations. On the other hand, in this presentation of computing environment 100, the detailed discussion focuses on a single computer, specifically computer 101, to keep the presentation as simple as possible. Computer 101 may be located in the cloud, even if it is... Figure 1 The computer 101 is not shown in the cloud. On the other hand, except to any extent that can be definitively indicated, the computer 101 is not required to be in the cloud.

[0032] Processor set 110 includes one or more computer processors of any type now known or to be developed in the future. Processing circuitry 120 may be distributed across multiple packages, such as multiple coordinated integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is memory located within the processor chip package(s) and is typically used for data or code that should be readily accessible by the threads or cores running on processor set 110. Cache memory is typically organized into multiple levels based on its relative proximity to the processing circuitry. Alternatively, some or all of the cache for the processor set may be located “off-chip.” In some computing environments, processor set 110 may be designed to work with qubits and perform quantum computing.

[0033] Computer-readable program instructions are typically loaded onto computer 101 to cause the processor set 110 of computer 101 to perform a series of operational steps to implement a computer-implemented method, such that the instructions thus executed instantiate the method specified in the flowcharts and / or descriptive descriptions of the computer-implemented method included in this document (collectively, the “method of the invention”). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the method of the invention. In computing environment 100, at least some of the instructions for performing the method of the invention may be stored in persistent storage 113 in block 150.

[0034] Communication structure 111 is a signal transmission path that allows various components of computer 101 to communicate with each other. Typically, this structure consists of switches and conductive paths, such as switches and conductive paths forming buses, bridges, physical input / output ports, etc. Other types of signal communication paths can be used, such as fiber optic communication paths and / or wireless communication paths.

[0035] Volatile memory 112 is any type of volatile memory now known or to be developed in the future. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory is characterized by random access, but this is not necessary unless explicitly indicated. In computer 101, volatile memory 112 is located in a single package and is internal to computer 101; however, alternatively or additionally, volatile memory may be distributed across multiple packages and / or located externally relative to computer 101.

[0036] Persistent storage device 113 is any form of non-volatile storage device for a computer, now known or to be developed in the future. The non-volatility of this storage device means that the stored data is maintained regardless of whether power is supplied to the computer 101 and / or directly to the persistent storage device 113. Persistent storage device 113 may be a read-only memory (ROM), but typically, at least a portion of the persistent storage device allows data to be written, deleted, and rewritten. Some common forms of persistent storage include hard disks and solid-state storage devices. Operating system 122 may take several forms, such as various known proprietary operating systems or open-source portable operating system interface type operating systems employing a kernel. The code included in box 150 generally includes at least some of the computer code involved in performing the methods of the present invention.

[0037] Peripheral device set 114 includes a collection of peripheral devices for computer 101. Data communication connections between peripheral devices and other components of computer 101 can be implemented in various ways, such as Bluetooth connectivity, near field communication (NFC) connectivity, connections via cables (such as Universal Serial Bus (USB) type cables), plug-in connections (e.g., Secure Digital (SD) cards), connections via local area communication networks, and even connections via wide area networks such as the Internet. In various embodiments, UI device set 123 may include components such as displays, speakers, microphones, wearable devices (e.g., goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Storage device 124 is an external storage device (such as an external hard drive) or a pluggable storage device (such as an SD card). Storage device 124 may be persistent and / or volatile. In some embodiments, storage device 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments requiring computer 101 to have substantial storage (e.g., where computer 101 locally stores and manages a large database), this storage can be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that can be used in IoT applications. For example, one sensor could be a thermometer, while another could be a motion detector.

[0038] Network module 115 is a collection of computer software, hardware, and firmware that allows computer 101 to communicate with other computers via WAN 102. Network module 115 may include hardware such as a modem or Wi-Fi transceiver, software for packetizing and / or unpacking data for transmission over a communication network, and / or web browser software for transmitting data over the Internet. In some embodiments, the network control and network forwarding functions of network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN), the control and forwarding functions of network module 115 are performed on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for performing the methods of the present invention can typically be downloaded to computer 101 from an external computer or external storage device via a network adapter card or network interface included in network module 115.

[0039] WAN 102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances using any technology now known or to be developed in the future for transmitting computer data. In some embodiments, a WAN may be replaced by and / or supplemented by a local area network (LAN) designed to transmit data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0040] End User Equipment (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of the enterprise operating computer 101) and can take any of the forms discussed above in connection with computer 101. EUD 103 typically receives helpful and useful data from the operation of computer 101. For example, assuming computer 101 is designed to provide recommendations to the end user, these recommendations are typically transmitted from network module 115 of computer 101 to EUD 103 via WAN 102. In this way, EUD 103 can display or otherwise present the recommendations to the end user. In some embodiments, EUD 103 can be a client device, such as a thin client, a heavy client, a mainframe computer, a desktop computer, etc.

[0041] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 can be controlled and used by the same entity operating computer 101. Remote server 104 represents (multiple) machines that collect and store helpful and useful data for use by other computers, such as computer 101. For example, if computer 101 is designed and programmed to provide recommendations based on historical data, that historical data can be provided to computer 101 from a remote database 132 of remote server 104.

[0042] Public cloud 105 is any computer system available to multiple entities, providing on-demand availability of computer system resources and / or other computing capabilities, particularly data storage (cloud storage) and computing power, without direct active management by users. Cloud computing typically leverages resource sharing to achieve consistency and economies of scale. Direct and active management of the computing resources of public cloud 105 is performed by the computer hardware and / or software of cloud orchestration module 131. The computing resources provided by public cloud 105 are typically implemented by virtual computing environments running on various computers constituting host physical machine group 142, which is a universe of physical computers in and / or available to public cloud 105. Virtual computing environments typically take the form of virtual machines from virtual machine set 143 and / or containers from container set 144. It should be understood that these VCEs can be stored as images and can be transferred between various physical machine hosts as images or after the VCEs are instantiated. Cloud orchestration module 131 manages the transfer and storage of images, deploys new instantiations of VCEs, and manages active instantiation of VCE deployments. Gateway 130 is a collection of computer software, hardware, and firmware that allow public cloud 105 to communicate via WAN 102.

[0043] Now, we will provide some further explanation of Virtualized Computing Environments (VCEs). A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from an image. Two familiar types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature where the kernel allows multiple isolated user-space instances, called containers, to exist. From the perspective of the programs running within them, these isolated user-space instances typically behave like real computers. Computer programs running on a regular operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running within a container can only use the contents of the container and the devices allocated to the container; this is a feature known as containerization.

[0044] Private cloud 106 is similar to public cloud 105, except that computing resources are only available for use by a single enterprise. While private cloud 106 is depicted communicating with WAN 102, in other embodiments, private cloud may be completely disconnected from the internet and accessible only via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), typically implemented by different vendors. Each of the multiple clouds remains a separate and discrete entity, but the larger hybrid cloud architecture is bound together by standardization or proprietary technology that enables orchestration, management, and / or data / application portability across the multiple component clouds. In this embodiment, public cloud 105 and private cloud 106 are both part of a larger hybrid cloud.

[0045] For example, one or more embodiments described herein can utilize machine learning techniques to perform prediction and / or classification tasks. In one or more embodiments, machine learning functions can be implemented using artificial neural networks (ANNs) that have the ability to be trained to perform functions. In machine learning and cognitive science, ANNs are a family of statistical learning models inspired by the biological neural networks of animals, particularly the brain. ANNs can be used to estimate or approximate systems and functions that depend on a large number of inputs. Convolutional neural networks (CNNs) are a class of deep feedforward ANNs that are particularly useful in tasks such as, but not limited to, analyzing visual images and natural language processing (NLP). Recurrent neural networks (RNNs) are another class of deep feedforward ANNs and are particularly useful in tasks such as, but not limited to, handwriting recognition and speech recognition without segmented connections. Other types of neural networks are also known and can be used according to one or more embodiments described herein.

[0046] An ANN can be embodied as a "neuromorphic" system of interconnected processor elements that act as analog "neurons" and exchange "messages" with each other in the form of electronic signals. Similar to the so-called "plasticity" of synaptic neurotransmitter connections that carry messages between biological neurons, the connections in an ANN that carry electronic messages between analog neurons are provided with numerical weights corresponding to the strength of a given connection. These weights can be adjusted and tuned empirically, allowing the ANN to adapt to the input and learn. For example, an ANN for handwriting recognition is defined by a set of input neurons that can be activated by pixels of an input image. After being weighted and transformed by a function determined by the network designer, the activations of these input neurons are then passed to other downstream neurons, often referred to as "hidden" neurons. This process is repeated until an output neuron is activated. The activated output neuron determines which character was input.

[0047] Containers are Virtual CEs (VCEs) that utilize operating system-level virtualization. This refers to an operating system feature where the kernel allows multiple isolated user-space instances, called containers, to exist. From the perspective of a program running within one of these instances, they typically behave like a real computer. A computer program running on a regular operating system can utilize all of that computer's resources, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running within a container can only use the contents of the container and the devices allocated to it; this is a characteristic known as containerization.

[0048] Now for reference Figure 2 The diagram illustrates a block diagram of the components of a machine learning training and inference system 200. According to one or more embodiments of the invention, the machine learning training and inference system 200 can utilize machine learning techniques to perform tasks, such as generating a prediction of the next frame for a video. Embodiments of the invention utilize AI, which includes various so-called machine learning techniques. The phrase "machine learning" broadly describes the function of an electronic system that learns from data. Machine learning systems, engines, or modules can include trainable machine learning algorithms, which can be trained, for example, in an external cloud environment, to learn functional relationships between inputs and outputs, and the resulting model (sometimes referred to as a "trained neural network," a "trained model," and / or a "trained machine learning model") can be used to manage information. In one or more embodiments of the invention, machine learning functions can be implemented using artificial neural networks (ANNs) with the ability to be trained to perform functions. In machine learning and cognitive science, ANNs are a family of statistical learning models inspired by the biological neural networks of animals, particularly the brain. ANNs can be used to estimate or approximate systems and functions that depend on a large number of inputs. Convolutional neural networks (CNNs) are a class of deep feedforward ANNs that are particularly useful in tasks such as, but not limited to, analyzing visual images and natural language processing (NLP). Recurrent neural networks (RNNs) are another class of deep feedforward ANNs and are particularly useful in tasks such as, but not limited to, handwriting recognition and speech recognition without segmented connections. Other types of neural networks are also known and can be used according to one or more embodiments of the invention described herein.

[0049] An ANN can be embodied as a "neuromorphic" system of interconnected processor elements that act as analog "neurons" and exchange "messages" with each other in the form of electronic signals. Similar to the so-called "plasticity" of synaptic neurotransmitter connections that carry messages between biological neurons, the connections in an ANN that carry electronic messages between analog neurons are provided with numerical weights corresponding to the strength of a given connection. These weights can be empirically adjusted and tuned, allowing the ANN to adapt to inputs and learn. For example, an ANN for handwriting recognition is defined by a set of input neurons that can be activated by pixels of an input image. After being weighted and transformed by a function determined by the network designer, the activations of these input neurons are then passed to other downstream neurons, often referred to as "hidden" neurons. This process is repeated until an output neuron is activated. The activated output neuron determines which character was input. It should be understood that these same techniques can be applied to locating a target object from a set of images with similar visual elements as described herein, represented by a combination of expressions.

[0050] Generative Adversarial Networks (GANs) are an exciting new innovation in machine learning and typically employ generative modeling methods that leverage deep learning approaches such as CNNs. A GAN is a clever way to train generative models by constructing the problem as a supervised learning problem with two sub-models: a generator model trained to generate new examples and a discriminator model that attempts to classify examples as real (from the domain) or fake (generated). These two models are trained together in a zero-sum adversarial game until the discriminator model is fooled approximately half the time, meaning the generator model is generating seemingly plausible examples.

[0051] Machine learning training and inference system 200 performs training 202 and inference 204. During training 202, training engine 216 trains a model (e.g., trained model 218) to perform a task. Inference 204 is the process of implementing trained model 218 to perform a task in the context of a larger system (e.g., system 226).

[0052] Training 202 begins with training data 212, which can be structured or unstructured data. Training engine 216 receives training data 212 and model form 214. Model form 214 represents the untrained base model. Model form 214 can have preset weights and biases, which can be adjusted during training. It should be understood that model form 214 can be selected from many different model forms depending on the task to be performed. For example, in the case where training 202 is training a model to perform image classification, model form 214 can be a CNN (Convolutional Neural Network) model form. Training 202 can be supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, etc., including combinations and / or multiple thereof. For example, supervised learning can be used to train a machine learning model to classify objects of interest in images. For this purpose, training data 212 includes labeled images, including images of objects of interest with associated labels (ground truth) and other images excluding objects of interest with associated labels. In this example, training engine 216 takes training images from training data 212 as input, makes a prediction for classifying the images, and compares that prediction with known labels. Training engine 216 then adjusts the model's weights and / or biases based on the comparison results (e.g., by using backpropagation). Training 202 can be performed multiple times (referred to as "epochs") until a suitable model is trained (e.g., training model 218).

[0053] Once trained, the trained model 218 can be used to perform inference 204 to perform a task. The inference engine 220 applies the trained model 218 to new data 222 (e.g., real-world, non-training data). For example, if the trained model 218 is trained to classify images of specific objects such as chairs, the new data 222 could be images of chairs that are not part of the training data 212. In this way, the new data 222 represents data that the model 218 has not yet been exposed to. The inference engine 220 makes a prediction 224 (e.g., classification of objects in the images of the new data 222) and passes the prediction 224 to the system 226. The system 226 can take actions, perform operations, perform analyses, etc., based on the prediction 224, including combinations and / or multiple such actions. In some embodiments of the invention, the system 226 can add to and / or modify the new data 222 based on the prediction 224.

[0054] According to one or more embodiments of the present invention, the predictions 224 generated by the inference engine 220 are periodically monitored and validated to ensure that the inference engine 220 operates as expected. Based on the validation, additional training 202 can be performed using the trained model 218 as a starting point. Additional training 202 may include all or a subset of the original training data 212 and / or new training data 212. According to one or more embodiments of the present invention, training 202 includes updating the trained model 218 to account for changes in the expected input data.

[0055] Video prediction is the task of predicting future frames given past video frames and is an integral part of the decompression process used for video playback. The ability to predict, anticipate, and reason about future outcomes is a key component of intelligent decision-making systems. Given the success of deep learning in computer vision, deep learning-based video prediction has become a promising research direction. Video prediction, defined as a self-supervised learning task, represents a suitable framework for learning because it demonstrates the potential to extract meaningful representations of underlying patterns in natural videos. In many cases, next frame prediction can be viewed as a spatiotemporal problem. That is, given a sequence of images in continuous time steps, the ability to predict the next frame of a video through time series learning.

[0056] Next-frame prediction architectures can be broadly categorized into two main types: sequence-to-one (I / P) architectures and sequence-to-multiple (S / P) architectures. In a sequence-to-one architecture, the sequence of previous frames (i.e., I / P: frame t; O / P: frame t+1) is used to predict the immediately following frame. In a sequence-to-multiple architecture, the sequence of previous frames (i.e., I / P: frame t; O / P: frame t+k) is used to predict the next few frame sequences. Currently, it can be seen that multiple network structures are used to construct next-frame prediction models. Autoencoders and generative adversarial networks (GANs) are widely used to generate entirely new videos (e.g., in video synthesis) or to predict and generate the next frame given the current and previous frames of a video (e.g., in video prediction).

[0057] Now for reference Figure 3 The illustration shows a schematic diagram of training multiple machine learning models for performing next-frame prediction according to one or more embodiments of the present invention. As shown, multiple trained machine learning models 302-1, 302-2, 302-3, and 302-4 (collectively referred to as trained machine learning models 302) are trained using a set of training data 310 to perform next-frame prediction. In exemplary embodiments, such as Figure 2The training system 202 shown is used to train each trained machine learning model 302. In an exemplary embodiment, each of the trained machine learning models 302-1, 302-2, 302-3, and 302-4 has a corresponding accuracy 304-1, 304-2, 304-3, and 304-4, collectively referred to as accuracy 304. In an exemplary embodiment, after training the machine learning model based on a standardized set of input videos, the accuracy 304 of the trained machine learning model is calculated. In an exemplary embodiment, the trained machine learning model includes a Long Short-Term Memory (LSTM) model, a Convolutional Neural Network (CNN) model, and a Gated Recurrent Unit (GRU) model. The trained machine learning model may also include a model created by combining one or more of the LSTM, CNN, and GRU models.

[0058] Now for reference Figure 4 The diagram illustrates a block diagram of a system 400 for predicting the next frame of a video using integration, according to one or more embodiments of the present invention. As shown, the system 400 includes a computing system 410 configured to receive a frame sequence 422 of a video 420 and, in response, generate a predicted next frame 424, wherein the predicted next frame 424 is a frame immediately following the frame sequence 422.

[0059] In an exemplary embodiment, the computing system 410 is embodied in the computer 101, such as Figure 1 The computer shown. The computing system 410 includes multiple trained machine learning models 412, each trained to generate a next-frame prediction based on an input sequence of video frames. Each trained machine learning model 412 has a corresponding accuracy 414, reflecting the computational accuracy of the machine learning model 412 for a predefined set of video frames. The computing system 410 also includes a model scoring engine 416 configured to calculate a score for each trained machine learning model 412, reflecting the accuracy of the trained machine learning model 412 for a given frame sequence 422.

[0060] In an exemplary embodiment, the model scoring engine 416 is configured to analyze the performance of each trained machine learning model 412 for frame sequence 422 or a subset of frame sequence 422. In one embodiment, the model scoring engine 416 computes an individual loss value for one or more frames in frame sequence 422 by comparing ground truth frames from frame sequence 422 with corresponding predicted frames created using the trained machine learning model 412.

[0061] In one example, the frame sequence comprises five consecutive frames (frame 1, frame 2, frame 3, frame 4, and frame 5), and the model scoring engine 416 is configured to compute an individual loss value corresponding to each trained machine learning model 412 for frames 2, 3, 4, and 5. The individual loss value of a frame is calculated based on the similarity between the predicted frame generated using each trained machine learning model 412 and the corresponding ground truth frame. In one example, if the similarity between two frames is 95%, the individual loss value will be 5%. For example, the first trained machine learning model 412 (model A) is configured to generate predicted frames 2_A, 3_A, 4_A, and 5_A, which are compared with frames 2, 3, 4, and 5, respectively, to compute the individual loss value for each of frames 2 through 5 of model A.

[0062] In an exemplary embodiment, the model scoring engine 416 is configured to calculate the overall loss value for each trained machine learning model 412 based on the individual loss values ​​of the frame sequences 422 of the corresponding trained machine learning model 412. In one embodiment, the overall loss value for each trained machine learning model 412 is the sum of the individual loss values ​​of the corresponding trained machine learning model 412's frame sequences 422. In another embodiment, the overall loss value for each trained machine learning model 412 is the average of the individual loss values ​​of the corresponding trained machine learning model 412's frame sequences 422. In yet another embodiment, the overall loss value for each trained machine learning model 412 is a weighted average of the individual loss values ​​for the sequence of frames 422 of the corresponding trained machine learning model 412, wherein newer frames are given higher weights.

[0063] In an exemplary embodiment, the model scoring engine 416 is configured to generate a score for each trained machine learning model 412 that reflects the accuracy of each trained machine learning model 412 in predicting the next frame of the provided frame sequence 422 of the video 420. In one embodiment, the score of the trained machine learning model 412 is calculated by multiplying the model accuracy 414 corresponding to the trained machine learning model 412 by the total loss value calculated for each trained machine learning model 412.

[0064] In an exemplary embodiment, once the model scoring engine 416 is configured to identify the trained model with the highest score from a plurality of trained machine learning models 412, the computing system 410 is configured to utilize the identified trained machine learning model to generate a predicted next frame 424 of video 420 based on the identification of the trained model obtained from the model scoring engine 416. In one embodiment, video 420 may include metadata including the identified machine learning model 426 best suited for generating a predicted next frame for each frame.

[0065] In an exemplary embodiment, the process of evaluating the best model for the frame sequence 422 across video 420 is performed continuously. For example, the video may include one hundred frames, and these frames are continuously evaluated to identify the most suitable machine learning model for generating predictions for the next frame for each frame. In one example, a first model (model A) may be identified as the best model for generating the next frames for frames 1-20 and 78-100, a second model (model B) may be identified as the best model for generating the next frames for frames 21-50, and a third model (model C) may be identified as the best model for generating the next frames for frames 51-77. This data may be stored in the metadata of video 420. As a result, when the processing system is playing back video 420 and frames are corrupted or lost (accidentally or intentionally), the processing system will be able to identify the best machine learning model for generating frames.

[0066] Now for reference Figure 5 The diagram illustrates a flowchart of a computer-implemented method 500 for predicting the next frame of a video using integration, according to one or more embodiments of the present invention. In an exemplary embodiment, method 500 is performed by a computing system 410 (such as...). Figure 4 The computational system shown in the diagram performs the operation. At box 502, method 500 begins by obtaining a video with a sequence of frames. Next, method 500 includes obtaining multiple machine learning models (MLMs) trained to perform predictions for the next frame, as shown in box 504. In an exemplary embodiment, the multiple machine learning models include a Long Short-Term Memory (LSTM) model, a Convolutional Neural Network (CNN) model, and a Gated Recurrent Unit (GRU) model. In one embodiment, the multiple machine learning models include a combination of two or more of the LSTM, CNN, and GRU models.

[0067] At box 506, method 500 includes inputting a subset of the frame sequence into each of a plurality of MLMs. In one example, the subset includes a predetermined number of frames in the frame sequence. For example, when the video comprises hundreds of frames, the subset may include only ten or fifteen frames of the video. The subset of frames is the frame immediately preceding the desired frame to be predicted. For example, the subset of frames in a video having frames 1-100 could be frames 51-61 for the desired prediction of frame 62.

[0068] At block 508, method 500 includes computing a score for each of a plurality of trained machine learning models for a subset of the frame sequence. In an exemplary embodiment, the computed score of each trained machine learning model reflects how well the trained machine learning model performed in generating frames corresponding to frames provided in the subset of the frame sequence. Figure 6 This includes a more detailed discussion of methods for calculating the score of each of the multiple trained machine learning models.

[0069] At box 510, method 500 includes using the MLM with the highest score among a plurality of MLMs to generate the next frame, where the next frame is the frame immediately following a subset of the frame sequence. Continuing the example above, the next frame would be frame 62.

[0070] Now for reference Figure 6 The diagram illustrates a flowchart of a computer-implemented method 600 for calculating a score of a machine learning model trained for predicting the next frame of a video, according to one or more embodiments of the present invention. In an exemplary embodiment, method 600 is comprised of a model scoring engine 416 (such as...). Figure 4 The method (as shown in the model scoring engine) is executed. At box 602, method 600 includes obtaining the accuracy of the trained machine learning model. Next, as shown in box 604, method 600 includes calculating a frame loss value for each frame in the frame sequence by comparing ground truth frames obtained from the frame sequence with corresponding predicted frames generated by the trained machine learning model. Method 600 also includes calculating a total loss value for the trained machine learning model based on the frame loss values ​​for each subset of the frame sequence corresponding to the MLM, as shown in box 606. In various embodiments, the total loss value may be the average of the frame loss values, the sum of the frame loss values, a weighted average of the frame loss values, etc. At box 608, method 600 includes determining a score for the trained machine learning model by multiplying the total loss value of the trained machine learning model by the accuracy of the trained machine learning model. In one embodiment, the score for the trained machine learning model is determined by multiplying the total loss value of the trained machine learning model by the accuracy corresponding to the trained machine learning model.

[0071] In an exemplary embodiment, once the best-fit machine learning model for training each frame of the generated video is determined, this information can be included in the video's metadata. In some cases, this information can be used to select frames that can be intentionally discarded from the video to reduce its file size.

[0072] Various embodiments of the invention are described herein with reference to the accompanying drawings. Alternative embodiments of the invention can be devised without departing from its scope. Various connections and positional relationships (e.g., above, below, adjacent, etc.) between elements are illustrated in the following description and drawings. Unless otherwise stated, these connections and / or positional relationships can be direct or indirect, and the invention is not intended to be limiting in this respect. Therefore, coupling of entities can refer to direct or indirect coupling, and positional relationships between entities can be direct or indirect positional relationships. Furthermore, the various tasks and process steps described herein can be incorporated into a more comprehensive program or process with additional steps or functions not described in detail herein.

[0073] The following definitions and abbreviations are used to interpret the claims and specification. As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” “contains,” or “containing,” or any other variation thereof, are intended to cover non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or apparatus.

[0074] Additionally, the term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms "at least one" and "one or more" can be understood to include any integer greater than or equal to one, i.e., one, two, three, four, etc. The term "multiple" can be understood to include any integer greater than or equal to two, i.e., two, three, four, five, etc. The term "connection" can include both indirect "connection" and direct "connection."

[0075] The terms “approximately,” “substantially,” “approximately,” and their variations are intended to include the degree of error associated with a measurement based on a specific quantity of equipment available at the time of application submission. For example, “approximately” could include a range of ±8%, 5%, or 2% of a given value.

[0076] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to techniques found in the market, or to enable those skilled in the art to understand the embodiments described herein.

Claims

1. A computer-implemented method for predicting the next frame of a video, the computer-implemented method comprising: Obtain the video having a frame sequence; Obtain multiple machine learning models (MLMs) trained to perform predictions for the next frame; A subset of the frame sequence is input into each of the plurality of MLMs; For the subset of the frame sequence, calculate the score for each of the plurality of MLMs; as well as The next frame is generated using the MLM with the highest score among the plurality of MLMs, wherein the next frame is the frame immediately following the subset of the frame sequence.

2. The computer-implemented method according to claim 1, wherein, Each of the plurality of MLMs has accuracy.

3. The computer-implemented method according to claim 2, wherein, Calculating the score for each of the plurality of MLMs includes: calculating the frame loss value for each of the subsets of the frame sequence by comparing ground truth frames from the subset of the frame sequence with predicted frames generated by the corresponding MLMs among the plurality of MLMs.

4. The computer-implemented method according to claim 3, wherein, Calculating the score for each of the plurality of MLMs further includes: calculating the total loss value for each of the plurality of MLMs based on the frame loss value of each of the subsets of the frame sequences corresponding to the MLMs among the plurality of MLMs.

5. The computer-implemented method according to claim 3 or 4, wherein, Calculating the score for each of the plurality of MLMs further includes: calculating the total loss value for each of the plurality of MLMs by summing the frame loss values ​​for each of the subsets of the frame sequences corresponding to the MLMs in the plurality of MLMs.

6. The computer-implemented method according to claim 4 or 5, wherein, Calculating the score for each of the plurality of MLMs further includes multiplying the total loss value corresponding to the MLM among the plurality of MLMs by the accuracy value corresponding to the MLM among the plurality of MLMs.

7. The computer-implemented method according to any one of the preceding claims, wherein, The multiple MLMs include Long Short-Term Memory (LSTM) models, Convolutional Neural Network (CNN) models, and Gated Recurrent Unit (GRU) models.

8. The computer-implemented method according to any one of the preceding claims, updating the metadata of the video to include an indication of the MLM among the plurality of MLMs having the highest score for predicting the next frame.

9. A computer program product having one or more computer-readable storage media, the one or more computer-readable storage media having computer-readable program code commonly stored on the one or more computer-readable storage media, the computer-readable program code being executed by a processor of a computer system to cause the computer system to perform operations, the operations including: Obtain a video with a frame sequence; Obtain multiple machine learning models (MLMs) trained to perform predictions for the next frame; A subset of the frame sequence is input into each of the plurality of MLMs; For the subset of the frame sequence, calculate the score for each of the plurality of MLMs; as well as The next frame is generated using the MLM with the highest score among the plurality of MLMs, wherein the next frame is the frame immediately following the subset of the frame sequence.

10. The computer program product according to claim 9, wherein, Each of the plurality of MLMs has accuracy.

11. The computer program product according to claim 10, wherein, Calculating the score for each of the plurality of MLMs includes: calculating the frame loss value for each of the subsets of the frame sequence by comparing ground truth frames from the subset of the frame sequence with predicted frames generated by the corresponding MLMs among the plurality of MLMs.

12. The computer program product according to claim 11, wherein, Calculating the score for each of the plurality of MLMs further includes: calculating the total loss value for each of the plurality of MLMs based on the frame loss value of each of the subsets of the frame sequences corresponding to the MLMs among the plurality of MLMs.

13. The computer program product according to claim 11 or claim 12, wherein, Calculating the score for each of the plurality of MLMs further includes: calculating the total loss value for each of the plurality of MLMs by summing the frame loss values ​​for each of the subsets of the frame sequences corresponding to the MLMs in the plurality of MLMs.

14. The computer program product according to claim 12 or 13, wherein, Calculating the score for each of the plurality of MLMs further includes multiplying the total loss value corresponding to the MLM among the plurality of MLMs by the accuracy value corresponding to the MLM among the plurality of MLMs.

15. The computer program product according to any one of claims 9 to 14, wherein, The multiple MLMs include Long Short-Term Memory (LSTM) models, Convolutional Neural Network (CNN) models, and Gated Recurrent Unit (GRU) models.

16. The computer program product according to any one of claims 9 to 15, updating the metadata of the video to include an indication of the MLM among the plurality of MLMs having the highest score for predicting the next frame.

17. A computing system, comprising: processor; Memory coupled to the processor; as well as One or more computer-readable storage media coupled to the processor, the one or more computer-readable storage media collectively containing instructions that are executed by the processor via the memory to cause the processor to perform operations, the operations including: Obtain a video with a frame sequence; Obtain multiple machine learning models (MLMs) trained to perform predictions for the next frame; A subset of the frame sequence is input into each of the plurality of MLMs; For the subset of the frame sequence, calculate the score for each of the plurality of MLMs; and The next frame is generated using the MLM with the highest score among the plurality of MLMs, wherein the next frame is the frame immediately following the subset of the frame sequence.

18. The computing system according to claim 17, wherein, Each of the plurality of MLMs has accuracy.

19. The computing system according to claim 18, wherein, Calculating the score for each of the plurality of MLMs includes: calculating the frame loss value for each of the subsets of the frame sequence by comparing ground truth frames from the subset of the frame sequence with predicted frames generated by the corresponding MLMs among the plurality of MLMs.

20. The computing system according to claim 19, wherein, Calculating the score for each of the plurality of MLMs further includes: calculating the total loss value for each of the plurality of MLMs based on the frame loss value of each of the subsets of the frame sequences corresponding to the MLMs among the plurality of MLMs.

Images