Using Interpolation to Generate Video from Still Images
The method improves video generation from still images by using filters and machine learning models to select and interpolate frames, addressing inefficiencies in existing technologies.
Patent Information
- Application Number
- JP2023528278
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-05-18
- Filing Date
- 2021-12-30
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2041-12-30
AI Technical Summary
Existing methods for generating videos from still images lack efficiency and precision in selecting suitable image pairs and interpolating frames, leading to suboptimal video creation from large collections.
A method involving a filter that selects candidate image pairs based on temporal, motion, quality, and semantic criteria, using machine learning models to generate intermediate images and create videos, incorporating depth prediction and camera effects.
Enhances the quality and efficiency of video generation from still images by selecting relevant pairs and interpolating frames effectively, balancing speed and precision in the process.
Smart Images

Figure 0007595163000001 
Figure 0007595163000002 
Figure 0007595163000003
Abstract
Description
[Technical field]
[0001] REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 190,234, filed May 18, 2021, and entitled “USE OF INTERPOLATION TO GENERATE VIDEO FROM STILL IMAGES,” which is incorporated herein in its entirety. [Background technology]
[0002] background Users of devices such as smartphones or other digital cameras take many pictures and videos and store them in an image library. Users utilize such libraries to browse their photos and videos to reminisce about various events such as birthdays, weddings, vacations, trips, etc. Users may have large image libraries that contain thousands of images taken over an extended period of time.
[0003] The discussion of the background art provided herein is intended to provide a general context for the present disclosure. To the extent that it is described in this background section, the work of the presently named inventors is not expressly or impliedly admitted as prior art to the present disclosure, as well as descriptions that do not qualify as prior art at the time of filing. Summary of the Invention [Means for solving the problem]
[0004] overview The method includes selecting candidate image pairs from an image collection associated with the user account, each pair including a first still image and a second still image from the user account. The method further includes applying a filter to select a particular image pair from the candidate image pairs. The method further includes generating one or more intermediate images based on the particular image pair with an image interpolator. The method includes generating one or more intermediate images based on the particular image pair with an image interpolator and generating a video including three or more frames arranged in a sequence, a first frame of the sequence is a first still image and a last frame of the sequence is a second still image, and each image of the one or more intermediate images is a corresponding intermediate frame of the sequence between the first frame and the last frame.
[0005] In some embodiments, the filter includes a time filter that excludes one or more of the candidate image pairs if a time difference between respective timestamps associated with a first still image of the pair and a second still image of the pair is greater than a time threshold. In some embodiments, each of the one or more intermediate images is associated with a respective timestamp having a value between the timestamp of the first still image and the timestamp of the second still image, and the position of each intermediate image in the sequence is based on the respective timestamp. In some embodiments, the time threshold is 2 seconds. In some embodiments, the filter includes a motion filter that excludes one or more of the candidate image pairs by estimating motion between the first still image and the second still image and determining that the motion between the first still image and the second still image is less than a minimum motion threshold. In some embodiments, the filter further excludes one or more of the candidate image pairs by determining that the motion between the first still image and the second still image is greater than a maximum motion threshold. In some embodiments, the filter includes a filter machine learning module that excludes one or more of the candidate image pairs by generating feature vectors representing a first still image and a second still image in each of the candidate image pairs and eliminating one or more of the candidate image pairs corresponding to each of the feature vectors if a distance between each of the feature vectors is greater than a threshold vector distance, the feature vectors being mathematical representations, the mathematical representations of the similar images being closer in vector space than the mathematical representations of the dissimilar images. In some embodiments, the feature vectors are first feature vectors and the filter machine learning module is further operable to perform the following operations, the operations including receiving an intermediate image of the one or more intermediate images as an input, generating one or more second feature vectors corresponding to the intermediate images, and eliminating one or more intermediate images corresponding to each of the feature vectors if a distance from each of the feature vectors of the intermediate images to each of the feature vectors of the corresponding candidate image pair is greater than a threshold vector distance.In some embodiments, the image interpolator includes an interpolation machine learning model that receives a first still image and a second still image as input and generates one or more intermediate images. In some embodiments, generating one or more intermediate images based on the particular image pair includes generating a plurality of candidate intermediate images and evaluating each candidate intermediate image, where evaluating each candidate intermediate image is performed by generating a candidate video including the first still image as a first frame, the candidate intermediate image as a second frame, and the second still image as a third frame, and selecting the candidate intermediate image as one of the one or more intermediate images if the candidate video does not include a frame interpolation failure. In some embodiments, the method further includes determining the frame interpolation failure using a classifier machine learning model trained to determine whether the input image is a generated image, and the candidate intermediate image is selected if the classifier machine learning model determines that the candidate intermediate image is indistinguishable from the generated image. In some embodiments, the method further includes determining that a frame interpolation failure occurs if the filter excludes one or more intermediate images, and generating an additional one or more intermediate images in response to the occurrence of the frame interpolation failure. In some embodiments, generating the video includes generating a three-dimensional representation of a scene in the first still image based on a depth prediction of the first still image using a depth machine learning model, where the depth machine learning model is a classifier that receives the first still image as an input, and the video includes camera effects generated based on the three-dimensional representation of the scene. In some embodiments, the camera effects include at least one of zooming, panning, or rotation.
[0006] This specification advantageously describes a method for creating a video from a pair of images by synthesizing the motion between the pair of images and filling the gaps with newly generated frames. This specification describes a media application for eliminating candidate image pairs using filters, generating intermediate images by performing frame interpolation of a particular image pair, and generating a video from the particular image pair and the intermediate images. This specification advantageously describes a method for balancing a set of filters with a required high reproducibility and faster computation time with a set of filters with a required high accuracy and slower computation time. [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 is a block diagram illustrating an example network environment in accordance with certain embodiments described herein. [Diagram 2] 1 is a block diagram illustrating an exemplary computing device in accordance with some embodiments described herein. [Diagram 3] 2 is a block diagram illustrating in detail a filtering module and an image interpolator in accordance with certain embodiments described herein. [Figure 4] 5A-5C are diagrams illustrating examples of different filters applied to an image pair in accordance with certain embodiments described herein. [Diagram 5] FIG. 2 illustrates an example of a computer-generated interpolation method for inserting two intermediate images between a first image and a second image, according to certain embodiments described herein. [Figure 6] 1A-1C are diagrams illustrating examples of three-dimensional rotation of a still image, according to certain embodiments described herein. [Figure 7A] FIG. 2 illustrates a flowchart for generating a video from a pair of images in accordance with some embodiments described herein. [Figure 7B] FIG. 2 illustrates a flowchart for generating a video from a pair of images in accordance with some embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] Detailed Description Exemplary Environment 100 FIG. 1 shows a block diagram of an exemplary environment 100. In some embodiments, the environment 100 includes a media server 101, a user device 115a, and a user device 115n, all connected to a network 105. Users 125a, 125n may be associated with user devices 115a, 115n, respectively. In some embodiments, the environment 100 may include other servers or devices not shown in FIG. 1, or may not include a media server 101. In FIG. 1 and other figures, a letter following a reference number, e.g., "115a," denotes a reference to the element having that particular reference number. A reference number in the text without a letter following it, e.g., "115," denotes a general reference to an embodiment of the element having that reference number.
[0009] The media server 101 may include a processor, memory, and network communication hardware. In some embodiments, the media server 101 is a hardware server. The media server 101 is communicatively connected to a network 105 via signal line 102. The signal line 102 may be a wired connection, such as Ethernet, coaxial cable, fiber optic cable, or a wireless connection, such as Wi-Fi, Bluetooth, or other wireless technology. In some embodiments, the media server 101 transmits data to and receives data from one or more user devices 115a, 115n via the network 105. The media server 101 may include a media application 103a and a database 199.
[0010] The media application 103a may include code and routines (including one or more trained machine learning models) operable to cause a user interface to generate a video with motion from at least two still images. In some embodiments, the media application 103a may be implemented using hardware including a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, the media application 103a may be implemented using a combination of hardware and software.
[0011] The database 199 can store media collections (e.g., still images, images with motion such as animated GIFs or videos, videos, etc.) associated with user accounts. The database 199 can store media items that are indexed and associated with identities of users 125 of user devices 115. The database 199 can also store social network data related to users 125, user preferences for users 125, etc.
[0012] The user device 115 may be a computing device that includes a memory and a hardware processor. For example, the user device 115 may include a desktop computer, a mobile device, a tablet computer, a mobile phone, a wearable device, a head mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or another electronic device that can access the network 105.
[0013] In the illustrated implementation, user device 115a is connected to network 105 via signal line 108, and user device 115n is connected to network 105 via signal line 110. Media application 103 may be stored on user device 115a as media application 103b or on user device 115n as media application 103c. Signal lines 108 and 110 may be wired connections such as Ethernet, coaxial cable, fiber optic cable, or wireless connections such as Wi-Fi, Bluetooth, or other wireless technologies. User devices 115a, 115n are utilized by users 125a, 125n, respectively. User devices 115a, 115n in FIG. 1 are used as examples. Although FIG. 1 shows two user devices 115a and 115n, the present disclosure applies to a system architecture including one or more user devices 115.
[0014] In some embodiments, a user account includes an image collection: for example, a user takes images and videos from their camera (e.g., a smartphone or other camera), uploads images from a digital single-lens reflex (DSLR) camera, and adds media captured and shared by other users to the image collection.
[0015] Although the following examples are described with pairs of still images, the media application 103 can generate videos with at least two still images. The media application 103 selects candidate image pairs from an image collection. Each image pair can include a first still image and a second still image from the library.
[0016] The media application 103 applies filters to select particular image pairs from the candidate image pairs. For example, the filters can filter out candidate image pairs that are not close duplicates, candidate pairs that do not contain artifacts, candidate pairs of poor quality, a large number of candidate pairs, candidate pairs that are not chronologically close, candidate pairs that are outside an acceptable time range (e.g., within 2 seconds of image capture, more than 0.1 seconds of each other, etc.), candidate pairs that are dissimilar (have a similarity below a threshold), candidate pairs with excessive motion (have motion above a threshold), candidate pairs that do not have bidirectional optical flow, and / or candidate pairs that are visually distinct (e.g., visual enhancements or filters have been applied to one of the images to make it different from the other image). For example, the media application 103 filters out pairs where the estimated motion between the first and second still images exceeds a first motion threshold (e.g., there is too much motion between the two still images that needs to be depicted through an interpolated image) and / or pairs where the estimated motion between the first and second still images is below a minimum motion threshold (e.g., there is too little motion between the two still images). In some embodiments, the media application 103 selects particular image pairs where the motion between the image pairs is greater than a first threshold and less than a second threshold if the images depict a particular type of subject (e.g., face, pet, person, etc.), if the images meet a quality threshold (e.g., not blurry, bright enough), if the images depict an interesting emotion (e.g., smiling), and / or if the images depict a particular type of activity (e.g., sports, dancing, etc.).
[0017] The media application 103 can use an image interpolator to generate one or more intermediate images based on the two still images. Each of the one or more intermediate images can be associated with a respective timestamp having a value between the timestamp of the first still image and the timestamp of the second still image. The position of each image in the sequence can be based on the respective timestamp.
[0018] In some embodiments, generating one or more intermediate images based on two still images includes generating a plurality of candidate intermediate images and evaluating each candidate intermediate image, where evaluating each candidate intermediate image is performed by generating a candidate video including the first still image as a first frame, the candidate intermediate image as a second frame, and the second still image as a third frame, and if the candidate video does not include a frame interpolation failure, selecting the candidate intermediate image as one of the one or more intermediate images.
[0019] In some embodiments, the image interpolator is an interpolating machine learning model (e.g., a generative model) that receives as input a first still image and a second still image and generates one or more intermediate images to simulate motion between the first image and the second image. The interpolating machine learning model may be trained on a training set of video that includes motion. For example, the interpolating machine learning model may receive a subset of video frames as input and generate one or more missing frames as output. The model may be trained using a cost function based on the difference between the generated frames and the corresponding original frames (excluded from the subset). Examples of such loss functions include, but are not limited to, L2 or L1 loss per pixel between one or more generated frames and the corresponding original frames. The training may include determining updates to the parameters of the interpolating machine learning model by applying an optimization routine, such as stochastic gradient descent, to the loss function. The optimization routine may be applied until a threshold condition is met. The threshold condition may include reaching a threshold number of training iterations and / or a threshold performance on a test dataset. For example, the model may be considered sufficiently trained when the cost function is minimized, e.g., when the generated frames and the corresponding original frames are indistinguishable. The classifier may be trained to compare the generated frames with the corresponding original frames. The training set of videos, acquired with the permission of the user, may include videos of various types, for example, videos showing facial movements such as smiling or opening and closing of the eyes, body movements such as walking, dancing, jumping, pet movements, etc.
[0020] The media application 103 can use the depth machine learning model to generate a three-dimensional representation of a scene in the first image based on a prediction of the depth of the first image. For example, the depth machine learning model can be a classifier that receives the first image as an input and outputs a prediction of the depth in the first image. The prediction of the depth can include depth coordinates of objects / features in the image, e.g., z-axis coordinates of objects in the image if the image is in an xy plane. The video can include camera effects generated based on the three-dimensional representation of the scene. The camera effects can include zooming and / or panning.
[0021] In some embodiments, the deep machine learning model includes a neural network. In some embodiments, the neural network includes a convolutional neural network. For example, the convolutional neural network can generate a three-dimensional image by extracting features from an input image, forming a low-resolution image, and iteratively refining the three-dimensional image. In some embodiments, the deep machine learning (ML) model can be trained using a training set of images and a corresponding depth map. For example, the deep ML model can be trained to predict the depth of an image in the training set. The prediction can be compared to a ground-truth depth in the depth map, and the difference can be used as feedback during training to update parameters of the deep machine learning model. The comparison of the predicted depth to the ground-truth depth map can be performed using a loss function, such as an L1 or L2 loss between the predicted depth and the corresponding ground-truth depth. The parameter updates can be determined by applying an optimization routine to the loss function.
[0022] The media application 103 can display a user interface that includes the video. The media application 103 can also notify the user that a video is available. The media application 103 can generate a video of any images that are not filtered out by the filter. The media application 103 can generate the video periodically, e.g., once a month, once a week, daily, etc.
[0023] Exemplary Computing Device 200 2 is a block diagram of an example computing device 200 that may be used to implement one or more features described herein. Computing device 200 may be any suitable computer system, server, or other electronic or hardware device. In one example, computing device 200 is a user device 115 used to run a media application 103. In another example, computing device 200 is a media server 101. In yet another example, media application 103 is located partially on user device 115 and partially on media server 101.
[0024] One or more methods described herein can be implemented as a standalone program running on any type of computing device, a program running on a web browser, or a mobile application (app) running on a mobile computing device (e.g., a mobile phone, a smartphone, a tablet computer, a wearable device (e.g., a watch, an armband, jewelry, headwear, virtual reality goggles or glasses, augmented reality goggles or glasses, head mounted displays), a laptop computer). In a primary example, all computations are performed in the mobile application on the mobile computing device. However, a client / server architecture can be used. For example, the mobile computing device sends user input data to a server device and receives and outputs (e.g., displays) final output data from the server. In another example, computations may be shared between the mobile computing device and one or more server devices.
[0025] In some embodiments, computing device 200 includes a processor 235, a memory 237, an I / O interface 239, a display 241, a camera 243, and storage 245. Processor 235 may be connected to bus 218 via signal line 222. Memory 237 may be connected to bus 218 via signal line 224. I / O interface 239 may be connected to bus 218 via signal line 226. Display 241 may be connected to bus 218 via signal line 228. Camera 243 may be connected to bus 218 via signal line 230. Storage 245 may be connected to bus 218 via signal line 232.
[0026] The processor 235 may be one or more processors and / or processing circuits for executing program code and controlling basic operations of the computing device 200. A "processor" includes any suitable hardware system, mechanism, or component for processing data, signals, or other information. A processor may include a general-purpose central processing unit (CPU) having one or more cores (e.g., single-core, dual-core, or multi-core configurations), multiple processing units (e.g., having a multi-processor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit for achieving a function, a dedicated processor for performing processing based on a neural network model, a neural circuit, a system with a processor optimized for performing matrix calculations (e.g., matrix multiplication), or other systems. In some implementations, the processor 235 may include one or more co-processors for performing neural network processing. In some implementations, the processor 235 may be a processor that generates a probabilistic output by processing data. For example, the output generated by the processor 235 may be inaccurate or accurate within a range of output expectations. Processing need not be limited to a particular geographic location, nor limited in time. For example, a processor may perform functions in real-time, offline, or in batch mode. Portions of processing may be performed by different (or the same) processing systems at different times and in different locations. A computer may be any processor in communication with a memory.
[0027] Memory 237 is typically provided within computing device 200 for use by processor 235 and may be any suitable processor-readable storage medium for storing instructions executed by the processor or set of processors, such as random access memory (RAM), read only memory (ROM), electrically erasable read only memory (EEPROM), flash memory, etc. Memory 237 may be located separately from and / or integrated with processor 235. Memory 237 may store software (including media application 103) executed on computing device 200 by processor 235.
[0028] Memory 237 can include an operating system 262, other applications 264, and application data 266. Other applications 264 can include, for example, a camera application, an image library application, an image management application, an image gallery application, a media viewing application, a communication application, a web hosting engine or application, a mapping application, a media sharing application, etc. One or more methods disclosed herein may be implemented on a number of environments and platforms, for example, as a standalone computer program capable of running on any type of computing device, as a web application having web pages, as a mobile application ("app") running on a mobile computing device.
[0029] Application data 266 may be data generated by other applications 264 or hardware of computing device 200. For example, application data 266 may include images captured by camera 243, user actions identified by other applications 264 (e.g., social networking applications), etc.
[0030] The I / O interface 239 can provide functionality that allows the computing device 200 to interface with other systems and devices. The interfacing devices may be included as part of the computing device 200 or may be separate but in communication with the computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or database 199), and input / output devices can communicate through the I / O interface 239. In some embodiments, the I / O interface 239 can connect to interface devices, such as input devices (keyboards, pointing devices, touch screens, microphones, cameras, scanners, sensors, etc.) and / or output devices (display devices, speaker devices, printers, monitors, etc.). For example, when a user provides touch input, the I / O interface 239 transmits data to the media application 103.
[0031] Some exemplary interface connection devices that can be connected to the I / O interface 239 may include a display 241 that can be used to display the content described herein, e.g., images, videos, and / or user interfaces of output applications, and receive touch (or gesture) input from a user. For example, the display 241 can be used to display a user interface that includes a subset of candidate image pairs. The display 241 may include any suitable display device, e.g., a liquid crystal display (LCD), light emitting diode (LED), or plasma display screen, a cathode ray tube (CRT), a television, a monitor, a touch screen, a three-dimensional display screen, or other visual display device. For example, the display 241 may be a flat display screen provided on a mobile device, multiple display screens embedded in a glasses form factor or headset device, or a monitor screen of a computing device.
[0032] Camera 243 may be any type of image capture device capable of capturing images and / or video. In some embodiments, camera 243 captures images or video that I / O interface 239 transmits to media application 103.
[0033] Storage 245 stores data related to media application 103. For example, storage 245 may store image collections associated with a user account, training sets for machine learning models, videos, etc. In embodiments in which media application 103 is part of media server 101, storage 245 is the same as database 199 of FIG.
[0034] Exemplary Media Application 103 2 illustrates an exemplary media application 103. The media application 103 includes a filtering module 202, an image interpolator 204, and a user interface module 206.
[0035] The filtering module 202 applies a filter to select a particular image pair from the candidate image pairs. In some embodiments, the filtering module 202 includes a set of instructions executable by the processor 235 to apply the filter. In some embodiments, the filtering module 202 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.
[0036] In some embodiments, the filtering module 202 includes a time filter that filters out candidate image pairs where the images are known for too long, where time refers to the time of image capture. For example, the time filter filters out candidate image pairs where the time difference between the timestamps associated with the first and second still images is greater than a time threshold. The time threshold may be any time value, such as 2 seconds, 1 minute, 1 day, etc.
[0037] In some embodiments, the temporal filter filters out candidate image pairs that are not in chronological order. In some embodiments, the temporal filter reorders any candidate pairs that are not in chronological order and resubmits them to the filtering module 202 for analysis. For example, even if the candidate image pairs are in chronological order, the filter module 202 may filter out the candidate image pairs if, for example, the time between the first and second still images exceeds a time threshold.
[0038] In some embodiments, the filtering module 202 includes a motion filter that filters out candidate image pairs that have too much or too little motion. For example, the motion filter may estimate the motion between a first still image and a second still image and filter out a candidate image pair if the motion between the first still image and the second still image is less than a minimum motion threshold. As another example, the motion filter may filter out a candidate image pair if the motion between the first still image and the second still image is greater than a maximum motion threshold.
[0039] In some embodiments, the filtering module 202 includes a quality filter that filters out candidate image pairs that have a quality below a quality threshold. For example, the quality filter filters out candidate image pairs if one or both of the candidate images are blurry, noisy, violate the rule of thirds (dividing an image into thirds and placing an object in one third of the image), etc.
[0040] In some embodiments, the filtering module 202 includes a semantic filter that filters out candidate image pairs that do not include subjects determined to be of interest to the user. For example, the user's interest may be determined based on (a) explicit preferences provided by the user (e.g., specific people or pets that the user is interested in), (b) the user's behavior (with permission, determining the people that appear most frequently in the user's captured images, the images the user frequently views, and the images for which the user has provided an indication of approval (e.g., thumbs up / down, like, +1, etc.)). Such a determination may include technically comparing the current image to known user interest attributes (e.g., depicting person A). The filtering module 202 may only make such a determination if it has received permission from the user to collect user data.
[0041] The filtering module 202 may include a list of acceptable objects that are personalized for the user or more general. For example, a user may consistently indicate approval of certain objects, such as the user's daughter, dogs, landscapes, etc. In another example, the semantic module may automatically filter out images of objects such as receipts, screenshots of memes, etc.
[0042] In some embodiments, the filtering module 202 also filters out images that have incompatible image sizes. For example, the filtering module 202 can filter out images that do not meet a predetermined image size (e.g., less than a particular resolution, e.g., at least 400 pixels wide and at least 500 pixels high).
[0043] 3, a detailed example 300 of the filtering module 202 and image interpolator 204 is shown. In some embodiments, the filtering module 202 includes one or both of a filter 302 and a filter machine learning module 304. In some embodiments, the filter 302 includes one or more of a temporal filter, a motion filter, a quality filter, a semantic filter, etc.
[0044] In some embodiments, the filter machine learning module 304 includes a machine learning model trained to generate feature vectors from candidate images and filter the candidate images based on the feature vectors. In some embodiments, the filter machine learning module 304 includes a set of instructions executable by the processor 235 to generate the feature vectors. In some embodiments, the filter machine learning module 304 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.
[0045] In some embodiments, the filter machine learning module 304 can generate feature vectors that represent the first and second still images of each candidate pair in a multi-dimensional feature space. Images with similar features may have similar feature vectors. For example, the feature vectors are mathematical expressions, and the mathematical expressions of similar images are closer in vector space than the mathematical expressions of dissimilar images. The vector space may be a function of various elements of the images, such as the subject matter depicted (objects detected from the image), the composition of the image, color information, image orientation, image metadata, specific objects recognized from the image (e.g., known faces, with user permission), etc.
[0046] In some embodiments, the filter machine learning module 304 can use training data (obtained with permission for training purposes) to generate a trained model, specifically, a filter machine learning model. For example, the training data can include ground truth data in the form of image pairs associated with visual similarity descriptions for the image pairs. In some embodiments, the visual similarity descriptions can include feedback from a user as to whether the image pairs are related. In some embodiments, the visual similarity descriptions can be added automatically by image analysis. The training data can be obtained from any source, e.g., a data repository specified for training, data that has been given permission to be used as training data for machine learning. In some embodiments, the training can be performed on the media server 101 providing the training data directly to the user device 115, locally on the user device 115, or a combination of both.
[0047] In some embodiments, the training data may include synthetic data generated for training purposes, e.g., data that is not based on activity in the situation being trained, e.g., data generated from simulated or computer-generated images / videos. In some embodiments, the filter machine learning module 304 uses weights obtained and unedited / transferred from another application. For example, in these embodiments, the trained model may be generated, e.g., on a different device and provided as part of the media application 103. In various embodiments, the trained model may be provided as a data file that includes a model structure or morphology (e.g., defining the number and type of neural network nodes, the connections between the nodes, and the organization of the nodes into multiple layers) and associated weights. The filter machine learning module 304 can read the trained model data file and implement a neural network, including node connections, layers, and weights, based on the model structure or morphology specified in the trained model.
[0048] The filter machine learning module 304 generates a trained model, referred to herein as a filter machine learning model. In some embodiments, the filter machine learning module 304 is configured to apply the filter machine learning model to data, such as application data 266 (e.g., candidate image pairs), to identify one or more features of each candidate image and generate feature vectors (embeddings) that represent each candidate image pair. In some embodiments, the filter machine learning module 304 may include software code executed by the processor 235. In some embodiments, the filter machine learning module 304 may specify circuitry (e.g., a programmable processor, a field programmable gate array (FPGA)) that enables the processor 235 to apply the filter machine learning model. In some embodiments, the filter machine learning module 304 may include software instructions, hardware instructions, or a combination thereof. In some embodiments, the filter machine learning module 304 may provide an application programming interface (API). The operating system 262 and / or other applications 264 may utilize this API to call the filter machine learning module 304 to, for example, output feature vectors for each candidate image pair by applying the filter machine learning model to the application data 266. In some embodiments, the mathematical representations of the candidate image pairs retained by the filter machine learning module 304 are closer in vector space than the mathematical representations of the candidate image pairs rejected by the filter machine learning module 304 .
[0049] In some embodiments, the filter machine learning model includes a classifier that receives the candidate image pairs as input. Examples of classifiers include neural networks, support vector machines, k-nearest neighbors, logistic regression, naive Bayes, decision trees, perceptrons, etc.
[0050] In some embodiments, the filter machine learning model may include one or more model forms or structures. For example, the model forms or structures may include any type of neural network, such as a linear network, a deep neural network that implements multiple layers (e.g., a "hidden layer" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (CNN) (e.g., a network that splits or partitions input data into multiple portions or tiles, processes each tile separately using one or more neural network layers, and aggregates results from the processing of each tile), or a sequence-to-sequence neural network (e.g., a network that receives sequential data as input, such as words in a sentence or frames in a video, and produces a sequence of results as output).
[0051] The model form or structure may specify the connections between various nodes and the organization of the nodes into layers. For example, the nodes in the first layer (e.g., input layer) may receive data as input data or application data 266. For example, when using a filter machine learning model to analyze input images, e.g., candidate image pairs, such data may include, for example, one or more pixels per node. Subsequent intermediate layers may receive as input the output of the nodes of the previous layer according to the connections specified in the model form or structure. These layers may also be referred to as hidden layers. The final layer (e.g., output layer) generates an output of the filter machine learning model. For example, the output may be a feature vector of the candidate image pair. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.
[0052] The features output by the filter machine learning module 304 may include the subject (e.g., a sunset vs. a particular person), the colors present in the image (green hills vs. a blue lake), color balance, lighting source, angle and intensity, the location of objects in the image (e.g., adhering to the rule of thirds), the relative location of objects (e.g., depth of field), the location of the shot, the focus (foreground vs. background), or shadows. While the aforementioned features are human understandable, the output features may be embeddings or other numerical values that are representative of the image and not human analyzable (e.g., individual feature values may not correspond to specific features such as colors present, object locations, etc.). However, the trained model is robust to images, so it outputs similar features for similar images and different features for significantly different images. Examples of such models include the encoders of autoencoder models.
[0053] In some embodiments, the model form is a CNN that includes network layers, each of which extracts image features at a different level of extraction. The CNN used to identify features in an image may be used to classify the image. The model architecture may include layer combinations and sequences of multidimensional convolutions, average pooling, max pooling, activation functions, normalization, regularization, and other layers and modules actually used in applied deep neural networks.
[0054] In different embodiments, the filter machine learning model may include one or more models. The one or more models may include a plurality of nodes arranged in layers according to a model structure or morphology. In some embodiments, the node may be, for example, a computational node without memory configured to process a unit of input and generate a unit of output. The computation performed by the node may include, for example, multiplying each of a plurality of node inputs by a weight, obtaining a weighted sum, and generating a node output by adjusting the weighted sum with a bias value or an intercept value. For example, the filter machine learning module 304 may adjust each weight based on feedback in response to automatically updating one or more parameters of the filter machine learning model.
[0055] In some embodiments, the computations performed by the nodes may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function such as a ReLU function, a sigmoid function, a tanh function, etc. In various embodiments, such computations may include operations such as matrix multiplication. In some embodiments, the computations performed by the multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, using individual processing units of a graphical processing unit (GPU), or using dedicated neural circuitry. In some embodiments, the nodes may include memory. The nodes may, for example, store one or more previous inputs and use one or more previous inputs in processing subsequent inputs. For example, a node with memory may include a long short-term memory (LSTM) node. The LSTM node may use the memory to maintain state that allows the node to operate like a finite state machine (FSM). A model including such nodes may be useful in processing sequential data, such as multiple words in a sentence or paragraph, a series of images, frames in a video, conversation, or other audio, etc. For example, the heuristics-based model used in the gating model may remember one or more features previously generated for previous images.
[0056] In some embodiments, the filter machine learning model may include embeddings or weights for individual nodes. For example, the filter machine learning model may be initialized as a number of nodes organized into layers as specified by the model topology or structure. At initialization, a respective weight may be applied to the connection between each pair of nodes connected according to the model topology, e.g., each pair of nodes in successive layers of a neural network. For example, each weight may be randomly assigned or initialized to a default value. The filter machine learning model may then be trained, e.g., using a training set of image pairs to generate results. In some embodiments, a subset of the entire architecture may be reused from other machine learning applications as a transfer learning method to leverage the pre-trained weights.
[0057] For example, the training may include applying a supervised learning technique. In supervised learning, the training data may include a plurality of inputs (e.g., image pairs from an image collection associated with a user account) and an expected output corresponding to each image pair (e.g., image embeddings for the image pairs). For example, the weight values are automatically adjusted based on a comparison of the output of the filter machine learning model to the expected output, such that the filter machine learning model increases the probability of generating the expected output when provided with similar inputs. The comparison may be performed using a loss function, and the adjusted weight values are determined by applying an optimization routine to the loss function.
[0058] In some embodiments, training can include applying unsupervised learning techniques, in which only input data (e.g., image pairs from an image collection associated with a user account) may be provided and a filter machine learning model may be trained to discriminate between the data, e.g., to cluster image pairs into distinct groups.
[0059] In various embodiments, the trained model includes a set of weights that correspond to the model structure. In embodiments that omit a training set, the filter machine learning module 304 may generate the filter machine learning model based on prior training, such as by a developer of the filter machine learning module 304 or a third party. In some embodiments, the filter machine learning model may include a fixed set of weights downloaded from a server that provides the weights.
[0060] In some embodiments, the filter machine learning module 304 may be implemented in an offline manner. Implementing the filter machine learning module 304 may include using a static training set that does not include updates when data in the static training set changes. This advantageously results in increased efficiency of processing performed by the computing device 200 and reduced power consumption of the processing device 200. In these embodiments, the filter machine learning model may be generated in a first stage and provided as part of the filter machine learning module 304. In some embodiments, small updates to the filter machine learning model may be implemented in an online manner where updates to the training data are included as part of the training of the filter machine learning model. Small updates are updates having a size smaller than a threshold size. The size of the update is related to the number of variables in the filter machine learning model that are affected by the update. In such embodiments, an application that invokes the filter machine learning module 304 (e.g., the operating system 262, one or more other applications 264, etc.) may utilize image embeddings of candidate image pairs to identify visually similar clusters. The filter machine learning module 304 may also generate system logs periodically, such as hourly, monthly, or quarterly. The system logs may be used to update the filter machine learning model, for example to update the embeddings of the filter machine learning model.
[0061] In some embodiments, the filter machine learning module 304 may be implemented in a manner that is adapted to the particular configuration of the computing device 200 on which the filter machine learning module 304 executes. For example, the filter machine learning module 304 may determine a computation graph that utilizes available computational resources, such as the processor 235. If the filter machine learning module 304 is implemented as a distributed application on multiple devices, such as when the media server 101 includes multiple media servers 101, the filter machine learning module 304 may determine the computations to be performed on each device to optimize the computations. In another example, if the filter machine learning module 304 determines that the processor 235 includes a GPU with a particular number (e.g., 1000) of GPU cores, the filter machine learning module 304 may implement the filter machine learning module 304 (e.g., as 1000 separate processes or threads).
[0062] In some embodiments, the filter machine learning module 304 can implement a set of trained models. For example, the filter machine learning model can include multiple trained models, each applicable to the same input data. In these embodiments, the filter machine learning module 304 can select a particular trained model based on, for example, available computational resources, success rate using previous inferences, etc.
[0063] In some embodiments, the filter machine learning module 304 may execute multiple trained models. In these embodiments, the filter machine learning module 304 may combine the outputs, for example, using a majority vote that scores the outputs obtained by applying each trained model, or by selecting one or more particular outputs. In some embodiments, such a selector is part of the model itself and serves as a connection layer between the trained models. Furthermore, in these embodiments, the filter machine learning module 304 may apply a time threshold (e.g., 0.5 ms) for applying the individual trained models and utilize only the individual outputs that are available within the time threshold. Outputs that are not received within the time threshold may not be utilized, e.g., filtered out. For example, such an approach may be appropriate when there is a time limit specified between invoking the filter machine learning module 304, e.g., by the operating system 262 or one or more other applications 264. In this way, the responsiveness of the media application 103 can be improved by limiting the maximum time it takes for the filter machine learning module 304 to perform a task, such as identifying one or more features of a candidate image pair and generating feature vectors (embeddings) representing each of the candidate image pairs, thereby enabling the filter machine learning module 304 to provide the best classification in real time.
[0064] In some embodiments, the filter machine learning module 304 operates to eliminate one or more candidate pairs corresponding to each feature vector if the distance between the respective feature vectors is greater than a threshold vector distance that is eliminated.
[0065] In some embodiments, the filter machine learning module 304 receives feedback. For example, the filter machine learning module 304 can receive feedback from a user or a set of users via the user interface module 206. The feedback can include, for example, that the candidate image pairs are too dissimilar to be used to generate a video. If a user provides feedback, the filter machine learning module 304 provides the feedback to the filter machine learning model, which uses the feedback to update parameters of the filter machine learning model to modify the output image embeddings of the set of candidate image pairs. If a set of users provides feedback, the filter machine learning module 304 provides aggregate feedback to the filter machine learning model, which uses the aggregate feedback to update parameters of the filter machine learning model to modify the output image embeddings of the set of candidate image pairs. For example, the aggregate feedback can include a subset of the videos and the user's reactions to the subset of the videos. User responses may include viewing only one video and refusing to view the remaining videos, viewing all videos in the subset, sharing the video, providing an indication of approval or disapproval of the video (e.g., thumbs up / down, like, +1, etc.) The filter machine learning module 304 may modify the set of candidate image pairs based on updating parameters of the filter machine learning model.
[0066] In some embodiments, the filtering module 202 determines a subset of different filters to be applied to the candidate image pairs. With reference to FIG. 4, different options 400 of filters to be applied to a pair of images are shown. The filters can be classified as early, mid-term, and late filters. Early filters, e.g., temporal filters that filter out candidate image pairs that are too far apart in time, can perform analysis of image metadata. Mid-term filters, e.g., motion filters that filter out candidate image pairs that have too much motion, can perform analysis of image data. Late filters can include image data and test renderings. A balance between different filters may be achieved with filters that are faster to execute but have higher repeatability and filters that are slower but have higher accuracy. In some embodiments, the filtering module 202 can determine a subset of different filters based on the need to generate the video quickly. For example, if a video is generated once a month, processing time may not be that important. However, if a user requests a video, the processing time needs to be minimized in order to provide a response to the user in a short time so that the application providing the video is responsive.
[0067] In some embodiments, the filtering module 202 selects a particular image pair from the plurality of candidate image pairs. For example, the particular image pair is an image pair that was not excluded by the filtering module 202. In some embodiments, the particular image pair is selected as the first image pair of the plurality of candidate image pairs that was not excluded by the filtering module 202.
[0068] In some embodiments, the filter machine learning module 304 receives an intermediate image of the one or more intermediate images from the image interpolator 204 as an input to the filter machine learning module, generates a feature vector corresponding to the intermediate image, and excludes the intermediate image if it is too dissimilar to the corresponding first still image or second still image. For example, the image interpolator 204 generates a feature vector from the intermediate image, compares the feature vector to the feature vectors of the corresponding candidate image pair, and excludes the intermediate image if the distance between the feature vector of the intermediate image and any of the feature vectors of the candidate image pair is greater than a threshold vector distance.
[0069] The image interpolator 204 generates one or more intermediate images based on a particular image pair. In some embodiments, the image interpolator 204 includes a set of instructions executable by the processor 235 to generate one or more intermediate images. In some embodiments, the image interpolator 204 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.
[0070] In some embodiments, the image interpolator 204 generates one or more intermediate images that include intermediate steps. The one or more intermediate images are inserted as frames between a first static frame and a second static frame. When these frames are displayed consecutively as a video, the one or more intermediate images are inserted to form a smooth animation. For example, FIG. 5 shows an example of an interpolation method that inserts two intermediate frames 550a, 550b between a first frame 500 and a second frame 575. In this example, the first frame 500 shows eyes open and a mouth closed, and the second frame 575 shows a smile with eyes partially closed and a mouth partially open. The image interpolator 204 generates the intermediate frames 550a, 550b that include the actions of closing the eyes and moving the mouth. When the frames are displayed sequentially as a video (in the order 500, 550a, 550b, 575), the intermediate frames 550a, 550b allow the viewer to perceive smooth motion between the first still image 500 and the second still image 575.
[0071] While FIG. 5 shows two intermediate images, in different implementations, the image interpolator 204 generates one, two, three, or more intermediate images. In some embodiments, the number of intermediate images may be a function of the total motion between frames 500-575. In this case, more motion occurs, more intermediate images are generated. In some embodiments, the image interpolator 204 generates intermediate images based on available computing power. If the computing power is high, more intermediate images are generated, and if the computing power is low, fewer intermediate images are generated. In some embodiments, the image interpolator 204 may generate intermediate images based on image resolution. If the selected image pair has a high resolution, the image interpolator 204 generates more intermediate images. In some embodiments, the image interpolator 204 generates more intermediate images based on the refresh rate of the user device 115. For example, a higher refresh rate (e.g., 120 Hz) requires more intermediate images than a lower refresh rate (e.g., 50 Hz).
[0072] In some embodiments, the image interpolator 204 receives multiple consecutive image pairs and generates multiple intermediate images. For example, the image interpolator 204 may receive specific image pairs a, b, c, and d, where a and b, b and c, and c and d constitute image pairs. The image interpolator 204 may generate one or more intermediate image pairs for each specific image pair.
[0073] In some embodiments, each intermediate image is associated with a respective timestamp having a value between the timestamp of the first still image and the timestamp of the second still image. In some embodiments, the image interpolator 204 compiles the video based on the timestamps of each of the first still image, the one or more intermediate images, and the second still image.
[0074] In some embodiments, the image interpolator 204 generates the one or more intermediate images by generating a candidate intermediate image, evaluating each candidate intermediate image by generating a candidate video including the first still image as a first frame, including the candidate intermediate image as a second frame, and including the second still image as a third frame, and selecting the candidate intermediate image as one of the one or more intermediate images if the candidate video does not include a frame interpolation failure. The frame interpolation failure may occur in response to providing the intermediate image to the filtering module 202 or based on a failure detected by the image interpolator 204 as discussed in more detail below with reference to the classifier machine learning module 308.
[0075] In some embodiments, the image interpolator 204 sends each intermediate image to a filtering module 202, e.g., a filter machine learning module 304, to verify that the intermediate image is sufficiently similar to the particular image pair. If the filtering module 202 does not exclude an intermediate image, the image interpolator 204 generates the video. If the filter machine learning module 304 excludes an intermediate image, the exclusion is considered a frame interpolation failure, and the image interpolator 204 generates one or more additional intermediate images.
[0076] 3, a detailed example 300 of the filtering module 202 and the image interpolator 204 is shown. In some embodiments, the image interpolator 204 includes an interpolation machine learning module 306, a classifier machine learning module 308, a depth machine learning module 310, and a video generator 312.
[0077] In some embodiments, one or more of the interpolation machine learning module 306, the classifier machine learning module 308, and the depth machine learning module 310 may be layers / blocks in a neural network or may be separate neural networks. For example, the interpolation machine learning module 306 may receive a particular image pair as input and output one or more intermediate images that are input to the classifier machine learning module 208. If the one or more intermediate images do not include frame interpolation failures, the classifier machine learning module 208 may output one or more intermediate images. The one or more intermediate images may be input to the video generator 312. Also, the depth machine learning module 310 may receive a first still image as input and output a three-dimensional representation of the scene to the video generator 312. Other embodiments are possible. For example, the interpolation machine learning module 306 and the classifier machine learning module 308 may be layers in a neural network, or the interpolation machine learning module 306 and the depth machine learning module 310 may be layers in a neural network. In yet another example, the interpolation machine learning module 306 can operate alone and provide one or more intermediate images directly to the video generator 312 as output.
[0078] In some embodiments, the interpolation machine learning module 306 includes an interpolation machine learning model that is trained to receive a first still image and a second still image as input and generate one or more intermediate images as output. The interpolation machine learning module 306 can include any type of generative machine learning model trained to generate images from an input image pair. In some embodiments, the interpolation machine learning module 306 includes a set of instructions executable by the processor 235 to generate one or more intermediate images. In some embodiments, the interpolation machine learning module 306 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.
[0079] In some embodiments, the interpolation machine learning module 306 can use training data (obtained with permission for training) to generate a trained model, specifically an interpolation machine learning model. For example, the training data can include ground truth data in the form of image pairs and intermediate images associated with visual similarity descriptions of the image pairs and intermediate images. In some embodiments, the visual similarity descriptions can be added automatically by image analysis. The training data can be obtained from any source, e.g., a data repository specified for training, data that has been given permission to be used as training data for machine learning. In some embodiments, the training can be performed on the media server 101 providing the training data directly to the user device 115, locally on the user device 115, or a combination of both.
[0080] In some embodiments, the training data may include synthetic data generated for training purposes, e.g., data that is not based on activity in the situation being trained, e.g., data generated from simulated or computer-generated images / videos. In some embodiments, the interpolation machine learning module 306 uses weights obtained and unedited / transferred from another application. For example, in these embodiments, the trained model may be generated, e.g., on a different device and provided as part of the media application 103. In various embodiments, the trained model may be provided as a data file that includes a model structure or morphology (e.g., defining the number and type of neural network nodes, the connections between the nodes, and the organization of the nodes into multiple layers) and associated weights. The interpolation machine learning module 306 can read the trained model data file and implement a neural network, including node connections, layers, and weights, based on the model structure or morphology specified in the trained model.
[0081] In some embodiments, the interpolation machine learning module 306 is configured to apply an interpolation machine learning model to data, such as application data 266 (e.g., selected image pairs), to generate one or more intermediate images that may approximate the positions of different objects between the first and second still images. For example, given a first still image and a second still image, the interpolation machine learning module 306 outputs a first intermediate image, followed by a series of intermediate images. In this example, the first and second still images depict a toddler and an infant. Both children's heads are rotated differently between the first and second still images. In this example, the interpolation machine learning module 306 generates one intermediate image in which the child's head is in a position between the first and second still images. In some embodiments, the interpolation machine learning module 306 generates additional intermediate images in which the child's head is in a position between the first and second still images.
[0082] In some embodiments, the interpolation machine learning module 306 may include software code executed by the processor 235. In some embodiments, the interpolation machine learning module 306 may specify a circuit configuration (e.g., a programmable processor, a field programmable gate array (FPGA)) that enables the processor 235 to apply the interpolation machine learning model. In some embodiments, the interpolation machine learning module 306 may include software instructions, hardware instructions, or a combination thereof. In some embodiments, the interpolation machine learning module 306 may provide an application programming interface (API). The operating system 262 and / or other applications 264 may utilize this API to call the interpolation machine learning module 306 to, for example, apply the interpolation machine learning model to application data 266 to output one or more intermediate images.
[0083] In some embodiments, the interpolating machine learning model includes a classifier that receives the selected image pairs as input. Examples of classifiers include neural networks, support vector machines, k-nearest neighbors, logistic regression, naive Bayes, decision trees, perceptrons, etc.
[0084] In some embodiments, the interpolation machine learning model may include one or more model forms or structures. For example, the model forms or structures may include a linear network, a deep neural network implementing multiple layers (e.g., a "hidden layer" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (CNN) (e.g., a network that splits or partitions input data into multiple portions or tiles, processes each tile separately with one or more neural network layers, and aggregates results from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that receives sequential data as input, such as words in a sentence or frames in a video, and produces a sequence of results as output).
[0085] The model form or structure may specify the connections between the various nodes and the organization of the nodes organized into layers. For example, the nodes of the first layer (e.g., input layer) may receive data as input data or application data 266. For example, when using an interpolation machine learning model to analyze input images, e.g., a selected image pair, such data may include, for example, one or more pixels per node. Subsequent intermediate layers may receive as input the output of the nodes of the previous layer according to the connections specified in the model form or structure. For example, a first intermediate image in the middle of the first still image and the second still image may be part of the first intermediate layer. These layers may also be referred to as hidden layers. The final layer (e.g., output layer) generates the output of the interpolation machine learning model. For example, the output may be a series of intermediate images based on the first still image, the second still image, and the first intermediate image. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.
[0086] In some embodiments, the model form is a CNN that includes network layers, each of which extracts image features at a different level of extraction. The CNN used to identify features in an image may be used to classify the image. The model architecture may include layer combinations and sequences of multidimensional convolutions, average pooling, max pooling, activation functions, normalization, regularization, and other layers and modules actually used in applied deep neural networks.
[0087] In different embodiments, the interpolation machine learning model may include one or more models. The one or more models may include a plurality of nodes arranged in layers according to a model structure or morphology. In some embodiments, the node may be, for example, a computational node without memory configured to process a unit of input and generate a unit of output. The computation performed by the node may include, for example, multiplying each of a plurality of node inputs by a weight, obtaining a weighted sum, and generating a node output by adjusting the weighted sum with a bias value or an intercept value. For example, the interpolation machine learning module 306 may adjust each weight based on feedback in response to automatically updating one or more parameters of the interpolation machine learning model.
[0088] In some embodiments, the computations performed by the nodes may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function such as a ReLU function, a sigmoid function, a tanh function, etc. In various embodiments, such computations may include operations such as matrix multiplication. In some embodiments, the computations performed by the multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, using individual processing units of a graphical processing unit (GPU), or using dedicated neural circuitry. In some embodiments, the nodes may include memory. The nodes may, for example, store one or more previous inputs and use one or more previous inputs in processing subsequent inputs. For example, a node with memory may include a long short-term memory (LSTM) node. The LSTM node may use the memory to maintain state that allows the node to operate like a finite state machine (FSM). A model including such nodes may be useful in processing sequential data, such as multiple words in a sentence or paragraph, a series of images, frames in a video, conversation, or other audio, etc. For example, the heuristics-based model used in the gating model may remember one or more features previously generated for previous images.
[0089] In some embodiments, the interpolation machine learning model may include embeddings or weights for individual nodes. For example, the interpolation machine learning model may be initialized as a number of nodes organized into layers as specified by the model topology or structure. At initialization, a respective weight may be applied to the connection between each pair of nodes connected according to the model topology, e.g., each pair of nodes in successive layers of a neural network. For example, each weight may be randomly assigned or initialized to a default value. The interpolation machine learning model may then be trained, e.g., using a training set of image pairs to generate results. In some embodiments, a subset of the entire architecture may be reused from other machine learning applications as a transfer learning method to leverage the pre-trained weights.
[0090] For example, training can include applying supervised learning techniques. In supervised learning, the training data can include multiple inputs (e.g., image pairs from an image collection associated with a user account) and expected outputs (e.g., one or more intermediate images) corresponding to each image pair. For example, weight values are automatically adjusted based on a comparison of the output of the interpolating machine learning model to the expected output to increase the probability that the interpolating machine learning model will generate the expected output when provided with similar inputs.
[0091] In some embodiments, training can include applying unsupervised learning techniques, in which only input data (e.g., image pairs from an image collection associated with a user account) may be provided and an interpolation machine learning model may be trained to discriminate between the data, e.g., to cluster image pairs into distinct groups.
[0092] In various embodiments, the trained model includes a set of weights that correspond to the model structure. In embodiments that omit a training set, the interpolation machine learning module 306 may generate the interpolation machine learning model based on prior training, such as by a developer of the interpolation machine learning module 306 or a third party. In some embodiments, the interpolation machine learning model may include a set of fixed weights downloaded from a server that provides the weights.
[0093] In some embodiments, the interpolation machine learning module 306 may be implemented in an offline manner. Implementing the interpolation machine learning module 306 may include using a static training set that does not include updates when data in the static training set changes. This advantageously results in increased efficiency of processing performed by the computing device 200 and reduced power consumption of the processing device 200. In these embodiments, the interpolation machine learning model may be generated in a first stage and provided as part of the interpolation machine learning module 306. In some embodiments, small updates to the interpolation machine learning model may be implemented in an online manner where updates to the training data are included as part of training the interpolation machine learning model. Small updates are updates having a size smaller than a threshold size. The size of the update is related to the number of variables in the interpolation machine learning model that are affected by the update. In such embodiments, an application that invokes the interpolation machine learning module 306 (e.g., the operating system 262, one or more other applications 264, etc.) may utilize image embeddings of candidate image pairs to identify visually similar clusters. The interpolation machine learning module 306 may also generate system logs periodically, such as hourly, monthly, or quarterly. The system logs may be used to update the interpolated machine learning model, for example to update the embeddings of the interpolated machine learning model.
[0094] In some embodiments, the interpolation machine learning module 306 may be implemented in a manner that is compatible with the particular configuration of the computing device 200 on which the interpolation machine learning module 306 executes. For example, the interpolation machine learning module 306 may determine a computation graph that utilizes available computational resources, such as the processor 235. If the interpolation machine learning module 306 is implemented as a distributed application on multiple devices, such as when the media server 101 includes multiple media servers 101, the interpolation machine learning module 306 may determine the computations to be performed on each device to optimize the computations. In another example, if the interpolation machine learning module 306 determines that the processor 235 includes a GPU with a particular number (e.g., 1000) of GPU cores, the interpolation machine learning module 306 may implement the interpolation machine learning module 306 (e.g., as 1000 separate processes or threads).
[0095] In some embodiments, the interpolation machine learning module 306 can implement a set of trained models. For example, the interpolation machine learning model can include multiple trained models, each applicable to the same input data. In these embodiments, the interpolation machine learning module 306 can select a particular trained model based on, for example, available computational resources, success rate using previous inferences, etc.
[0096] In some embodiments, the interpolation machine learning module 306 may execute multiple trained models. In these embodiments, the interpolation machine learning module 306 may combine the outputs, for example, using a majority vote that scores the outputs obtained by applying each trained model, or by selecting one or more particular outputs. In some embodiments, such a selector is part of the model itself and serves as a connection layer between the trained models. Furthermore, in these embodiments, the interpolation machine learning module 306 may apply a time threshold (e.g., 0.5 ms) for applying each trained model and utilize only those individual outputs that are available within the time threshold. Outputs that are not received within the time threshold may not be utilized, e.g., filtered out. For example, such an approach may be appropriate when there is a time limit specified between invoking the interpolation machine learning module 306, e.g., by the operating system 262 or one or more other applications 264. In this way, the responsiveness of the media application 103 can be improved by limiting the maximum time it takes for the interpolation machine learning module 306 to perform a task, such as identifying one or more features of a selected image pair and generating feature vectors (embeddings) representing each candidate image pair, thereby enabling the interpolation machine learning module 306 to provide the best generation of one or more intermediate images in real time.
[0097] In some embodiments, the interpolation machine learning module 306 receives feedback. For example, the interpolation machine learning module 306 can receive feedback from a user or a set of users via the user interface module 206. The feedback can include, for example, that the intermediate images are too dissimilar to the particular image pair used to generate the video. If a user provides feedback, the interpolation machine learning module 306 provides the feedback to the interpolation machine learning model, which uses the feedback to update parameters of the interpolation machine learning model to modify one or more intermediate images of the output. If a set of users provides feedback, the interpolation machine learning module 306 provides aggregate feedback to the interpolation machine learning model, which uses the aggregate feedback to update parameters of the interpolation machine learning model to modify the intermediate images of the output. For example, the aggregate feedback can include a subset of the video and the user's reaction to the subset of the video. User responses may include viewing only one video and declining to view the remaining videos, viewing all videos in the subset, sharing a video, providing an indication of approval or disapproval of the video (e.g., thumbs up / down, like, +1, etc.), etc.
[0098] In some embodiments, the classifier machine learning module 308 includes a classifier machine learning model that is trained to receive as input one or more intermediate images and one or more of the first still image and the second still image and output a likelihood that the one or more intermediate images are generated images. In some embodiments, the interpolation machine learning module 306 includes a set of instructions executable by the processor 235 to output a likelihood that the one or more intermediate images are generated images. In some embodiments, the interpolation machine learning module 306 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.
[0099] In some embodiments, the discriminator machine learning module 308 can use training data (obtained with permission for training) to generate a trained model, specifically, a discriminator machine learning model. For example, the training data can include ground truth data in the form of pairs of images and intermediate images associated with a description of whether the intermediate image is a generated image or not. In some embodiments, the description of the intermediate image can be added automatically by image analysis. The training data can be obtained from any source, for example, a data repository specified for training, data that has been given permission to be used as training data for machine learning. In some embodiments, the training can be performed on the media server 101 that provides the training data directly to the user device 115, the training can be performed locally on the user device 115, or a combination of both. In some embodiments, a generative adversarial approach can be used to train the discriminator model together with the interpolation machine learning model.
[0100] In some embodiments, the classifier machine learning module 308 is configured to apply a classifier machine learning model to data such as application data 266 (e.g., the intermediate image and one or more of the first still image and the second still image) and generate a likelihood indicating that the intermediate image is visually indistinguishable from one or more of the first still image and the second still image. In some embodiments, if the likelihood does not meet a threshold, the intermediate image is rejected. In some embodiments, if the likelihood exceeds a threshold, the intermediate image is accepted and provided as an input to the depth machine learning module 310 or the video generator 312, depending on the embodiment.
[0101] Additional information regarding how the classifier machine learning module 308 uses training data, uses the processor 235, functions as different types of machine learning models, etc., is provided above and will not be repeated here.
[0102] In some embodiments, the depth machine learning module 310 includes a depth machine learning model that is trained to receive a first still image as an input and output a three-dimensional representation of a scene in the first still image based on a prediction of the depth of the first still image. In some embodiments, the interpolation machine learning module 306 includes a set of instructions executable by the processor 235 to output a likelihood that one or more intermediate images are generative images. In some embodiments, the interpolation machine learning module 306 may be stored in the memory 237 of the computing device 200 and accessible and executable by the processor 235.
[0103] In some embodiments, the deep machine learning module 310 can use training data (obtained with permission for training) to generate a trained model, specifically, a deep machine learning model. For example, the training data can include ground truth data in the form of images of a scene and a three-dimensional representation that includes the scene. The training data can be obtained from any source, for example, a data repository specified for training, data that has been given permission to be used as training data for machine learning.
[0104] In some embodiments, the depth machine learning module 310 is configured to apply a depth machine learning model to data, such as the application data 266 (e.g., the first still image), to generate a three-dimensional representation of the scene. In some embodiments, the depth machine learning model is a classifier that receives the first still image as input and generates the three-dimensional representation of the scene. The three-dimensional representation of the scene can include camera effects, such as zooming, panning, rotation, or combinations thereof.
[0105] FIG. 6 illustrates an example of a 3D zoom of a still image. The first image 600 may be the same as the first still image received as an input to the depth machine learning model as described above. The depth machine learning model generates as output a 3D representation of the scene with a zoom camera effect. This 3D representation is shown as a second example 650 and a third example 675. In the example of FIG. 6, sequences 600-675 provide the viewer with a video in which the person and foliage move closer together in the frame, thereby enhancing the depth of the image.
[0106] Additional information regarding how the deep machine learning module 310 uses training data, uses the processor 235, functions as different types of machine learning models, etc., is provided above and will not be repeated here.
[0107] The video generator 312 generates the video from the first still image, the second still image, and the one or more intermediate images. In some embodiments, three or more frames are arranged in a sequence, where the first frame of the sequence is the first still image, the last frame of the sequence is the second still image, and each image of the one or more intermediate images is a corresponding intermediate frame of the sequence between the first frame and the last frame.
[0108] The user interface module 206 generates a user interface. In some embodiments, the user interface module 206 includes a set of instructions executable by the processor 235 to generate a user interface. In some embodiments, the user interface module 206 may be stored in memory 237 of the computing device 200 and accessible and executable by the processor 235.
[0109] Exemplary Flowchart 7A-7B are flow charts illustrating an example method 700 for generating video from candidate image pairs according to some embodiments. The method illustrated in flow chart 700 may be performed by computing device 200 of FIG.
[0110] Method 700 may begin at block 702 with generating a request for access to a media item collection associated with a user account. In some embodiments, the request is generated by the user interface module 206. Block 702 may be followed by block 704.
[0111] In block 704, a permission interface element is displayed. For example, the user interface module 206 may display a user interface that includes a permission interface element to request that a user provide access permissions to the media item collection. Block 704 may be followed by block 706.
[0112] At block 706, it is determined whether permission for access to the media item collection has been granted by the user. In some embodiments, block 806 is performed by the user interface module 206. If the user has not provided permission, the method ends. If the user has provided permission, block 708 may be performed after block 706.
[0113] Candidate image pairs are selected from the image collection at block 708. For example, the candidate pairs are selected as occurring within a limited time period, such as last week, this month, etc. Alternatively, the candidate pairs may be received after a user creates the candidate pairs from images acquired from the camera 243. Block 710 may be performed after block 708.
[0114] In block 710, it is determined whether a filter excludes the candidate image pair. The filter includes a time filter, a motion filter, etc. If the determination is YES, the candidate image pair is excluded. If the determination is NO, any remaining candidate image pair may be considered as the particular image pair. Block 712 may be executed after block 710.
[0115] In block 712, an image interpolator generates one or more intermediate images based on the particular image pair. Block 712 may be followed by block 714.
[0116] In block 714, it is determined whether the filter or image interpolator excludes one or more intermediate images. If the determination is YES, the one or more intermediate images are excluded. If the determination is NO, the one or more intermediate images are provided to a video generator. Block 716 may be executed after block 714.
[0117] At block 716, a video is generated that includes three or more frames arranged in a sequence, where a first frame of the sequence is a first image, a last frame of the sequence is a second image, and each image of the one or more intermediate images is a corresponding intermediate frame of the sequence between the first frame and the last frame.
[0118] In addition to the above, the system, program or functionality described herein may give the user control to choose whether and when to enable collection of user information (e.g., information about the user's social network, social behavior or activity, occupation, viewing preferences for image-based creations, user preferences such as settings for hiding people or pets, user interface preferences, or information about the user's current location) and to transmit content or information from the server. Furthermore, certain data may be processed to remove identifiable personal information in one or more ways before being stored or used. For example, the user's identity may be processed so that the user's personal information cannot be identified. Also, when obtaining location information (e.g., to the city, zip code, or state level), the user's geographic location may be generalized so that the user's location cannot be identified. Thus, the user may control the user information that is collected, the use of the information, and the information provided to the user.
[0119] In the above description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the various described embodiments. However, it will be apparent to those skilled in the art that the various described embodiments can be implemented without these specific details. In some cases, structures and devices are shown in block diagrams to avoid obscuring the description. For example, the embodiments may be described above primarily with reference to user interfaces and specific hardware. However, the embodiments may be applied to any type of computing device capable of receiving data and commands, and any peripheral device that provides services.
[0120] References herein to "some embodiments" or "some instances" mean that a particular feature, structure, or characteristic described in connection with the embodiments or instances may be included in at least one implementation of the description. The appearances of the phrase "in some embodiments" in various places in the specification do not necessarily all refer to the same embodiments.
[0121] Some portions of the above detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is herein generally conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated. It is convenient at times, primarily for reasons of common usage, to refer to these data as bits, values, elements, symbols, characters, terms, numbers, or the like.
[0122] It should be understood that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise noted, or as will be apparent from the discussion, throughout the description, discussions utilizing terms including "processing," "operating," "calculating," "determining," or "displaying," etc., refer to operations and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical quantities in the computer system memory, registers or other information storage devices, transmission devices, or display devices.
[0123] The embodiments herein also relate to a processor for performing one or more steps of the methods described above. The processor may be a dedicated processor selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on any type of disk including, but not limited to, optical disk, ROM, CD-ROM, magnetic disk, RAM, EPROM, EEPROM, magnetic or optical card, flash memory including USB key with non-volatile memory, or non-transitory computer readable storage medium including any type of medium suitable for storing electronic instructions, each connected to a computer system bus.
[0124] This specification may include some entirely hardware embodiments, some entirely software embodiments, or some embodiments containing both hardware and software elements, hi some embodiments, this specification is implemented in software, including but not limited to firmware, resident software, microcode, etc.
[0125] Furthermore, the descriptions may take the form of a computer program product accessible from a computer usable or computer readable medium providing program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer usable or computer readable medium may be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, machine, or device.
[0126] A data processing system suitable for storing or executing program code includes at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory used during the actual execution of the program code, mass storage devices, and cache memory that provides temporary storage of at least some of the program code to reduce the number of times the code must be retrieved from mass storage devices during execution.
Claims
1. 1. A computer-implemented method comprising: selecting candidate image pairs from an image collection associated with a user account, each candidate image pair including a first still image and a second still image from the user account, the method further comprising: applying a filter to select certain image pairs from the candidate image pairs, the filter excluding image pairs from the candidate image pairs that satisfy a certain condition, the method further comprising: generating one or more intermediate images based on the particular image pair using an image interpolator; generating a video comprising three or more frames arranged in a sequence, a first frame of the sequence being the first still image, a last frame of the sequence being the second still image, and each image of the one or more intermediate images being a corresponding intermediate frame of the sequence between the first frame and the last frame.
2. A computer-implemented method comprising: selecting candidate image pairs from an image collection associated with a user account, each candidate image pair including a first still image and a second still image from the user account, the method further comprising: applying a filter to select particular image pairs from the candidate image pairs, the filter including a temporal filter that excludes one or more of the candidate image pairs if a time difference between respective timestamps associated with the first still image of the candidate image pair and the second still image of the candidate image pair is greater than a time threshold; and the method further comprises: generating one or more intermediate images based on the particular image pair using an image interpolator; generating a video comprising three or more frames arranged in a sequence, a first frame of the sequence being the first still image, a last frame of the sequence being the second still image, and each image of the one or more intermediate images being a corresponding intermediate frame of the sequence between the first frame and the last frame.
3. each of the one or more intermediate images is associated with a respective timestamp having a value between a timestamp of the first still image and a timestamp of the second still image; The method of claim 2 , wherein the position of each intermediate image in the sequence is based on the respective timestamp.
4. The method of claim 2 or 3, wherein the time threshold is 2 seconds.
5. A computer-implemented method comprising: selecting candidate image pairs from an image collection associated with a user account, each candidate image pair including a first still image and a second still image from the user account, the method further comprising: applying a filter to select particular image pairs from the candidate image pairs, the filter including a motion filter that eliminates one or more of the candidate image pairs by estimating motion between the first still image and the second still image and determining that the motion between the first still image and the second still image is less than a minimum motion threshold, the method further comprising: generating one or more intermediate images based on the particular image pair using an image interpolator; generating a video comprising three or more frames arranged in a sequence, a first frame of the sequence being the first still image, a last frame of the sequence being the second still image, and each image of the one or more intermediate images being a corresponding intermediate frame of the sequence between the first frame and the last frame.
6. The method of claim 5 , wherein the filter further eliminates one or more of the candidate image pairs by determining that the motion between the first still image and the second still image exceeds a maximum motion threshold.
7. A computer-implemented method comprising: selecting candidate image pairs from an image collection associated with a user account, each candidate image pair including a first still image and a second still image from the user account, the method further comprising: applying a filter to select particular image pairs from the candidate image pairs, the filter comprising a filter machine learning module that excludes one or more of the candidate image pairs by generating feature vectors representing the first and second still images in each of the candidate image pairs and eliminating one or more of the candidate image pairs corresponding to each of the feature vectors if a distance between each of the feature vectors is greater than a threshold vector distance, the feature vectors being mathematical representations, the mathematical representations of similar images being closer in vector space than the mathematical representations of dissimilar images, the method further comprising: generating one or more intermediate images based on the particular image pair using an image interpolator; generating a video comprising three or more frames arranged in a sequence, a first frame of the sequence being the first still image, a last frame of the sequence being the second still image, and each image of the one or more intermediate images being a corresponding intermediate frame of the sequence between the first frame and the last frame.
8. The feature vector is a first feature vector, and the filter machine learning module is further operable to: The operation includes: receiving as an input an intermediate image of the one or more intermediate images; generating one or more second feature vectors corresponding to the intermediate images; and eliminating the one or more intermediate images corresponding to each of the feature vectors if a distance from each of the feature vectors of the intermediate images to each of the feature vectors of a corresponding candidate image pair is greater than the threshold vector distance.
9. 9. The method of claim 1, wherein the image interpolator includes an interpolation machine learning model that receives the first still image and the second still image as input and generates the one or more intermediate images.
10. A computer-implemented method comprising: selecting candidate image pairs from an image collection associated with a user account, each candidate image pair including a first still image and a second still image from the user account, the method further comprising: applying a filter to select particular image pairs from the candidate image pairs; and generating one or more intermediate images based on the particular image pair using an image interpolator, wherein generating the one or more intermediate images based on the particular image pair includes: generating a plurality of candidate intermediate images; and evaluating each candidate intermediate image. Evaluating each candidate intermediate image includes: generating a candidate video including the first still image as a first frame, the candidate intermediate image as a second frame, and the second still image as a third frame; selecting the candidate intermediate image as one of the one or more intermediate images if the candidate video does not include a frame interpolation failure, the method further comprising:
1. A method comprising: generating a video comprising three or more frames arranged in a sequence, a first frame of the sequence being the first still image, a last frame of the sequence being the second still image, and each image of the one or more intermediate images being a corresponding intermediate frame of the sequence between the first frame and the last frame.
11. 11. The method of claim 10, further comprising determining the frame interpolation failure using a classifier machine learning model trained to determine whether an input image is a generated image, and wherein the candidate intermediate image is selected if the classifier machine learning model determines that the candidate intermediate image is indistinguishable from the generated image.
12. determining that a frame interpolation failure occurs when the filter excludes the one or more intermediate images; The method of claim 7 or 8, further comprising: generating an additional one or more intermediate images in response to the occurrence of the frame interpolation failure.
13. A computer-implemented method comprising: selecting candidate image pairs from an image collection associated with a user account, each candidate image pair including a first still image and a second still image from the user account, the method further comprising: applying a filter to select particular image pairs from the candidate image pairs; generating one or more intermediate images based on the particular image pair using an image interpolator; generating a video including three or more frames arranged in a sequence, a first frame of the sequence being the first still image, a last frame of the sequence being the second still image, and each image of the one or more intermediate images being a corresponding intermediate frame of the sequence between the first frame and the last frame; The method, wherein generating the video includes generating a three-dimensional representation of a scene in the first still image based on a prediction of depth of the first still image using a depth machine learning model, the depth machine learning model being a classifier that receives the first still image as input, and the video including camera effects generated based on the three-dimensional representation of the scene.
14. The method of claim 13 , wherein the camera effects include at least one of a zoom, a pan, or a rotate.
15. A computer program which, when executed by one or more computers, causes the one or more computers to carry out the method of any one of claims 1 to 14.
16. one or more processors; A computing device comprising: a memory coupled to the one or more processors and storing a computer program according to claim 15.
17. 1. A system comprising:
15. A system comprising one or more processors and a memory storing computer readable instructions that, when executed by the one or more processors, cause the system to perform the operations recited in any one of claims 1 to 14.
Citation Information
Patent Citations
Method and apparatus for interpolating image
JP2003141559A
Image creation program and information processing device
JP2021010099A
Methods and systems for obtaining still images corresponding to video
US20120311623A1
Apparatus and methods for artifact detection and removal using frame interpolation techniques
US20180315174A1
Efficient CNN-based solution for video frame interpolation
US20200356827A1
