Systems and methods for a user interface for temporal data capture, data verification, and data synthesis

Abstract

An example method includes displaying, by an interactive graphical user interface (GUI) of a computing device, an image. The method also includes detecting, by the interactive GUI, user interaction with the displayed image, wherein the user interaction comprises modifying one or more image characteristics using at least one image adjustment control provided by the GUI. The method further includes generating, by the computing device and based on the user interaction, a synthetic sequence of temporal images that simulate a video based on the image. The method also includes providing, by the computing device, the synthetic sequence of temporal images.

Description

[0001] SYSTEMS AND METHODS FOR A USER INTERFACE FOR TEMPORAL DATA

[0002] CAPTURE, DATA VERIFICATION, AND DATA SYNTHESIS

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] [1] This present application claims priority to U.S. provisional application serial no. 63 / 587,606 filed October 3, 2023, and to U.S. provisional application serial no. 63 / 587,639 filed October 3, 2023, the full disclosures of which are incorporated herein by reference.

[0005] BACKGROUND

[0006] [2] Camera autofocus (AF) is a hybrid system that involves multiple modules, such as, for example, a phase-difference autofocus (PDAF) module, to achieve an optimal focus.

[0007] SUMMARY

[0008] [3] This application generally relates to providing an interactive user interface to facilitate temporal data capture, data verification, and data synthesis. Such data may be used to train machine learning models to achieve higher accuracies for AF functions.

[0009] [4] In one aspect, a computer-implemented method is provided. The method includes displaying, by an interactive graphical user interface (GUI) of a computing device, an image. The method also includes detecting, by the interactive GUI, user interaction with the displayed image, wherein the user interaction comprises modifying one or more image characteristics using at least one image adjustment control provided by the GUI. The method further includes generating, by the computing device and based on the user interaction, a synthetic sequence of temporal images that simulate a video based on the image. The method also includes providing, by the computing device, the synthetic sequence of temporal images.

[0010] [5] In a second aspect, a device is provided. The device includes one or more processors and data storage. The data storage has stored thereon computer-executable instructions that, when executed by one or more processors, cause the device to carry out functions. The functions include: displaying, by an interactive graphical user interface (GUI) of a computing device, an image; detecting, by the interactive GUI, user interaction with the displayed image, wherein the user interaction comprises modifying one or more image characteristics using at least one image adjustment control provided by the GUI; generating, by the computing device and based on the user interaction, a synthetic sequence of temporal images that simulate a video based on the image; and providing, by the computing device, the synthetic sequence of temporal images. [6] In a third aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to carry out functions. The functions include: displaying, by an interactive graphical user interface (GUI) of a computing device, an image; detecting, by the interactive GUI, user interaction with the displayed image, wherein the user interaction comprises modifying one or more image characteristics using at least one image adjustment control provided by the GUI; generating, by the computing device and based on the user interaction, a synthetic sequence of temporal images that simulate a video based on the image; and providing, by the computing device, the synthetic sequence of temporal images.

[0011] [7] In a fourth aspect, an article of manufacture is provided. The article of manufacture includes one or more computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a device, cause the device to carry out functions. The functions include: displaying, by an interactive graphical user interface (GUI) of a computing device, an image; detecting, by the interactive GUI, user interaction with the displayed image, wherein the user interaction comprises modifying one or more image characteristics using at least one image adjustment control provided by the GUI; generating, by the computing device and based on the user interaction, a synthetic sequence of temporal images that simulate a video based on the image; and providing, by the computing device, the synthetic sequence of temporal images.

[0012] [8] In a fifth aspect, a system is provided. The system includes means for displaying, by an interactive graphical user interface (GUI) of a computing device, an image; means for detecting, by the interactive GUI, user interaction with the displayed image, wherein the user interaction comprises modifying one or more image characteristics using at least one image adjustment control provided by the GUI; means for generating, by the computing device and based on the user interaction, a synthetic sequence of temporal images that simulate a video based on the image; and means for providing, by the computing device, the synthetic sequence of temporal images.

[0013] [9] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the figures and the following detailed description and the accompanying drawings. BRIEF DESCRIPTION OF THE FIGURES

[0014]

[0010] FIG. 1 is a block diagram illustrating data capture and synthesis, in accordance with example embodiments.

[0015]

[0011] FIG. 2 illustrates an example lighting panel to adjust a lux level, in accordance with example embodiments.

[0016]

[0012] FIG. 3 is a table illustrating example calibrated exposure times and gains per lux and ISO level, in accordance with example embodiments.

[0017]

[0013] FIG. 4 is a table illustrating an example temporal focus sweep dataset, in accordance with example embodiments.

[0018]

[0014] FIG. 6 illustrates example histograms for different signal-to-noise ratio (SNR) levels, in accordance with example embodiments.

[0019]

[0015] FIG. 6 illustrates example unpacked images, in accordance with example embodiments.

[0020]

[0016] FIG. 7 illustrates example unpacked images, in accordance with example embodiments.

[0021]

[0017] FIG. 8A is an example of an interactive graphical user interface (GUI), in accordance with example embodiments.

[0022]

[0018] FIG. 8B is another example of an interactive graphical user interface (GUI), in accordance with example embodiments.

[0023]

[0019] FIG. 8C is another example of an interactive graphical user interface (GUI), in accordance with example embodiments.

[0024]

[0020] FIG. 9A illustrates an example output of a PD calibration tool, in accordance with example embodiments.

[0025]

[0021] FIG. 9B illustrates example graphical representations of outputs of a PD calibration tool, in accordance with example embodiments.

[0026]

[0022] FIG. 9C illustrates an example output displaying disparity values per VCM, in accordance with example embodiments.

[0027]

[0023] FIG. 10 illustrates example phase-difference (PD) frames with corresponding mask metadata, in accordance with example embodiments.

[0028]

[0024] FIG. 11 illustrates example ground truth disparity values, in accordance with example embodiments.

[0029]

[0025] FIG. 12 is a diagram illustrating training and inference phases of a machine learning model, in accordance with example embodiments.

[0030]

[0026] FIG. 13 depicts a distributed computing architecture, in accordance with example embodiments.

[0027] FIG. 14 is a block diagram of a computing device, in accordance with example embodiments.

[0031]

[0028] FIG. 15 depicts a network of computing clusters arranged as a cloud-based server system, in accordance with example embodiments.

[0032]

[0029] FIG. 16 is a flowchart of a method, in accordance with example embodiments.

[0033] DETAILED DESCRIPTION

[0034] Overview

[0035]

[0030] Camera autofocus (AF) is a hybrid system that involves multiple modules to achieve an optimal focus. Such modules are configured to work together to leverage the capturing conditions for more accurate focus parameters. Camera AF may employ three modules: phasedifference autofocus (PDAF), contrast-detection autofocus (CDAF), and time-of-flight (ToF). Each algorithm has its own advantages and disadvantages, and when properly tuned, they may work together to improve the overall camera AF.

[0036]

[0031] PDAF may be a primary AF algorithm because of efficiency and accuracy. Generally, PDAF is based on one frame to estimate a target lens position that achieves a target focus. However, PDAF may be unreliable in challenging conditions, such as extremely low-light scenes. In such situations, the AF system may leverage CDAF or ToF, depending on the design tuning and capturing conditions. CDAF may be more accurate than PDAF under certain conditions, but it can be slower because it performs an image focus search that requires capturing multiple images at different focus distances to determine a target focus distance. In the event that CDAF is unreliable, ToF can be a complementary system for both PDAF and CDAF. ToF emits an actual signal to the scene and measures the round-trip time to obtain the object distance and then determine a target focus. ToF can be highly accurate under normal conditions, but it has a limited distance range and may sometimes be unreliable in bright scenes.

[0037]

[0032] In such a hybrid configuration, algorithm evaluation and testing can be useful for each module in order to utilize the relative strengths. Additionally, evaluation may be useful for improvement and for determining optimal tuning parameters. Single image focus evaluation may be performed through capturing a focus sweep of a textured chart under good conditions and then determining the ground truth by running any of the aforementioned algorithms.

[0038]

[0033] Determining ground truth for some scenes, such as extreme low light, multiple depth scenes, scenes that lack adequate texture, and so forth, may be challenging. In addition to photo evaluation settings, such issues may arise when capturing data for video AF and obtaining ground truth for video data. Video AF data with dynamic ground truth for each frame can be extremely challenging to collect. This is because collection of such data may depend on obtaining a focus sweep at each video frame, and camera setups are not configured to facilitate synchronization in real time, as capturing a focus sweep takes time and involves moving the lens. Developing an AF video evaluation framework with accurate AF ground truth can be challenging for several reasons:

[0039]

[0034] Cameras are not configured to provide a focus sweep for each video frame: This can make it highly challenging to capture such data directly. One approach may be to use a stopmotion approach for a static scene, capturing a focus sweep and then moving the scene slightly. However, such an approach can be time-consuming, and the output video may not be smooth and may lack realistic frame rate granularity.

[0040]

[0035] Using an assistant depth sensor holds multiple challenges: obtaining accurate ground truth data may be hindered because of accumulated errors at each data preparation stage.

[0041]

[0036] Data collection: collecting real-world data is generally challenging because it must be large-scale (e.g., at least 10k videos) and include a wide variety of scenes, objects, conditions, camera parameters, and so on. This can involve significant human effort, especially for large projects like camera AF projects.

[0042]

[0037] Accordingly, there is a need to solve the aforementioned technical challenges of data collection, data verification, and data synthesis. As described herein, a comprehensive framework may be configured for temporal data capture, verification, ground truth preparation, and data synthesis for video AF evaluation and model training. For example, a data capture procedure that captures a variety of representative camera capture settings, including multiple verification procedures and ground truth disparity calculation, is provided. Also, for example, a tool for manual data verification to generate accurate ground truth is described. As another example, a temporal data synthesis framework is described that may be used to generate and verify temporal testing and training data.

[0043]

[0038] In some aspects, a generic framework for evaluating and training video AF algorithms is described. This data framework is a temporal data generation engine for evaluating and training the temporal dual-depth (TDD) model. Training a temporal machine learning model based on temporal data with accurate ground truth may be achieved using the data synthesized as described herein. PDAF output may be significantly improved. The framework does not depend on additional hardware and / or additional cameras. Additionally, the data capture process is automated, which minimizes human involvement and effort. Furthermore, with the data synthesis module, an unlimited number of samples with accurate ground truth may be generated. This can result in a significant improvement in model accuracy and eliminates a need for directly collecting a large dataset.

[0044]

[0039] FIG. 1 is a block diagram 100 illustrating data capture and synthesis, in accordance with example embodiments. Data capture 105 can involve unpacking 110, scaling up for visualization, and histogram analysis. Data tensor 115 can involve creating a large tensor that includes the data for faster browsing. Verification 120 may be performed by visually inspecting the data via an interactive sweep data browser.

[0045]

[0040] Ground truth disparity 125 may involve a calibration tool 130 to determine a slope and a cross ambiguity function (CAF) peak per distance. Ground truth verification 135 may be performed through plots. A disparity per VCM / distance 140 may be determined.

[0046]

[0041] Temporal data synthesis 145 may involve temporal dual-depth (TDD) or temporal single-depth (TSD) synthesis 150. Verification 155 may involve checking the generated videos visually. Verification 160 may involve checking the generated data size and count.

[0047] Data capture

[0048]

[0042] Some embodiments involve collecting a set of 8k / 4k images of random textured high- contrast single-depth patterns. Finalized printing details for a poster may involve single sided color, styrene material, with a matte finishing and a size of 48x27 inches.

[0049]

[0043] In some embodiments, a variety of scenes may be captured at different conditions and optical image stabilization (OIS) positions. For example, scenes may be captured at different lux levels and signal to noise ratio (SNR) levels by adjusting the exposure and gain for each capture. Also, for example, focus sweeps may be captured at multiple OIS locations to obtain a variety of optical distortions and radial gain patterns.

[0050]

[0044] In some embodiments, a focus sweep may be captured by sweeping the lens from 0 to 324 logical lens positions in steps of 6. A total of 108 images may be captured per sweep, consisting of 54 PD interleaved views and 54 RAW Bayer images. Generally, RAW Bayer images include a single color per pixel. Both PD data and RAW Bayer data may be captured. Additionally, a focus sweep may be captured for various SNR levels, OIS positions, and distances. For example, a focus sweep may be captured for six SNR levels and four OIS positions, composing a total of 24 sweeps per distance per scene. Each scene may be captured at four distances: 080cm, 120cm, 360cm, and 560cm; and a laser meter may be used to measure distance between the camera and the poster. In some embodiments, 20 scenes may be captured, totaling 311,040 images. Although the captured full-frame images may be about 300k, millions of training batches may be extracted because PDAF uses a region of interest (ROI) to determine disparity, instead of the full frame.

[0045] In some embodiments, the SNR levels may be calibrated to reflect an equivalent lux level and gain. This may be performed by histogram matching, which matches lux / gain with exposure time to define the SNR levels. In some embodiments, the lab lighting and lux level may be fixed for data capture (e.g., 1420 lux at the poster plane). For example, the lux level may be adjusted by controlling the lighting panel.

[0051]

[0046] FIG. 2 illustrates an example lighting panel to adjust a lux level, in accordance with example embodiments. First lighting panel 205 corresponds to a high light setting at 1420 lux. Second lighting panel 210 corresponds to a low light setting at 50 lux. Third lighting panel 215 corresponds to an extremely low light setting at 5 lux.

[0052]

[0047] FIG. 3 is a table 300 illustrating example calibrated exposure times and gains per lux and ISO level, in accordance with example embodiments. For example, the top row 305 corresponds to the lux levels illustrated in FIG. 2. Clean data configurations for each of the lux levels are provided in second row 310. Noisy data configurations for each of the lux levels are provided in third row 315.

[0053]

[0048] In some embodiments, an Octopus Python script may be configured to control the camera. For the OIS position, the sensor may be allowed to move along the x and y axis in integer linear steps from 0 to 15. In some embodiments, the first four OIS positions may be fixed and predefined as (7,7), (8,7), (7,6), and (8,6). The next four positions may be randomly selected within a distance range from the current OIS position. The distance range may be determined based on a distance between the camera and the poster. Data may be captured at multiple OIS positions to mimic real-time OIS movements and a variety of optical radial distortions and gain maps may be obtained. For each focus sweep, the sensor temperature may be stored to investigate potential temperature effects on lens position drift.

[0054]

[0049] In some embodiments, the selection of posters may be based on a variety of edge patterns, rather than scene content and semantics. The disparity of PD sensors is mainly determined by the edges and boundaries of the captured images. For PD sensors, edges are the meaning of scene semantics. Another reason for such selection may be to avoid confusion for a machine learning model to be trained on the data. For example, edge patterns can prevent models from learning depth semantics from single depth data.

[0055]

[0050] FIG. 4 is a table illustrating an example temporal focus sweep dataset, in accordance with example embodiments. Thumbnail images of each scene are displayed along with capturing details such as OIS positions, distances, SNR levels, a total number of sweeps, and a total number of images. Twenty example patterns are illustrated in the table.

[0056] Data verification Unpacking

[0057]

[0051] Once the data is captured, the captured data may be verified so as to meet the intended capturing settings (e.g., correct voice coil motor (VCM), gain, exposure, OIS, and scene, and so forth). To do so, the binary files for the PD data, and the RAW Bayer image data may be unpacked. The term “unpacking” as used herein, generally refers to converting an image from bits to an image format in bytes by taking the raw binary data (the bits) that represent the image and interpreting it according to a specific image file format (e.g., JPEG, PNG, or GIF). This interpretation structures the data into a format that image viewers and editors can understand. A first check intensity histogram of the captured SNR levels may be applied.

[0058]

[0052] FIG. 6 illustrates example histograms 600 for different signal-to-noise ratio (SNR) levels, in accordance with example embodiments. Each histogram comprises intensity values along the horizontal axis, and counts along the vertical axis. Also, for example, the higher gain (ISO) decreases from the histograms in the top row to the histograms in the bottom row. As another example, the exposure increases for histograms arranged left to right.

[0059]

[0053] In some embodiments, a histogram analysis may be performed. Histogram analysis helps ensure that the pixel intensities are distributed around an expected mean. A histogram essentially counts the number of pixels at each discrete intensity level / range, representing an intensity distribution of an image. By plotting the histogram, it may be determined as to whether the distribution is as expected. In addition to histogram analysis, images may be stored in .png format with scaling up the intensity for visual assessment. In some embodiments, both the left and right PD views may be stored, as well as averaged green channels, as single-channel images.

[0060]

[0054] FIG. 6 illustrates example unpacked images 600, in accordance with example embodiments.

[0061]

[0055] FIG. 7 illustrates example unpacked images 700, in accordance with example embodiments.

[0062] Preparing data for data browser

[0063]

[0056] Data verification may be performed subsequent to data being unpacked and saved as .png files. However, given the large number of images (e.g., more than 300k images), it may be challenging to browse through every image to make sure that the data has been captured correctly. Accordingly, an interactive tool may be configured and provided that enables users to browse the data (e.g., with an interactive slider). Such a slider may be used to browse through the focus sweeps, change the SNR level, and / or a distance, for each scene.

[0057] In some embodiments, the scene data may be stored in a single, higher-dimensional tensor. This allows for efficient browsing of the data, as an entire scene may be loaded into the main memory. This can facilitate real-time and smooth browsing.

[0064] Interactive data browser

[0065]

[0058] FIG. 8A is an example of an interactive graphical user interface (GUI) 800A, in accordance with example embodiments. Some embodiments involve displaying, by an interactive graphical user interface (GUI) of a computing device, an image. For example, a green channel preview 805 of the image may be displayed. One or more image adjustment controls may be provided by the GUI. For example, a focus sweep slider 810 and an OIS position slider 815 may be provided. Also, for example, a first contrast autofocus (CAF) 820 for a green channel and a second CAF 825 for the PD data may be displayed. Additional information may be provided, such as a PD data preview 830, alternate PD data 835, scene list 840, distance 845, and / or SNR level 850.

[0066]

[0059] The focus sweep data may be verified by sliding the focus sweep slider 810. OIS position slider 815 may be used to verify that images have been captured at the appropriate OIS position. The green channel preview 805 and PD data 835 may be updated as a user adjusts the image.

[0067]

[0060] Also, for example, the gradient energy value along with the CAF curves for both the green channel and PD data may be provided at each VCM position. These curves may be useful for verifying the data, as a single-peak curve is expected at the sharpest focus slice in the sweep.

[0068]

[0061] As previously described, each scene may be captured at six SNR levels and four distances. Accordingly, the data may be verified at each SNR level 850 and for each distance 845. This may be achieved by selecting the distance by clicking the distance button 845, and the previews may be updated with the new distance data.

[0069]

[0062] SNR levels 850 may be displayed as calibrated lux levels, which may be controlled by exposure time and gain, as illustrated in table 300 of FIG. 3. With three lux levels and two ISO (or noise) levels, a combination of these may create an SNR level 850, which is displayed on the preview.

[0070]

[0063] FIG. 8B is another example of an interactive graphical user interface (GUI) 800B, in accordance with example embodiments. GUI 800B illustrates an instance of an animated change of SNR level for a given distance, VCM position, and OIS location.

[0071]

[0064] Referring again to FIG. 8A, the dynamic and interactive GUI 800A allows for efficient data verification prior to preparing ground truth disparity and data synthesis. The PD disparity may be verified by alternating the PD views at a fixed lens position. PD data may be verified by selecting the “Swap PD” button under the PD data preview 830 to alternate the views and ensure maintaining of desired PD disparity according to the defocus value. Generally, there is a correlation between disparity and defocus, or circle of confusion (CoC) size. The disparity is generally larger when the defocus is large and may vary per VCM position.

[0072]

[0065] FIG. 8C is another example of an interactive graphical user interface (GUI) 800C, in accordance with example embodiments. GUI 800B illustrates an instance of an animated view illustrating that the disparity is larger when the defocus is large and can vary based on VCM position.

[0073]

[0066] The scenes may be selected using the “Scene” button that displays a scene list 840. By selecting a scene (e.g., from a drop-down menu) and clicking the button “Run”, a new scene may be loaded. Scenes may be verified by (1) selecting a scene to be verified. Subsequently, (2) a distance, lux, and ISO may be selected by clicking respective buttons. (3) Various combinations of distance, lux, and ISO may be verified for accuracy. In the example illustration, this may correspond to a total of 4 * 3 * 2 = 24 sweeps. (4) For each combination, the respective CAF curve displayed may be verified so that unusual spikes do not appear. Generally, at a desired SNR level, CAF curve is likely to have a single peak, as it is a single depth scene. Also, for example, the data may be verified by sliding the focus sweep slider 810 so that there are no abrupt image focus changes. For example, it is desirable that the focus changes smoothly and gradually. The PD defocus disparity at the PD data preview 830 may be randomly verified by alternating the views. (5) Steps (2) and (3) may be repeated for each OIS position. (6) Steps ( 1 )-(4) may be repeated for each scene.

[0074] Ground truth disparity

[0075]

[0067] In some embodiments, ground truth (GT) disparity may be determined (e.g., subsequent to data capture) based on the VCM position and the scene depth. For example, a disparity intercept (peak CAF), slope, and offset may be determined for a given focus sweep (e.g., by using a PD calibration tool). Multiple focus sweeps may be captured under good lighting conditions at a fixed distance for high contrast ROI. An average value may be determined across multiple sweeps for a more accurate GT. In some embodiments, scenes with the highest SNR may be selected from each scene and the GT disparity may be determined for four distances (80 cm, 200 cm, 360 cm, and 560 cm).

[0076] PD calibration tool

[0077]

[0068] In some embodiments, in order to run a PD calibration tool, the calibration parameters may be provided in a proto file. The calibration parameters may include tile rows, tile cols, and ml model. Tile rows and tile cols are respectively the number of rows and columns used to divide an image into a grid. In some embodiments, tile rows may be set to 4 and tile cols may be set to 3 (e.g., a 3x4 grid) for a higher SNR for each tile, and center tiles may be used for GT determination as they have a lower disparity offset. The ml model may be used to specify a machine learning (ML) based disparity model. For example, one model may be based on a Block Matching Algorithm (BMA), and a second model may be the learning-based model for disparity calculation. The calibration parameters may also include sensor_params.

[0078]

[0069] Running the calibration tool and specifying the parameters may generate an output sheet with the values for each tile per VCM.

[0079]

[0070] FIG. 9A illustrates an example output 900A of a PD calibration tool, in accordance with example embodiments. For example, values for each tile per VCM are displayed as columns A-H, with tilelD, VCM position, disparity, confidence, defocus, CAF, CAF x, and CAF_y.

[0080] The VCM values may be 0 and 6.

[0081] PD ground truth verification

[0082]

[0071] In general, it may be challenging to verify the output values in the sheet (illustrated in FIG. 9A) by visual inspection. One verification approach is to plot the values for each tile and visually inspect the plots instead. The VCM with a peak CAF is likely to correspond to a sharpest slice in the focus sweep.

[0083]

[0072] FIG. 9B illustrates example graphical representations 900B of outputs of a PD calibration tool, in accordance with example embodiments. As illustrated in the plots, the curves appear normal and appear to have a linear disparity per tile.

[0084]

[0073] After verifying the intercept, slope, and offset, outliers may be filtered out for the intercept, slope, and offset by running a median filter with a filter threshold that may be defined based on the value range. After filtering out the outliers using the median filter, an average value for each intercept, slope, and offset may be determined. A disparity equation may be used to calculate the disparity for each VCM per distance. current vcm ■■■ intercept + offse disparity slope

[0085] (Eqn. 1)

[0074] Another approach to determine the GT disparity (e.g., for large intercept variation), may be to use a data browser tool to select the CAF peak and confirm the GT disparity visually for each distance. Eqn. 1 may be simplified to current vcra - vcra at caf eak disp rarit Jy = - slope

[0086] (Eqn. 2)

[0075] Subsequent to determining and verifying the disparity ground truth (GT) for each distance, another output sheet may be generated for each distance that contains the disparity per VCM.

[0087]

[0076] FIG. 9C illustrates an example output 900C displaying disparity values per VCM, in accordance with example embodiments.

[0088] Temporal Data Synthesis

[0089]

[0077] Subsequent to data capture and verification, along with determination of the ground truth disparities, temporal data may be synthesized (e.g. training data for ML model training). As described herein, although the captured data involved a single depth data set, multiple-depth data may be synthesized from the dataset. The dual-depth data may be referred to as temporal dual-depth (TDD) data. Also, for example, temporal single-depth (TSD) data may be synthesized. For illustrative purposes, TDD data synthesis is described herein, as a TSD dataset is a subset of a TDD dataset.

[0090]

[0078] In general, synthesizing temporal data may not be straightforward as additional dimensions (e.g., the time dimension, temporal dynamics, how the spatial dimensions at a single frame are going to evolve overtime, etc.) may be involved.

[0091] Simulating lens position change and scene motion

[0092]

[0079] Simulation of lens movement, focus blur, and focus breathing (which changes the FoV) may be used for synthesizing realistic temporal sequences. Object and camera motion may be imported to mimic real-world dynamics captured through cameras. In general, the TDD data lens movements and motion synthesis steps are summarized herein.

[0093]

[0080] A starting frame at a certain VCM position may be selected. A target lens position with zero disparity (the focal plane to be at the scene depth) may be selected so that there is a sufficient delta lens difference between the starting position and the target lens position. For example, a zero disparity of a single depth plane at 80cm may be at a VCM of 66 based on the ground-truth. The possible starting point may be selected to be 66 + / - delta. The value of step size delta may be set to be at least 30 logical lens positions (i.e., 5 steps from the target). The goal is to traverse the lens position from the starting point to the zero disparity lens position and capture frames at different lens positions as a sequence. Such a sequence can reflect the temporal dynamics of PDAF systems, in which the lens keeps moving until reaching the zero disparity lens position. Some related parameters may be a max disparity margin that indicates a maximum lens travel range allowed, and repeat fr that indicates a number of repeated frames of the same lens position.

[0081] In some embodiments, patches may be cropped from the entire image frame. Such patches represent the AF ROI. In some embodiments, a spatial shift of the cropped patch and a random change in the size may be introduced. The ROI shift can simulate motion in real scenes and a change in size can reflect the ROI size dynamics (e.g., a change in face size). Data may be synthesized with a fixed default ROI size and touch ROI. Some related parameters may include ROI size, ROI center shift, and drawing ROI for visualization.

[0094]

[0082] For the ROI size, default roi, may be a Boolean flag indicating a fixed default ROI size. Also, for example, roi size ratio change may indicate a range of an initial ROI size ratio with respect to the entire frame size. The initial ROI size ratio may be uniformly drawn randomly from a range for each sequence. As another example, roi h offset range may indicate a width offset range with respect to height in order to avoid a fixed aspect ratio. This may also be randomly picked for the initial ROI size of each sequence. Also, for example, roi aspect ratio range may indicate an allowed range of the width / height aspect ratio range. As another example, roi change range may indicate a range of the change percentage of the ROI size between the consecutive frames. The change between the frames may be randomly picked.

[0095]

[0083] For the ROI center shift, roi_pixel_shift_range may indicate an upper limit of pixels to shift between the consecutive frames. The shift amount may be uniformly randomly drawn from the range. The roi_pixel_shift_range may also be depth-dependent. For example, the upper limit may be lower for objects that are farther away, as closer objects may maintain a larger shift due to motion parallax. Another ROI center shift parameter may be a pixel drift threshold indicating a maximum allowed number of pixels that can drift from the initial center. The pixel drift threshold may be introduced to avoid the ROI from continuously being shifted and changing dramatically from the original.

[0096]

[0084] For drawing ROI for visualization, a parameter roi color may indicate a color of the ROI bounding box. This parameter may be used to draw the ROI for visualization and verification purposes. Another parameter may be roi thickness that indicates a thickness of a ROI bounding box.

[0097]

[0085] In some embodiments, data synthesis may involve a temporal VCM sweep, ROI shift, and / or a size change. The ROI can be dynamic, and the VCM may change until it reaches a zero disparity lens position, which makes the ROI sharper and in focus. In some embodiments, stacked red-green-green-blue (RGGB) color channels may be used for visualization purposes as the PD data has a lower resolution and limited color channels. However, the PD views may be synthesized accordingly by projecting the VCM and ROI coordinates to a sparse PD space.

[0086] As described herein, data may be captured at multiple OIS positions and an OIS change may be introduced in the data synthesis process. In addition to increasing dataset size, multiple OIS positions can simulate different radial distortion patterns by changing the relative sensor location with respect to the optics. Additionally, in real camera systems, the OIS position is also moving for video stabilization purposes. Accordingly, the multiple OIS positions enable simulation of realistic video data. For example, the OIS position may be changed to a random position based on a certain VCM change rate.

[0098]

[0087] Some of the OIS motion parameters may be ois that indicate predefined OIS coordinates. For example, fixed OIS positions such as (7,7), (8,7), (7,6), and (8,6) may be used. Another parameter may be ois change rate indicating a rate of OIS change based on the number of VCM position changes. Also, for example, another parameter may be ois dis shift indicating a desired compensation for pixel shift due to OIS change (e.g., to avoid a significant shift in the ROI content, making the data unrealistic).

[0099]

[0088] In some embodiments, temporal PD sequences may be generated by incorporating OIS position change. As described, stacked RGGB may be used for visualization. As the VCM and ROI change, the OIS also changes, simulating a behavior of real camera systems during AF convergence.

[0100] Simulating motion blur, exposure change, and noise

[0101]

[0089] In addition to simulating ROI and OIS dynamics, additional and / or alternative image degradations may be synthesized to further simulate the real dynamics of a camera in temporal sequences.

[0102]

[0090] For example, an image degradation corresponding to a continuous exposure change over time may be introduced. This may be represented by changes in brightness and / or noise levels. Also, for example, an image degradation related to scene and / or camera motion may be introduced to simulate motion blur.

[0103]

[0091] For example, motion blur may be added based on a direction and magnitude of the ROI and OIS motion. A 2D linear motion blur kernel may be constructed, and the input image may be convolved with the constructed kernel. The motion blur model may be determined as follows:

[0104] I* = K

[0105] (Eqn. 3)

[0106]

[0092] where I is an input clean image, I* is a blurry image, K is a square 2D linear kernel, and the dimension (or magnitude) may be determined based on a ROI shift amount in pixels between consecutive frames. A related parameter may be add motion blur that is a Boolean flag to simulate motion blur.

[0107]

[0093] In some embodiments, exposure change may be introduced. For example, an intensity value may be multiplied by a brightness scale (e.g., that may be randomly picked). Related parameters include brightness change that is a Boolean flag to enable brightness change. Another parameter may be roi bright change ratio indicating a lower bound of a linear intensity scaling.

[0108]

[0094] Some embodiments involve adding Gaussian noise (e.g., Heteroscedastic Gaussian noise) including shot and read noise. The noise model may be described as follows:

[0109] P = I + S O I + R

[0110] (Eqn. 4) S~ A(g = 0, <JS)

[0111] (Eqn. 5) / ?~ A(g = 0, <JR)

[0112] (Eqn. 6)

[0113]

[0095] where I is an input clean image, / * is a noisy image, and S and R are the shot and read noise, respectively. Both S and R may follow a zero-mean normal distribution. O represents the element wise multiplication operator. The code related parameters may include add noise that is a Boolean flag to enable adding noise, noise_per_frame_change that is a Boolean flag to change a noise distribution for consecutive frames. When this parameter value is False, the same initial random distribution is maintained and a noise sample is randomly drawn for each frame. Another parameter may be shot noise range that indicates a range of as, and read noise range that indicates a range of oR.

[0114]

[0096] With image degradation and ROI / OIS dynamics incorporated, the domain gap may become smaller between synthesized temporal data and the actual data sequences in real cameras.

[0115] Synthesizing dual-depth

[0116]

[0097] As previously described, a technical solution to an AF problem for multi-depth cases is provided herein. However, single-depth data may also be captured and synthesized. For illustrative purposes, synthesizing dual-depth temporal sequences from the single-depth data is described herein.

[0117]

[0098] In some embodiments, a dual-depth frame may be synthesized by concatenating two depth layers into a single frame. This may be achieved by splitting the frame into two equal regions, where each half is from a different depth. A simplified case, for example, may have a background (BG) (left-hand side) at 560 cm and a foreground (FG) (right-hand side) at 200 cm. The single-depth temporal sequence synthesis described in the previous sections may be applied to each layer independently.

[0118]

[0099] However, the above example may be a simplified example compared to real-world scenarios, where arbitrary-shaped depth layers may appear. Also, for example, the depth layers may not be connected and may have arbitrary area ratios. In addition to spatial simplification, there may be temporal dynamics, such as where to converge (e.g., coverage AF on FG or BG) and how to introduce dynamics for a FG mask. To further bridge the gap with real -world multidepth data, a segmentation mask dataset may be used. For example, a SYNDOF segmentation mask dataset may be used. The SYNDOF dataset provides realistic arbitrary-shaped masks of real objects (e.g., human, face, flower, vehicle, etc.). The SYNDOF dataset provides binary masks of objects segmented based on depth, and is useful for defocus detection purposes. The FG mask in the SYNDOF dataset may not be centered and may not provide the FG center or FG area percentage. Accordingly, the data may be processed for higher control of the data synthesis process. The mask may be center cropped based on FG data. The FG pixel percentage may be determined and categorized based on FG ratio.

[0119]

[0100] In some embodiments, a randomly selected mask with a specific ratio may be assigned to the FG based on a depth layer order, and the rest of the frame may be associated with the BG. A new mask dynamics may be introduced, where the mask dynamics are independent of the ROI and OIS dynamics. This may enable random scaling, translation, and rotation of the mask within the ROI. For temporal convergence, three example scenarios may be considered: (1) the initial lens position is beyond the BG, which makes both the FG and BG have negative disparities; (2) the initial lens position is between the FG and BG, one would be positive and the other would be negative disparity; and (3) the initial lens position is in front of the FG, which makes both the FG and BG disparity positive.

[0120]

[0101] In some embodiments, a random initial lens position may be introduced that covers the three cases with enough delta lens distance (e.g., between an initial lens position and a zero disparity lens position) to generate sufficiently long temporal sequences. In AF, one approach may be to prioritize FG focus over BG. Accordingly, sequences that are converging to FG zero disparity may be generated. Synthesizing dual-depth data may be performed using parameters such as described below.

[0121]

[0102] A mask translation range parameter may be a range of allowed translation in pixels, both vertically and horizontally. The translation amount may be randomly picked between two consecutive frames.

[0103] A mask rotation range parameter may be a range of allowed rotation in pixels clockwise and counterclockwise. The rotation amount may be randomly picked between two consecutive frames.

[0122]

[0104] A mask scale range parameter may correspond to a scaling range to scale the mask either up or down. This may be randomly picked between two consecutive frames.

[0123]

[0105] A mask non center parameter may be a Boolean variable to enable saving noncentered FG masks.

[0124]

[0106] A mask non center ratio parameter may be a range of the FG mask initial center away from the ROI center to allow non-centered FG mask. The FG mask center may be initialized randomly.

[0125]

[0107] A fg ratio parameter may indicate a range of the FG mask ratio with respect to the BG. For each sequence, the starting FG ratio may be picked randomly.

[0126]

[0108] A save mask parameter may be a Boolean flag to enable saving the mask metadata for each frame in the sequence, including the mask itself, the (x, y) coordinates of the FG mask center, and the FG ratio.

[0127]

[0109] FIG. 10 illustrates example phase-difference (PD) frames 1000 with corresponding mask metadata, in accordance with example embodiments. For example, FIG. 10 illustrates an example of a PD frame with its corresponding mask metadata at different frames in the sequence.

[0128] [HO] In some embodiments, an animation may be generated to illustrate an example of a TDD sequence with the synthesized dual-depth, along with the ROI, OIS, and FG mask dynamics. Generally, the FG may be continuously scaling, translating, and rotating, while the entire ROI may be changing dynamically and independently.

[0129] Synthetic TDD ground truth, visualization, & verification

[0130] [Hl] Each TDD synthesized sequence may be saved in a folder with a corresponding ground truth disparity file.

[0131]

[0112] FIG. 11 illustrates example ground truth disparity values 1100, in accordance with example embodiments. As illustrated, the columns in a saved ground truth table may include a frame indicating a frame number, VCM indicating a VCM position, disparity _0 indicating FG disparity, disparity l indicating BG disparity, pd roi start (x,y) indicating the upper left corner of the ROI from the entire PD frame, pd roi end (x,y) indicating the lower right corner of the ROI from the entire PD frame.

[0113] After TDD data is synthesized and ground truth is generated, it is desirable to verify that the synthesized data is accurate and is aligned with the ground truth disparity. This verification may be performed in two stages.

[0132]

[0114] For example, an RGB video of the PD sequence may be saved, where each frame corresponds to a synthesized PD frame. The video may be visually inspected to ensure that the colored high-resolution sequence is running as intended. In some embodiments, Bayer RGGB pixels may be used as they have a higher resolution and provide colored information for better visual assessment and verification. This may be achieved by averaging the two green channels and stacking the R, averaged G, and B channels as RGB image frames.

[0133]

[0115] Also, for example, the PD data may be packed into a binary .bin file. Each frame may have a different width and height when the default roi is disabled. The PD views may be unpacked and their per-frame coordinates may be used to ensure that a width times a height is equal to a size in bytes. It may be verified that a number of frames matches a number of folders saved. This may be iterated through the frames, unpacking them and ensuring that a size and dimension match a size in the GT sheet.

[0134] Training Machine Learning Models for Generating Inferences / Predictions

[0135]

[0116] FIG. 12 shows diagram 1200 illustrating a training phase 1202 and an inference phase 1204 of trained machine learning model(s) 1232, in accordance with example embodiments. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions about (patterns in the) training data. The resulting trained machine learning algorithm can be termed as a trained machine learning model. For example, FIG. 12 shows training phase 1202 where one or more machine learning algorithms 1220 are being trained on training data 1210 to become trained machine learning model(s) 1232. Then, during inference phase 1204, trained machine learning model(s) 1232 can receive input data 1230 and one or more inference / prediction requests 1240 (perhaps as part of input data 1230) and responsively provide as an output one or more inferences and / or prediction(s) 1250.

[0136]

[0117] As such, trained machine learning model(s) 1232 can include one or more models of one or more machine learning algorithms 1220. Machine learning algorithm(s) 1220 may include, but are not limited to: an artificial neural network (e.g., a herein-described convolutional neural networks, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system). Machine learning algorithm(s) 1220 may be supervised or unsupervised, and may implement any suitable combination of online and offline learning. Examples of the models may include multi-layer perceptron (MLP) models, convolutional neural networks (CNN), long short-term memory (LSTM) algorithms, generative adversarial networks (GAN), K-means clustering, Gaussian mixture models (GMM), or any other of a number of machinelearning training techniques known to a person of ordinary skill in the art. The models may comprise a single type of model or multiple types, as well as various combinations of types including combinations of single types and of multiple types.

[0137]

[0118] In some examples, machine learning algorithm(s) 1220 and / or trained machine learning model(s) 1232 can be accelerated using on-device coprocessors, such as graphic processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application specific integrated circuits (ASICs). Such on-device coprocessors can be used to speed up machine learning algorithm(s) 1220 and / or trained machine learning model(s) 1232. In some examples, trained machine learning model(s) 1232 can be trained, reside and execute to provide inferences on a particular computing device, and / or otherwise can make inferences for the particular computing device.

[0138]

[0119] During training phase 1202, machine learning algorithm(s) 1220 can be trained by providing at least training data 1210 as training input using unsupervised, supervised, semisupervised, and / or reinforcement learning techniques. Training data 1210 may include the TDD data synthesized herein. Unsupervised learning involves providing a portion (or all) of training data 1210 to machine learning algorithm(s) 1220 and machine learning algorithm(s) 1220 determining one or more output inferences based on the provided portion (or all) of training data 1210. Supervised learning involves providing a portion of training data 1210 to machine learning algorithm(s) 1220, with machine learning algorithm(s) 1220 determining one or more output inferences based on the provided portion of training data 1210, and the output inference(s) are either accepted or corrected based on correct results associated with training data 1210. In some examples, supervised learning of machine learning algorithm(s) 1220 can be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or set of labels may be used to correct inferences of machine learning algorithm(s) 1220.

[0139]

[0120] For example, the training models may use supervised learning methods, where the training data may be labeled and outputs may be graded based on their fidelity to a “truth” output. The training models may use an unsupervised learning method, where there may not be labels on the training data and the training models may classify correlations without reference to a “truth” value. The training models may combine supervised and unsupervised techniques.

[0140]

[0121] Semi-supervised learning involves having correct results for part, but not all, of training data 1210. During semi-supervised learning, supervised learning is used for a portion of training data 1210 having correct results, and unsupervised learning is used for a portion of training data 1210 not having correct results. Reinforcement learning involves machine learning algorithm(s) 1220 receiving a reward signal regarding a prior inference, where the reward signal can be a numerical value. During reinforcement learning, machine learning algorithm(s) 1220 can output an inference and receive a reward signal in response, where machine learning algorithm(s) 1220 are configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing an expected total of the numerical values provided by the reward signal over time. In some examples, machine learning algorithm(s) 1220 and / or trained machine learning model(s) 1232 can be trained using other machine learning techniques, including but not limited to, incremental learning and curriculum learning.

[0141]

[0122] In some examples, machine learning algorithm(s) 1220 and / or trained machine learning model(s) 1232 can use transfer learning techniques. For example, transfer learning techniques can involve trained machine learning model(s) 1232 being pre-trained on one set of data and additionally trained using training data 1210. More particularly, machine learning algorithm(s) 1220 can be pre-trained on data from one or more computing devices and a resulting trained machine learning model provided to computing device CD1, where CD1 is intended to execute the trained machine learning model during inference phase 1204. Then, during training phase 1202, the pre-trained machine learning model can be additionally trained using training data 1210, where training data 1210 can be derived from kernel and non-kernel data of computing device CD1. This further training of the machine learning algorithm(s) 1220 and / or the pretrained machine learning model using training data 1210 of CDl’s data can be performed using either supervised or unsupervised learning. Once machine learning algorithm(s) 1220 and / or the pre-trained machine learning model has been trained on at least training data 1210, training phase 1202 can be completed. The trained resulting machine learning model can be utilized as at least one of trained machine learning model(s) 1232.

[0142]

[0123] In particular, once training phase 1202 has been completed, trained machine learning model(s) 1232 can be provided to a computing device, if not already on the computing device. Inference phase 1204 can begin after trained machine learning model(s) 1232 are provided to computing device CD1.

[0124] In the context of this disclosure, a ML model may be trained. Training data 1210 may be image data. In this example, the training data 1210 are first processed by a CNN. The CNN may extract features from the training data 1210. The CNN may be a disconnected neural network, meaning all members of the neural network, such as perceptrons, may not be in a causal relationship with one another. It should be mentioned that other feature extraction techniques besides CNNs may be equally employed in lieu of the CNN described herein. An MLP neural network may receive the extracted feature outputs from the CNN as inputs. The MLP may be a fully connected network, which may comprise all perceptrons in a causal relationship with one another. The MLP may have as an output a detection probability prediction 1250. The prediction 1250 may be, for example, the probability that one or more focus ranges are correct for a camera input, or some other prediction related to AF, PDAF, or similar.

[0143]

[0125] During inference phase 1204, trained machine learning model(s) 1232 can receive input data 1230 and generate and output one or more corresponding inferences and / or prediction(s) 1250 about input data 1230. Input data 1230 may be data from one or more sensors. As such, input data 1230 can be used as an input to trained machine learning model(s) 1232 for providing corresponding inference(s) and / or prediction(s) 1250 to kernel components and non-kernel components. For example, trained machine learning model(s) 1232 can generate inference(s) and / or prediction(s) 1250 in response to one or more inference / prediction requests 1240. In some examples, trained machine learning model(s) 1232 can be executed by a portion of other software. For example, trained machine learning model(s) 1232 can be executed by an inference or prediction daemon to be readily available to provide inferences and / or predictions upon request. Input data 1230 can include data from computing device CD1 executing trained machine learning model(s) 1232 and / or input data from one or more computing devices other than CD1. For example, input data 1230 can include images, prompts. Input data 1230 can also include prompts labeled with context data, device data, browsing history, search history, and so forth. Other types of input data are possible as well.

[0144]

[0126] The trained machine learning model(s) 1232 may use a matrix containing depth of field calculations, an array of input imagery, or one or more numbers between 0 and 1 as the prediction 1250. Such an ML model may have one or more feature extraction layers, such as the CNN, and one or more fully connected layers, such as the MLP.

[0145]

[0127] Inference(s) and / or prediction(s) 1250 can include output prompts, output partial prompts, output prompt completions, numerical values, and / or other output data produced by trained machine learning model(s) 1232 operating on input data 1230 (and training data 1210). In some examples, trained machine learning model(s) 1232 can use output inference(s) and / or prediction(s) 1250 as input feedback 1260. Trained machine learning model(s) 1232 can also rely on past inferences as inputs for generating new inferences.

[0146]

[0128] In some examples, one computing device CD SOLO can include the trained version of the neural network, perhaps after training. Then, computing device CD SOLO can receive a request to predict a caption, and use the trained version of the neural network to predict the caption.

[0147]

[0129] In some examples, two or more computing devices CD CLI and CD SRV can be used to provide output captions; e.g., a first computing device CD CLI can generate and send requests to predict a caption to a second computing device CD SRV. Then, CD SRV can use the trained version of the neural network, to predict the caption, and respond to the requests from CD CLI for the predicted caption. Then, upon reception of responses to the requests, CD CLI can provide the requested predicted caption (e.g., using a multimodal user interface and / or a display).

[0148]

[0130] An ML model, as discussed in this disclosure, refers to a computer model that has been trained using one or more machine-learning techniques. In general, this training may be done by providing training inputs to one or more training models, which in turn may provide an output. The output may be in the form of a prediction, a confidence score, or other probabilitybased metrics.

[0149]

[0131] The aforementioned are merely examples of the types of machine-learning training that may be done to produce an ML model for AF. Many other combinations and techniques may be employed, which do not alter the scope of the inventive concept. Examples such as employing a CNN, use of different data types than those listed, alone or in combination with the data types listed, having a fully connected portion of the machine-learning training, etc. are not meant to limit the scope of the disclosed methods and apparatuses, but rather to serve as examples of how an ML model for AF may be accomplished.

[0150]

[0132] By way of example, training for the ML model may equivalently be accomplished by incorporating, as either a part of or the complete machine-learning training phase 1202, a long short-term memory (LSTM) algorithm. The LSTM algorithm is a type of recombinant neural network (RNN), which may process data in a time-indexed fashion. The LSTM may solve a so-called “vanishing gradient problem,” in which gradients used in fitting may tend to zero and, thus, may not yield useful fit parameters for a given model (e.g., weights and biases). During the training phase 1202, the LSTM may allow for persistent gradients used to fit when the gradients may otherwise go to zero (e.g., in a traditional RNN). For the purpose of the present disclosure, an example of implementing the LSTM in the machine-learning training phase 1202 may be to replace a MLP with the LSTM or to include the LSTM before or after the MLP.

[0151] Example Data Network

[0152]

[0133] FIG. 13 depicts a distributed computing architecture 1300, in accordance with example embodiments. Distributed computing architecture 1300 includes server devices 1308, 1310 that are configured to communicate, via network 1306, with programmable devices 1304a, 1304b, 1304c, 1304d, 1304e. Network 1306 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public Internet, or any other type of network configured to provide a communications path between networked computing devices. Network 1306 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public Internet.

[0153]

[0134] Although FIG. 13 only shows five programmable devices, distributed application architectures may serve tens, hundreds, or thousands of programmable devices. Moreover, programmable devices 1304a, 1304b, 1304c, 1304d, 1304e (or any additional programmable devices) may be any sort of computing device, such as a mobile computing device, desktop computer, wearable computing device, head-mountable device (HMD), network terminal, a mobile computing device, and so on. In some examples, such as illustrated by programmable devices 1304a, 1304b, 1304c, 1304e, programmable devices can be directly connected to network 1306. In other examples, such as illustrated by programmable device 1304d, programmable devices can be indirectly connected to network 1306 via an associated computing device, such as programmable device 1304c. In this example, programmable device 1304c can act as an associated computing device to pass electronic communications between programmable device 1304d and network 1306. In other examples, such as illustrated by programmable device 1304e, a computing device can be part of and / or inside a vehicle, such as a car, a truck, a bus, a boat or ship, an airplane, etc. In other examples not shown in FIG. 13, a programmable device can be both directly and indirectly connected to network 1306.

[0154]

[0135] Server devices 1308, 1310 can be configured to perform one or more services, as requested by programmable devices 1304a-1304e. For example, server device 1308 and / or 1310 can provide content to programmable devices 1304a-1304e. The content can include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. The content can include compressed and / or uncompressed content. The content can be encrypted and / or unencrypted. Other types of content are possible as well.

[0136] As another example, server device 1308 and / or 1310 can provide programmable devices 1304a-1304e with access to software for database, search, computation, graphical, audio, video, World Wide Web / Internet utilization, and / or other functions. Many other examples of server devices are possible as well.

[0155] Computing Device Architecture

[0156]

[0137] FIG. 14 is a block diagram of an example computing device 1400, in accordance with example embodiments. In particular, computing device 1400 shown in FIG. 14 can be configured to perform at least one function of and / or related to the machine learning models, and / or method 1600.

[0157]

[0138] Computing device 1400 may include a user interface module 1401, a network communications module 1402, one or more processors 1403, data storage 1404, one or more camera(s) 1412, one or more sensors 1414, and power system 1416, all of which may be linked together via a system bus, network, or other connection mechanism 1405.

[0158]

[0139] User interface module 1401 can be operable to send data to and / or receive data from external user input / output devices. For example, user interface module 1401 can be configured to send and / or receive data to and / or from user input devices such as a touch screen, a computer mouse, a keyboard, a keypad, a touch pad, a trackball, a joystick, a voice recognition module, and / or other similar devices. User interface module 1401 can also be configured to provide output to user display devices, such as one or more cathode ray tubes (CRT), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, either now known or later developed. User interface module 1401 can also be configured to generate audible outputs, with devices such as a speaker, speaker jack, audio output port, audio output device, earphones, and / or other similar devices. User interface module 1401 can further be configured with one or more haptic devices that can generate haptic outputs, such as vibrations and / or other outputs detectable by touch and / or physical contact with computing device 1400. In some examples, user interface module 1401 can be used to provide a graphical user interface (GUI) for utilizing computing device 1400, such as, for example, a graphical user interface of a mobile phone device.

[0159]

[0140] Network communications module 1402 can include one or more devices that provide one or more wireless interface(s) 1407 and / or one or more wireline interface(s) 1408 that are configurable to communicate via a network. Wireless interface(s) 1407 can include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth™ transceiver, a Zigbee® transceiver, a Wi-Fi™ transceiver, a WiMAX™ transceiver, an LTE™ transceiver, and / or other type of wireless transceiver configurable to communicate via a wireless network. Wireline interface(s) 1408 can include one or more wireline transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or similar transceiver configurable to communicate via a twisted pair wire, a coaxial cable, a fiberoptic link, or a similar physical connection to a wireline network.

[0160]

[0141] In some examples, network communications module 1402 can be configured to provide reliable, secured, and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., guaranteed message delivery) can be provided, perhaps as part of a message header and / or footer (e.g., packet / message sequencing information, encapsulation headers and / or footers, size / time information, and transmission verification information such as cyclic redundancy check (CRC) and / or parity check values). Communications can be made secure (e.g., be encoded or encrypted) and / or decry pted / decoded using one or more cryptographic protocols and / or algorithms, such as, but not limited to, Data Encryption Standard (DES), Advanced Encryption Standard (AES), a Rivest-Shamir-Adelman (RSA) algorithm, a Diffie-Hellman algorithm, a secure sockets protocol such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms can be used as well or in addition to those listed herein to secure (and then decry pt / decode) communications.

[0161]

[0142] One or more processors 1403 can include one or more general purpose processors, and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). One or more processors 1403 can be configured to execute computer-readable instructions 606 that are contained in data storage 1404 and / or other instructions as described herein.

[0162]

[0143] Data storage 1404 can include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of one or more processors 1403. The one or more computer-readable storage media can include volatile and / or non-volatile storage components, such as optical, magnetic, organic or other memory or disc storage, which can be integrated in whole or in part with at least one of one or more processors 1403. In some examples, data storage 1404 can be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disc storage unit), while in other examples, data storage 1404 can be implemented using two or more physical devices.

[0144] Data storage 1404 can include computer-readable instructions 1406 and perhaps additional data. In some examples, data storage 1404 can include storage required to perform at least part of the herein-described methods, scenarios, and techniques and / or at least part of the functionality of the herein-described devices and networks. In some examples, data storage 1404 can include storage for a trained neural network model 1410 (e.g., trained RNN, trained LSTM, etc.). In particular of these examples, computer-readable instructions 1406 can include instructions that, when executed by one or more processors 1403, enable computing device 1400 to provide for some or all of the functionality of trained neural network model 1410.

[0163]

[0145] In some examples, computing device 1400 can include one or more camera(s) 1412. Camera(s) 1412 can include one or more image capture devices, such as still and / or video cameras, equipped to capture light and record the captured light in one or more images; that is, camera(s) 1412 can generate image(s) of captured light. The one or more images can be one or more still images and / or one or more images utilized in video imagery. Camera(s) 1412 can capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light.

[0164]

[0146] In some examples, computing device 1400 can include one or more sensors 1414. Sensors 1414 can be configured to measure conditions within computing device 1400 and / or conditions in an environment of computing device 1400 and provide data about these conditions. For example, sensors 1414 can include one or more of: (i) sensors for obtaining data about computing device 1400, such as, but not limited to, a thermometer for measuring a temperature of computing device 1400, a battery sensor for measuring power of one or more batteries of power system 1416, and / or other sensors measuring conditions of computing device 1400; (ii) an identification sensor to identify other objects and / or devices, such as, but not limited to, a Radio Frequency Identification (RFID) reader, proximity sensor, one-dimensional barcode reader, two-dimensional barcode (e.g., Quick Response (QR) code) reader, and a laser tracker, where the identification sensors can be configured to read identifiers, such as RFID tags, barcodes, QR codes, and / or other devices and / or object configured to be read and provide at least identifying information; (iii) sensors to measure locations and / or movements of computing device 1400, such as, but not limited to, a tilt sensor, a gyroscope, an accelerometer, a Doppler sensor, a GPS device, a sonar sensor, a radar device, a laser-displacement sensor, and a compass; (iv) an environmental sensor to obtain data indicative of an environment of computing device 1400, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a movement sensor, a microphone, a sound sensor, an ultrasound sensor and / or a smoke sensor; and / or (v) a force sensor to measure one or more forces (e.g., inertial forces and / or G-forces) acting about computing device 1400, such as, but not limited to one or more sensors that measure: forces in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor that identifies ZMPs and / or locations of the ZMPs. Many other examples of sensors 1414 are possible as well.

[0165]

[0147] Power system 1416 can include one or more batteries 1418 and / or one or more external power interfaces 1420 for providing electrical power to computing device 1400. Each battery of the one or more batteries 1418 can, when electrically coupled to the computing device 1400, act as a source of stored electrical power for computing device 1400. One or more batteries 1418 of power system 1416 can be configured to be portable. Some or all of one or more batteries 1418 can be readily removable from computing device 1400. In other examples, some or all of one or more batteries 1418 can be internal to computing device 1400, and so may not be readily removable from computing device 1400. Some or all of one or more batteries 1418 can be rechargeable. For example, a rechargeable battery can be recharged via a wired connection between the battery and another power supply, such as by one or more power supplies that are external to computing device 1400 and connected to computing device 1400 via the one or more external power interfaces. In other examples, some or all of one or more batteries 1418 can be non-rechargeable batteries.

[0166]

[0148] One or more external power interfaces 1420 of power system 1416 can include one or more wired-power interfaces, such as a USB cable and / or a power cord, that enable wired electrical power connections to one or more power supplies that are external to computing device 1400. One or more external power interfaces 1420 can include one or more wireless power interfaces, such as a Qi wireless charger, that enable wireless electrical power connections, such as via a Qi wireless charger, to one or more external power supplies. Once an electrical power connection is established to an external power source using one or more external power interfaces 1420, computing device 1400 can draw electrical power from the external power source the established electrical power connection. In some examples, power system 1416 can include related sensors, such as battery sensors associated with the one or more batteries or other types of electrical power sensors.

[0167] Cloud-Based Servers

[0168]

[0149] FIG. 15 depicts a cloud-based server system in accordance with an example embodiment. In FIG. 15, functionality of a neural network, and / or a computing device can be distributed among computing clusters 1509a, 1509b, and 1509c. Computing cluster 1509a can include one or more computing devices 1500a, cluster storage arrays 1510a, and cluster routers 1511a connected by a local cluster network 1512a. Similarly, computing cluster 1509b can include one or more computing devices 1500b, cluster storage arrays 1510b, and cluster routers 1511b connected by a local cluster network 1512b. Likewise, computing cluster 1509c can include one or more computing devices 1500c, cluster storage arrays 1510c, and cluster routers 1511c connected by a local cluster network 1512c.

[0169]

[0150] In some embodiments, computing clusters 1509a, 1509b, and 1509c can be a single computing device residing in a single computing center. In other embodiments, computing clusters 1509a, 1509b, and 1509c can include multiple computing devices in a single computing center, or even multiple computing devices located in multiple computing centers located in diverse geographic locations. For example, FIG. 15 depicts each of computing clusters 1509a, 1509b, 1509c residing in different physical locations.

[0170]

[0151] In some embodiments, data and services at computing clusters 1509a, 1509b, 1509c can be encoded as computer readable information stored in non-transitory, tangible computer readable media (or computer readable storage media) and accessible by other computing devices. In some embodiments, computing clusters 1509a, 1509b, 1509c can be stored on a single disk drive or other tangible storage media, or can be implemented on multiple disk drives or other tangible storage media located at one or more diverse geographic locations.

[0171]

[0152] In some embodiments, each of computing clusters 1509a, 1509b, and 1509c can have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. In other embodiments, however, each computing cluster can have different numbers of computing devices, different numbers of cluster storage arrays, and different numbers of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster can depend on the computing task or tasks assigned to each computing cluster.

[0172]

[0153] In computing cluster 1509a, for example, computing devices 1500a can be configured to perform various computing tasks of a neural network, and / or a computing device. In one embodiment, the various functionalities of a neural network, and / or a computing device can be distributed among one or more of computing devices 1500a, 1500b, and 1500c. Computing devices 1500b and 1500c in respective computing clusters 1509b and 1509c can be configured similarly to computing devices 1500a in computing cluster 1509a. On the other hand, in some embodiments, computing devices 1500a, 1500b, and 1500c can be configured to perform different functions.

[0154] In some embodiments, computing tasks and stored data associated with a neural network, and / or a computing device can be distributed across computing devices 1500a, 1500b, and 1500c based at least in part on the processing requirements of a neural network, and / or a computing device, the processing capabilities of computing devices 1500a, 1500b, 1500c, the latency of the network links between the computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that can contribute to the cost, speed, fault-tolerance, resiliency, efficiency, and / or other design goals of the overall system architecture.

[0173]

[0155] Cluster storage arrays 1510a, 1510b, 1510c of computing clusters 1509a, 1509b, and 1509c can be data storage arrays that include disk array controllers configured to manage read and write access to groups of hard disk drives. The disk array controllers, alone or in conjunction with their respective computing devices, can also be configured to manage backup or redundant copies of the data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0174]

[0156] Similar to the manner in which the functions of a conditioned, axial self-attention based neural network, and / or a computing device can be distributed across computing devices 1500a, 1500b, 1500c of computing clusters 1509a, 1509b, 1509c, various active portions and / or backup portions of these components can be distributed across cluster storage arrays 1510a, 1510b, 1510c. For example, some cluster storage arrays can be configured to store one portion of the data of a first layer of a neural network, and / or a computing device, while other cluster storage arrays can store other portion(s) of data of second layer of a neural network, and / or a computing device. Also, for example, some cluster storage arrays can be configured to store the data of an encoder of a neural network, while other cluster storage arrays can store the data of a decoder of a neural network. Additionally, some cluster storage arrays can be configured to store backup versions of data stored in other cluster storage arrays.

[0175]

[0157] Cluster routers 1511a, 1511b, 1511c in computing clusters 1509a, 1509b, and 1509c can include networking equipment configured to provide internal and external communications for the computing clusters. For example, cluster routers 151 la in computing cluster 1509a can include one or more internet switching and routing devices configured to provide (i) local area network communications between computing devices 1500a and cluster storage arrays 1510a via local cluster network 1512a, and (ii) wide area network communications between computing cluster 1509a and computing clusters 1509b and 1509c via wide area network link 1513a to network 1306. Cluster routers 1511b and 1511c can include network equipment similar to cluster routers 1511a, and cluster routers 1511b and 1511c can perform similar networking functions for computing clusters 1509b and 1509b that cluster routers 1511a perform for computing cluster 1509a.

[0176]

[0158] In some embodiments, the configuration of cluster routers 1511a, 1511b, 1511c can be based at least in part on the data communication requirements of the computing devices and cluster storage arrays, the data communications capabilities of the network equipment in cluster routers 1511a, 1511b, 1511c, the latency and throughput of local cluster networks 1512a, 1512b, 1512c, the latency, throughput, and cost of wide area network links 1513a, 1513b, 1513c, and / or other factors that can contribute to the cost, speed, fault-tolerance, resiliency, efficiency and / or other design criteria of the moderation system architecture.

[0177] Example Methods of Operation

[0178]

[0159] FIG. 16 is a flowchart of a method 1600, in accordance with example embodiments. Method 1600 can be executed by a computing device, such as computing device 1400. Method 1600 can begin at block 1610, where the method involves displaying, by an interactive graphical user interface (GUI) of a computing device, an image.

[0179]

[0160] At block 1620, the method involves detecting, by the interactive GUI, user interaction with the displayed image, wherein the user interaction comprises modifying one or more image characteristics using at least one image adjustment control provided by the GUI.

[0180]

[0161] At block 1630, the method involves generating, by the computing device and based on the user interaction, a synthetic sequence of temporal images that simulate a video based on the image.

[0181]

[0162] At block 1640, the method involves providing, by the computing device, the synthetic sequence of temporal images.

[0182]

[0163] In some embodiments, the displayed image may be captured at a first lens position. Such embodiments involve receiving, by the interactive GUI, user indication of a target lens position. The generating of the synthetic sequence of temporal images involves capturing image frames by simulating lens traversal from the first lens position to the target lens position.

[0183]

[0164] In some embodiments, the target lens position may be a zero-disparity lens position for the displayed image.

[0184]

[0165] Some embodiments involve providing, by the interactive GUI, an indication of a region of interest (ROI) in the displayed image. The modifying of the one or more image characteristics involves altering one or more of a position of the ROI, or a size of the ROI.

[0166] Some embodiments involve adjusting, by the computing device, optical image stabilization (OIS) positions in the displayed image to simulate one or more radial distortion patterns.

[0185]

[0167] In some embodiments, the modifying of the one or more image characteristics involves adding a motion blur to the displayed image based on a direction or magnitude of a ROI.

[0186]

[0168] In some embodiments, the modifying of the one or more image characteristics involves adding a motion blur to the displayed image based on a direction or magnitude of an OIS motion.

[0187]

[0169] In some embodiments, the modifying of the one or more image characteristics involves changing an exposure level of the displayed image.

[0188]

[0170] In some embodiments, the modifying of the one or more image characteristics involves adding Gaussian noise.

[0189]

[0171] In some embodiments, the displayed image may be a single-depth image, and the generating of the synthetic sequence of temporal images involves generating multi-depth images from the single-depth image by concatenating generating of the synthetic sequence of temporal images into the single-depth image.

[0190]

[0172] In some embodiments, the generating of the synthetic sequence of temporal images involves applying the synthetic sequence of temporal images independently to each layer of the generating of the synthetic sequence of temporal images.

[0191]

[0173] In some embodiments, the at least one image adjustment control includes a sliding control for adjusting a focus sweep.

[0192]

[0174] In some embodiments, the at least one image adjustment control includes a distance control for adjusting a distance for a scene captured in the displayed image.

[0193]

[0175] In some embodiments, the at least one image adjustment control includes a signal-to- noise ratio (SNR) control for changing a SNR level.

[0194]

[0176] Some embodiments involve receiving a plurality of displayed images. Such embodiments also involve generating respective synthetic sequences of temporal images from each of the plurality of displayed images. Such embodiments further involve providing a training dataset comprising the respective synthetic sequences of temporal images for training a machine learning model to predict whether one or more input focus ranges are valid for a camera input.

[0195]

[0177] Some embodiments involve training the machine learning model based on the training dataset.

[0178] In some embodiments, the machine learning model may be a recurrent neural network (RNN).

[0196]

[0179] In some embodiments, the RNN may be a long short-term memory (LSTM) network.

[0197]

[0180] The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its spirit and scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those enumerated herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.

[0198]

[0181] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

[0199]

[0182] With respect to any or all of the ladder diagrams, scenarios, and flow charts in the figures and as discussed herein, each block and / or communication may represent a processing of information and / or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, functions described as blocks, transmissions, communications, requests, responses, and / or messages may be executed out of order from that shown or discussed, including substantially concurrent or in reverse order, depending on the functionality involved. Further, more or fewer blocks and / or functions may be used with any of the ladder diagrams, scenarios, and flow charts discussed herein, and these ladder diagrams, scenarios, and flow charts may be combined with one another, in part or in whole.

[0200]

[0183] A block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and / or related data may be stored on any type of computer readable medium such as a storage device including a disk or hard drive or other storage medium.

[0201]

[0184] The computer readable medium may also include non-transitory computer readable media such as non-transitory computer-readable media that stores data for short periods of time like register memory, processor cache, and random access memory (RAM). The computer readable media may also include non-transitory computer readable media that stores program code and / or data for longer periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or nonvolatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

[0202]

[0185] Moreover, a block that represents one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.

[0203]

[0186] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for explanatory purposes and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method, comprising: displaying, by an interactive graphical user interface (GUI) of a computing device, an image; detecting, by the interactive GUI, user interaction with the displayed image, wherein the user interaction comprises modifying one or more image characteristics using at least one image adjustment control provided by the GUI; generating, by the computing device and based on the user interaction, a synthetic sequence of temporal images that simulate a video based on the image; and providing, by the computing device, the synthetic sequence of temporal images.

2. The computer-implemented method of claim 1 , wherein the displayed image is captured at a first lens position, and further comprising: receiving, by the interactive GUI, user indication of a target lens position, and wherein the generating of the synthetic sequence of temporal images comprises capturing image frames by simulating lens traversal from the first lens position to the target lens position.

3. The computer-implemented method of claim 2, wherein the target lens position is a zero-disparity lens position for the displayed image.

4. The computer-implemented method of any of claims 1-3, further comprising: providing, by the interactive GUI, an indication of a region of interest (ROI) in the displayed image, and wherein the modifying of the one or more image characteristics comprises altering one or more of a position of the ROI, or a size of the ROI.

5. The computer-implemented method of any of claims 1-4, further comprising: adjusting, by the computing device, optical image stabilization (OIS) positions in the displayed image to simulate one or more radial distortion patterns.

6. The computer-implemented method of any of claims 1-5, wherein the modifying of the one or more image characteristics comprises: adding a motion blur to the displayed image based on a direction or magnitude of a ROI.

7. The computer-implemented method of any of claims 1-6, wherein the modifying of the one or more image characteristics comprises: adding a motion blur to the displayed image based on a direction or magnitude of an OIS motion.

8. The computer-implemented method of any of claims 1-7, wherein the modifying of the one or more image characteristics comprises: changing an exposure level of the displayed image.

9. The computer-implemented method of any of claims 1-8, wherein the modifying of the one or more image characteristics comprises adding Gaussian noise.

10. The computer-implemented method of any of claims 1-9, wherein the displayed image is a single-depth image, and wherein the generating of the synthetic sequence of temporal images further comprises: generating multi-depth images from the single-depth image by concatenating generating of the synthetic sequence of temporal images into the single-depth image.

11. The computer-implemented method of claim 10, wherein the generating of the synthetic sequence of temporal images further comprises: applying the synthetic sequence of temporal images independently to each layer of the generating of the synthetic sequence of temporal images.

12. The computer-implemented method of any of claims 1-11, wherein the at least one image adjustment control comprises a sliding control for adjusting a focus sweep.

13. The computer-implemented method of any of claims 1-12, wherein the at least one image adjustment control comprises a distance control for adjusting a distance for a scene captured in the displayed image.

14. The computer-implemented method of any of claims 1-13, wherein the at least one image adjustment control comprises a signal-to-noise ratio (SNR) control for changing a SNR level.

15. The computer-implemented method of any of claims 1-14, further comprising: receiving a plurality of displayed images; generating respective synthetic sequences of temporal images from each of the plurality of displayed images; and providing a training dataset comprising the respective synthetic sequences of temporal images for training a machine learning model to predict whether one or more input focus ranges are valid for a camera input.

16. The computer-implemented method of claim 15, further comprising: training the machine learning model based on the training dataset.

17. The computer-implemented method of claims 15 or 16, wherein the machine learning model is a recurrent neural network (RNN).

18. The computer-implemented method of any of claims 15-17, wherein the RNN is a long short-term memory (LSTM) network.

19. A computing device, comprising: one or more processors; and data storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out functions comprising the computer-implemented method of any one of claims 1-18.

20. The computing device of claim 19, wherein the computing device is a mobile device.

21. A computer program comprising instructions that, when executed by a computer, cause the computer to perform steps in accordance with the method of any one of claims 1-18.

22. An article of manufacture comprising one or more non-transitory computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out functions that comprise the computer-implemented method of any one of claims 1-18.

23. A system, comprising: means for carrying out the computer-implemented method of any one of claims 1-18.

Images