Systems and methods for temporal dual-depth machine learning based phase-difference autofocus (PDAF) model

Abstract

An example method includes receiving, by a computing device, a training dataset comprising a plurality of images based on sparse sensor data. The method also includes training, based on the training dataset, a dual-depth machine learning (ML) model to predict at least two disparity values for a phase-difference autofocus (PDAF) adjustment to a lens position for a camera. The method further includes providing, by the computing device, the trained dual-depth ML model.

Description

SYSTEMS AND METHODS FOR TEMPORAL DUAL-DEPTH MACHINE LEARNING BASED PHASE-DIFFERENCE AUTOFOCUS (PDAF) MODELCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This present application claims priority to U.S. provisional application serial no. 63 / 587,606 filed October 3, 2023, and to U.S. provisional application serial no. 63 / 587,639 filed October 3, 2023, the full disclosures of which are incorporated herein by reference.BACKGROUND

[0002] Camera autofocus (AF) is a hybrid system that involves multiple modules, such as, for example, a phase-difference autofocus (PDAF) module, to achieve an optimal focus.SUMMARY

[0003] This application generally relates to the training and inference of a temporal dual-depth machine learning based PDAF model.

[0004] In one aspect, a computer-implemented method is provided. The method includes receiving, by a computing device, a training dataset comprising a plurality of images based on sparse sensor data. The method also includes training, based on the training dataset, a dualdepth machine learning (ML) model to predict at least two disparity values for a phasedifference autofocus (PDAF) adjustment to a lens position for a camera. The method further includes providing, by the computing device, the trained dual-depth ML model.

[0005] In a second aspect, a device is provided. The device includes one or more processors and data storage. The data storage has stored thereon computer-executable instructions that, when executed by one or more processors, cause the device to carry out functions. The functions include: receiving, by a computing device, a training dataset comprising a plurality of images based on sparse sensor data; training, based on the training dataset, a dual-depth machine learning (ML) model to predict at least two disparity values for a phase-difference autofocus (PDAF) adjustment to a lens position for a camera; and providing, by the computing device, the trained dual-depth ML model.

[0006] In a third aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to carry out functions. The functions include: receiving, by a computing device, a training dataset comprising a plurality of images based on sparse sensor data; training, based on the training dataset, a dual-depth machine learning (ML) model to predict at least two disparity values for a phase-differenceautofocus (PDAF) adjustment to a lens position for a camera; and providing, by the computing device, the trained dual-depth ML model.

[0007] In a fourth aspect, an article of manufacture is provided. The article of manufacture includes one or more computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a device, cause the device to carry out functions. The functions include: receiving, by a computing device, a training dataset comprising a plurality of images based on sparse sensor data; training, based on the training dataset, a dual-depth machine learning (ML) model to predict at least two disparity values for a phase-difference autofocus (PDAF) adjustment to a lens position for a camera; and providing, by the computing device, the trained dual-depth ML model.

[0008] In a fifth aspect, a system is provided. The system includes means for receiving, by a computing device, a training dataset comprising a plurality of images based on sparse sensor data; means for training, based on the training dataset, a dual-depth machine learning (ML) model to predict at least two disparity values for a phase-difference autofocus (PDAF) adjustment to a lens position for a camera; and means for providing, by the computing device, the trained dual-depth ML model.

[0009] In a sixth aspect, a computer-implemented method is provided. The method includes receiving image data from an image sensor of a camera, wherein the image sensor is configured to output sparse sensor data. The method also includes predicting, based on a trained dualdepth machine learning (ML) model, at least two disparity values for a phase-difference autofocus (PDAF) adjustment to a lens position for the camera. The method further includes providing the predicted at least two disparity values.

[0010] In a seventh aspect, a device is provided. The device includes one or more processors and data storage. The data storage has stored thereon computer-executable instructions that, when executed by one or more processors, cause the device to carry out functions. The functions include: receiving image data from an image sensor of a camera, wherein the image sensor is configured to output sparse sensor data; predicting, based on a trained dual-depth machine learning (ML) model, at least two disparity values for a phase-difference autofocus (PDAF) adjustment to a lens position for the camera; and providing the predicted at least two disparity values.

[0011] In an eighth aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to carry out functions. The functions include: receiving image data from an image sensor of a camera, wherein the image sensor is configured to output sparse sensor data; predicting, based on a trained dual-depthmachine learning (ML) model, at least two disparity values for a phase-difference autofocus (PDAF) adjustment to a lens position for the camera; and providing the predicted at least two disparity values.

[0012] In a ninth aspect, an article of manufacture is provided. The article of manufacture includes one or more computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a device, cause the device to carry out functions. The functions include: receiving image data from an image sensor of a camera, wherein the image sensor is configured to output sparse sensor data; predicting, based on a trained dual-depth machine learning (ML) model, at least two disparity values for a phasedifference autofocus (PDAF) adjustment to a lens position for the camera; and providing the predicted at least two disparity values.

[0013] In a tenth aspect, a system is provided. The system includes means for receiving image data from an image sensor of a camera, wherein the image sensor is configured to output sparse sensor data; means for predicting, based on a trained dual-depth machine learning (ML) model, at least two disparity values for a phase-difference autofocus (PDAF) adjustment to a lens position for the camera; and means for providing the predicted at least two disparity values.

[0014] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the figures and the following detailed description and the accompanying drawings.BRIEF DESCRIPTION OF THE FIGURES

[0015] FIG. 1 is an illustration of front, right-side, and rear views of a digital camera device 100, in accordance with example embodiments.

[0016] FIG. 2 is an example illustration of an image sensor with a sparse phase-difference (PD) pixels distribution, in accordance with example embodiments.

[0017] FIG. 3 illustrates a comparison between non-temporal and temporal machine learning (ML) based approaches for phase-difference autofocus (PDAF), in accordance with example embodiments.

[0018] FIG. 4 is an example architecture for a dual-disparity machine learning (ML) based approach for phase-difference autofocus (PDAF), in accordance with example embodiments.

[0019] FIG. 5 illustrates an example training process for a recurrent neural network (RNN) using randomly trimmed fixed-size frames, in accordance with example embodiments.

[0020] FIG. 6 illustrates an example training process to capture frame sequences with scene changes, in accordance with example embodiments.

[0021] FIG. 7 illustrates performance comparisons between non-temporal and temporal machine learning (ML) based approaches for phase-difference autofocus (PDAF), in accordance with example embodiments.

[0022] FIG. 8 illustrates autofocus (AF) stability performance comparisons between various approaches for phase-difference autofocus (PDAF), in accordance with example embodiments.

[0023] FIG. 9 illustrates example image comparisons for single-depth and multi-depth machine learning (ML) based approaches for phase-difference autofocus (PDAF), in accordance with example embodiments.

[0024] FIG. 10 is a diagram illustrating training and inference phases of a machine learning model, in accordance with example embodiments.

[0025] FIG. 11 depicts a distributed computing architecture, in accordance with example embodiments.

[0026] FIG. 12 is a block diagram of a computing device, in accordance with example embodiments.

[0027] FIG. 13 depicts a network of computing clusters arranged as a cloud-based server system, in accordance with example embodiments.

[0028] FIG. 14 is a flowchart of a method, in accordance with example embodiments.

[0029] FIG. 15 is another flowchart of a method, in accordance with example embodiments.DETAILED DESCRIPTION

[0030] Example methods, devices, and systems are described herein. It should be understood that the words “example” and “exemplary” are used herein to mean “serving as an example, instance, or illustration.” Any embodiment or feature described herein as being an “example” or “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or features. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein.

[0031] Thus, the example embodiments described herein are not meant to be limiting. Aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

[0032] Further, unless context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be generally viewed ascomponent aspects of one or more overall embodiments, with the understanding that not all illustrated features are necessary for each embodiment.Overview

[0033] Camera autofocus (AF) is a hybrid system that involves multiple modules to achieve an optimal focus. Such modules are configured to work together to leverage the capturing conditions for more accurate focus parameters. Camera AF may employ three modules: phasedifference autofocus (PDAF), contrast-detection autofocus (CDAF), and time-of-flight (ToF). Each algorithm has its own advantages and disadvantages, and when properly tuned, they may work together to improve the overall camera AF.

[0034] PDAF measures the phase difference (PD) between two images of the same scene taken from slightly different angles. The calculated PD phase is then used to move the lens accordingly to the optimal focus that achieves zero disparity. PDAF may be a primary AF algorithm because of efficiency and accuracy. Generally, PDAF is based on one frame to estimate a target lens position that achieves a target focus. However, PDAF may be unreliable in challenging conditions, such as extremely low-light scenes. In such situations, the AF system may leverage CDAF or ToF, depending on the design tuning and capturing conditions. CDAF may be more accurate than PDAF under certain conditions, but it can be slower because it performs an image focus search that requires capturing multiple images at different focus distances to determine a target focus distance. In the event that CDAF is unreliable, ToF can be a complementary system for both PDAF and CDAF. ToF emits an actual signal to the scene and measures the round-trip time to obtain the obj ect distance and then determine a target focus. ToF can be highly accurate under normal conditions, but it has a limited distance range and may sometimes be unreliable in bright scenes.Example Camera Systems

[0035] As image capture devices, such as cameras, become more popular, they may be employed as standalone hardware devices or integrated into various other types of devices. For instance, still and video cameras are now regularly included in wireless computing devices (e.g., mobile devices, such as mobile phones), tablet computers, laptop computers, video game interfaces, home automation devices, and even automobiles and other types of vehicles.

[0036] The physical components of a camera may include one or more apertures through which light enters, one or more recording surfaces for capturing the images represented by the light, and lenses positioned in front of each aperture to focus at least part of the image on therecording surface(s). The apertures may be of a fixed size or may be adjustable. In an analog camera, the recording surface may be a photographic film. In a digital camera, the recording surface may include an electronic image sensor (e.g., a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) sensor) to transfer and / or store captured images in a data storage unit (e.g., memory).

[0037] One or more shutters may be coupled to, or positioned near, the lenses or the recording surfaces. Each shutter may either be in a closed position, in which it blocks light from reaching the recording surface, or an open position, in which light is allowed to reach the recording surface. The position of each shutter may be controlled by a shutter button. For instance, a shutter may be in the closed position by default. When the shutter button is triggered (e.g., pressed), the shutter may change from the closed position to the open position for a period of time, known as the shutter cycle. During the shutter cycle, an image may be captured on the recording surface. At the end of the shutter cycle, the shutter may change back to the closed position.

[0038] Alternatively, the shuttering process may be electronic. For example, before an electronic shutter of a CCD image sensor is “opened,” the sensor may be reset to remove any residual signal in its photodiodes. While the electronic shutter remains open, the photodiodes may accumulate charge. When or after the shutter closes, these charges may be transferred to longer-term data storage. Combinations of mechanical and electronic shuttering may also be possible.

[0039] Regardless of type, a shutter may be activated and / or controlled by something other than a shutter button. For instance, the shutter may be activated by a softkey, a timer, or some other trigger. Herein, the term “capture” may refer to any mechanical and / or electronic shuttering process that results in one or more images being recorded, regardless of how the shuttering process is triggered or controlled.

[0040] The exposure of a captured image may be determined by a combination of the size of the aperture, the brightness of the light entering the aperture, and the length of the shutter cycle (also referred to as the shutter length, the exposure length, or the exposure time). Additionally, a digital and / or analog gain (e.g., based on an ISO setting) may be applied to the image, thereby influencing the exposure. In some embodiments, the term “exposure length,” “exposure time,” or “exposure time interval” may refer to the shutter length multiplied by the gain for a particular aperture size. Thus, these terms may be used somewhat interchangeably, and should be interpreted as possibly being a shutter length, an exposure time, and / or any other metric that controls the amount of signal response that results from light reaching the recording surface.

[0041] In some implementations or modes of operation, a camera may capture one or more still images each time image capture is triggered. In other implementations or modes of operation, a camera may capture a video image by continuously capturing images at a particular rate (e.g., 24 frames per second) as long as image capture remains triggered (e.g., while the shutter button is held down). Some cameras, when operating in a mode to capture a still image, may open the shutter when the camera device or application is activated, and the shutter may remain in this position until the camera device or application is deactivated. While the shutter is open, the camera device or application may capture and display a representation of a scene on a viewfinder (sometimes referred to as displaying a “preview frame”). When image capture is triggered, one or more distinct payload images of the current scene may be captured.

[0042] Cameras, including digital and analog cameras, may include software to control one or more camera functions and / or settings, such as aperture size, exposure time, gain, and so on. Additionally, some cameras may include software that digitally processes images during or after image capture. While the description above refers to cameras in general, it may be particularly relevant to digital cameras. Digital cameras may be standalone devices (e.g., a DSLR camera) or may be integrated with other devices.

[0043] Either or both of a front-facing camera and a rear-facing camera may include or be associated with an ambient light sensor (ALS) that may continuously or from time to time determine the ambient brightness of a scene that the camera can capture. In some devices, the ALS can be used to adjust the display brightness of a screen associated with the camera (e.g., a viewfinder). When the determined ambient brightness is high, the brightness level of the screen may be increased to make the screen easier to view. When the determined ambient brightness is low, the brightness level of the screen may be decreased, also to make the screen easier to view as well as to potentially save power. Additionally, the ambient light sensor’s input may be used to determine an exposure time of an associated camera, or to help in this determination.

[0044] FIG. 1 is an illustration of front, right-side, and rear views of a digital camera device 100, in accordance with example embodiments. Digital camera device 100 may be, for example, a mobile device (e.g., a mobile phone), a tablet computer, or a wearable computing device. However, other embodiments are possible. Digital camera device 100 may include various elements, such as a body 102, a front-facing camera 104, a multi-element display 106, a shutter button 108, and other buttons 110. Digital camera device 100 could further include one or more rear-facing cameras 112, 114. Front-facing camera 104 may be positioned on a side of body 102 typically facing a user while in operation, or on the same side as multi-elementdisplay 106. Rear-facing cameras 112, 114 may be positioned on a side of body 102 opposite front-facing camera 104. Referring to the cameras as front-facing and rear-facing is arbitrary, and digital camera device 100 may include multiple cameras positioned on various sides of body 102.

[0045] Multi-element display 106 could represent a cathode ray tube (CRT) display, a lightemitting diode (LED) display, a liquid crystal display (LCD), a plasma display, or any other type of display known in the art. In some embodiments, multi-element display 106 may display a digital representation of the current image being captured by front-facing camera 104 and / or rear-facing cameras 112, 114, or an image that could be captured or was recently captured by either or both of these cameras. Thus, multi-element display 106 may serve as a viewfinder for either camera. Multi-element display 106 may also support touchscreen and / or presencesensitive functions that may be able to adjust the settings and / or configuration of any aspect of digital camera device 100.

[0046] Multi-element display 106 may include additional features related to a camera application. For example, multiple modes may be available for a user, including, a motion mode, portrait mode, video mode, video bokeh mode, and so forth. The camera application may be in camera mode and provide additional features, such as a reverse icon to activate reverse camera view, a trigger button to capture a previewed image, and a photo stream icon to access a database of captured images. Also for example, a magnification ratio slider may be displayed and a user can move a virtual object along the magnification ratio slider to select a magnification ratio. In some embodiments, a user may use the multi-element display 106, also referred to herein as the display screen, to adjust the magnification ratio (e.g., by moving two fingers on display screen in an outward motion away from each other), and magnification ratio slider may automatically display the magnification ratio.

[0047] Front-facing camera 104 may include an image sensor and associated optical elements such as lenses. Front-facing camera 104 may offer zoom capabilities or could have a fixed focal length. In other embodiments, interchangeable lenses could be used with front-facing camera 104. Front-facing camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. Front-facing camera 104 also could be configured to capture still images, video images, or both. Further, front-facing camera 104 could represent a monoscopic, stereoscopic, or multiscopic camera. Rear-facing cameras 112, 114 may be similarly or differently arranged. Additionally, front-facing camera 104, rear-facing cameras 112, 114, or both, may be an array of one or more cameras.

[0048] Either or both of front-facing camera 104 and rear-facing cameras 112, 114 may include or be associated with an illumination component that provides a light field to illuminate a target object. For instance, an illumination component could provide flash or constant illumination of the target object (e.g., using one or more LEDs). An illumination component could also be configured to provide a light field that includes one or more of structured light, polarized light, and light with specific spectral content. Other types of light fields known and used to recover three-dimensional (3D) models from an object are possible within the context of the embodiments herein.

[0049] In some digital camera devices 100, either or both of front-facing camera 104 and rearfacing cameras 112, 114 may include or be associated with an ambient light sensor that may continuously or from time to time determine the ambient brightness of a scene that the camera can capture. In some devices, the ambient light sensor can be used to adjust the display brightness of a screen associated with the camera (e.g., a viewfinder). When the determined ambient brightness is high, the brightness level of the screen may be increased to make the screen easier to view. When the determined ambient brightness is low, the brightness level of the screen may be decreased, also to make the screen easier to view as well as to potentially save power. Additionally, the ambient light sensor’s input may be used to determine an exposure time of an associated camera, or to help in this determination.

[0050] Digital camera device 100 could be configured to use multi-element display 106 and either front-facing camera 104 or rear-facing cameras 112, 114 to capture images of a target object (e.g., a subject within a scene). The captured images could be a plurality of still images or a video image (e.g., a series of still images captured in rapid succession with or without accompanying audio captured by a microphone). The image capture could be triggered by activating shutter button 108, pressing a softkey on multi-element display 106, or by some other mechanism. Depending upon the implementation, the images could be captured automatically at a specific time interval, for example, upon pressing shutter button 108, upon appropriate lighting conditions of the target object, upon moving digital camera device 100 a predetermined distance, or according to a predetermined capture schedule.

[0051] As noted above, the functions of digital camera device 100 (or another type of digital camera) may be integrated into a computing device, such as a wireless computing device, cell phone, tablet computer, laptop computer, and so on. For example, a camera controller may be integrated with the digital camera device 100 to control one or more functions of the digital camera device 100.

[0052] Although some cameras may be equipped with a full resolution PD sensor, other cameras may employ a GM5 imaging sensor that provides sparse PD pixels.

[0053] FIG. 2 illustrates an imaging sensor 100 with sparse PD pixels, in accordance with example embodiments. For example, a PD L / R pair may appear at every 8 x 8 pixel block.

[0054] Such low-resolution sparse PD data may make it challenging for typical block matching algorithms (BMA) to accurately calculate disparity and focus the lens. To improve accuracy, a machine learning (ML) based approach may be used, such as, for example, based on a convolutional neural network (CNN), to directly estimate disparity from the sparse PD data.

[0055] Although an ML-based PDAF model can outperform BMA, there may be challenges with temporal consistency. This is because the ML-based PDAF model processes frame by frame and does not utilize historical focus values. Furthermore, the model is generally configured to assume that the ROI is single depth, which may result in an average focus when there is an ROI of multiple scene depth. For example, when both the foreground (FG) and background (BG) are at an average focus, both may appear blurry. However, it is desirable to have an image with a sharper FG and appropriate bokeh effect in the BG.

[0056] To overcome the aforementioned technical challenges, a dual-depth model that estimates two disparity values, one for the FG and another for the rest of the scene or the BG, is desirable. Such a model may be configured to be FG-biased and may be designed to have good FG / BG separation to improve sharpness at multi-depth ROIs. However, estimating two disparities instead of one is a more challenging problem and may result in lower stability as compared to a non-temporal approach. It may also be challenging to use the same model capacity and design when the number of pixels is reduced to half on average when splitting the ROI into two estimates.

[0057] Another challenge is that some cameras may have a wider aperture (e.g., F-number of 2.8) than other cameras (e.g., F-number of 3.5). A wider aperture enables a shallower depth of field (DoF), which can be beneficial for portrait photography as it helps to isolate the subject from the background. However, a shallow DoF can also make AF more difficult, as a slight drift in the lens position can result in the subject being out of focus. Accordingly, developing a dual-depth model for a camera may pose challenges, such as where the camera uses a sparse PD sensor (OCL L / R pair every 8x8 pixel block), which has a much lower resolution than a full-resolution PD pixels used in other cameras. Such a lower resolution PD pixel may significantly affect the quality of PDAF algorithms. Another challenge may involve estimating two disparities with less pixels for each disparity. This can be more difficult than estimating one disparity, and can also lead to increased model instability. Another challenge relates to ashallower depth of field (DoF). For example, a camera may have a shallower DoF, which means that even small errors in focus may be more pronounced.

[0058] It is desirable to solve the aforementioned technical problems without increasing model capacity or buffering frames. To address such issues, one approach may involve replacing the last few layers of a machine learning model with a recurrent neural network (RNN) unit. By doing so, the model may be configured to maintain an internal state and pass it to a next frame inference, which allows it to be temporally trained to preserve consistency with previous frames, thereby increasing stability. In some embodiments, a long short-term memory (LSTM) may be deployed as a recurrent network, since LSTM units may be better at preserving longterm memory than standard RNNs. In general, real camera systems are always streaming, and the historical data they generate may be utilized. Therefore, temporal models can be very useful for these systems.

[0059] However, to utilize a LSTM internal state and temporal consistency, a special type of data is needed that can hold temporal dependency and can reflect the temporal dynamics of real AF systems. For example, data that consists of static focus sweeps, which do not contain temporal information may not be useful. For focus ML-based models, focus sweeps that capture images at different blur and PD disparity levels may be desirable. However, it can be challenging to capture a focus sweep of a video in real time because it involves moving the lens in discrete steps to cover the entire lens motion range for a static scene. One way to achieve this may be to stop motion, but this is a tedious process and may be limited by the number of scenarios and short sequences that can be captured. This makes it hard to capture a large-scale dataset. To enable LSTM training and leverage temporal capabilities, an AF temporal data capture and synthesis framework may be used. Such a data framework may facilitate capture and synthesis of dual-depth temporal (TDD) sequence data with accurate ground truth, which can then be used for LSTM model training.

[0060] As described herein, a learning-based PDAF framework for an image sensor with sparse PD pixels may be introduced. A holistic framework that utilizes synthesized temporal data is described, including a network design to achieve the goal of consistent PDAF dual-disparity values over time.

[0061] Accordingly, a TDD framework is described that has significantly improved over other models for faces, low-light, high zoom (remosiac), and multi-depth scenes. Specifically, the model described herein reduces the failure rate from 22% to 0.67% for normal scenes, and from 85.71% to 2.56% for challenging scenes. Additionally, model evaluation demonstrates that the model described herein outperforms state-of-the-art competitor cameras.Temporal dual-depth (TDD) approach

[0062] As mentioned in the overview, a PDAF framework is described that can provide two or more disparities for sharper FG captures and clear FG / BG DoF separation. To maintain power and inference time constraints, the rich temporal dependency in PD sequences can be leveraged to enable an accurate dual-disparity inference.

[0063] Designing a PDAF algorithm that captures the temporal dependency can be significant for maintaining consistent disparity change over time (e.g., smooth lens transition and reduced lens fluctuations).

[0064] FIG. 3 illustrates a comparison between non-temporal and temporal machine learning (ML) based approaches for phase-difference autofocus (PDAF), in accordance with example embodiments. A non-temporal PDAF approach does not capture temporal dependency and does not utilize the rich historical information in previous PD disparities for more consistent disparity change (or smooth lens transition). As a result, inconsistencies in lens transition may be more likely to arise with the non-temporal PDAF approach. FIG. 3 depicts the difference between the non-temporal approach and the temporal approach.Temporal MobileNet (TMobileNet)

[0065] FIG. 4 is an example architecture for a dual-disparity machine learning (ML) based approach for phase-difference autofocus (PDAF), in accordance with example embodiments. In some embodiments, a recurrent dual-disparity model may be designed by adding LSTM units before a dense output layer. Also, for example, the output dense layer of the disparity model may be modified to provide two disparity values (i.e., dFGand dBGfor FG and BG disparities, respectively). The output of a confidence model may also be configured to provide two confidence values. The number of unnecessary nodes in both the disparity and confidence models may be reduced.

[0066] The TMobileNet may generally be a recurrent neural network that does not require buffering frames to maintain temporal consistency. Instead, a hidden internal state H may be shared from a previous inference to a next inference. This enables the network to learn longterm temporal dependencies and improve the accuracy of its predictions. The model may be configured to process one frame at a time and does not introduce any latency. However, the model can be configured to leverage the learned internal state weights to increase accuracy over time. The model can also handle variable ROI sizes. If the input ROI size is different from a default, the input may be first scaled to the default ROI size before being fed to the model. The input and output of the TmobileNet may be described as:dn= PDAF Rn.Ln. Hn- ,(Eqn. 1)

[0067] where dnis a disparity value at time n, Rn, Lnare the right and left PD views of selectedis an internal state that is passed on to time n from time n — 1.

[0068] In some embodiments, an MAE loss function may be used. The Mean Absolute Error (MAE) loss may be used for training the model. In some embodiments, a center-weighted MAE loss function may be used. For example, the center weight term may be added to penalize more for small disparities around a zero disparity lens position (i.e., ±10% of the disparity range). This may emphasize sharper AF responses. The center weight MAE loss may be defined as follows:(Eqn. 2) if digt< 0.1 * disparityrangeotherwise(Eqn. 3)

[0069] where df1and df are the ground truth and estimated disparities, respectively,is a center weight term. In some embodiments, the confidence model may be adapted to indicate how confident the disparity prediction may be. In some embodiments, the two subnetworks, disparity model and confidence model, may be trained in two phases: (1) train disparity only; (2) then freeze the pre-trained disparity model and train confidence based on the disparity error. The confidence values output by the confidence network may range from 0 to 1. The higher the error, the lower the confidence.TDD training

[0070] Designing a temporal model architecture may only be the first step. The more challenging task may be obtaining temporal data for training and developing a training procedure that takes into account the temporal dimension. This involves temporal dynamics and multiple training decisions and experimentation.Data in SStables for model training

[0071] With the temporal data being verified, synthesized, and stored, training directly on that data may not be efficient as it doesn’t utilize a TensorFlow (TF) infrastructure. In some embodiments, the data may be stored in an SSTable format, which is a specific data storage format for TF. The SSTable format may facilitate efficient data loading during training. Each entry in the SSTable may be a tf. example object that contains a temporal sequence of left / rightPD views (e.g., in PNG format), as well as the corresponding two disparities ground truth for each frame (e.g., FG and BG disparities for each frame). The path for each frame may be saved, along with some additional metadata for training purposes (e.g., VCM). Also, for example, the corresponding FG / BG mask and mask metadata (i.e., center coordinates and area ratio) may be saved for each frame for experimental training purposes. In some embodiments, the data may be split into train, validation, and test sets in the ratio 80: 10: 10, respectively.

[0072] In some embodiments, the data may be stored in CNS for faster data loading. Additionally, the script may be adjusted for different datasets, such as the input path of focus sweeps, the dataset type, and a number of shards of the SSTable, and some other parameters. The number of SSTable shards may need to be balanced based on the dataset size. Specifically, the number of shards may be selected to be larger than the number of parallel workers, and it may be desirable for a final output shard to be within 100-200 MB in size for efficient data loading.Temporal training strategy

[0073] Temporal models have a time dimension that should be taken into account during training. This is because temporal models can learn temporal dependencies between frames, which would be difficult to capture if the frames were trained independently. For example, training a temporal model with individual frames that are not temporally related would likely not be useful. The temporal data may be stored in SSTables, where each entry is a variablelength sequence of frames. The sequence length varies from 10 to 90 frames, and may be determined from the data synthesis stage.

[0074] Recurrent neural networks such as LSTM units can take variable-length sequences. However, for simplicity, the TDD model may be trained using fixed-size frames that may be randomly trimmed from a full sequence at each training iteration. Specifically, at each epoch, an iteration may be performed over the samples and a smaller sequence of frames (e.g., 10 frames) may be randomly trimmed from each sample. Temporal mini -patches of these min- sequences (may be performed, and multiple models with different mini-patch sizes (e.g., 32, 48, and 64 temporal mini-samples) may be trained, and a best model may be selected based on a performance on a validation set.

[0075] FIG. 5 illustrates an example training process for a recurrent neural network (RNN) using randomly trimmed fixed-size frames, in accordance with example embodiments. FIG. 5 shows the procedure for trimming mini-sequences from the variable-length sequences that are synthesized and stored in SSTables.Running model training

[0076] To train the TDD (or TMobile) model, there are several hyperparameters that need to be set for training. Also, for example, the training may be parallelized by utilizing flume and xManager to train multiple models concurrently with different hyperparameters.

[0077] For example, the xManager training may be launched with multiple parallel jobs, each with a different set of hyperparameters. As described earlier, there are two models that need to be trained in two stages: the disparity model, followed by the confidence model.

[0078] In some embodiments, a single GPU (num_gpus=l) may be used to train the two models. For example, 16 work units may be run in parallel (parallel_work_units=16), with a maximum number of trials of 32 (max_num_trials=32) in case any of the workers fail. The epoch number may be set to 200 to train the disparity model and to 120 to train the confidence model, as both tend to converge around these numbers. The parameter train temporal model may be set to True to train a temporal model, and data augmentation may be disabled at training time, as the augmentation / simulation may be performed at the data synthesis phase.

[0079] In some embodiments, the learning rate range may be set from 5e-4 to le-2 and a minibatch size may be set to 32 and 48. Also, for example, pretrained models may be fine-tuned rather than starting from scratch, such as by setting FINE TUNE MODEL to True and providing the pretrained model path.

[0080] Data augmentation can play a significant role in stretching a boundary of the data domain space by introducing different image manipulation techniques (e.g., noise, color, scaling, flipping, rotation, and so forth). Many ML-based methods apply augmentation on the fly during training. In some embodiments, data augmentation may be applied during the temporal data synthesis stage.

[0081] In some embodiments, offline data synthesis may be more suitable for the framework described herein. For example, multiple verification steps may be performed at different stages of offline data synthesis. This can be very challenging to perform on the fly during training runtime to ensure that the data is completely accurate. Another important consideration may be the training time. Involving data synthesis in training would likely increase the training time by at least a factor of three.

[0082] Training temporal models is a challenging task because of the temporal dynamics involved. Additionally, temporal models are sensitive to data distribution and can easily become biased.Long-memory state may lead to focus freeze

[0083] One advantage with LSTM is a long-term memory (internal state) that can be passed to future frames. In some embodiments, a balance may need to be struck between keeping theinternal state when it is useful and dropping it when it is not. For example, if there is no significant scene change or target focus change, it can be beneficial to keep the internal state for more consistent prediction. However, having a persistent and always long-term memory may increase a delay in lens transition for the target focus when the scene changes.

[0084] Based on the data synthesis framework, the sequences may be synthesized without scene change data, and each sample may represent one scene dynamic. Additionally, each sequence may converge to a zero disparity lens position. These two biases can cause the LSTM to have longer states and tend to converge to zero disparity, regardless of the scene change. This may result in freezing of the lens position in some videos, even when the scene changes.

[0085] One approach to solve this issue may be via explicit training with scene change data. To enable this approach, the data synthesis framework may be modified and sequences with scene changes and disparity magnitude and direction change may be included. The model may be retrained with these sequences to allow it to adaptively detect scene changes and reset its internal state accordingly, without any delays or loss of internal state. In some embodiments, this may be achieved by splitting the sequences into smaller segments, shuffling the segments, and mixing the segments segment- wise with other sequences randomly to create new sequences that contain within-sequence scene changes.

[0086] FIG. 6 illustrates an example training process to capture frame sequences with scene changes, in accordance with example embodiments. Using this approach, the LSTM model may be configured to adaptively reset its internal state based on scene changes, and the focus freeze issue may be successfully resolved.Splitting longer sequences

[0087] To create scene change sequences, two sequences may be interleaved segment-wise, which doubles the length of the sequence. For example, prior to the scene-change data, the sequence length ranges from 10 to 90 frames. After the scene change, the sequence length ranges from 20 to 180 frames. In some embodiments, at runtime, when training the model, the entire sequence may be loaded and then a short sequence may be randomly trimmed for minibatch training. As mentioned earlier, a mini-batch size of 48 may be used for training. This means loading 48 sequences to randomly trim mini-sequences at each iteration. Technically, this doubles the mini-batch size and may cause Flume data loading pipeline and memory issues, especially when parallelizing the process on multiple jobs.

[0088] To address this issue, longer sequences may be split into multiple shorter sequences. For example, three sequence categories may be defined based on length: less than 80 frames, greater than 80 frames but less than 120 frames, and greater than 120 frames. Sequences in thefirst category may not be split. Sequences in the second and third categories may be split into two and three equally smaller sequences, respectively.Challenging foreground (FG) in the ROI

[0089] One of the biggest challenges for dual-disparity models is when the FG does not have enough frequencies and becomes problematic. The FG in the AF ROI can be problematic for many reasons, such as being small, lacking in texture related detail, or having disconnected shapes (e.g., chain fence). Such problematic FG can result in an unstable FG disparity prediction. Since the FG disparity is generally preferred for the target lens position, this can lead to focus problems.

[0090] For example, temporal single-depth (TSD) tends to average focus, and temporal dualdepth (TDD) prefers FG disparity. Accordingly, TDD may have a step transition between FG and BG, whereas TSD may have a gradual focus change that results in average focus when the ROI is mixed FG / BG. One approach to handle problematic FG is at the disparity model level. For example, the model may be trained to avoid FG disparity and instead consider BG disparity when FG is problematic. For example, the problematic FG may be defined as a small area with a non-central location. Mask metadata saved during data synthesis may be used to train a model to check FG area and location in every frame. In some embodiments, the model may be configured to ignore FG disparity when the FG area is below an area threshold ratio and the FG center is away from the ROI center for a given pixel distance threshold. Models generally demonstrate a lens hunting point that occurs around a boundary, which may depend on the scene and the model threshold. Problematic FG can depend on FG frequencies and / or shape, in addition area and location.

[0091] In some embodiments, a post-inference hunting prevention unit may be added to mitigate lens hunting. This unit may be configured to clamp the signal when a peak lens hunting event is detected, and / or when the lens direction is changing. Also, for example, BG disparity may be used instead of FG disparity when hunting is detected. Generally, the hunting prevention unit may be configured to clamp the lens position when lens target position (tPos) hunting is detected.TDD evaluation and results

[0092] Qualitative and quantitative evaluations may be performed by measuring AF stability metrics, MAE for accuracy, failure rate, AF score, and visually assess lens motion curves with respect to ground truth curves.AF Evaluation Metrics

[0093] To measure AF stability, stability of the lens position motion overtime may be analyzed based on multiple factors. This can be helpful for A / B comparisons. Such metrics may include a histogram energy of a standard deviation of local smaller segments, as well as other metrics like total lens movements.

[0094] In some embodiments, a number of total lens movements (LM) may be used as an AF evaluation metric. This may involve summing up the lens-movement differences between consecutive frames. This metric also measures the relative stability when testing two AF modules under the same conditions. A lower LM is indicative of a lower power consumption and higher stability.

[0095] In some embodiments, a number of lens position changes (LC) may be used as an AF evaluation metric. In some embodiments, LC may be added to LM in case the number of lens changes between consecutive frames is to be determined. LC is indicative of stability across video frames. For example, LC for a static scene may be very low.

[0096] In some embodiments, a fluctuation histogram energy (E) may be used as an AF evaluation metric. This may involve measuring the fluctuation amount for lens target position (tpos). Also, for example, a histogram distribution for multiple fluctuation categories (low to high) may be determined. Such histogram bins may be indicative of potential AF hunting.

[0097] In some embodiments, a number of non-PDAF calls (NPC) may be used as an AF evaluation metric. There may be multiple different AF search types (e.g., SType), and this metric counts the number of non-PDAF SType calls. A lower NPC is indicative of an efficient AF system.

[0098] In some embodiments, a PDAF coverage (PC) may be used as an AF evaluation metric. PC is a percentage of PDAF usage across the video. PC helps in optimizing PDAF confidence and is an indicator for stability. CDAF is configured to perform a scan to locate the position of maximum sharpness and may not provide the best user experience for AF, especially in video since this may be perceived as hunting. A higher PC is indicative of better AF performance.Temporal model and history data

[0099] Generally speaking, real camera systems stream frames, and historical data can be available when the camera application is opened. One evaluation approach involves a comparison of how useful the historical sequence is to temporal models and whether increasing a number of frames would likely increase the accuracy of temporal models.

[0100] FIG. 7 illustrates performance comparisons between non-temporal and temporal machine learning (ML) based approaches for phase-difference autofocus (PDAF), in accordance with example embodiments. As illustrated in the first row of the plot in FIG. 7, thetemporal model has an overall MAE on sequence data that is about two times lower than the non-temporal model. Additionally, the second row results illustrate that an average MAE is minimized. Indeed, the estimate is more accurate as more frames are added to the temporal model inference (blue curve). In contrast, the non-temporal model has an independent estimate accuracy for each frame because there is no temporal association between the frames (red curve). These results show that the temporal model can leverage the continuous camera stream and has superior performance over the non-temporal model.AF stability evaluation

[0101] FIG. 8 illustrates autofocus (AF) stability performance comparisons between various approaches for phase-difference autofocus (PDAF), in accordance with example embodiments. Stability tests may be performed for comparisons between a HK3 Tele (corresponding to a GM5 sparse PD sensor) and CIO Tele. Table 800 reports the AF stability performance of 18 videos captured side-by-side from CIO and HK3. The arrow next to each metric in the table indicates whether a lower or higher value is better. Best numbers are highlighted in green and on-par results are highlighted in yellow.

[0102] The results in table 800 indicate that the HK3 temporal model significantly outperforms the CIO model for all metrics. For example, the PDAF coverage (PC) of HK3 is 95.05%, which is significantly higher than the 69.66% of CIO. This means that HK3 uses PDAF a majority of the time, which predicts lens position from a single image and does not rely on other AF systems like CDAF, which can be slow and require lens position search. Another significant improvement is that the total lens movements (LM) of HK3 are about 7 times lower (i.e., 1223 forHK3 vs. 8535 for CIO). This is a significant reduction of unnecessary lens motion, which makes AF more stable and also reduces power consumption.

[0103] FIG. 9 illustrates example image comparisons 900 for single-depth and multidepth machine learning (ML) based approaches for phase-difference autofocus (PDAF), in accordance with example embodiments. When the ML based approach is based on an ROI that is at a single depth, an average focus is determined for scenes with ROI of multiple scene depth. FIG. 9 illustrates an example of average focus when the ROI is at multiple depths. The singledepth model (left-hand side) has both the foreground (FG) and background (BG) at average focus, which makes both slightly blurry. The multi-depth model (right-hand side) would have a sharper FG and nice bokeh in the BG.Training Machine Learning Models for Generating Inferences / Predictions

[0104] FIG. 10 shows diagram 1000 illustrating a training phase 1002 and an inference phase 1004 of trained machine learning model(s) 1032, in accordance with example embodiments. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions about (patterns in the) training data. The resulting trained machine learning algorithm can be termed as a trained machine learning model. For example, FIG. 10 shows training phase 1002 where one or more machine learning algorithms 1020 are being trained on training data 1010 to become trained machine learning model(s) 1032. Then, during inference phase 1004, trained machine learning model(s) 1032 can receive input data 1030 and one or more inference / prediction requests 1040 (perhaps as part of input data 1030) and responsively provide as an output one or more inferences and / or predict! on(s) 1050.

[0105] As such, trained machine learning model(s) 1032 can include one or more models of one or more machine learning algorithms 1020. Machine learning algorithm(s) 1020 may include, but are not limited to: an artificial neural network (e.g., a herein-described convolutional neural networks, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system). Machine learning algorithm(s) 1020 may be supervised or unsupervised, and may implement any suitable combination of online and offline learning. Examples of the models may include multi-layer perceptron (MLP) models, convolutional neural networks (CNN), long short-term memory (LSTM) algorithms, generative adversarial networks (GAN), K-means clustering, Gaussian mixture models (GMM), or any other of a number of machinelearning training techniques known to a person of ordinary skill in the art. The models may comprise a single type of model or multiple types, as well as various combinations of types including combinations of single types and of multiple types.

[0106] In some examples, machine learning algorithm(s) 1020 and / or trained machine learning model(s) 1032 can be accelerated using on-device coprocessors, such as graphic processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application specific integrated circuits (ASICs). Such on-device coprocessors can be used to speed up machine learning algorithm(s) 1020 and / or trained machine learning model(s) 1032. In some examples, trained machine learning model(s) 1032 can be trained, reside and execute to provide inferences on a particular computing device, and / or otherwise can make inferences for the particular computing device.

[0107] During training phase 1002, machine learning algorithm(s) 1020 can be trained by providing at least training data 1010 as training input using unsupervised, supervised, semisupervised, and / or reinforcement learning techniques. Training data 1010 may include the TDD data synthesized herein. Unsupervised learning involves providing a portion (or all) of training data 1010 to machine learning algorithm(s) 1020 and machine learning algorithm(s) 1020 determining one or more output inferences based on the provided portion (or all) of training data 1010. Supervised learning involves providing a portion of training data 1010 to machine learning algorithm(s) 1020, with machine learning algorithm(s) 1020 determining one or more output inferences based on the provided portion of training data 1010, and the output inference(s) are either accepted or corrected based on correct results associated with training data 1010. In some examples, supervised learning of machine learning algorithm(s) 1020 can be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or set of labels may be used to correct inferences of machine learning algorithm(s) 1020.

[0108] For example, the training models may use supervised learning methods, where the training data may be labeled and outputs may be graded based on their fidelity to a “truth” output. The training models may use an unsupervised learning method, where there may not be labels on the training data and the training models may classify correlations without reference to a “truth” value. The training models may combine supervised and unsupervised techniques.

[0109] Semi-supervised learning involves having correct results for part, but not all, of training data 1010. During semi-supervised learning, supervised learning is used for a portion of training data 1010 having correct results, and unsupervised learning is used for a portion of training data 1010 not having correct results. Reinforcement learning involves machine learning algorithm(s) 1020 receiving a reward signal regarding a prior inference, where the reward signal can be a numerical value. During reinforcement learning, machine learning algorithm(s) 1020 can output an inference and receive a reward signal in response, where machine learning algorithm(s) 1020 are configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing an expected total of the numerical values provided by the reward signal over time. In some examples, machine learning algorithm(s) 1020 and / or trained machine learning model(s) 1032 can be trained using other machine learning techniques, including but not limited to, incremental learning and curriculum learning.

[0110] In some examples, machine learning algorithm(s) 1020 and / or trained machine learning model(s) 1032 can use transfer learning techniques. For example, transfer learningtechniques can involve trained machine learning model(s) 1032 being pre-trained on one set of data and additionally trained using training data 1010. More particularly, machine learning algorithm(s) 1020 can be pre-trained on data from one or more computing devices and a resulting trained machine learning model provided to computing device CD1, where CD1 is intended to execute the trained machine learning model during inference phase 1004. Then, during training phase 1002, the pre-trained machine learning model can be additionally trained using training data 1010, where training data 1010 can be derived from kernel and non-kernel data of computing device CD1. This further training of the machine learning algorithm(s) 1020 and / or the pre-trained machine learning model using training data 1010 of CDl’s data can be performed using either supervised or unsupervised learning. Once machine learning algorithm(s) 1020 and / or the pre-trained machine learning model has been trained on at least training data 1010, training phase 1002 can be completed. The trained resulting machine learning model can be utilized as at least one of trained machine learning model(s) 1032.

[0111] In particular, once training phase 1002 has been completed, trained machine learning model(s) 1032 can be provided to a computing device, if not already on the computing device. Inference phase 1004 can begin after trained machine learning model(s) 1032 are provided to computing device CD1.

[0112] In the context of this disclosure, a ML model may be trained. Training data 1010 may be image data. In this example, the training data 1010 are first processed by a CNN. The CNN may extract features from the training data 1010. The CNN may be a disconnected neural network, meaning all members of the neural network, such as perceptrons, may not be in a causal relationship with one another. It should be mentioned that other feature extraction techniques besides CNNs may be equally employed in lieu of the CNN described herein. An MLP neural network may receive the extracted feature outputs from the CNN as inputs. The MLP may be a fully connected network, which may comprise all perceptrons in a causal relationship with one another. The MLP may have as an output a detection probability prediction 1050. The prediction 1050 may be, for example, the probability that one or more focus ranges are correct for a camera input, or some other prediction related to AF, PDAF, or similar.

[0113] During inference phase 1004, trained machine learning model(s) 1032 can receive input data 1030 and generate and output one or more corresponding inferences and / or prediction(s) 1050 about input data 1030. Input data 1030 may be data from one or more sensors. As such, input data 1030 can be used as an input to trained machine learning model(s) 1032 for providing corresponding inference(s) and / or predict! on(s) 1050 to kernel componentsand non-kernel components. For example, trained machine learning model(s) 1032 can generate inference(s) and / or predict! on(s) 1050 in response to one or more inference / prediction requests 1040. In some examples, trained machine learning model(s) 1032 can be executed by a portion of other software. For example, trained machine learning model(s) 1032 can be executed by an inference or prediction daemon to be readily available to provide inferences and / or predictions upon request. Input data 1030 can include data from computing device CD1 executing trained machine learning model(s) 1032 and / or input data from one or more computing devices other than CD1. For example, input data 1030 can include images, prompts. Input data 1030 can also include prompts labeled with context data, device data, browsing history, search history, and so forth. Other types of input data are possible as well.

[0114] The trained machine learning model(s) 1032 may use a matrix containing depth of field calculations, an array of input imagery, or one or more numbers between 0 and 1 as the prediction 1050. Such an ML model may have one or more feature extraction layers, such as the CNN, and one or more fully connected layers, such as the MLP.

[0115] Inference(s) and / or predict! on(s) 1050 can include output prompts, output partial prompts, output prompt completions, numerical values, and / or other output data produced by trained machine learning model(s) 1032 operating on input data 1030 (and training data 1010). In some examples, trained machine learning model(s) 1032 can use output inference(s) and / or predict! on(s) 1050 as input feedback 1060. Trained machine learning model(s) 1032 can also rely on past inferences as inputs for generating new inferences.

[0116] In some examples, one computing device CD SOLO can include the trained version of the neural network, perhaps after training. Then, computing device CD SOLO can receive a request to predict a caption, and use the trained version of the neural network to predict the caption.

[0117] In some examples, two or more computing devices CD CLI and CD SRV can be used to provide output captions; e.g., a first computing device CD CLI can generate and send requests to predict a caption to a second computing device CD SRV. Then, CD SRV can use the trained version of the neural network, to predict the caption, and respond to the requests from CD CLI for the predicted caption. Then, upon reception of responses to the requests, CD CLI can provide the requested predicted caption (e.g., using a multimodal user interface and / or a display).

[0118] An ML model, as discussed in this disclosure, refers to a computer model that has been trained using one or more machine-learning techniques. In general, this training may be done by providing training inputs to one or more training models, which in turn may providean output. The output may be in the form of a prediction, a confidence score, or other probability-based metrics.

[0119] The aforementioned are merely examples of the types of machine-learning training that may be done to produce an ML model for AF. Many other combinations and techniques may be employed, which do not alter the scope of the inventive concept. Examples such as employing a CNN, use of different data types than those listed, alone or in combination with the data types listed, having a fully connected portion of the machine-learning training, etc. are not meant to limit the scope of the disclosed methods and apparatuses, but rather to serve as examples of how an ML model for AF may be accomplished.

[0120] By way of example, training for the ML model may equivalently be accomplished by incorporating, as either a part of or the complete machine-learning training phase 1002, a long short-term memory (LSTM) algorithm. The LSTM algorithm is a type of recombinant neural network (RNN), which may process data in a time-indexed fashion. The LSTM may solve a so-called “vanishing gradient problem,” in which gradients used in fitting may tend to zero and, thus, may not yield useful fit parameters for a given model (e.g., weights and biases). During the training phase 1002, the LSTM may allow for persistent gradients used to fit when the gradients may otherwise go to zero (e.g., in a traditional RNN). For the purpose of the present disclosure, an example of implementing the LSTM in the machine-learning training phase 1002 may be to replace a MLP with the LSTM or to include the LSTM before or after the MLP.Example Data Network

[0121] FIG. 11 depicts a distributed computing architecture 1100, in accordance with example embodiments. Distributed computing architecture 1100 includes server devices 1108, 1110 that are configured to communicate, via network 1106, with programmable devices 1104a, 1104b, 1104c, 1104d, 1104e. Network 1106 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public Internet, or any other type of network configured to provide a communications path between networked computing devices. Network 1106 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public Internet.

[0122] Although FIG. 11 only shows five programmable devices, distributed application architectures may serve tens, hundreds, or thousands of programmable devices. Moreover, programmable devices 1104a, 1104b, 1104c, 1104d, 1104e (or any additional programmable devices) may be any sort of computing device, such as a mobile computingdevice, desktop computer, wearable computing device, head-mountable device (HMD), network terminal, a mobile computing device, and so on. In some examples, such as illustrated by programmable devices 1104a, 1104b, 1104c, 1104e, programmable devices can be directly connected to network 1106. In other examples, such as illustrated by programmable device 1104d, programmable devices can be indirectly connected to network 1106 via an associated computing device, such as programmable device 1104c. In this example, programmable device 1104c can act as an associated computing device to pass electronic communications between programmable device 1104d and network 1106. In other examples, such as illustrated by programmable device 1104e, a computing device can be part of and / or inside a vehicle, such as a car, a truck, a bus, a boat or ship, an airplane, etc. In other examples not shown in FIG. 11, a programmable device can be both directly and indirectly connected to network 1106.

[0123] Server devices 1108, 1110 can be configured to perform one or more services, as requested by programmable devices 1104a-1104e. For example, server device 1108 and / or 1110 can provide content to programmable devices 1104a-l 104e. The content can include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. The content can include compressed and / or uncompressed content. The content can be encrypted and / or unencrypted. Other types of content are possible as well.

[0124] As another example, server device 1108 and / or 1110 can provide programmable devices 1104a-1104e with access to software for database, search, computation, graphical, audio, video, World Wide Web / Internet utilization, and / or other functions. Many other examples of server devices are possible as well.Computing Device Architecture

[0125] FIG. 12 is a block diagram of an example computing device 1200, in accordance with example embodiments. In particular, computing device 1200 shown in FIG. 12 can be configured to perform at least one function of and / or related to the machine learning models, and / or methods 1400, and / or 1300.

[0126] Computing device 1200 may include a user interface module 1201, a network communications module 1202, one or more processors 1203, data storage 1204, one or more camera(s) 1212, one or more sensors 1214, and power system 1216, all of which may be linked together via a system bus, network, or other connection mechanism 1205.

[0127] User interface module 1201 can be operable to send data to and / or receive data from external user input / output devices. For example, user interface module 1201 can be configured to send and / or receive data to and / or from user input devices such as a touch screen,a computer mouse, a keyboard, a keypad, a touch pad, a trackball, a joystick, a voice recognition module, and / or other similar devices. User interface module 1201 can also be configured to provide output to user display devices, such as one or more cathode ray tubes (CRT), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, either now known or later developed. User interface module 1201 can also be configured to generate audible outputs, with devices such as a speaker, speaker jack, audio output port, audio output device, earphones, and / or other similar devices. User interface module 1201 can further be configured with one or more haptic devices that can generate haptic outputs, such as vibrations and / or other outputs detectable by touch and / or physical contact with computing device 1200. In some examples, user interface module 1201 can be used to provide a graphical user interface (GUI) for utilizing computing device 1200, such as, for example, a graphical user interface of a mobile phone device.

[0128] Network communications module 1202 can include one or more devices that provide one or more wireless interface(s) 1207 and / or one or more wireline interface(s) 1208 that are configurable to communicate via a network. Wireless interface(s) 1207 can include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth™ transceiver, a Zigbee® transceiver, a Wi-Fi™ transceiver, a WiMAX™ transceiver, an LTE™ transceiver, and / or other type of wireless transceiver configurable to communicate via a wireless network. Wireline interface(s) 1208 can include one or more wireline transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or similar transceiver configurable to communicate via a twisted pair wire, a coaxial cable, a fiber-optic link, or a similar physical connection to a wireline network.

[0129] In some examples, network communications module 1202 can be configured to provide reliable, secured, and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., guaranteed message delivery) can be provided, perhaps as part of a message header and / or footer (e.g., packet / message sequencing information, encapsulation headers and / or footers, size / time information, and transmission verification information such as cyclic redundancy check (CRC) and / or parity check values). Communications can be made secure (e.g., be encoded or encrypted) and / or decry pted / decoded using one or more cryptographic protocols and / or algorithms, such as, but not limited to, Data Encryption Standard (DES), Advanced Encryption Standard (AES), a Rivest-Shamir-Adelman (RSA) algorithm, a Diffie-Hellman algorithm, a secure sockets protocol such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS),and / or Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms can be used as well or in addition to those listed herein to secure (and then decry pt / decode) communications.

[0130] One or more processors 1203 can include one or more general purpose processors, and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). One or more processors 1203 can be configured to execute computer- readable instructions 1206 that are contained in data storage 1204 and / or other instructions as described herein.

[0131] Data storage 1204 can include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of one or more processors 1203. The one or more computer-readable storage media can include volatile and / or non-volatile storage components, such as optical, magnetic, organic or other memory or disc storage, which can be integrated in whole or in part with at least one of one or more processors 1203. In some examples, data storage 1204 can be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disc storage unit), while in other examples, data storage 1204 can be implemented using two or more physical devices.

[0132] Data storage 1204 can include computer-readable instructions 1206 and perhaps additional data. In some examples, data storage 1204 can include storage required to perform at least part of the herein-described methods, scenarios, and techniques and / or at least part of the functionality of the herein-described devices and networks. In some examples, data storage 1204 can include storage for a trained neural network model 1210 (e.g., trained RNN, trained LSTM, etc.). In particular of these examples, computer-readable instructions 1206 can include instructions that, when executed by one or more processors 1203, enable computing device 1200 to provide for some or all of the functionality of trained neural network model 1210.

[0133] In some examples, computing device 1200 can include one or more camera(s) 1212. Camera(s) 1212 can include one or more image capture devices, such as still and / or video cameras, equipped to capture light and record the captured light in one or more images; that is, camera(s) 1212 can generate image(s) of captured light. The one or more images can be one or more still images and / or one or more images utilized in video imagery. Camera(s) 1212 can capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light.

[0134] In some examples, computing device 1200 can include one or more sensors 1214. Sensors 1214 can be configured to measure conditions within computing device 1200and / or conditions in an environment of computing device 1200 and provide data about these conditions. For example, sensors 1214 can include one or more of: (i) sensors for obtaining data about computing device 1200, such as, but not limited to, a thermometer for measuring a temperature of computing device 1200, a battery sensor for measuring power of one or more batteries of power system 1216, and / or other sensors measuring conditions of computing device 1200; (ii) an identification sensor to identify other objects and / or devices, such as, but not limited to, a Radio Frequency Identification (RFID) reader, proximity sensor, one-dimensional barcode reader, two-dimensional barcode (e.g., Quick Response (QR) code) reader, and a laser tracker, where the identification sensors can be configured to read identifiers, such as RFID tags, barcodes, QR codes, and / or other devices and / or object configured to be read and provide at least identifying information; (iii) sensors to measure locations and / or movements of computing device 1200, such as, but not limited to, a tilt sensor, a gyroscope, an accelerometer, a Doppler sensor, a GPS device, a sonar sensor, a radar device, a laser-displacement sensor, and a compass; (iv) an environmental sensor to obtain data indicative of an environment of computing device 1200, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a movement sensor, a microphone, a sound sensor, an ultrasound sensor and / or a smoke sensor; and / or (v) a force sensor to measure one or more forces (e.g., inertial forces and / or G-forces) acting about computing device 1200, such as, but not limited to one or more sensors that measure: forces in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor that identifies ZMPs and / or locations of the ZMPs. Many other examples of sensors 1214 are possible as well.

[0135] Power system 1216 can include one or more batteries 1218 and / or one or more external power interfaces 1220 for providing electrical power to computing device 1200. Each battery of the one or more batteries 1218 can, when electrically coupled to the computing device 1200, act as a source of stored electrical power for computing device 1200. One or more batteries 1218 of power system 1216 can be configured to be portable. Some or all of one or more batteries 1218 can be readily removable from computing device 1200. In other examples, some or all of one or more batteries 1218 can be internal to computing device 1200, and so may not be readily removable from computing device 1200. Some or all of one or more batteries 1218 can be rechargeable. For example, a rechargeable battery can be recharged via a wired connection between the battery and another power supply, such as by one or more power supplies that are external to computing device 1200 and connected to computing device1200 via the one or more external power interfaces. In other examples, some or all of one or more batteries 1218 can be non-rechargeable batteries.

[0136] One or more external power interfaces 1220 of power system 1216 can include one or more wired-power interfaces, such as a USB cable and / or a power cord, that enable wired electrical power connections to one or more power supplies that are external to computing device 1200. One or more external power interfaces 1220 can include one or more wireless power interfaces, such as a Qi wireless charger, that enable wireless electrical power connections, such as via a Qi wireless charger, to one or more external power supplies. Once an electrical power connection is established to an external power source using one or more external power interfaces 1220, computing device 1200 can draw electrical power from the external power source the established electrical power connection. In some examples, power system 1216 can include related sensors, such as battery sensors associated with the one or more batteries or other types of electrical power sensors.Cloud-Based Servers

[0137] FIG. 13 depicts a cloud-based server system in accordance with an example embodiment. In FIG. 13, functionality of a neural network, and / or a computing device can be distributed among computing clusters 1309a, 1309b, and 1309c. Computing cluster 1309a can include one or more computing devices 1300a, cluster storage arrays 1310a, and cluster routers 1311a connected by a local cluster network 1312a. Similarly, computing cluster 1309b can include one or more computing devices 1300b, cluster storage arrays 1310b, and cluster routers 1311b connected by a local cluster network 1312b. Likewise, computing cluster 1309c can include one or more computing devices 1300c, cluster storage arrays 1310c, and cluster routers 1311c connected by a local cluster network 1312c.

[0138] In some embodiments, computing clusters 1309a, 1309b, and 1309c can be a single computing device residing in a single computing center. In other embodiments, computing clusters 1309a, 1309b, and 1309c can include multiple computing devices in a single computing center, or even multiple computing devices located in multiple computing centers located in diverse geographic locations. For example, FIG. 13 depicts each of computing clusters 1309a, 1309b, 1309c residing in different physical locations.

[0139] In some embodiments, data and services at computing clusters 1309a, 1309b, 1309c can be encoded as computer readable information stored in non-transitory, tangible computer readable media (or computer readable storage media) and accessible by other computing devices. In some embodiments, computing clusters 1309a, 1309b, 1309c can bestored on a single disk drive or other tangible storage media, or can be implemented on multiple disk drives or other tangible storage media located at one or more diverse geographic locations.

[0140] In some embodiments, each of computing clusters 1309a, 1309b, and 1309c can have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. In other embodiments, however, each computing cluster can have different numbers of computing devices, different numbers of cluster storage arrays, and different numbers of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster can depend on the computing task or tasks assigned to each computing cluster.

[0141] In computing cluster 1309a, for example, computing devices 1300a can be configured to perform various computing tasks of a neural network, and / or a computing device. In one embodiment, the various functionalities of a neural network, and / or a computing device can be distributed among one or more of computing devices 1300a, 1300b, and 1300c. Computing devices 1300b and 1300c in respective computing clusters 1309b and 1309c can be configured similarly to computing devices 1300a in computing cluster 1309a. On the other hand, in some embodiments, computing devices 1300a, 1300b, and 1300c can be configured to perform different functions.

[0142] In some embodiments, computing tasks and stored data associated with a neural network, and / or a computing device can be distributed across computing devices 1300a, 1300b, and 1300c based at least in part on the processing requirements of a neural network, and / or a computing device, the processing capabilities of computing devices 1300a, 1300b, 1300c, the latency of the network links between the computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that can contribute to the cost, speed, fault-tolerance, resiliency, efficiency, and / or other design goals of the overall system architecture.

[0143] Cluster storage arrays 1310a, 1310b, 1310c of computing clusters 1309a, 1309b, and 1309c can be data storage arrays that include disk array controllers configured to manage read and write access to groups of hard disk drives. The disk array controllers, alone or in conjunction with their respective computing devices, can also be configured to manage backup or redundant copies of the data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0144] Similar to the manner in which the functions of a conditioned, axial selfattention based neural network, and / or a computing device can be distributed across computingdevices 1300a, 1300b, 1300c of computing clusters 1309a, 1309b, 1309c, various active portions and / or backup portions of these components can be distributed across cluster storage arrays 1310a, 1310b, 1310c. For example, some cluster storage arrays can be configured to store one portion of the data of a first layer of a neural network, and / or a computing device, while other cluster storage arrays can store other portion(s) of data of second layer of a neural network, and / or a computing device. Also, for example, some cluster storage arrays can be configured to store the data of an encoder of a neural network, while other cluster storage arrays can store the data of a decoder of a neural network. Additionally, some cluster storage arrays can be configured to store backup versions of data stored in other cluster storage arrays.

[0145] Cluster routers 1311a, 1311b, 1311c in computing clusters 1309a, 1309b, and 1309c can include networking equipment configured to provide internal and external communications for the computing clusters. For example, cluster routers 1311a in computing cluster 1309a can include one or more internet switching and routing devices configured to provide (i) local area network communications between computing devices 1300a and cluster storage arrays 1310a via local cluster network 1312a, and (ii) wide area network communications between computing cluster 1309a and computing clusters 1309b and 1309c via wide area network link 1313a to network 1106. Cluster routers 1311b and 1311c can include network equipment similar to cluster routers 1311a, and cluster routers 1311b and 1311c can perform similar networking functions for computing clusters 1309b and 1309b that cluster routers 1311a perform for computing cluster 1309a.

[0146] In some embodiments, the configuration of cluster routers 1311a, 1311b, 1311c can be based at least in part on the data communication requirements of the computing devices and cluster storage arrays, the data communications capabilities of the network equipment in cluster routers 1311a, 1311b, 1311c, the latency and throughput of local cluster networks 1312a, 1312b, 1312c, the latency, throughput, and cost of wide area network links 1313a, 1313b, 1313c, and / or other factors that can contribute to the cost, speed, fault-tolerance, resiliency, efficiency and / or other design criteria of the moderation system architecture.Example Methods of Operation

[0147] FIG. 14 is a flowchart of a method 1400, in accordance with example embodiments. Method 1400 can be executed by a computing device, such as computing device 1200. Method 1400 can begin at block 1410, where the method involves receiving, by a computing device, a training dataset comprising a plurality of images based on sparse sensor data.

[0148] At block 1420, the method involves training, based on the training dataset, a dual-depth machine learning (ML) model to predict at least two disparity values for a phasedifference autofocus (PDAF) adjustment to a lens position for a camera.

[0149] At block 1430, the method involves providing, by the computing device, the trained dual-depth ML model.

[0150] In some embodiments, the at least two disparity values include a first disparity value for a foreground of an image, and a second disparity value for a background for the image.

[0151] In some embodiments, the sparse sensor data includes one left-right pixel pair in an 8 X 8 pixel block.

[0152] In some embodiments, the training of the ML model involves learning temporal dependencies to maintain a consistent change in disparity over time.

[0153] In some embodiments, the ML model may be a recurrent neural network (RNN) configured to maintain an internal state that is shared from a previous inference to a next inference.

[0154] In some embodiments, the RNN includes a plurality of long short-term memory (LSTM) units. In some embodiments, the LSTM includes 16 units.

[0155] In some embodiments, the LSTM units are positioned prior to a dense output layer of the RNN.

[0156] In some embodiments, the training of the ML model may be based on a Mean Absolute Error (MAE) loss.

[0157] In some embodiments, the MAE loss may be based on a center weight term adjustment for disparities around a zero disparity lens position.

[0158] In some embodiments, the ML model includes a confidence subnetwork to predict at least two confidence values associated with the at least two predicted disparity values, wherein a confidence value for a predicted disparity value is indicative of an error in the predicted disparity value.

[0159] In some embodiments, the training of the ML model involves pre-training the ML model to predict the at least two predicted disparity values. Such embodiments also involve training the confidence subnetwork based on the at least two predicted disparity values output by the pre-trained ML model.

[0160] In some embodiments, the training dataset includes synthetically generated focus sweeps that capture images at different blur and phase-difference (PD) disparity levels.

[0161] FIG. 15 is a flowchart of a method 1500, in accordance with example embodiments. Method 1500 can be executed by a computing device, such as computing device 1200. Method 1500 can begin at block 1510, where the method involves receiving image data from an image sensor of a camera, wherein the image sensor is configured to output sparse sensor data.

[0162] At block 1520, the method involves predicting, based on a trained dual-depth machine learning (ML) model, at least two disparity values for a phase-difference autofocus (PDAF) adjustment to a lens position for the camera.

[0163] At block 1530, the method involves providing the predicted at least two disparity values.

[0164] Some embodiments involve adjusting the lens position for the camera based on the predicted at least two disparity values.

[0165] In some embodiments, the at least two disparity values include a first disparity value for a foreground of an image, and a second disparity value for a background for the image.

[0166] In some embodiments, the sparse sensor data includes one left-right pixel pair in a 8 X 8 pixel block.

[0167] In some embodiments, the image data includes one or more regions of interest (ROIs).

[0168] In some embodiments, the ROIs may be located at different depth layers.

[0169] In some embodiments, the ROIs may have different sizes.

[0170] In some embodiments, the image data includes an image of a scene, and wherein an ambient light for the scene is below a threshold brightness.

[0171] In some embodiments, the trained dual-depth ML model having been trained to learn temporal dependencies to maintain a consistent change in disparity over time.

[0172] In some embodiments, the trained dual-depth ML model may be a recurrent neural network (RNN) configured to maintain an internal state that is shared from a previous inference to a next inference.

[0173] In some embodiments, the RNN includes a plurality of long short-term memory (LSTM) units.

[0174] The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its spirit and scope, as will be apparent to those skilled in the art. Functionally equivalent methods andapparatuses within the scope of the disclosure, in addition to those enumerated herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.

[0175] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

[0176] With respect to any or all of the ladder diagrams, scenarios, and flow charts in the figures and as discussed herein, each block and / or communication may represent a processing of information and / or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, functions described as blocks, transmissions, communications, requests, responses, and / or messages may be executed out of order from that shown or discussed, including substantially concurrent or in reverse order, depending on the functionality involved. Further, more or fewer blocks and / or functions may be used with any of the ladder diagrams, scenarios, and flow charts discussed herein, and these ladder diagrams, scenarios, and flow charts may be combined with one another, in part or in whole.

[0177] A block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and / or related data may be stored on any type of computer readable medium such as a storage device including a disk or hard drive or other storage medium.

[0178] The computer readable medium may also include non-transitory computer readable media such as non-transitory computer-readable media that stores data for shortperiods of time like register memory, processor cache, and random access memory (RAM). The computer readable media may also include non-transitory computer readable media that stores program code and / or data for longer periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

[0179] Moreover, a block that represents one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.

[0180] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for explanatory purposes and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method, comprising: receiving, by a computing device, a training dataset comprising a plurality of images based on sparse sensor data; training, based on the training dataset, a dual-depth machine learning (ML) model to predict at least two disparity values for a phase-difference autofocus (PDAF) adjustment to a lens position for a camera; and providing, by the computing device, the trained dual-depth ML model.

2. The computer-implemented method of claim 1 , wherein the at least two disparity values comprises a first disparity value for a foreground of an image, and a second disparity value for a background for the image.

3. The computer-implemented method of any of claims 1 or 2, wherein the sparse sensor data comprises one left-right pixel pair in an 8 X 8 pixel block.

4. The computer-implemented method of any of claims 1-3, wherein the training of the ML model comprises learning temporal dependencies to maintain a consistent change in disparity over time.

5. The computer-implemented method of claim 4, wherein the ML model is a recurrent neural network (RNN) configured to maintain an internal state that is shared from a previous inference to a next inference.

6. The computer-implemented method of claim 5, wherein the RNN comprises a plurality of long short-term memory (LSTM) units.

7. The computer-implemented method of claim 6, wherein the LSTM comprises 16 units.

8. The computer-implemented method of claim 6, wherein the LSTM units are positioned prior to a dense output layer of the RNN.

9. The computer-implemented method of any of claims 1-8, wherein the training of the ML model is based on a Mean Absolute Error (MAE) loss.

10. The computer-implemented method of claim 9, wherein the MAE loss is based on a center weight term adjustment for disparities around a zero disparity lens position.

11. The computer-implemented method of any of claims 1-10, wherein the ML model comprises a confidence subnetwork to predict at least two confidence values associated with the at least two predicted disparity values, wherein a confidence value for a predicted disparity value is indicative of an error in the predicted disparity value.

12. The computer-implemented method of claim 11, wherein the training of the ML model further comprises: pre-training the ML model to predict the at least two predicted disparity values; and training the confidence subnetwork based on the at least two predicted disparity values output by the pre-trained ML model.

13. The computer-implemented method of any of claims 1-12, wherein the training dataset comprises synthetically generated focus sweeps that capture images at different blur and phasedifference (PD) disparity levels.

14. A computer-implemented method, comprising: receiving image data from an image sensor of a camera, wherein the image sensor is configured to output sparse sensor data; predicting, based on a trained dual-depth machine learning (ML) model, at least two disparity values for a phase-difference autofocus (PDAF) adjustment to a lens position for the camera; and providing the predicted at least two disparity values.

15. The computer-implemented method of claim 14, further comprising: adjusting the lens position for the camera based on the predicted at least two disparity values.

16. The computer-implemented method of any of claims 14 or 15, wherein the at least two disparity values comprises a first disparity value for a foreground of an image, and a second disparity value for a background for the image.

17. The computer-implemented method of any of claims 14-16, wherein the sparse sensor data comprises one left-right pixel pair in a 8 X 8 pixel block.

18. The computer-implemented method of any of claims 14-17, wherein the image data comprises one or more regions of interest (ROIs).

19. The computer-implemented method of claim 18, wherein the ROIs are located at different depth layers.

20. The computer-implemented method of claim 18, wherein the ROIs comprise different sizes.

21. The computer-implemented method of any of claims 14-20, wherein the image data comprises an image of a scene, and wherein an ambient light for the scene is below a threshold brightness.

22. The computer-implemented method of any of claims 14-21, the trained dual-depth ML model having been trained to learn temporal dependencies to maintain a consistent change in disparity over time.

23. The computer-implemented method of claim 22, wherein the trained dual-depth ML model is a recurrent neural network (RNN) configured to maintain an internal state that is shared from a previous inference to a next inference.

24. The computer-implemented method of claim 23, wherein the RNN comprises a plurality of long short-term memory (LSTM) units.

25. A computing device, comprising: one or more processors; anddata storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out functions comprising the computer-implemented method of any one of claims 1-24.

26. The computing device of claim 25, wherein the computing device is a mobile device.

27. A computer program comprising instructions that, when executed by a computer, cause the computer to perform steps in accordance with the method of any one of claims 1-24.

28. An article of manufacture comprising one or more non-transitory computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out functions that comprise the computer-implemented method of any one of claims 1-24.

29. A system, comprising: means for carrying out the computer-implemented method of any one of claims 1-24.

Images