Systems and methods for temporal single-depth machine learning based phase-difference autofocus (PDAF) model

Abstract

An example method includes receiving, by a computing device, a training dataset comprising a plurality of images based on dense sensor data. The method also includes training, based on the training dataset, a temporal, single-depth machine learning (ML) model to predict a disparity value for a phase-difference autofocus (PDAF) adjustment to a lens position for a camera, wherein the training of the ML model comprises learning temporal dependencies to maintain a consistent change in disparities over time. The method further includes providing, by the computing device, the trained temporal, single-depth ML model.

Description

SYSTEMS AND METHODS FOR TEMPORAL SINGLE-DEPTH MACHINELEARNING BASED PHASE-DIFFERENCE AUTOFOCUS (PDAF) MODELCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This present application claims priority to U.S. provisional application serial no. 63 / 587,606 filed October 3, 2023, and to U.S. provisional application serial no. 63 / 587,639 filed October 3, 2023, the full disclosures of which are incorporated herein by reference.BACKGROUND

[0002] Camera autofocus (AF) is a hybrid system that involves multiple modules, such as, for example, a phase-difference autofocus (PDAF) module, to achieve an optimal focus.SUMMARY

[0003] This application generally relates to the training and inference of a temporal singledepth (TSD) machine learning based PDAF (MLPD) model.

[0004] In one aspect, a computer-implemented method is provided. The method includes receiving, by a computing device, a training dataset comprising a plurality of images based on dense sensor data. The method also includes training, based on the training dataset, a temporal, single-depth machine learning (ML) model to predict a disparity value for a phase-difference autofocus (PDAF) adjustment to a lens position for a camera, wherein the training of the ML model comprises learning temporal dependencies to maintain a consistent change in disparities over time. The method further includes providing, by the computing device, the trained temporal, single-depth ML model.

[0005] In a second aspect, a device is provided. The device includes one or more processors and data storage. The data storage has stored thereon computer-executable instructions that, when executed by one or more processors, cause the device to carry out functions. The functions include: receiving, by a computing device, a training dataset comprising a plurality of images based on dense sensor data; training, based on the training dataset, a temporal, singledepth machine learning (ML) model to predict a disparity value for a phase-difference autofocus (PDAF) adjustment to a lens position for a camera, wherein the training of the ML model comprises learning temporal dependencies to maintain a consistent change in disparities over time; and providing, by the computing device, the trained dual-depth ML model.

[0006] In a third aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to carry out functions. Thefunctions include: receiving, by a computing device, a training dataset comprising a plurality of images based on dense sensor data; training, based on the training dataset, a temporal, singledepth machine learning (ML) model to predict a disparity value for a phase-difference autofocus (PDAF) adjustment to a lens position for a camera, wherein the training of the ML model comprises learning temporal dependencies to maintain a consistent change in disparities over time; and providing, by the computing device, the trained dual-depth ML model.

[0007] In a fourth aspect, an article of manufacture is provided. The article of manufacture includes one or more computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a device, cause the device to carry out functions. The functions include: receiving, by a computing device, a training dataset comprising a plurality of images based on dense sensor data; training, based on the training dataset, a temporal, single-depth machine learning (ML) model to predict a disparity value for a phase-difference autofocus (PDAF) adjustment to a lens position for a camera, wherein the training of the ML model comprises learning temporal dependencies to maintain a consistent change in disparities over time; and providing, by the computing device, the trained dual-depth ML model.

[0008] In a fifth aspect, a system is provided. The system includes means for receiving, by a computing device, a training dataset comprising a plurality of images based on dense sensor data; means for training, based on the training dataset, a temporal, single-depth machine learning (ML) model to predict a disparity value for a phase-difference autofocus (PDAF) adjustment to a lens position for a camera, wherein the training of the ML model comprises learning temporal dependencies to maintain a consistent change in disparities over time; and means for providing, by the computing device, the trained dual-depth ML model.

[0009] In a sixth aspect, a computer-implemented method is provided. The method includes receiving image data from an image sensor of a camera, wherein the image sensor is configured to operate in a dual mode, wherein a first mode of the dual mode is configured to provide dense sensor data, and wherein a second mode of the dual mode is configured to provide sparse sensor data. The method also includes predicting, based on the image data and by a trained temporal, single-depth machine learning (ML) model, a disparity value for a phase-difference autofocus (PDAF) adjustment to a lens position for the camera, the ML model having been trained to learn temporal dependencies to maintain a consistent change in disparities over time. The method further includes providing the predicted disparity value.

[0010] In a seventh aspect, a device is provided. The device includes one or more processors and data storage. The data storage has stored thereon computer-executable instructions that,when executed by one or more processors, cause the device to carry out functions. The functions include: receiving image data from an image sensor of a camera, wherein the image sensor is configured to operate in a dual mode, wherein a first mode of the dual mode is configured to provide dense sensor data, and wherein a second mode of the dual mode is configured to provide sparse sensor data; predicting, based on the image data and by a trained temporal, single-depth machine learning (ML) model, a disparity value for a phase-difference autofocus (PDAF) adjustment to a lens position for the camera, the ML model having been trained to learn temporal dependencies to maintain a consistent change in disparities over time; and providing the predicted disparity value.

[0011] In an eighth aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to carry out functions. The functions include: receiving image data from an image sensor of a camera, wherein the image sensor is configured to operate in a dual mode, wherein a first mode of the dual mode is configured to provide dense sensor data, and wherein a second mode of the dual mode is configured to provide sparse sensor data; predicting, based on the image data and by a trained temporal, single-depth machine learning (ML) model, a disparity value for a phase-difference autofocus (PDAF) adjustment to a lens position for the camera, the ML model having been trained to learn temporal dependencies to maintain a consistent change in disparities over time; and providing the predicted disparity value.

[0012] In a ninth aspect, an article of manufacture is provided. The article of manufacture includes one or more computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a device, cause the device to carry out functions. The functions include: receiving image data from an image sensor of a camera, wherein the image sensor is configured to operate in a dual mode, wherein a first mode of the dual mode is configured to provide dense sensor data, and wherein a second mode of the dual mode is configured to provide sparse sensor data; predicting, based on the image data and by a trained temporal, single-depth machine learning (ML) model, a disparity value for a phasedifference autofocus (PDAF) adjustment to a lens position for the camera, the ML model having been trained to learn temporal dependencies to maintain a consistent change in disparities over time; and providing the predicted disparity value.

[0013] In a tenth aspect, a system is provided. The system includes means for receiving image data from an image sensor of a camera, wherein the image sensor is configured to operate in a dual mode, wherein a first mode of the dual mode is configured to provide dense sensor data, and wherein a second mode of the dual mode is configured to provide sparse sensor data; meansfor predicting, based on the image data and by a trained temporal, single-depth machine learning (ML) model, a disparity value for a phase-difference autofocus (PDAF) adjustment to a lens position for the camera, the ML model having been trained to learn temporal dependencies to maintain a consistent change in disparities over time; and means for providing the predicted disparity value.

[0014] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the figures and the following detailed description and the accompanying drawings.BRIEF DESCRIPTION OF THE FIGURES

[0015] FIG. 1 is an illustration of front, right-side, and rear views of a digital camera device 100, in accordance with example embodiments.

[0016] FIG. 2 illustrates example resolutions of an image sensor for different camera lens, in accordance with example embodiments.

[0017] FIG. 3 illustrates an arrangement of left / right pixels in sparse phase-difference (PD) mode, in accordance with example embodiments.

[0018] FIG. 4 illustrates an example right shielded view in sparse phase-difference (PD) mode, in accordance with example embodiments.

[0019] FIG. 5 illustrates an example training process to capture frame sequences with scene changes, in accordance with example embodiments.

[0020] FIG. 6 illustrates an example ground truth (GT) disparity table, in accordance with example embodiments.

[0021] FIG. 7 illustrates an example sparse disparity estimation using sparse PD input data and dense ground truth (GT) disparity data, in accordance with example embodiments.

[0022] FIG. 8 illustrates an example architecture for a temporal, single-depth machine learning (ML) model, in accordance with example embodiments.

[0023] FIG. 9 illustrates example mean absolute error (MAE) values between dense GT and sparse GT, in accordance with example embodiments.

[0024] FIG. 10 is a diagram illustrating training and inference phases of a machine learning model, in accordance with example embodiments.

[0025] FIG. 11 depicts a distributed computing architecture, in accordance with example embodiments.

[0026] FIG. 12 is a block diagram of a computing device, in accordance with example embodiments.

[0027] FIG. 13 depicts a network of computing clusters arranged as a cloud-based server system, in accordance with example embodiments.

[0028] FIG. 14 is a flowchart of a method, in accordance with example embodiments.

[0029] FIG. 15 is another flowchart of a method, in accordance with example embodiments.DETAILED DESCRIPTION

[0030] Example methods, devices, and systems are described herein. It should be understood that the words “example” and “exemplary” are used herein to mean “serving as an example, instance, or illustration.” Any embodiment or feature described herein as being an “example” or “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or features. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein.

[0031] Thus, the example embodiments described herein are not meant to be limiting. Aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

[0032] Further, unless context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be generally viewed as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are necessary for each embodiment.Overview

[0033] Camera autofocus (AF) is a hybrid system that involves multiple modules to achieve an optimal focus. Such modules are configured to work together to leverage the capturing conditions for more accurate focus parameters. Camera AF may employ three modules: phasedifference autofocus (PDAF), contrast-detection autofocus (CDAF), and time-of-flight (ToF). Each algorithm has its own advantages and disadvantages, and when properly tuned, they may work together to improve the overall camera AF.

[0034] PDAF measures the phase difference (PD) between two images of the same scene taken from slightly different angles. The calculated PD phase is then used to move the lens accordingly to the optimal focus that achieves zero disparity. PDAF may be a primary AF algorithm because of efficiency and accuracy. Generally, PDAF is based on one frame toestimate a target lens position that achieves a target focus. However, PDAF may be unreliable in challenging conditions, such as extremely low-light scenes. In such situations, the AF system may leverage CDAF or ToF, depending on the design tuning and capturing conditions. CDAF may be more accurate than PDAF under certain conditions, but it can be slower because it performs an image focus search that requires capturing multiple images at different focus distances to determine a target focus distance. In the event that CDAF is unreliable, ToF can be a complementary system for both PDAF and CDAF. ToF emits an actual signal to the scene and measures the round-trip time to obtain the obj ect distance and then determine a target focus. ToF can be highly accurate under normal conditions, but it has a limited distance range and may sometimes be unreliable in bright scenes.Example Camera Systems

[0035] As image capture devices, such as cameras, become more popular, they may be employed as standalone hardware devices or integrated into various other types of devices. For instance, still and video cameras are now regularly included in wireless computing devices (e.g., mobile devices, such as mobile phones), tablet computers, laptop computers, video game interfaces, home automation devices, and even automobiles and other types of vehicles.

[0036] The physical components of a camera may include one or more apertures through which light enters, one or more recording surfaces for capturing the images represented by the light, and lenses positioned in front of each aperture to focus at least part of the image on the recording surface(s). The apertures may be of a fixed size or may be adjustable. In an analog camera, the recording surface may be a photographic film. In a digital camera, the recording surface may include an electronic image sensor (e.g., a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) sensor) to transfer and / or store captured images in a data storage unit (e.g., memory).

[0037] One or more shutters may be coupled to, or positioned near, the lenses or the recording surfaces. Each shutter may either be in a closed position, in which it blocks light from reaching the recording surface, or an open position, in which light is allowed to reach the recording surface. The position of each shutter may be controlled by a shutter button. For instance, a shutter may be in the closed position by default. When the shutter button is triggered (e.g., pressed), the shutter may change from the closed position to the open position for a period of time, known as the shutter cycle. During the shutter cycle, an image may be captured on the recording surface. At the end of the shutter cycle, the shutter may change back to the closed position.

[0038] Alternatively, the shuttering process may be electronic. For example, before an electronic shutter of a CCD image sensor is “opened,” the sensor may be reset to remove any residual signal in its photodiodes. While the electronic shutter remains open, the photodiodes may accumulate charge. When or after the shutter closes, these charges may be transferred to longer-term data storage. Combinations of mechanical and electronic shuttering may also be possible.

[0039] Regardless of type, a shutter may be activated and / or controlled by something other than a shutter button. For instance, the shutter may be activated by a softkey, a timer, or some other trigger. Herein, the term “capture” may refer to any mechanical and / or electronic shuttering process that results in one or more images being recorded, regardless of how the shuttering process is triggered or controlled.

[0040] The exposure of a captured image may be determined by a combination of the size of the aperture, the brightness of the light entering the aperture, and the length of the shutter cycle (also referred to as the shutter length, the exposure length, or the exposure time). Additionally, a digital and / or analog gain (e.g., based on an ISO setting) may be applied to the image, thereby influencing the exposure. In some embodiments, the term “exposure length,” “exposure time,” or “exposure time interval” may refer to the shutter length multiplied by the gain for a particular aperture size. Thus, these terms may be used somewhat interchangeably, and should be interpreted as possibly being a shutter length, an exposure time, and / or any other metric that controls the amount of signal response that results from light reaching the recording surface.

[0041] In some implementations or modes of operation, a camera may capture one or more still images each time image capture is triggered. In other implementations or modes of operation, a camera may capture a video image by continuously capturing images at a particular rate (e.g., 24 frames per second) as long as image capture remains triggered (e.g., while the shutter button is held down). Some cameras, when operating in a mode to capture a still image, may open the shutter when the camera device or application is activated, and the shutter may remain in this position until the camera device or application is deactivated. While the shutter is open, the camera device or application may capture and display a representation of a scene on a viewfinder (sometimes referred to as displaying a “preview frame”). When image capture is triggered, one or more distinct payload images of the current scene may be captured.

[0042] Cameras, including digital and analog cameras, may include software to control one or more camera functions and / or settings, such as aperture size, exposure time, gain, and so on. Additionally, some cameras may include software that digitally processes images during or after image capture. While the description above refers to cameras in general, it may beparticularly relevant to digital cameras. Digital cameras may be standalone devices (e.g., a DSLR camera) or may be integrated with other devices.

[0043] Either or both of a front-facing camera and a rear-facing camera may include or be associated with an ambient light sensor (ALS) that may continuously or from time to time determine the ambient brightness of a scene that the camera can capture. In some devices, the ALS can be used to adjust the display brightness of a screen associated with the camera (e.g., a viewfinder). When the determined ambient brightness is high, the brightness level of the screen may be increased to make the screen easier to view. When the determined ambient brightness is low, the brightness level of the screen may be decreased, also to make the screen easier to view as well as to potentially save power. Additionally, the ambient light sensor’s input may be used to determine an exposure time of an associated camera, or to help in this determination.

[0044] FIG. 1 is an illustration of front, right-side, and rear views of a digital camera device 100, in accordance with example embodiments. Digital camera device 100 may be, for example, a mobile device (e.g., a mobile phone), a tablet computer, or a wearable computing device. However, other embodiments are possible. Digital camera device 100 may include various elements, such as a body 102, a front-facing camera 104, a multi-element display 106, a shutter button 108, and other buttons 110. Digital camera device 100 could further include one or more rear-facing cameras 112, 114. Front-facing camera 104 may be positioned on a side of body 102 typically facing a user while in operation, or on the same side as multi-element display 106. Rear-facing cameras 112, 114 may be positioned on a side of body 102 opposite front-facing camera 104. Referring to the cameras as front-facing and rear-facing is arbitrary, and digital camera device 100 may include multiple cameras positioned on various sides of body 102.

[0045] Multi-element display 106 could represent a cathode ray tube (CRT) display, a lightemitting diode (LED) display, a liquid crystal display (LCD), a plasma display, or any other type of display known in the art. In some embodiments, multi-element display 106 may display a digital representation of the current image being captured by front-facing camera 104 and / or rear-facing cameras 112, 114, or an image that could be captured or was recently captured by either or both of these cameras. Thus, multi-element display 106 may serve as a viewfinder for either camera. Multi-element display 106 may also support touchscreen and / or presencesensitive functions that may be able to adjust the settings and / or configuration of any aspect of digital camera device 100.

[0046] Multi-element display 106 may include additional features related to a camera application. For example, multiple modes may be available for a user, including, a motion mode, portrait mode, video mode, video bokeh mode, and so forth. The camera application may be in camera mode and provide additional features, such as a reverse icon to activate reverse camera view, a trigger button to capture a previewed image, and a photo stream icon to access a database of captured images. Also for example, a magnification ratio slider may be displayed and a user can move a virtual object along the magnification ratio slider to select a magnification ratio. In some embodiments, a user may use the multi-element display 106, also referred to herein as the display screen, to adjust the magnification ratio (e.g., by moving two fingers on display screen in an outward motion away from each other), and magnification ratio slider may automatically display the magnification ratio.

[0047] Front-facing camera 104 may include an image sensor and associated optical elements such as lenses. Front-facing camera 104 may offer zoom capabilities or could have a fixed focal length. In other embodiments, interchangeable lenses could be used with front-facing camera 104. Front-facing camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. Front-facing camera 104 also could be configured to capture still images, video images, or both. Further, front-facing camera 104 could represent a monoscopic, stereoscopic, or multiscopic camera. Rear-facing cameras 112, 114 may be similarly or differently arranged. Additionally, front-facing camera 104, rear-facing cameras 112, 114, or both, may be an array of one or more cameras.

[0048] Either or both of front-facing camera 104 and rear-facing cameras 112, 114 may include or be associated with an illumination component that provides a light field to illuminate a target object. For instance, an illumination component could provide flash or constant illumination of the target object (e.g., using one or more LEDs). An illumination component could also be configured to provide a light field that includes one or more of structured light, polarized light, and light with specific spectral content. Other types of light fields known and used to recover three-dimensional (3D) models from an object are possible within the context of the embodiments herein.

[0049] In some digital camera devices 100, either or both of front-facing camera 104 and rearfacing cameras 112, 114 may include or be associated with an ambient light sensor that may continuously or from time to time determine the ambient brightness of a scene that the camera can capture. In some devices, the ambient light sensor can be used to adjust the display brightness of a screen associated with the camera (e.g., a viewfinder). When the determined ambient brightness is high, the brightness level of the screen may be increased to make thescreen easier to view. When the determined ambient brightness is low, the brightness level of the screen may be decreased, also to make the screen easier to view as well as to potentially save power. Additionally, the ambient light sensor’s input may be used to determine an exposure time of an associated camera, or to help in this determination.

[0050] Digital camera device 100 could be configured to use multi-element display 106 and either front-facing camera 104 or rear-facing cameras 112, 114 to capture images of a target object (e.g., a subject within a scene). The captured images could be a plurality of still images or a video image (e.g., a series of still images captured in rapid succession with or without accompanying audio captured by a microphone). The image capture could be triggered by activating shutter button 108, pressing a softkey on multi-element display 106, or by some other mechanism. Depending upon the implementation, the images could be captured automatically at a specific time interval, for example, upon pressing shutter button 108, upon appropriate lighting conditions of the target object, upon moving digital camera device 100 a predetermined distance, or according to a predetermined capture schedule.

[0051] As noted above, the functions of digital camera device 100 (or another type of digital camera) may be integrated into a computing device, such as a wireless computing device, cell phone, tablet computer, laptop computer, and so on. For example, a camera controller may be integrated with the digital camera device 100 to control one or more functions of the digital camera device 100.

[0052] Although some cameras may be equipped with a full resolution PD sensor, other cameras may employ an imaging sensor that is configured with two PD formats and layouts referred to as dense and sparse (or partial) PD. For example, a sensor may be configured with two PD modes for 30 frames per second (fps) and 60fps, each utilizing different PD formats and layouts, and referred to herein as dense and sparse (or partial) PD formats, respectively. The image data output from a dense or sparse PD format is referred to herein as dense data or sparse data, respectively. Generally, the disparity between dense and sparse may be primarily correlated with a resolution difference.

[0053] FIG. 2 illustrates example resolutions 200 of an image sensor for different camera lenses, in accordance with example embodiments. For example, in both ultra- wide (UW) and telephoto (Tele) modes, the dense PD resolution may be 4032 x 756 (width interleaved), where the single PD view resolution may be 2016 x 756. The sparse PD resolution may be 504 x 560 (height interleaved), with the single PD view resolution being 504 x 280.

[0054] Low-resolution sparse PD data may make it challenging for typical block matching algorithms (BMA) to accurately calculate disparity and focus the lens. To improve accuracy, a machine learning (ML) based approach may be used, such as, for example, based on a convolutional neural network (CNN), to directly estimate disparity. Although an ML-based PDAF model can outperform BMA, there may be challenges with temporal consistency. This is because the ML-based PDAF model processes frame by frame and does not utilize historical focus values.

[0055] It is desirable to solve the aforementioned technical problems without increasing model capacity or buffering frames. To address such issues, one approach may involve replacing the last few layers of a machine learning model with a recurrent neural network (RNN) unit. By doing so, the model may be configured to maintain an internal state and pass it to a next frame inference, which allows it to be temporally trained to preserve consistency with previous frames, thereby increasing stability. In some embodiments, a long short-term memory (LSTM) may be deployed as a recurrent network, since LSTM units may be better at preserving longterm memory than standard RNNs. In general, real camera systems are always streaming, and the historical data they generate may be utilized. Therefore, temporal models can be very useful for these systems.

[0056] However, to utilize a LSTM internal state and temporal consistency, a special type of data is needed that can hold temporal dependency and can reflect the temporal dynamics of real AF systems. For example, data that consists of static focus sweeps, which do not contain temporal information may not be useful. For focus ML-based models, focus sweeps that capture images at different blur and PD disparity levels may be desirable. However, it can be challenging to capture a focus sweep of a video in real time because it involves moving the lens in discrete steps to cover the entire lens motion range for a static scene. One way to achieve this may be to stop motion, but this is a tedious process and may be limited by the number of scenarios and short sequences that can be captured. This makes it hard to capture a large-scale dataset. To enable LSTM training and leverage temporal capabilities, an AF temporal data capture and synthesis framework may be used. Such a data framework may facilitate capture and synthesis of dual-depth temporal (TDD) sequence data with accurate ground truth, which can then be used for LSTM model training.

[0057] A dual mode sensor may pose additional challenges as described herein.

[0058] FIG. 3 illustrates an arrangement of left / right pixels in sparse phase-difference (PD) mode, in accordance with example embodiments. The streamed PD data format of a dual mode sensor may not be in an interleaved format. For example, within a macro block (16 X 32) 305,the sensor tail mode data 310 may not be in the format of LR LR... Instead, it may have multiple R pixels adjacent to each other. Generally, the definition of L in a dual mode sensor such as, for example, IMX858 tail mode, is left-shielded.

[0059] For example, the sparse PD format of the dual mode sensor (e.g., IMX858) may include L / R shielded pixels 315 within the macro block 305. Some example shielded L / R pixels are highlighted in the macro block 305, and their format is illustrated in tail mode PD image 310. Such a format, when unpacked and rearranged, may result in a distinct pattern (e.g., distinct from that of a GM5-OCL image sensor). For example, the right shielded PD image may be rearranged as illustrated in first rearrangement 320, and the left shielded PD image may be rearranged as illustrated in second rearrangement 325. Specifically, the L and R pixels may not be adjacent, which can lead to a type of distortion known as sawtooth distortion.

[0060] FIG. 4 illustrates an example right shielded view 400 in sparse phase-difference (PD) mode, in accordance with example embodiments. When capturing a color checker, unlike a GM5-OCL sensor, the vertical and horizontal edges may not be straight and may exhibit a zigzag pattern when visualized (referred to herein as a sawtooth distortion).

[0061] The sparse PD mode of the IMX858 sensor in P24 is utilized for both Tele and UW cameras. When the PD view is unpacked and rearranged, the resulting resolution may be described as follows. For example, the dense PD mode has a resolution of 4032 x 756 (interleaved width), with a single PD view resolution of 2016 x 756. Also, for example, the sparse PD mode features a resolution of 504 x 560 (interleaved height), with a single PD view resolution of 504 x 280. Compared to a GM5 sensor, the width of the unpacked PD view in the dual mode IMX858 is the same, but the height is five times smaller, resulting in a total pixel count that is also five times lower. This can have a resultant effect in a loss of pixels for regions of interest (ROIs).

[0062] Also, for example, the pattern and unpacking procedures for the different sensors may differ. For example, for the dual mode sensor, the process may involve interleaved data for height or width, pixel rearrangement, and extraction of left and right PD views. Such an altered PD view extraction pipeline results in a flipped PD view with a reversed direction of disparity.

[0063] In general, sparse PD sensors offer lower resolution compared to dense PD sensors. For instance, the sparse PD pixels used in 60fps video mode may be 9% less in resolution compared to dense PD pixels used in 30fps video mode. This poses a significant challenge for ML models that benefit from higher resolution data inputs, as they provide richer information for the model.

[0064] Another resolution challenge is that the sparse PD pixel count in IMX858 may be five times lower compared to sparse PD in GM5. Therefore, the MLPD development procedures have to be different due to the significant resolution difference.

[0065] Also, for example, the sparse PD arrangement and pattern in IMX858 may differ from that in GM5, leading to a spatial distortion referred to as a sawtooth distortion. This may occur because the left and right pixels are not vertically and horizontally aligned within each image column and row, respectively. Therefore, the MLPD development procedures have to be different due to the significant difference in pixel arrangement.

[0066] Also, for example, the Tele camera may have a large effective focal length (EFL) and aperture size, as well as a small pixel size. These specifications collectively result in the Tele camera having a shallower depth of field (DoF). Generally, for AF systems, having a camera with a shallow depth of field (DoF) presents a challenge because the granularity of defocus blur increases. This makes even slight deviations from the focal plane more pronounced. Accordingly, designing a PDAF system for such a Tele camera is more challenging.

[0067] Another challenge is related to the AF system in general, where AF typically employs contrast detection AF (CDAF) and additional assisting accessories such as time-of-flight (ToF) sensors. One significant limitation of ToF sensors is their restricted signal emission range, which effectively limits their operation to distances up to 3 meters for indoor scenes. Outdoor scenes pose greater challenges for ToF due to the high ambient light levels, which introduce considerable noise to the estimation process and limit the effective distance range (< 80cm). For tele cameras with 5x zoom, many use cases involve distant scenes exceeding 3 meters. Unlike the main camera and the ultra-wide (UW) camera, the tele camera is generally the least capable of utilizing ToF distance accuracy effectively.

[0068] As described herein, the sensor for Tele video may operate in two modes: dense PD for 30fps and sparse PD for 60fps. Dense data generally offers richer information, resulting in more accurate PD disparity measurements. Empirical evidence indicates a strong correlation between dense and sparse PD horizontal disparity, which may be related to the difference in resolution width. As described herein, dense PD data may be utilized for calculating disparity ground truth and such information may be propagated to the corresponding sparse PD data.

[0069] In some embodiments, the input ROI may be resized to match the MLPD input size. For illustrative purposes, the original PD pixels may be referred to herein as "authentic pixels." It is desirable to maximize a number of authentic PD pixels prior to resizing to maintain the quality of the data input to the MLPD. Given the fivefold reduction in pixel count in the dual mode sensor, the number of authentic pixels may be increased in several ways.

[0070] For example, the image frame may have a width roughly twice the height. Such a difference in aspect ratio may be resolved with a readjustment of the ROI tuning ratios to better align with the new aspect ratio and a resultant increase in the authentic pixel count.

[0071] Another approach to increasing the authentic pixel count may be to increase a ratio of the ROI width to height. This may be achieved by modifying the tuning parameters.

[0072] Also, for example, in AF systems, a default ROI may be widely used. Since input resizing is performed to match the MLPD, in some embodiments, it may be more efficient to match the MLPD input size to the default ROI to avoid resizing. Such an approach can reduce computations and minimize resizing artifacts.

[0073] In data synthesis, training data may be generated based on a default ROI size distribution. The size distribution generally follows a Gaussian model where the default size represents the mean and reflects a typical occurrence of a default ROI in real AF systems. With the new default ROI size in a dual mode sensor, an adjustment may be made to the ROI coordinates during data synthesis to accurately simulate the AF system for the sparse PD in a Tele camera.

[0074] As described herein, a learning-based PDAF framework for an image sensor with a dual mode may be introduced. Utilizing dense PD ground truth and increasing the pixel count alone may not be sufficient, due to a significant reduction in pixel count (i.e., five times lower in IMX858 as compared to GM5). Therefore, a single-disparity model may be designed. Such an approach may be less complex than a dual-depth approach due to the challenges outlined herein, and may demonstrate more stable performance, while still providing acceptable focus in multi-depth ROI cases.

[0075] Accordingly, a TDD framework is described that has significantly improved over other models for faces, low-light, high zoom (remosiac), and multi-depth scenes. Specifically, the model described herein reduces the failure rate from 22% to 0.67% for normal scenes, and from 85.71% to 2.56% for challenging scenes. Additionally, model evaluation demonstrates that the model described herein outperforms state-of-the-art competitor cameras.Sparse Temporal Data Synthesis

[0076] Subsequent to data capture and verification, along with the ground truth disparity, the temporal data may be synthesized to facilitate model training. As the ML model is a singledepth MLPD, the data synthesized is temporal single-depth (TSD) data. Generally, synthesizing temporal data is not straightforward because the time dimension is significant in determining how the spatial dimension evolves over time across frames.Simulating Lens Position Change and Scene Motion

[0077] Simulating lens movement, focus blur, and focus breathing (which changes the field of view (FoV)) may be significant considerations for synthesizing realistic temporal sequences. Object and camera motion may be imported to mimic real -world dynamics captured through cameras. In general, the TDD data lens movements and motion synthesis steps may be summarized as follows.

[0078] One approach involves picking a starting frame at a certain voice coil motor (VCM) position and allowing for a sufficient delta lens difference between the starting position and a target lens position (e.g., corresponding to a zero disparity lens position with the focal plane to be at the scene depth. For example, the zero disparity of a single depth plane at 80cm may be at a VCM of 66 based on the ground-truth. One example starting point may be be 66 + / - delta. In some embodiments, delta may be set to be at least 30 logical lens positions (i.e., 5 steps from the target lens position). The goal of this step is to traverse the lens position from the starting point to the zero disparity and take frames at different lens positions as a sequence. This reflects the temporal dynamics of PDAF systems, in which the lens keeps moving until reaching the zero disparity lens position. Some related parameters may be a max disparity margin indicative of a maximum lens travel range allowed, and repeat fr indicative of a number of repeated frames of the same lens position.

[0079] In some embodiments, patches may be cropped from the entire image frame. Such patches represent the AF ROI. In some embodiments, a spatial shift of the cropped patch and a random change in the size may be introduced. The ROI shift can simulate motion in real scenes and a change in size can reflect the ROI size dynamics (e.g., a change in face size). Data may be synthesized with a fixed default ROI size and touch ROI. Some related parameters may include ROI size, ROI center shift, and drawing ROI for visualization.

[0080] For the ROI size, default roi, may be a Boolean flag indicating a fixed default ROI size. Also, for example, roi size ratio change may indicate a range of an initial ROI size ratio with respect to the entire frame size. The initial ROI size ratio may be uniformly drawn randomly from a range for each sequence. As another example, roi h offset range may indicate a width offset range with respect to height in order to avoid a fixed aspect ratio. This may also be randomly picked for the initial ROI size of each sequence. Also, for example, roi aspect ratio range may indicate an allowed range of the width / height aspect ratio range. As another example, roi change range may indicate a range of the change percentage of the ROI size between the consecutive frames. The change between the frames may be randomly picked.

[0081] For the ROI center shift, roi_pixel_shift_range may indicate an upper limit of pixels to shift between the consecutive frames. The shift amount may be uniformly randomly drawn from the range. The roi_pixel_shift_range may also be depth-dependent. For example, the upper limit may be lower for objects that are farther away, as closer objects may maintain a larger shift due to motion parallax. Another ROI center shift parameter may be a pixel drift threshold indicating a maximum allowed number of pixels that can drift from the initial center. The pixel drift threshold may be introduced to avoid the ROI from continuously being shifted and changing dramatically from the original.

[0082] For drawing ROI for visualization, a parameter roi color may indicate a color of the ROI bounding box. This parameter may be used to draw the ROI for visualization and verification purposes. Another parameter may be roi thickness that indicates the thickness of a ROI bounding box.

[0083] In some embodiments, data synthesis may involve a temporal VCM sweep, ROI shift, and / or a size change. The ROI can be dynamic, and the VCM may change until it reaches a zero disparity lens position, which makes the ROI sharper and in focus. In some embodiments, stacked red-green-green-blue (RGGB) color channels may be used for visualization purposes as the PD data has a lower resolution and limited color channels. However, the PD views may be synthesized accordingly by projecting the VCM and ROI coordinates to a sparse PD space.

[0084] As described herein, data may be captured at multiple OIS positions and an OIS change may be introduced in the data synthesis process. In addition to increasing dataset size, multiple OIS positions can simulate different radial distortion patterns by changing the relative sensor location with respect to the optics. Additionally, in real camera systems, the OIS position is also moving for video stabilization purposes. Accordingly, the multiple OIS positions enable simulation of realistic video data. For example, the OIS position may be changed to a random position based on a certain VCM change rate.

[0085] Some of the OIS motion parameters may be ois that indicate predefined OIS coordinates. For example, fixed OIS positions such as (7,7), (8,7), (7,6), and (8,6) may be used. Another parameter may be ois change rate indicating a rate of OIS change based on the number of VCM position changes. Also, for example, another parameter may be ois dis shift indicating a desired compensation for pixel shift due to OIS change (e.g., to avoid a significant shift in the ROI content, making the data unrealistic).

[0086] In some embodiments, temporal PD sequences may be generated by incorporating OIS position change. As described, stacked RGGB may be used for visualization. As the VCM andROI change, the OIS also changes, simulating the behavior of real camera systems during AF convergence.Simulating motion blur, exposure change, and noise

[0087] In addition to simulating ROI and OIS dynamics, additional and / or alternative image degradations may be synthesized to further simulate the real dynamics of a camera in temporal sequences.

[0088] For example, an image degradation corresponding to a continuous exposure change over time may be introduced. This may be represented by changes in brightness and / or noise levels. Also, for example, an image degradation related to scene and / or camera motion may be introduced to simulate motion blur.

[0089] For example, motion blur may be added based on a direction and magnitude of the ROI and OIS motion. A 2D linear motion blur kernel may be constructed, and the input image may be convolved with the constructed kernel. The motion blur model may be determined as follows: / * = / * / <(Eqn. 1)

[0090] where I is an input clean image, I* is a blurry image, K is a square 2D linear kernel, and the dimension (or magnitude) may be determined based on a ROI shift amount in pixels between consecutive frames. A related parameter may be add motion blur that is a Boolean flag to simulate motion blur.

[0091] In some embodiments, exposure change may be introduced. For example, an intensity value may be multiplied by a brightness scale (e.g., that may be randomly picked). Related parameters include brightness change that is a Boolean flag to enable brightness change. Another parameter may be roi bright change ratio indicating a lower bound of a linear intensity scaling.

[0092] Some embodiments involve adding Gaussian noise (e.g., Heteroscedastic Gaussian noise) including shot and read noise. The noise model may be described as follows: = 1 + S Q I + R(Eqn. 2) S~ N( i = 0, <JS)(Eqn. 3) / ?~ A(g = 0, <JR)(Eqn. 4)

[0093] where I is an input clean image, / * is a noisy image, and S and R are the shot and read noise, respectively. Both S and R may follow a zero-mean normal distribution. O represents the element wise multiplication operator. The code related parameters may include add noise that is a Boolean flag to enable adding noise, noise_per_frame_change that is a Boolean flag to change a noise distribution for consecutive frames. When this parameter value is False, the same initial random distribution is maintained and a noise sample is randomly drawn for each frame. Another parameter may be shot noise range that indicates a range of as, and read noise range that indicates a range of aR.

[0094] With image degradation and ROI / OIS dynamics incorporated, the domain gap may become smaller between synthesized temporal data and the actual data sequences in real cameras.Introducing Scene change

[0095] One advantage with LSTM is a long-term memory (internal state) that can be passed to future frames. In some embodiments, a balance may need to be struck between keeping the internal state when it is useful and dropping it when it is not. For example, if there is no significant scene change or target focus change, it can be beneficial to keep the internal state for more consistent prediction. However, having a persistent and always long-term memory may increase a delay in lens transition for the target focus when the scene changes.

[0096] Based on the data synthesis framework, the sequences may be synthesized without scene change data, and each sample may represent one scene dynamic. Additionally, each sequence may converge to a zero disparity lens position. These two biases can cause the LSTM to have longer states and tend to converge to zero disparity, regardless of the scene change. This may result in freezing of the lens position in some videos, even when the scene changes.

[0097] One approach to solve this issue may be via explicit training with scene change data. To enable this approach, the data synthesis framework may be modified and sequences with scene changes and disparity magnitude and direction change may be included. The model may be retrained with these sequences to allow it to adaptively detect scene changes and reset its internal state accordingly, without any delays or loss of internal state. In some embodiments, this may be achieved by splitting the sequences into smaller segments, shuffling the segments, and mixing the segments segment- wise with other sequences randomly to create new sequences that contain within-sequence scene changes.

[0098] FIG. 5 illustrates an example training process to capture frame sequences with scene changes, in accordance with example embodiments. Using this approach, the LSTM modelmay be configured to adaptively reset its internal state based on scene changes, and the focus freeze issue may be successfully resolved.Splitting longer sequences

[0099] To create scene change sequences, two sequences may be interleaved segment-wise, which doubles the length of the sequence. For example, prior to the scene-change data, the sequence length ranges from 10 to 90 frames. After the scene change, the sequence length ranges from 20 to 180 frames. In some embodiments, at runtime, when training the model, the entire sequence may be loaded and then a short sequence may be randomly trimmed for minibatch training. As mentioned earlier, a mini-batch size of 48 may be used for training. This means loading 48 sequences to randomly trim mini-sequences at each iteration. Technically, this doubles the mini-batch size and may cause Flume data loading pipeline and memory issues, especially when parallelizing the process on multiple jobs.

[0100] To address this issue, longer sequences may be split into multiple shorter sequences. For example, three sequence categories may be defined based on length: less than 80 frames, greater than 80 frames but less than 120 frames, and greater than 120 frames. Sequences in the first category may not be split. Sequences in the second and third categories may be split into two and three equally smaller sequences, respectively.Synthetic TSD ground truth, visualization, & verification

[0101] Each TSD synthesized sequence may be saved in a folder with a corresponding ground truth disparity file.

[0102] FIG. 6 illustrates an example ground truth (GT) disparity table 600, in accordance with example embodiments. As illustrated, the columns in a saved ground truth table may include a frame indicating a frame number, VCM indicating a VCM position, disparity _0 indicating FG disparity, disparity l indicating BG disparity, pd roi start (x,y) indicating the upper left corner of the ROI from the entire PD frame, pd roi end (x,y) indicating the lower right corner of the ROI from the entire PD frame.

[0103] After TSD data is synthesized and ground truth is generated, it is desirable to verify that the synthesized data is accurate and is aligned with the ground truth disparity. This verification may be performed in two stages.

[0104] For example, an RGB video of the PD sequence may be saved, where each frame corresponds to a synthesized PD frame. The video may be visually inspected to ensure that the colored high-resolution sequence is running as intended. In some embodiments, Bayer RGGB pixels may be used as they have a higher resolution and provide colored informationfor better visual assessment and verification. This may be achieved by averaging the two green channels and stacking the R, averaged G, and B channels as RGB image frames.

[0105] Also, for example, the PD data may be packed into a binary .bin file. Each frame may have a different width and height when the default roi is disabled. The PD views may be unpacked and their per-frame coordinates may be used to ensure that a width times a height is equal to a size in bytes. It may be verified that a number of frames matches a number of folders saved. This may be iterated through the frames, unpacking them and ensuring that a size and dimension match a size in the GT sheet.Temporal single-depth (TSD) approach

[0106] In some embodiments, an MLPD may be trained for sparse disparity estimation using sparse PD input data, and with a ground truth (GT) disparity derived from dense BMA. The dense BMA data is generally more accurate (especially in low-light conditions) and due to a high similarity between sparse and dense data up to a scale factor. However, such a training with dense GT means that the model has to rely on a dense DCC calibration instead of a sparse DCC calibration for the sparse MLPD, since the model output is aligned with dense PD disparity.

[0107] FIG. 7 illustrates an example sparse disparity estimation 700 using sparse PD input data and dense ground truth (GT) disparity data, in accordance with example embodiments. Dense PD data 705 is input to BMA 710 to obtain BMA dense disparity data 715. This is ground truth (GT) data 720 to train MLPD 730. Sparse PD data 725 may be received from a sparse mode of the image sensor and provided to MLPD 730. MLPD 730 may predict a disparity value 735, which may be provided to the tele camera for dense DCC calibration 740. The dense DCC calibration 740 may be provided to the VCM.

[0108] FIG. 8 illustrates an example architecture 800 for a temporal, single-depth (TSD) machine learning (ML) model, in accordance with example embodiments. A PDAF framework is described herein that incorporates LSTM units before the dense output layer. In some embodiments, an output dense layer of the disparity model may be configured to produce a single disparity value d (single-depth MLPD). The output of the confidence model may be configured to provide a single confidence value. In some embodiments, a number of unnecessary nodes may be reduced in both the disparity and confidence models. For example, the number of LSTM units may be reduced (e.g., to 8 or 4), as increasing the number of units may cause temporal overfitting in the TSD model. Also, for example, as previously described, the input pixel count obtained from a dual mode image sensor may be smaller.

[0109] Designing a PDAF algorithm that captures the temporal dependency can be significant for maintaining consistent disparity change over time (e.g., smooth lens transition and reduced lens fluctuations).

[0110] The TMobileNet may generally be a recurrent neural network that does not require buffering frames to maintain temporal consistency. Instead, a hidden internal state H may be shared from a previous inference to a next inference. This enables the network to learn long-term temporal dependencies and improve the accuracy of its predictions. The model may be configured to process one frame at a time and does not introduce any latency. However, the model can be configured to leverage the learned internal state weights to increase accuracy over time. The model can also handle variable ROI sizes. If the input ROI size is different from a default, the input may be first scaled to the default ROI size before being fed to the model. The input and output of the TMobileNet may be described as: dn= PDAF(Rn, Ln, Hn_f),(Eqn. 5)

[0111] where dnis a disparity value at time n, Rn, Lnare the right and left PD views of selected ROI, andis an internal state that is passed on to time n from time n — 1.

[0112] In some embodiments, an MAE loss function may be used. The Mean Absolute Error (MAE) loss may be used for training the model. In some embodiments, a center- weighted MAE loss function may be used. For example, the center weight term may be added to penalize more for small disparities around a zero disparity lens position (i.e., ±10% of the disparity range). This may emphasize sharper AF responses. The center weight MAE loss may be defined as follows:(Eqn. 6) if digt< 0.1 * disparityrangeotherwise(Eqn. 7)

[0113] where di9tand df are the ground truth and estimated disparities, respectively, is a center weight term. In some embodiments, the confidence model may be adapted to indicate how confident the disparity prediction may be. In some embodiments, the two subnetworks, disparity model and confidence model, may be trained in two phases: (1) train disparity only; (2) then freeze the pre-trained disparity model and train confidence based on the disparity error. The confidence value output by the confidence network may range from 0to 1. The higher the error, the lower the confidence. In some embodiments, the predicted disparity may not be used for AF in the event the confidence is below a threshold.Average GT disparity

[0114] As described herein, dense BMA disparity is used as GT for training the sparse MLPD. For MLPD training, a large synthetic dataset may be used, which includes both sparse and dense BMA disparity. The training dataset may be leveraged to validate the similarity between sparse and dense disparities. For example, the GT disparity may be determined by averaging the disparity per VCM for each distance using high lux level sweeps.

[0115] FIG. 9 illustrates example mean absolute error (MAE) values between dense GT and sparse GT, in accordance with example embodiments. First table 905 corresponds to GT for dense disparity at a distance of 80 cm. Second table 910 corresponds to GT for sparse disparity at a distance of 80 cm. A scale that minimizes the mean absolute error (MAE) between dense GT and sparse GT may be determined. For example, as indicated in plot 915, based on a search range from -3.5 to -4.5 with a step size of 0.00001, a minimum MAE appears to occur at a scale of 3.97815 with a disparity error of 0.0073, normalizing it by the range of 42 results in a 0.017% error rate.Training Machine Learning Models for Generating Inferences / Predictions

[0116] FIG. 10 shows diagram 1000 illustrating a training phase 1002 and an inference phase 1004 of trained machine learning model(s) 1032, in accordance with example embodiments. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions about (patterns in the) training data. The resulting trained machine learning algorithm can be termed as a trained machine learning model. For example, FIG. 10 shows training phase 1002 where one or more machine learning algorithms 1020 are being trained on training data 1010 to become trained machine learning model(s) 1032. Then, during inference phase 1004, trained machine learning model(s) 1032 can receive input data 1030 and one or more inference / prediction requests 1040 (perhaps as part of input data 1030) and responsively provide as an output one or more inferences and / or predict! on(s) 1050.

[0117] As such, trained machine learning model(s) 1032 can include one or more models of one or more machine learning algorithms 1020. Machine learning algorithm(s) 1020 may include, but are not limited to: an artificial neural network (e.g., a herein-described convolutional neural networks, a recurrent neural network, a Bayesian network, a hiddenMarkov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system). Machine learning algorithm(s) 1020 may be supervised or unsupervised, and may implement any suitable combination of online and offline learning. Examples of the models may include multi-layer perceptron (MLP) models, convolutional neural networks (CNN), long short-term memory (LSTM) algorithms, generative adversarial networks (GAN), K-means clustering, Gaussian mixture models (GMM), or any other of a number of machinelearning training techniques known to a person of ordinary skill in the art. The models may comprise a single type of model or multiple types, as well as various combinations of types including combinations of single types and of multiple types.

[0118] In some examples, machine learning algorithm(s) 1020 and / or trained machine learning model(s) 1032 can be accelerated using on-device coprocessors, such as graphic processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application specific integrated circuits (ASICs). Such on-device coprocessors can be used to speed up machine learning algorithm(s) 1020 and / or trained machine learning model(s) 1032. In some examples, trained machine learning model(s) 1032 can be trained, reside and execute to provide inferences on a particular computing device, and / or otherwise can make inferences for the particular computing device.

[0119] During training phase 1002, machine learning algorithm(s) 1020 can be trained by providing at least training data 1010 as training input using unsupervised, supervised, semisupervised, and / or reinforcement learning techniques. Training data 1010 may include the TDD data synthesized herein. Unsupervised learning involves providing a portion (or all) of training data 1010 to machine learning algorithm(s) 1020 and machine learning algorithm(s) 1020 determining one or more output inferences based on the provided portion (or all) of training data 1010. Supervised learning involves providing a portion of training data 1010 to machine learning algorithm(s) 1020, with machine learning algorithm(s) 1020 determining one or more output inferences based on the provided portion of training data 1010, and the output inference(s) are either accepted or corrected based on correct results associated with training data 1010. In some examples, supervised learning of machine learning algorithm(s) 1020 can be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or set of labels may be used to correct inferences of machine learning algorithm(s) 1020.

[0120] For example, the training models may use supervised learning methods, where the training data may be labeled and outputs may be graded based on their fidelity to a “truth” output. The training models may use an unsupervised learning method, where there may notbe labels on the training data and the training models may classify correlations without reference to a “truth” value. The training models may combine supervised and unsupervised techniques.

[0121] Semi-supervised learning involves having correct results for part, but not all, of training data 1010. During semi-supervised learning, supervised learning is used for a portion of training data 1010 having correct results, and unsupervised learning is used for a portion of training data 1010 not having correct results. Reinforcement learning involves machine learning algorithm(s) 1020 receiving a reward signal regarding a prior inference, where the reward signal can be a numerical value. During reinforcement learning, machine learning algorithm(s) 1020 can output an inference and receive a reward signal in response, where machine learning algorithm(s) 1020 are configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing an expected total of the numerical values provided by the reward signal over time. In some examples, machine learning algorithm(s) 1020 and / or trained machine learning model(s) 1032 can be trained using other machine learning techniques, including but not limited to, incremental learning and curriculum learning.

[0122] In some examples, machine learning algorithm(s) 1020 and / or trained machine learning model(s) 1032 can use transfer learning techniques. For example, transfer learning techniques can involve trained machine learning model(s) 1032 being pre-trained on one set of data and additionally trained using training data 1010. More particularly, machine learning algorithm(s) 1020 can be pre-trained on data from one or more computing devices and a resulting trained machine learning model provided to computing device CD1, where CD1 is intended to execute the trained machine learning model during inference phase 1004. Then, during training phase 1002, the pre-trained machine learning model can be additionally trained using training data 1010, where training data 1010 can be derived from kernel and non-kernel data of computing device CD1. This further training of the machine learning algorithm(s) 1020 and / or the pre-trained machine learning model using training data 1010 of CDl’s data can be performed using either supervised or unsupervised learning. Once machine learning algorithm(s) 1020 and / or the pre-trained machine learning model has been trained on at least training data 1010, training phase 1002 can be completed. The trained resulting machine learning model can be utilized as at least one of trained machine learning model(s) 1032.

[0123] In particular, once training phase 1002 has been completed, trained machine learning model(s) 1032 can be provided to a computing device, if not already on the computingdevice. Inference phase 1004 can begin after trained machine learning model(s) 1032 are provided to computing device CD1.

[0124] In the context of this disclosure, a ML model may be trained. Training data 1010 may be image data. In this example, the training data 1010 are first processed by a CNN. The CNN may extract features from the training data 1010. The CNN may be a disconnected neural network, meaning all members of the neural network, such as perceptrons, may not be in a causal relationship with one another. It should be mentioned that other feature extraction techniques besides CNNs may be equally employed in lieu of the CNN described herein. An MLP neural network may receive the extracted feature outputs from the CNN as inputs. The MLP may be a fully connected network, which may comprise all perceptrons in a causal relationship with one another. The MLP may have as an output a detection probability prediction 1050. The prediction 1050 may be, for example, the probability that one or more focus ranges are correct for a camera input, or some other prediction related to AF, PDAF, or similar.

[0125] During inference phase 1004, trained machine learning model(s) 1032 can receive input data 1030 and generate and output one or more corresponding inferences and / or prediction(s) 1050 about input data 1030. Input data 1030 may be data from one or more sensors. As such, input data 1030 can be used as an input to trained machine learning model(s) 1032 for providing corresponding inference(s) and / or predict! on(s) 1050 to kernel components and non-kernel components. For example, trained machine learning model(s) 1032 can generate inference(s) and / or predict! on(s) 1050 in response to one or more inference / prediction requests 1040. In some examples, trained machine learning model(s) 1032 can be executed by a portion of other software. For example, trained machine learning model(s) 1032 can be executed by an inference or prediction daemon to be readily available to provide inferences and / or predictions upon request. Input data 1030 can include data from computing device CD1 executing trained machine learning model(s) 1032 and / or input data from one or more computing devices other than CD1. For example, input data 1030 can include images, prompts. Input data 1030 can also include prompts labeled with context data, device data, browsing history, search history, and so forth. Other types of input data are possible as well.

[0126] The trained machine learning model(s) 1032 may use a matrix containing depth of field calculations, an array of input imagery, or one or more numbers between 0 and 1 as the prediction 1050. Such an ML model may have one or more feature extraction layers, such as the CNN, and one or more fully connected layers, such as the MLP.

[0127] Inference(s) and / or prediction(s) 1050 can include output prompts, output partial prompts, output prompt completions, numerical values, and / or other output data produced by trained machine learning model(s) 1032 operating on input data 1030 (and training data 1010). In some examples, trained machine learning model(s) 1032 can use output inference(s) and / or predict! on(s) 1050 as input feedback 1060. Trained machine learning model(s) 1032 can also rely on past inferences as inputs for generating new inferences.

[0128] In some examples, one computing device CD SOLO can include the trained version of the neural network, perhaps after training. Then, computing device CD SOLO can receive a request to predict a caption, and use the trained version of the neural network to predict the caption.

[0129] In some examples, two or more computing devices CD CLI and CD SRV can be used to provide output captions; e.g., a first computing device CD CLI can generate and send requests to predict a caption to a second computing device CD SRV. Then, CD SRV can use the trained version of the neural network, to predict the caption, and respond to the requests from CD CLI for the predicted caption. Then, upon reception of responses to the requests, CD CLI can provide the requested predicted caption (e.g., using a multimodal user interface and / or a display).

[0130] An ML model, as discussed in this disclosure, refers to a computer model that has been trained using one or more machine-learning techniques. In general, this training may be done by providing training inputs to one or more training models, which in turn may provide an output. The output may be in the form of a prediction, a confidence score, or other probability-based metrics.

[0131] The aforementioned are merely examples of the types of machine-learning training that may be done to produce an ML model for AF. Many other combinations and techniques may be employed, which do not alter the scope of the inventive concept. Examples such as employing a CNN, use of different data types than those listed, alone or in combination with the data types listed, having a fully connected portion of the machine-learning training, etc. are not meant to limit the scope of the disclosed methods and apparatuses, but rather to serve as examples of how an ML model for AF may be accomplished.

[0132] By way of example, training for the ML model may equivalently be accomplished by incorporating, as either a part of or the complete machine-learning training phase 1002, a long short-term memory (LSTM) algorithm. The LSTM algorithm is a type of recombinant neural network (RNN), which may process data in a time-indexed fashion. The LSTM may solve a so-called “vanishing gradient problem,” in which gradients used in fittingmay tend to zero and, thus, may not yield useful fit parameters for a given model (e.g., weights and biases). During the training phase 1002, the LSTM may allow for persistent gradients used to fit when the gradients may otherwise go to zero e.g., in a traditional RNN). For the purpose of the present disclosure, an example of implementing the LSTM in the machine-learning training phase 1002 may be to replace a MLP with the LSTM or to include the LSTM before or after the MLP.Example Data Network

[0133] FIG. 11 depicts a distributed computing architecture 1100, in accordance with example embodiments. Distributed computing architecture 1100 includes server devices 1108, 1110 that are configured to communicate, via network 1106, with programmable devices 1104a, 1104b, 1104c, 1104d, 1104e. Network 1106 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public Internet, or any other type of network configured to provide a communications path between networked computing devices. Network 1106 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public Internet.

[0134] Although FIG. 11 only shows five programmable devices, distributed application architectures may serve tens, hundreds, or thousands of programmable devices. Moreover, programmable devices 1104a, 1104b, 1104c, 1104d, 1104e (or any additional programmable devices) may be any sort of computing device, such as a mobile computing device, desktop computer, wearable computing device, head-mountable device (HMD), network terminal, a mobile computing device, and so on. In some examples, such as illustrated by programmable devices 1104a, 1104b, 1104c, 1104e, programmable devices can be directly connected to network 1106. In other examples, such as illustrated by programmable device 1104d, programmable devices can be indirectly connected to network 1106 via an associated computing device, such as programmable device 1104c. In this example, programmable device 1104c can act as an associated computing device to pass electronic communications between programmable device 1104d and network 1106. In other examples, such as illustrated by programmable device 1104e, a computing device can be part of and / or inside a vehicle, such as a car, a truck, a bus, a boat or ship, an airplane, etc. In other examples not shown in FIG. 11, a programmable device can be both directly and indirectly connected to network 1106.

[0135] Server devices 1108, 1110 can be configured to perform one or more services, as requested by programmable devices 1104a-1104e. For example, server device 1108 and / or 1110 can provide content to programmable devices 1104a-l 104e. The content can include, butis not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. The content can include compressed and / or uncompressed content. The content can be encrypted and / or unencrypted. Other types of content are possible as well.

[0136] As another example, server device 1108 and / or 1110 can provide programmable devices 1104a-1104e with access to software for database, search, computation, graphical, audio, video, World Wide Web / Internet utilization, and / or other functions. Many other examples of server devices are possible as well.Computing Device Architecture

[0137] FIG. 12 is a block diagram of an example computing device 1200, in accordance with example embodiments. In particular, computing device 1200 shown in FIG. 12 can be configured to perform at least one function of and / or related to the machine learning models, and / or methods 1400, and / or 1300.

[0138] Computing device 1200 may include a user interface module 1201, a network communications module 1202, one or more processors 1203, data storage 1204, one or more camera(s) 1212, one or more sensors 1214, and power system 1216, all of which may be linked together via a system bus, network, or other connection mechanism 1205.

[0139] User interface module 1201 can be operable to send data to and / or receive data from external user input / output devices. For example, user interface module 1201 can be configured to send and / or receive data to and / or from user input devices such as a touch screen, a computer mouse, a keyboard, a keypad, a touch pad, a trackball, a joystick, a voice recognition module, and / or other similar devices. User interface module 1201 can also be configured to provide output to user display devices, such as one or more cathode ray tubes (CRT), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, either now known or later developed. User interface module 1201 can also be configured to generate audible outputs, with devices such as a speaker, speaker jack, audio output port, audio output device, earphones, and / or other similar devices. User interface module 1201 can further be configured with one or more haptic devices that can generate haptic outputs, such as vibrations and / or other outputs detectable by touch and / or physical contact with computing device 1200. In some examples, user interface module 1201 can be used to provide a graphical user interface (GUI) for utilizing computing device 1200, such as, for example, a graphical user interface of a mobile phone device.

[0140] Network communications module 1202 can include one or more devices that provide one or more wireless interface(s) 1207 and / or one or more wireline interface(s) 1208 that are configurable to communicate via a network. Wireless interface(s) 1207 can include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth™ transceiver, a Zigbee® transceiver, a Wi-Fi™ transceiver, a WiMAX™ transceiver, an LTE™ transceiver, and / or other type of wireless transceiver configurable to communicate via a wireless network. Wireline interface(s) 1208 can include one or more wireline transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or similar transceiver configurable to communicate via a twisted pair wire, a coaxial cable, a fiber-optic link, or a similar physical connection to a wireline network.

[0141] In some examples, network communications module 1202 can be configured to provide reliable, secured, and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., guaranteed message delivery) can be provided, perhaps as part of a message header and / or footer (e.g., packet / message sequencing information, encapsulation headers and / or footers, size / time information, and transmission verification information such as cyclic redundancy check (CRC) and / or parity check values). Communications can be made secure (e.g., be encoded or encrypted) and / or decry pted / decoded using one or more cryptographic protocols and / or algorithms, such as, but not limited to, Data Encryption Standard (DES), Advanced Encryption Standard (AES), a Rivest-Shamir-Adelman (RSA) algorithm, a Diffie-Hellman algorithm, a secure sockets protocol such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms can be used as well or in addition to those listed herein to secure (and then decry pt / decode) communications.

[0142] One or more processors 1203 can include one or more general purpose processors, and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). One or more processors 1203 can be configured to execute computer- readable instructions 1206 that are contained in data storage 1204 and / or other instructions as described herein.

[0143] Data storage 1204 can include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of one or more processors 1203. The one or more computer-readable storage media can include volatile and / or non-volatile storage components, such as optical, magnetic, organic or other memory or disc storage, whichcan be integrated in whole or in part with at least one of one or more processors 1203. In some examples, data storage 1204 can be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disc storage unit), while in other examples, data storage 1204 can be implemented using two or more physical devices.

[0144] Data storage 1204 can include computer-readable instructions 1206 and perhaps additional data. In some examples, data storage 1204 can include storage required to perform at least part of the herein-described methods, scenarios, and techniques and / or at least part of the functionality of the herein-described devices and networks. In some examples, data storage 1204 can include storage for a trained neural network model 1210 (e.g., trained RNN, trained LSTM, etc.). In particular of these examples, computer-readable instructions 1206 can include instructions that, when executed by one or more processors 1203, enable computing device 1200 to provide for some or all of the functionality of trained neural network model 1210.

[0145] In some examples, computing device 1200 can include one or more camera(s) 1212. Camera(s) 1212 can include one or more image capture devices, such as still and / or video cameras, equipped to capture light and record the captured light in one or more images; that is, camera(s) 1212 can generate image(s) of captured light. The one or more images can be one or more still images and / or one or more images utilized in video imagery. Camera(s) 1212 can capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light.

[0146] In some examples, computing device 1200 can include one or more sensors 1214. Sensors 1214 can be configured to measure conditions within computing device 1200 and / or conditions in an environment of computing device 1200 and provide data about these conditions. For example, sensors 1214 can include one or more of: (i) sensors for obtaining data about computing device 1200, such as, but not limited to, a thermometer for measuring a temperature of computing device 1200, a battery sensor for measuring power of one or more batteries of power system 1216, and / or other sensors measuring conditions of computing device 1200; (ii) an identification sensor to identify other objects and / or devices, such as, but not limited to, a Radio Frequency Identification (RFID) reader, proximity sensor, one-dimensional barcode reader, two-dimensional barcode (e.g., Quick Response (QR) code) reader, and a laser tracker, where the identification sensors can be configured to read identifiers, such as RFID tags, barcodes, QR codes, and / or other devices and / or object configured to be read and provide at least identifying information; (iii) sensors to measure locations and / or movements of computing device 1200, such as, but not limited to, a tilt sensor, a gyroscope, an accelerometer, a Doppler sensor, a GPS device, a sonar sensor, a radar device, a laser-displacement sensor,and a compass; (iv) an environmental sensor to obtain data indicative of an environment of computing device 1200, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a movement sensor, a microphone, a sound sensor, an ultrasound sensor and / or a smoke sensor; and / or (v) a force sensor to measure one or more forces (e.g., inertial forces and / or G-forces) acting about computing device 1200, such as, but not limited to one or more sensors that measure: forces in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor that identifies ZMPs and / or locations of the ZMPs. Many other examples of sensors 1214 are possible as well.

[0147] Power system 1216 can include one or more batteries 1218 and / or one or more external power interfaces 1220 for providing electrical power to computing device 1200. Each battery of the one or more batteries 1218 can, when electrically coupled to the computing device 1200, act as a source of stored electrical power for computing device 1200. One or more batteries 1218 of power system 1216 can be configured to be portable. Some or all of one or more batteries 1218 can be readily removable from computing device 1200. In other examples, some or all of one or more batteries 1218 can be internal to computing device 1200, and so may not be readily removable from computing device 1200. Some or all of one or more batteries 1218 can be rechargeable. For example, a rechargeable battery can be recharged via a wired connection between the battery and another power supply, such as by one or more power supplies that are external to computing device 1200 and connected to computing device 1200 via the one or more external power interfaces. In other examples, some or all of one or more batteries 1218 can be non-rechargeable batteries.

[0148] One or more external power interfaces 1220 of power system 1216 can include one or more wired-power interfaces, such as a USB cable and / or a power cord, that enable wired electrical power connections to one or more power supplies that are external to computing device 1200. One or more external power interfaces 1220 can include one or more wireless power interfaces, such as a Qi wireless charger, that enable wireless electrical power connections, such as via a Qi wireless charger, to one or more external power supplies. Once an electrical power connection is established to an external power source using one or more external power interfaces 1220, computing device 1200 can draw electrical power from the external power source the established electrical power connection. In some examples, power system 1216 can include related sensors, such as battery sensors associated with the one or more batteries or other types of electrical power sensors.Cloud-Based Servers

[0149] FIG. 13 depicts a cloud-based server system in accordance with an example embodiment. In FIG. 13, functionality of a neural network, and / or a computing device can be distributed among computing clusters 1309a, 1309b, and 1309c. Computing cluster 1309a can include one or more computing devices 1300a, cluster storage arrays 1310a, and cluster routers 1311a connected by a local cluster network 1312a. Similarly, computing cluster 1309b can include one or more computing devices 1300b, cluster storage arrays 1310b, and cluster routers 1311b connected by a local cluster network 1312b. Likewise, computing cluster 1309c can include one or more computing devices 1300c, cluster storage arrays 1310c, and cluster routers 1311c connected by a local cluster network 1312c.

[0150] In some embodiments, computing clusters 1309a, 1309b, and 1309c can be a single computing device residing in a single computing center. In other embodiments, computing clusters 1309a, 1309b, and 1309c can include multiple computing devices in a single computing center, or even multiple computing devices located in multiple computing centers located in diverse geographic locations. For example, FIG. 13 depicts each of computing clusters 1309a, 1309b, 1309c residing in different physical locations.

[0151] In some embodiments, data and services at computing clusters 1309a, 1309b, 1309c can be encoded as computer readable information stored in non-transitory, tangible computer readable media (or computer readable storage media) and accessible by other computing devices. In some embodiments, computing clusters 1309a, 1309b, 1309c can be stored on a single disk drive or other tangible storage media, or can be implemented on multiple disk drives or other tangible storage media located at one or more diverse geographic locations.

[0152] In some embodiments, each of computing clusters 1309a, 1309b, and 1309c can have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. In other embodiments, however, each computing cluster can have different numbers of computing devices, different numbers of cluster storage arrays, and different numbers of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster can depend on the computing task or tasks assigned to each computing cluster.

[0153] In computing cluster 1309a, for example, computing devices 1300a can be configured to perform various computing tasks of a neural network, and / or a computing device. In one embodiment, the various functionalities of a neural network, and / or a computing device can be distributed among one or more of computing devices 1300a, 1300b, and 1300c. Computing devices 1300b and 1300c in respective computing clusters 1309b and 1309c can beconfigured similarly to computing devices 1300a in computing cluster 1309a. On the other hand, in some embodiments, computing devices 1300a, 1300b, and 1300c can be configured to perform different functions.

[0154] In some embodiments, computing tasks and stored data associated with a neural network, and / or a computing device can be distributed across computing devices 1300a, 1300b, and 1300c based at least in part on the processing requirements of a neural network, and / or a computing device, the processing capabilities of computing devices 1300a, 1300b, 1300c, the latency of the network links between the computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that can contribute to the cost, speed, fault-tolerance, resiliency, efficiency, and / or other design goals of the overall system architecture.

[0155] Cluster storage arrays 1310a, 1310b, 1310c of computing clusters 1309a, 1309b, and 1309c can be data storage arrays that include disk array controllers configured to manage read and write access to groups of hard disk drives. The disk array controllers, alone or in conjunction with their respective computing devices, can also be configured to manage backup or redundant copies of the data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0156] Similar to the manner in which the functions of a conditioned, axial selfattention based neural network, and / or a computing device can be distributed across computing devices 1300a, 1300b, 1300c of computing clusters 1309a, 1309b, 1309c, various active portions and / or backup portions of these components can be distributed across cluster storage arrays 1310a, 1310b, 1310c. For example, some cluster storage arrays can be configured to store one portion of the data of a first layer of a neural network, and / or a computing device, while other cluster storage arrays can store other portion(s) of data of second layer of a neural network, and / or a computing device. Also, for example, some cluster storage arrays can be configured to store the data of an encoder of a neural network, while other cluster storage arrays can store the data of a decoder of a neural network. Additionally, some cluster storage arrays can be configured to store backup versions of data stored in other cluster storage arrays.

[0157] Cluster routers 1311a, 1311b, 1311c in computing clusters 1309a, 1309b, and 1309c can include networking equipment configured to provide internal and external communications for the computing clusters. For example, cluster routers 1311a in computing cluster 1309a can include one or more internet switching and routing devices configured to provide (i) local area network communications between computing devices 1300a and clusterstorage arrays 1310a via local cluster network 1312a, and (ii) wide area network communications between computing cluster 1309a and computing clusters 1309b and 1309c via wide area network link 1313a to network 1106. Cluster routers 1311b and 1311c can include network equipment similar to cluster routers 1311a, and cluster routers 1311b and 1311c can perform similar networking functions for computing clusters 1309b and 1309b that cluster routers 1311a perform for computing cluster 1309a.

[0158] In some embodiments, the configuration of cluster routers 1311a, 1311b, 1311c can be based at least in part on the data communication requirements of the computing devices and cluster storage arrays, the data communications capabilities of the network equipment in cluster routers 1311a, 1311b, 1311c, the latency and throughput of local cluster networks 1312a, 1312b, 1312c, the latency, throughput, and cost of wide area network links 1313a, 1313b, 1313c, and / or other factors that can contribute to the cost, speed, fault-tolerance, resiliency, efficiency and / or other design criteria of the moderation system architecture.Example Methods of Operation

[0159] FIG. 14 is a flowchart of a method 1400, in accordance with example embodiments. Method 1400 can be executed by a computing device, such as computing device 1200. Method 1400 can begin at block 1410, where the method involves receiving, by a computing device, a training dataset comprising a plurality of images based on dense sensor data.

[0160] At block 1420, the method involves training, based on the training dataset, a temporal, single-depth machine learning (ML) model to predict a disparity value for a phasedifference autofocus (PDAF) adjustment to a lens position for a camera, wherein the training of the ML model comprises learning temporal dependencies to maintain a consistent change in disparities over time.

[0161] At block 1430, the method involves providing, by the computing device, the trained temporal, single-depth ML model.

[0162] In some embodiments, an image sensor for the camera may be configured to operate in a dual mode, wherein a first mode of the dual mode may be configured to provide dense sensor data, and wherein a second mode of the dual mode may be configured to provide sparse sensor data.

[0163] In some embodiments, the image sensor, when operating in the first mode, may output high resolution image data.

[0164] In some embodiments, the image sensor, when operating in the second mode, includes a left-right (L / R) shielded pixel arrangement.

[0165] In some embodiments, the image sensor, when operating in the second mode, may output low resolution image data.

[0166] In some embodiments, the training of the ML model involves determining disparity ground truth data based on the dense sensor data. Such embodiments involve providing the disparity ground truth data for use when operating in the first mode or the second mode.

[0167] In some embodiments, the ML model may be a recurrent neural network (RNN) configured to maintain an internal state that is shared from a previous inference to a next inference.

[0168] In some embodiments, the RNN includes a plurality of long short-term memory(LSTM) units. In some embodiments, the LSTM includes 4 units.

[0169] In some embodiments, the LSTM units may be positioned prior to a dense output layer of the RNN.

[0170] In some embodiments, the training of the ML model may be based on a Mean Absolute Error (MAE) loss.

[0171] In some embodiments, the MAE loss may be based on a center weight term adjustment for disparities around a zero disparity lens position.

[0172] In some embodiments, the ML model includes a confidence subnetwork to predict a confidence value associated with the predicted disparity value, wherein a confidence value for a predicted disparity value may be indicative of an error in the predicted disparity value.

[0173] In some embodiments, the training of the ML model involves pre-training the ML model to predict the disparity value. Such embodiments also involve training the confidence subnetwork based on the predicted disparity value output by the pre-trained ML model.

[0174] In some embodiments, the plurality of images include one or more regions of interest (ROIs). Such embodiments involve readjusting a size of an input for the temporal, single-depth machine learning (ML) model to conform to a size of the one or more ROIs.

[0175] In some embodiments, the training dataset may be synthetically generated based on a size distribution of the one or more ROIs.

[0176] In some embodiments, the training dataset includes synthetically generated focus sweeps that capture images at different blur and phase-difference (PD) disparity levels.

[0177] FIG. 15 is a flowchart of a method 1500, in accordance with example embodiments. Method 1500 can be executed by a computing device, such as computing device 1200. Method 1500 can begin at block 1510, where the method involves receiving image data from an image sensor of a camera, wherein the image sensor is configured to operate in a dual mode, wherein a first mode of the dual mode is configured to provide dense sensor data, and wherein a second mode of the dual mode is configured to provide sparse sensor data.

[0178] At block 1520, the method involves predicting, based on the image data and by a trained temporal, single-depth machine learning (ML) model, a disparity value for a phasedifference autofocus (PDAF) adjustment to a lens position for the camera, the ML model having been trained to learn temporal dependencies to maintain a consistent change in disparities over time.

[0179] At block 1530, the method involves providing the predicted disparity value.

[0180] Some embodiments involve adjusting the lens position for the camera based on the predicted disparity value.

[0181] In some embodiments, the image sensor, when operating in the first mode, may output high resolution image data.

[0182] In some embodiments, the image sensor, when operating in the second mode, includes a left-right (L / R) shielded pixel arrangement.

[0183] In some embodiments, the image sensor, when operating in the second mode, may output low resolution image data.

[0184] In some embodiments, the image data includes one or more regions of interest (ROIs).

[0185] Some embodiments involve readjusting a tuning ratio of a ROI of the one or more ROIs to align with an aspect ratio of an image frame.

[0186] In some embodiments, the one or more ROIs may be located within bounding boxes. Such embodiments involve increasing, for a bounding box for a ROI of the one or more ROIs, a ratio of width to height.

[0187] In some embodiments, the image data includes an image of a scene, and wherein an ambient light for the scene is below a threshold brightness.

[0188] In some embodiments, the trained temporal, single-depth ML model may be a recurrent neural network (RNN) configured to maintain an internal state that is shared from a previous inference to a next inference.

[0189] In some embodiments, the RNN includes a plurality of long short-term memory (LSTM) units.

[0190] In some embodiments, the trained ML model includes a trained confidence subnetwork to predict a confidence value associated with the predicted disparity value, wherein a confidence value for a predicted disparity value is indicative of an error in the predicted disparity value.

[0191] The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its spirit and scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those enumerated herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.

[0192] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

[0193] With respect to any or all of the ladder diagrams, scenarios, and flow charts in the figures and as discussed herein, each block and / or communication may represent a processing of information and / or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, functions described as blocks, transmissions, communications, requests, responses, and / or messages may be executed out of order from that shown or discussed, including substantially concurrent or in reverse order, depending on the functionality involved. Further, more or fewer blocks and / or functions may be used with any of the ladder diagrams, scenarios, and flow charts discussed herein, and these ladder diagrams, scenarios, and flow charts may be combined with one another, in part or in whole.

[0194] A block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described methodor technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and / or related data may be stored on any type of computer readable medium such as a storage device including a disk or hard drive or other storage medium.

[0195] The computer readable medium may also include non-transitory computer readable media such as non-transitory computer-readable media that stores data for short periods of time like register memory, processor cache, and random access memory (RAM). The computer readable media may also include non-transitory computer readable media that stores program code and / or data for longer periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

[0196] Moreover, a block that represents one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.

[0197] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for explanatory purposes and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method, comprising: receiving, by a computing device, a training dataset comprising a plurality of images based on dense sensor data; training, based on the training dataset, a temporal, single-depth machine learning (ML) model to predict a disparity value for a phase-difference autofocus (PDAF) adjustment to a lens position for a camera, wherein the training of the ML model comprises learning temporal dependencies to maintain a consistent change in disparities over time; and providing, by the computing device, the trained temporal, single-depth ML model.

2. The computer-implemented method of claim 1, wherein an image sensor for the camera is configured to operate in a dual mode, wherein a first mode of the dual mode is configured to provide dense sensor data, and wherein a second mode of the dual mode is configured to provide sparse sensor data.

3. The computer-implemented method of claim 2, wherein the image sensor, when operating in the first mode, outputs high resolution image data.

4. The computer-implemented method of claim 2, wherein the image sensor, when operating in the second mode, comprises a left-right (L / R) shielded pixel arrangement.

5. The computer-implemented method of claim 2, wherein the image sensor, when operating in the second mode, outputs low resolution image data.

6. The computer-implemented method of claim 2, wherein the training of the ML model further comprises: determining disparity ground truth data based on the dense sensor data; and providing the disparity ground truth data for use when operating in the first mode or the second mode.

7. The computer-implemented method of any of claims 1-6, wherein the ML model is a recurrent neural network (RNN) configured to maintain an internal state that is shared from a previous inference to a next inference.

8. The computer-implemented method of claim 7, wherein the RNN comprises a plurality of long short-term memory (LSTM) units.

9. The computer-implemented method of claim 8, wherein the LSTM comprises 4 units.

10. The computer-implemented method of claim 8, wherein the LSTM units are positioned prior to a dense output layer of the RNN.

11. The computer-implemented method of any of claims 1-10, wherein the training of the ML model is based on a Mean Absolute Error (MAE) loss.

12. The computer-implemented method of claim 11, wherein the MAE loss is based on a center weight term adjustment for disparities around a zero disparity lens position.

13. The computer-implemented method of any of claims 1-12, wherein the ML model comprises a confidence subnetwork to predict a confidence value associated with the predicted disparity value, wherein a confidence value for a predicted disparity value is indicative of an error in the predicted disparity value.

14. The computer-implemented method of claim 11, wherein the training of the ML model further comprises: pre-training the ML model to predict the disparity value; and training the confidence subnetwork based on the predicted disparity value output by the pre-trained ML model.

15. The computer-implemented method of any of claims 1-14, wherein the plurality of images comprise one or more regions of interest (ROIs), and further comprising: readjusting a size of an input for the temporal, single-depth machine learning (ML) model to conform to a size of the one or more ROIs.

16. The computer-implemented method of claim 15, wherein the training dataset is synthetically generated based on a size distribution of the one or more ROIs.

17. The computer-implemented method of any of claims 1-16, wherein the training dataset comprises synthetically generated focus sweeps that capture images at different blur and phasedifference (PD) disparity levels.

18. A computer-implemented method, comprising: receiving image data from an image sensor of a camera, wherein the image sensor is configured to operate in a dual mode, wherein a first mode of the dual mode is configured to provide dense sensor data, and wherein a second mode of the dual mode is configured to provide sparse sensor data; predicting, based on the image data and by a trained temporal, single-depth machine learning (ML) model, a disparity value for a phase-difference autofocus (PDAF) adjustment to a lens position for the camera, the ML model having been trained to learn temporal dependencies to maintain a consistent change in disparities over time; and providing the predicted disparity value.

19. The computer-implemented method of claim 18, further comprising: adjusting the lens position for the camera based on the predicted disparity value.

20. The computer-implemented method of any of claims 18 or 19, wherein the image sensor, when operating in the first mode, outputs high resolution image data.

21. The computer-implemented method of any of claims 18-20, wherein the image sensor, when operating in the second mode, comprises a left-right (L / R) shielded pixel arrangement.

22. The computer-implemented method of any of claims 18-21, wherein the image sensor, when operating in the second mode, outputs low resolution image data.

23. The computer-implemented method of any of claims 18-22, wherein the image data comprises one or more regions of interest (ROIs).

24. The computer-implemented method of claim 23, further comprising:readjusting a tuning ratio of a ROI of the one or more ROIs to align with an aspect ratio of an image frame.

25. The computer-implemented method of claim 23, wherein the one or more ROIs are located within bounding boxes, and further comprising: increasing, for a bounding box for a ROI of the one or more ROIs, a ratio of width to height.

26. The computer-implemented method of any of claims 18-25, wherein the image data comprises an image of a scene, and wherein an ambient light for the scene is below a threshold brightness.

27. The computer-implemented method of any of claims 18-26, wherein the trained temporal, single-depth ML model is a recurrent neural network (RNN) configured to maintain an internal state that is shared from a previous inference to a next inference.

28. The computer-implemented method of claim 27, wherein the RNN comprises a plurality of long short-term memory (LSTM) units.

29. The computer-implemented method of any of claims 18-28, wherein the trained ML model comprises a trained confidence subnetwork to predict a confidence value associated with the predicted disparity value, wherein a confidence value for a predicted disparity value is indicative of an error in the predicted disparity value.

30. A computing device, comprising: one or more processors; and data storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to carry out functions comprising the computer-implemented method of any one of claims 1-29.

31. The computing device of claim 29, wherein the computing device is a mobile device.

32. A computer program comprising instructions that, when executed by a computer, cause the computer to perform steps in accordance with the method of any one of claims 1-29.

33. An article of manufacture comprising one or more non-transitory computer readable media having computer-readable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to carry out functions that comprise the computer-implemented method of any one of claims 1-29.

34. A system, comprising: means for carrying out the computer-implemented method of any one of claims 1-29.

Images