Depth prediction based on dual pixel images

Abstract

Apparatus and methods related to determining a depth map of a bipixel image of an object using machine learning are provided. The computing device can receive a bipixel image of at least a foreground object. The bipixel image may include multiple bipixels. The bipixels of the multiple bipixels may include a left pixel and a right pixel, each representing light incident on a single bipixel element used to capture the bipixel image. The computing device can be used to train a machine learning system to determine a depth map associated with the bipixel image. The computing device can provide the trained machine learning system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 16 / 246,280, filed on January 11, 2019, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The inventive concepts of the present application provide apparatus and methods related to using machine learning to determine depth map correlation of dual-pixel images of an object. Background Art

[0004] Many modern computing devices (including mobile phones, personal computers, and tablets) include image capture devices, such as still cameras and / or video cameras. The image capture device can capture images, such as images including people, animals, landscapes, and / or objects.

[0005] Some image capture devices and / or computing devices correct or otherwise modify captured images. For example, some image capture devices may provide red-eye correction, which removes artifacts such as reddening of the eyes of people and animals that may be present in images captured using strong light (such as a flash). After the captured image is corrected, the corrected image may be saved, displayed, transmitted, printed onto paper, and / or used in other ways. Summary of the invention

[0006] In one aspect, a computer-implemented method is provided. A computing device receives a dual-pixel image of at least a foreground object. The dual-pixel image includes a plurality of dual-pixels. A dual-pixel in the plurality of dual-pixels includes a left pixel and a right pixel, each of the left pixel and the right pixel representing light incident on a single dual-pixel element for capturing the dual-pixel image. The computing device is used to train a machine learning system to determine a depth map associated with the dual-pixel image. The computing device provides a trained machine learning system.

[0007] In another aspect, a computing device is provided. The computing device includes one or more processors and a data store. The data store has stored thereon computer executable instructions that, when executed by the one or more processors, cause the computing device to perform functions. The functions include: receiving a dual-pixel image of at least one foreground object, the dual-pixel image including a plurality of dual-pixels, wherein the dual-pixels of the plurality of dual-pixels include a left pixel and a right pixel, the left pixel and the right pixel both representing light incident on a single dual-pixel element used to capture the dual-pixel image; training a machine learning system to determine a depth map associated with the dual-pixel image; and providing the trained machine learning system.

[0008] In another aspect, a product is provided. The product includes one or more computer-readable media having computer-readable instructions stored thereon, which, when executed by one or more processors of a computing device, cause the computing device to perform functions. The functions include: receiving a dual-pixel image of at least one foreground object, the dual-pixel image including a plurality of dual-pixels, wherein the dual-pixels of the plurality of dual-pixels include a left pixel and a right pixel, the left pixel and the right pixel both representing light incident on a single dual-pixel element used to capture the dual-pixel image; training a machine learning system to determine a depth map associated with the dual-pixel image; and providing the trained machine learning system.

[0009] In another aspect, a computing device is provided. The computing device includes: a method for receiving a dual-pixel image of at least a foreground object, the dual-pixel image including a plurality of dual-pixels, wherein the dual-pixels of the plurality of dual-pixels include a left pixel and a right pixel, the left pixel and the right pixel each representing light incident on a single dual-pixel element used to capture the dual-pixel image; a method for training a machine learning system to determine a depth map associated with the dual-pixel image; and a method for providing the trained machine learning system.

[0010] In another aspect, a computer-implemented method is provided. A computing device receives a dual-pixel image of at least a foreground object. The dual-pixel image includes a plurality of dual-pixels. A dual-pixel in the plurality of dual-pixels includes a left pixel and a right pixel, each of the left pixel and the right pixel representing light incident on a single dual-pixel element for capturing the dual-pixel image. The dual-pixel image is provided to a trained machine learning system to determine a depth map for the plurality of dual-pixels of the dual-pixel image. The computing device is configured to determine an output image based on the dual-pixel image and the depth map.

[0011] In another aspect, a computing device is provided. The computing device includes one or more processors and a data store. The data store has stored thereon computer executable instructions that, when executed by the one or more processors, cause the computing device to perform functions. The functions include: receiving a dual-pixel image of at least a foreground object, the dual-pixel image including a plurality of dual-pixels, wherein the dual-pixels of the plurality of dual-pixels include a left pixel and a right pixel, the left pixel and the right pixel both representing light incident on a single dual-pixel element for capturing the dual-pixel image; providing the dual-pixel image to a trained machine learning system to determine a depth map for the plurality of dual-pixels of the dual-pixel image; and determining an output image based on the dual-pixel image and the depth map.

[0012] In another aspect, a product is provided. The product includes one or more computer-readable media having computer-readable instructions stored thereon, which, when executed by one or more processors of a computing device, cause the computing device to perform functions. The functions include: receiving a dual-pixel image of at least one foreground object, the dual-pixel image including a plurality of dual-pixels, wherein the dual-pixels of the plurality of dual-pixels include a left pixel and a right pixel, and the left pixel and the right pixel both represent light incident on a single dual-pixel element for capturing the dual-pixel image; providing the dual-pixel image to a trained machine learning system to determine a depth map for the plurality of dual-pixels of the dual-pixel image; and determining an output image based on the dual-pixel image and the depth map.

[0013] In another aspect, a computing device is provided. The computing device includes: a method for receiving a dual-pixel image of at least one foreground object, the dual-pixel image including a plurality of dual-pixels, wherein the dual-pixels of the plurality of dual-pixels include a left pixel and a right pixel, and the left pixel and the right pixel both represent light incident on a single dual-pixel element for capturing the dual-pixel image; a method for providing the dual-pixel image to a trained machine learning system to determine a depth map for the plurality of dual-pixels of the dual-pixel image; and a method for determining an output image based on the dual-pixel map and the depth map.

[0014] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A computing device including a camera is shown according to an example embodiment.

[0016] Figure 2 A dual pixel image according to an example embodiment is shown.

[0017] Figure 3 is a schematic diagram illustrating the training and inference phases of a machine learning model according to an example embodiment.

[0018] Figure 4 is a schematic diagram including a block diagram of a system for partially blurring an image according to an example embodiment.

[0019] Figure 5 According to an example embodiment, Figure 4 The neural network of the system.

[0020] Fig. 6A According to an example embodiment, Figure 5 The encoder function of the neural network.

[0021] Figure 6B According to an example embodiment, Figure 5 Another encoder function for a neural network.

[0022] Figure 7 According to an example embodiment, Figure 5 The decoder function of the neural network.

[0023] Fig. 8A A multi-camera device according to an example embodiment is shown.

[0024] Figure 8B Further shown according to the example embodiment Fig. 8A Multi-camera equipment.

[0025] Fig. 9 is a diagram showing a method for making Fig. 8A Message flow diagram of the protocol for clock synchronization of multiple camera devices.

[0026] Fig.10 is a diagram showing a method of performing Fig. 8A Message flow diagram of the process of using repeated requests for a multi-camera device.

[0027] Fig.11 is a diagram showing a method of performing Fig. 8A Message flow diagram of the process of using a single frame request for a multi-camera device.

[0028] Fig.12 A distributed computing architecture according to an example embodiment is depicted.

[0029] Fig.13 is a block diagram of a computing device according to an example embodiment.

[0030] Fig.14 A network of computing clusters arranged as a cloud-based server system is depicted according to an example embodiment.

[0031] Fig.15 is a flowchart of a method according to an example embodiment.

[0032] Fig.16 is a flow chart of another method according to an example embodiment. DETAILED DESCRIPTION

[0033] The application describes the use of machine learning techniques (such as, but not limited to, neural network techniques) to selectively blur a portion of an image. After capturing an image of an object, such as a person, the captured image may be modified to further highlight the object. One technique for modifying an image to highlight an object depicted in the image is to blur other objects in the image—the resulting selectively blurred image shows a foreground object (or objects) in focus, and background objects and / or scenery that are out of focus, thereby highlighting the foreground object(s) within the image. Thus, a technical problem associated with image processing arises that involves selectively blurring a portion of an image that has already been obtained; for example, selectively blurring only a background portion of an image that has already been obtained.

[0034] A depth map (which may be a map or other data structure storing information about the distance from the surface of a scene object to a viewpoint) may be used to partially blur an image that has been acquired; for example, a depth map for an image captured by a camera may specify information about the distance from the camera to the surface of an object captured in the image, wherein the depth map may specify information about the image on a pixel-by-pixel (or other) basis. For example, a depth map may include a depth value for each pixel in the image, wherein a depth value DV1 of a depth map DM for a pixel PIX of an image IM represents the distance from the viewpoint to one or more objects depicted by the pixel PIX in the image IM. As another example, the image IM may be divided into regions (e.g., boxes of N×M pixels, where N and M are positive integers), and the depth map may include a depth value for each pixel region in the image; for example, a depth value DV2 of a depth map DM for a pixel region PIXR of an image IM represents the distance from the viewpoint to one or more objects depicted by the pixel region PIXR of the image IM. Other depth maps and correspondences between pixels of an image and depth values ​​of a depth map are also possible; for example, a depth value in a depth map for each dual pixel of a dual pixel image.

[0035] To partially blur the image, a portion of the image whose depth or distance from the camera is farther than the foreground object(s) determined by the depth data of the depth map can be at least partially blurred. Suitable blurring software can employ the depth map to perform a natural and pleasing depth-dependent blur on the background of the image while keeping the foreground objects in sharp focus. Furthermore, the depth map of the image can have other applications in computational photography, augmented reality, and image processing.

[0036] A common method for estimating a depth map of an image involves using two cameras. However, a depth map can be estimated from an image taken by a single camera using dual pixels on a light detection sensor; for example, a camera that provides an autofocus function. The dual pixels of an image can be considered as pixels that have been divided into two parts, such as a left pixel and a right pixel. Then, a dual pixel image is an image that includes dual pixels. For example, an image IMAGE1 with R rows and C columns of pixels can be and / or based on a dual pixel image DPI with R rows and C columns of dual pixels corresponding to pixels of the image IMAGE1.

[0037] To capture dual pixels, a camera may use a sensor that captures two slightly different views of a scene. When comparing the two views, foreground objects may appear to be stationary while background objects move vertically in an effect known as parallax. For example, a "selfie" or image of a person taken by that person typically has the person's face as the foreground object, and may have other objects in the background. Therefore, when comparing the two dual pixel views of the selfie, the person's face will appear to be stationary while the background objects will appear to move vertically.

[0038] Another common approach to estimating the depth map of an image involves using a dedicated depth sensor. For example, a time-of-flight camera can be used to estimate the depth map. However, this approach does not work when these dedicated depth sensors are not available.

[0039] Dual-pixel images can be used to estimate the depth map of an image taken by a single camera. A method for calculating depth from a dual-pixel image includes treating a dual-pixel image as two different single-pixel images and trying to match the two different single-pixel images. The depth of each point determines the degree of its movement between the two views. Therefore, we can estimate the depth by matching each point in one view with its corresponding point in the other view. This method is called binocular stereo vision. However, because the scene points hardly move between views, it is quite challenging to find these correspondences in dual-pixel images. Binocular stereo vision can be improved based on the observation that parallax is only one of the many depth clues (including semantics, defocus, and possibly other clues) present in the image. An example semantic clue is the reasoning that relatively close objects occupy more pixels in the image than relatively far objects. Defocus clues are clues based on the observation that points relatively far away from the observer (e.g., camera) appear less clear / blurry than relatively close points.

[0040] A proposed method for estimating and / or calculating depth from a dual-pixel image is to use a machine learning technique such as a neural network to predict depth from the dual-pixel image. In particular, the dual-pixel image can be provided to the neural network to train the neural network to predict a depth map of the input dual-pixel image. For example, the neural network can be and / or can include a convolutional neural network. The neural network can utilize disparity cues, semantic cues, and possibly other aspects of the dual-pixel image to predict a depth map of the input dual-pixel image.

[0041] The neural network can be trained on a relatively large image dataset (e.g., 50,000 or more). The dataset can include multiple photos of an object taken from different viewpoints substantially simultaneously to provide ground truth data for training the neural network to predict a depth map from a dual-pixel image. For example, a multi-camera device can be used to obtain multiple photos of an object taken by multiple cameras at slightly different angles to provide better ground truth depth data to train the neural network. In some examples, the multi-camera device can include multiple mobile computing devices, each equipped with a camera that can take dual-pixel images. The resulting dual-pixel images that serve as training data for the neural network are then similar to dual-pixel images taken using the same or similar type of camera on other mobile computing devices (e.g., a user's mobile computing device).

[0042] As a specific example, a multi-camera device may include five identical mobile computing devices; e.g., five identical smartphones. Dual-pixel images may be captured by the cameras of all five mobile computing devices simultaneously, substantially simultaneously; e.g., within a tolerance of 0.5-3 milliseconds (ms). Synchronization between the cameras ensures that depth can be calculated for dynamic scenes; e.g., scenes with moving objects.

[0043] Structure from motion and multi-view stereo techniques can be used to compute depth maps from dual-pixel images captured by a multi-camera device. Using five viewpoints eliminates the aperture problem because parallax exists in multiple directions. The arrangement of the cameras ensures that points in an image are usually visible in at least one other image, thereby reducing points without correspondence. In addition, the "baseline" or distance between the cameras of a multi-camera device is larger than the baseline of the sensor used to capture a typical dual-pixel image, resulting in more accurate depth estimates. Therefore, using five synchronized viewpoints can obtain a high-quality depth map, which is used as the ground truth depth map in training neural networks.

[0044] As part of training the neural network, a loss function can be used to evaluate the estimated depth data and / or estimated depth map provided by the neural network relative to the ground truth depth data. The loss function can be invariant to depth ambiguity in the dual pixel data, thereby enabling the neural network to learn from the ground truth depth data.

[0045] Once the neural network is trained, the trained neural network can receive as input an image of an object O having dual pixel data and estimate a depth map having estimated depth data for the input image. This depth map can then be provided to the blur software. The blur software can use the depth map to blur the background portion of the image without blurring the foreground portion of the image having the object O, thereby highlighting the object O within the image. Thus, the technical problem of image processing related to selectively blurring a portion of an already acquired image can be solved using the techniques described herein, which utilize machine learning / neural networks to estimate a depth map used in selectively blurring the image.

[0046] The techniques for selectively blurring images described herein may be performed by a mobile computing device, such as, but not limited to, a smartphone. For example, a trained neural network may be provided to a mobile computing device. Then, after the mobile computing device captures a dual-pixel image, the trained neural network may be called to estimate and provide a depth map for the dual-pixel image. Then, depth rendering software on the mobile computing device may use the depth map, and possibly object segmentation data (e.g., a mask outlining a person or other object of interest within the dual-pixel image) to selectively blur the dual-pixel image. For example, a user may apply blurring software to draw attention to an object (or objects) in the foreground by blurring the background. The selectively blurred image may then be displayed, transmitted, stored, and / or otherwise utilized; for example, printed onto paper.

[0047] The techniques described herein provide natural-looking enhanced images with foreground objects by selectively blurring background objects. Using machine learning techniques as described herein (such as the use of neural networks) it is possible to estimate a depth map that takes into account traditional depth cues such as parallax, as well as additional depth cues such as, but not limited to, semantic cues and defocus cues. Therefore, selectively blurred images generated using the machine learning techniques described herein using depth maps can look better than selectively blurred images using other techniques. Furthermore, the techniques described herein can be performed on a mobile device, allowing a user to selectively blur an image almost immediately after the image is captured. In examples where the same mobile device that captures the image can selectively blur the image, the techniques described herein can be used to selectively blur an image even without a network connection.

[0048] Machine learning technique for predicting depth from dual-pixel images

[0049] Figure 1 A computing device 100 is shown including a camera 110 according to an example embodiment. The camera 110 includes a dual pixel light sensor 120, which in turn has a grid of R rows x C columns of dual pixel elements (DPEs). Figure 1 It is shown that the first row of dual-pixel light sensors 120 includes C dual-pixel elements 130a, 130b...130c, the second row of dual-pixel light sensors 120 includes C dual-pixel elements 132a, 132b...132c, and so on, until reaching the Rth row of dual-pixel light sensors 120 including C dual-pixel elements 134a, 134b...134c.

[0050] When the camera 110 is directed to capture an image of the environment (e.g., an image of a face and / or other object in the environment), incident light 122 from the environment enters the camera 110 and reaches the dual pixel light sensor 120. The dual pixel sensor 120 then uses the dual pixel elements 130a...134c to capture and record information about the incident light 122. The camera 110 then generates an image of the environment using the recorded information about the incident light 122, which is displayed in FIG. Figure 1 It is shown as a dual pixel image 124 .

[0051] exist Figure 1 The lower portion of FIG. 1 shows an example of incident light 122 reaching a dual pixel element 134 c of a dual pixel light sensor 120. Each of the dual pixel elements in the dual pixel light sensor 120 (including the dual pixel element 134 c) may include two light sensors. In the example of the dual pixel element 134 c, the two light sensors are Figure 1 Shown in FIG. 1 are a left light sensor 140 and a right light sensor 142 .

[0052] Then, when the incident light 122 reaches the dual pixel element 134c, a first portion of the incident light 122 reaches and is incident on the left light sensor 140, and a second portion of the incident light 122 reaches and is incident on the right light sensor 142. Figure 1 A first portion of the incident light 122 reaching the left light sensor 140 is shown as left-side light 122a, and a second portion of the incident light 122 reaching the right light sensor 142 is shown as right-side light 122b. When the left-side light 122a reaches the left light sensor 140, the left light sensor 140 captures the left-side light 122a and thus records the captured light as the left-side pixel 124a. Similarly, when the right-side light 122b reaches the right light sensor 142, the right light sensor 142 captures the right-side light 122b and thus records the captured light as the right-side pixel 124b. And, the dual-pixel image 124 includes a left-side pixel and a right-side pixel from each dual-pixel element in the dual-pixel light sensor 120, which includes the left-side pixel 124a and the right-side pixel 124b.

[0053] The left side light 122a is different from the right side light 122b that reaches the right side light sensor 142 and reaches the left light sensor 140 at a (slightly) different angle than the right side light 122b that reaches the right light sensor 142. When the left side light 122a arrives at a different angle than the right side light 122b, the left side pixel 124a has a (slightly) different viewing angle than the right side pixel 124b. Therefore, the left side image of the dual pixel image 124 composed of all the left side pixels of the dual pixel image 124 will be (slightly) different from the right side image of the dual pixel image 124 composed of all the right side pixels of the dual pixel image 124. Therefore, parallax will exist between the left side image and the right side image of the dual pixel image 124.

[0054] Figure 2 A dual pixel image 200 according to an example embodiment is shown. Dual pixel image 200 includes a left image 210 and a right image 220. An image may have a foreground or portion of an image closest to an observer (e.g., a camera capturing the image), and a background, or portion of the image away from the observer. In some cases, the foreground and background are mutually exclusive portions that together constitute the entire image.

[0055] Figure 2 It is shown that the dual-pixel image 200 has a foreground and a background, and therefore, both the left image 210 and the right image 220 have a foreground and a background. For example, the left image 210 of the dual-pixel image 200 has a background 230a including buildings and trees and has a person as a foreground object 240a. Similarly, the right image 220 of the dual-pixel image 200 has a background 230b also including buildings and trees and also has a person as a foreground object 240b.

[0056] Both the left image 210 and the right image 220 show the same scene with the same person as a foreground object and the same buildings and trees in the background. However, parallax exists between the left image 210 and the right image 220, where the parallax may be most noticeable near the ear of the person shown as a foreground object between the left image 210 and the right image 220. As described above, at least in Figure 1 As indicated in the context of , the parallax is due to a (slight) difference in the light direction of the incident light captured for the left image 210 and the light direction of the incident light captured for the right image 220 .

[0057] Figure 3A system 300 is shown according to an example embodiment, which illustrates a training phase 302 and an inference phase 304 of a trained (multiple) machine learning model 332. Some machine learning techniques involve training one or more machine learning systems on an input training data set to recognize patterns in the training data and provide output inferences and / or predictions about (patterns in) the training data. The resulting trained machine learning system may be referred to as a trained machine learning model or machine learning model for short. For example, Figure 3 A training phase 302 is shown, in which one or more machine learning systems 320 are trained on training data 310 to become one or more trained machine learning models 332. Then, during an inference phase 304, the trained machine learning model 332 may receive input data 330 and one or more inference / prediction requests 340 (possibly as part of the input data 330), and responsively provide one or more inferences and / or predictions 350 as output.

[0058] The machine learning system(s) 320 may include, but are not limited to, artificial neural networks (e.g., convolutional neural networks described herein using confidence learning techniques described herein, recurrent neural networks), Bayesian networks, hidden Markov models, Markov decision processes, logistic regression functions, support vector machines, suitable statistical machine learning algorithms, and / or heuristic machine learning systems. During the training phase 302, the machine learning system(s) 320 may be trained using training techniques such as, but not limited to, unsupervised, supervised, semi-supervised, reinforcement learning, transfer learning, incremental learning, and / or curriculum learning techniques by providing at least training data 310 as training input.

[0059] Unsupervised learning involves providing a portion (or all) of the training data 310 to the machine learning system(s) 320. The machine learning system(s) 320 may then determine one or more output inferences based on the portion (or all) of the provided training data 310. Supervised learning may involve providing a portion of the training data 310 to the machine learning system(s) 320, wherein the machine learning system(s) 320 determines one or more output inferences based on the portion of the provided training data 310, and accepting or correcting the output inference(s) based on correct results associated with the training data 310. In some examples, supervised learning of the machine learning system(s) 320 may be governed by a set of rules and / or a set of labels for the training inputs, and the set of rules and / or the set of labels may be used to correct the inferences of the machine learning system(s) 320.

[0060] Semi-supervised learning may involve having correct results for some but not all of the training data 310. During semi-supervised learning, supervised learning is used for the portion of the training data 310 that has the correct results, and unsupervised learning is used for the portion of the training data 310 that does not have the correct results. Reinforcement learning involves (multiple) machine learning systems 320, which may involve receiving a reward signal about a previous inference, where the reward signal may be a numerical value. During reinforcement learning, (multiple) machine learning systems 320 may output an inference and receive a reward signal in response, where the (multiple) machine learning system 320 is configured to attempt to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing the expected sum of the numerical values ​​provided by the reward signal over time.

[0061] Transfer learning techniques may involve trained (multiple) machine learning models 332 that are pre-trained on a set of data and additionally trained using training data 310. More specifically, (multiple) machine learning systems 320 may be pre-trained on data from one or more computing devices, and the resulting trained machine learning model is provided to computing device CD1, where CD1 is intended to execute the trained machine learning model during the inference phase 304. Then, during the training phase 302, the pre-trained machine learning model may be additionally trained using training data 310, where training data 310 may be obtained from kernel and non-kernel data of computing device CD1. This further training of (multiple) machine learning systems 320 and / or pre-trained trained machine learning models using training data 310 of CD1's data may be performed using supervised or unsupervised learning. Once (multiple) machine learning systems 320 and / or pre-trained machine learning models have been trained on at least training data 310, the training phase 302 may be complete. The resulting trained machine learning model may be utilized as at least one of the trained (multiple) machine learning models 332.

[0062] Incremental learning techniques may involve providing input data to the trained machine learning model(s) 332 (and possibly the machine learning system(s) 320) that is used to continue to expand the knowledge of the trained machine learning model(s) 332. Curriculum learning techniques. May involve machine learning system(s) 320 having training data arranged in a particular order, such as providing relatively easy training examples first and proceeding with progressively more difficult training examples, e.g., similar to a school curriculum or course of study. Other techniques for training machine learning system(s) 320 and / or trained machine learning model(s) 332 are also possible.

[0063] In some examples, after the training phase 302 is completed but before the inference phase 304 begins, the trained (multiple) machine learning model 332 can be provided to the computing device CD1, where the trained (multiple) machine learning model 332 does not yet exist; for example, after the training phase 302 is completed, the trained (multiple) machine learning model 332 can be downloaded to the computing device CD1.

[0064] For example, computing device CD2 storing trained (multiple) machine learning models 332 may provide computing device CD1 with trained (multiple) machine learning models 332 in one or more of the following ways: communicating a copy of the trained (multiple) machine learning models 332 to computing device CD1, making a copy of the trained (multiple) machine learning models 332 for computing device CD1, providing computing device CD1 with access to the trained (multiple) machine learning models 332, and / or otherwise providing a trained machine learning system for computing device CD1. In some examples, the trained (multiple) machine learning models 332 may be used by computing device CD1 immediately after being provided by computing device CD2. In some examples, after providing the trained (multiple) machine learning models 332 to computing device CD1, the trained (multiple) machine learning models 332 may be installed and / or otherwise prepared for use before computing device CD1 can use the trained (multiple) machine learning models 332.

[0065] During the inference phase 304, the trained (multiple) machine learning model 332 can receive input data 330 and generate and output corresponding (multiple) inferences and / or (multiple) predictions 350 about the input data 330. Therefore, the input data 330 can be used as an input to the trained (multiple) machine learning model 332 to provide corresponding (multiple) inferences and / or (multiple) predictions 350 to the core components and non-core components. For example, the trained (multiple) machine learning model 332 can generate (multiple) inferences and / or (multiple) predictions 350 in response to (multiple) inference / prediction requests 340. In some examples, the trained (multiple) machine learning model 332 can be executed by part of other software. For example, the trained (multiple) machine learning model 332 can be executed by an inference or prediction daemon to provide inferences and / or predictions at any time on demand. The input data 330 may include data from the computing device CD1 executing the trained (multiple) machine learning model 332 and / or input data from one or more computing devices other than CD1.

[0066] In some examples, input data 330 may include a collection of images provided by one or more sources. The collection of images may include images of objects (such as faces, where the images of faces are taken under different lighting conditions), images of multiple objects, images residing on computing device CD1, and / or other images. In a specific example, the collection of images may include one or more dual-pixel images. Other types of input data are also possible.

[0067] Inference(s) and / or prediction(s) 350 may include output images, depth maps, numerical values, and / or other output data produced by trained machine learning model(s) 332 operating on input data 330 (and training data 310). In some examples, trained machine learning model(s) 332 may use output inference(s) and / or prediction(s) 350 as input feedback 360. Trained machine learning model(s) 332 may also rely on past inferences as input for generating new inferences.

[0068] In some examples, the machine learning system(s) 320 and / or the trained machine learning model(s) 332 may be executed and / or accelerated using one or more computer processors and / or terminal device coprocessors. The terminal device coprocessor may include, but is not limited to, one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), one or more digital signal processors (DSPs), and / or one or more application specific integrated circuit chips (ASICs). Such a terminal device coprocessor may accelerate the training of the machine learning system(s) 320 and / or the generation of inferences and / or predictions 350 by the trained machine learning model(s) 332. In some examples, the trained machine learning model(s) 332 may be trained, resident, and executed to provide inferences and / or predictions 350 on a particular computing device, and / or may otherwise be inferred for a particular computing device.

[0069] In some examples, a computing device CD-SOLO may include trained machine learning model(s) 332, possibly after training of machine learning system(s) 320 was performed on computing device CD_SOLO. Computing device CD_SOLO may then receive inference / prediction request(s) 340 to provide inference(s) and / or prediction(s) 350 and may operate on input data 330 using trained machine learning model(s) 332 to responsively provide inference(s) and / or prediction(s) 350, where inference(s) and / or prediction(s) 350 may be provided using a user interface and / or display, as one or more electronic communications, as one or more printed documents, etc.

[0070] In some examples, two or more computing devices CD_CLI and CD_SRV may be used to provide inference(s) and / or prediction(s) 350. For example, a first computing device CD_CLI may generate and send inference / prediction request(s) 340 to a second computing device CD_SRV. Upon receiving the inference / prediction request(s) 340 from CD_CLI, CD_SRV may operate on input data 330 using the trained machine learning model(s) 332, possibly after training the machine learning system(s) 320, and responsively determine inference(s) and / or prediction(s) 350. After determining the inference(s) and / or prediction(s) 350, CD-SRV responds to the request from CD_CLI by providing the inference(s) and / or prediction(s) 350 to CD_CLI.

[0071] Using binocular stereo vision technology, the depth of the surface of the object captured in the dual pixel image can be estimated based on the difference between the pixels in the left image 210 and the pixels in the right image 220. The binocular stereo vision technology estimates the depth of the point in the dual pixel image 200 based on the degree to which the point appears to move between the left image 210 and the right image 220. However, because the scene points hardly move between the left image 210 and the right image 220, it may be a challenge to find these correspondences in the dual pixel image 200. In addition, the binocular stereo vision technology may involve overcoming the aperture problem. That is, viewing the scene through a small aperture may make it difficult (if not impossible) to find the correspondence of the line parallel to the stereo baseline (i.e., the line connecting the two cameras). In other words, when viewing the vertical lines in the dual pixel image 200, any proposed shift of these lines in one view relative to other views looks roughly the same. In addition, the binocular stereo vision technology does not take into account optical effects, such as the point spread function of the lens, and ignores semantic clues that can be used to estimate depth.

[0072] Binocular stereo vision can be improved based on the observation that parallax cues are only one of many depth cues present in an image, including semantic cues and defocus cues. In addition, because humans learn to understand the rough size of everyday objects (such as people and buildings), they can accurately discern how far things are even when viewing images on a flat screen. Because a person in a selfie occupies more pixels than background objects (such as buildings), a person viewing the selfie can infer that the person is closer to the camera than the building.

[0073] Designing a handcrafted algorithm to estimate the depth of an object captured in an image (such as a dual-pixel image) using a combination of parallax cues, semantic cues, and possibly other cues can be difficult. However, a machine learning system can be trained and utilized to estimate the depth of an object captured in a dual-pixel image using semantic, traditional, and possibly other cues. For example, a convolutional neural network can be utilized and trained to receive a dual-pixel image as input and then generate a depth map that estimates the depth of an object captured in the dual-pixel image.

[0074] Figure 4 4 is a schematic diagram of a block diagram 400 including a system 402 for partially blurring an image according to an example embodiment. The system 402 may receive an original dual pixel image 410 as an input. The original dual pixel image 410 may be provided to a neural network 420, which may responsively generate a depth map 430 for the original dual pixel image 410. The original dual pixel 410 and the depth map 430 may be provided to a depth rendering software 440, which may use the depth map 430 to selectively blur the original dual pixel image 410; for example, by blurring background objects and unblurring foreground objects. That is, the depth rendering software 440 may generate a blurred output image 450 by selectively blurring the original dual pixel image 410 using the depth map 430.

[0075] The neural network 420 may be the one described above. Figure 3 4. An example of a machine learning system(s) 320 of a system 300 discussed in the context of FIG. 4. After training, the trained version of the neural network 420 can be an example of a trained machine learning model(s) 332. In this specific example, the input data 330 of the system 300 can include an original dual-pixel image 410, the inference / prediction request(s) 340 of the system 300 can include a request for a depth map of the original dual-pixel image 410, which can be a stand-alone request or part of a request to partially blur the original dual-pixel image 410; and the inference(s) and / or prediction(s) 350 can include a depth map 430 for the original dual-pixel image 410. The depth map 430 can then be stored, transmitted, and / or otherwise provided and / or can be used by the depth rendering software 440 to produce an output image 450.

[0076] Figure 4 Also shown is dual pixel image 460, which depicts a person as a foreground object 464 in front of a background that includes regions 462 and 466. In this example, dual pixel image 460 is provided as input to neural network 420; e.g., an instance of original dual pixel image 410.

[0077] Neural network 420 responsively generates a depth map 470 for dual pixel image 460. Figure 4 In the illustration of depth map 470 in FIG. 4 , lighter-colored pixels represent objects in the environment that are relatively close to / closer to the camera used to capture dual-pixel image 460 in the environment, and darker-colored pixels represent objects that are relatively far from the camera used to capture dual-pixel image 460 in the environment. For example, foreground object 474 in depth map 470 is shown using lighter-colored pixels, indicating that foreground object 464 of dual-pixel image 460 is relatively close to the camera used to capture dual-pixel image 460. As another example, regions 472 and 476 of depth map 470 are both shown using darker-colored pixels, indicating that regions 462 and 466 of dual-pixel image 460 are relatively far from the camera used to capture dual-pixel image 460. In addition, region 472 is shown using darker-colored pixels than the pixels used to show region 476, indicating that the object depicted in region 462 of dual-pixel image 460 is relatively far from the camera used to capture dual-pixel image 460 compared to the object depicted in region 472 of dual-pixel image 460.

[0078] The example continues by providing dual pixel image 460 and depth map 470 to depth rendering software 440, requesting that only background objects of the dual pixel image be blurred. Depth rendering software 440 then uses depth map 470 to generate a blurred output image 480 by blurring pixels in dual pixel image 460 whose corresponding depth values ​​in depth map 470 indicate that those pixels depict background objects; e.g., depth rendering software 440 blurs pixels in dual pixel image 460 whose depth values ​​in depth map 470 are in Figure 4 Darker pixels are used to depict the image.

[0079] The resulting blurred output image 480 shows a foreground object 484 (corresponding to foreground object 464) with little or no blur, and shows a region 482 (corresponding to region 462) having a relatively large amount of blur compared to a region 486 (corresponding to region 466). The relatively large amount of blur in region 482 relative to region 486 corresponds to the depth values ​​in the depth map 470 generated by the neural network 420, which determined that the object in region 462 of the dual-pixel image 460 is farther away from the camera used to capture the dual-pixel image 460 than the object in region 466 of the dual-pixel image 460. That is, the depth map 470 generated by the neural network 420 enables non-uniform blurring of background objects by the depth rendering software 440. In addition, the depth map 470 generated by the neural network 420 has fewer errors than conventional techniques (e.g., uniform blurring of background objects), and thus enables the depth rendering software 440 to provide a more natural and pleasing depth-related blur to the background objects in the blurred output image 480.

[0080] During training, neural network 420 may receive training images including images with dual-pixel depth data to generate an estimated depth map, such as depth map 430. For example, neural network 420 may be trained on a relatively large dataset (e.g., 50,000 or more) of training images. The dataset of training images may include dual-pixel images obtained by a device using one camera or by a multi-camera device, such as the multi-camera devices described herein.

[0081] In some examples, the training images used to train the neural network 420 may include one or more dual-pixel images of at least a foreground object, where the dual-pixel image may include a plurality of dual pixels. The dual pixels of the plurality of dual pixels may include a left pixel and a right pixel, each representing light incident on a single dual-pixel element used to capture the dual-pixel image (such as the left pixel 124a and the right pixel 124b generated by the dual-pixel element 134c discussed above).

[0082] In other examples, the dual pixel element can provide dual pixels that can have other orientations besides left pixel and right pixel; for example, some dual pixel sensors can provide upper and lower pixels instead of left pixel and right pixel. In other examples, the dual pixel element can provide more than two sub-pixels for each image pixel.

[0083] The neural network 420 may be trained to determine a depth map associated with the dual pixel image(s). The neural network 420 may be provided after training, such as described above. Figure 3 In some examples, the training images used to train the neural network 420 may include one or more dual-pixel images, which may include images with foreground objects having one or more specified object types; for example, images of people, images of faces, images of buildings, etc.

[0084] The estimated depth map generated by the neural network 420 can be evaluated (e.g., during training) using one or more loss functions. The estimated depth data and / or the estimated depth map generated by the neural network 420 with respect to the ground truth depth data can be evaluated using the loss function(s). The loss function(s) can be invariant to depth ambiguity in the dual pixel data, and thus enable the neural network to learn from the ground truth depth data. As an example of depth ambiguity, two images of different scenes with different depths can produce the same set of dual pixel images if the focal length or lens characteristics (e.g., lens tilt) also change between captures. These lens characteristics can vary from device to device. This means that a given dual pixel pair can correspond to a range of different depth maps; that is, the depth can only be predicted up to unknown scales and offsets. The loss functions described herein can be invariant to scale and offset values, so as long as the prediction is within the range of possible depth maps for a given dual pixel pair, the neural network is not penalized. The depth map provided by the trained neural network 420 can preserve the relative ordering of objects in the scene.

[0085] The neural network 420 may then be evaluated during training using one or more of equations (1), (2), (3), and (4), described below as loss functions; that is, one or more of equations (1), (2), (3), and (4) may be used to evaluate the correctness of a depth map generated by the neural network 420 compared to ground truth depth data.

[0086] To describe these loss functions, let f(I) be a function representing a predicted depth map generated by the neural network 420 when trained to operate on a dual-pixel data input I. Let G represent a ground truth depth map associated with the dual-pixel data input I. For example, the ground truth depth map G can be a depth map generated using the multi-camera device described herein. In order to properly train the neural network 420, the predicted depth map f(I) should be as close as possible to the ground truth depth map G. This requirement can be represented by the following equation (1), which can be implemented as a loss function to be minimized during the training of the neural network 420:

[0087]

[0088] Equation (1) indicates that the difference between the prediction f(I) and the ground truth G is taken at each pixel of the input I, and the difference is squared and summed over the entire input I.

[0089] However, equation (1) does not take into account the fact that the dual-pixel input data I depends not only on the depth of the scene but also on additional factors, such as, but not limited to, the focal length of the camera lens used to capture I, the point spread function of the camera lens, and / or the camera aperture. Due to these additional factors, there may be two different ground truth values ​​G and G' that produce the same dual-pixel data I. Since the neural network 420 is deterministic and only takes one input I, the neural network 420 cannot produce both G and G' using the same input I. In fact, if the training data for the neural network 420 has both (I, G) and (I, G') pairs, using equation (1) as the loss function will cause NN1 to predict a depth map that averages G and G', which will be incorrect.

[0090] Another loss function of the neural network 420 can be determined by assuming a thin lens model, under which G and G' are related by an affine mapping as shown in equation (2).

[0091] G′=aG+b (2)

[0092] where a and b are scalar values ​​that depend on the additional factors mentioned above; for example, focal length, point spread function, and / or camera aperture.

[0093] Equation (3) expresses the loss function based on the affine mapping of Equation (2).

[0094]

[0095] Similar to equation (1), equation (3) calculates the per-pixel difference squared and summed over the input I. However, in equation (3), the predicted depth map f(I) is modified by the affine mapping of equation (2), where a and b are scalar values ​​selected to give the minimum per-pixel difference squared and summed over the input I. That is, because the neural network 420 cannot distinguish between G and G′ given the same dual-pixel data I, equation (3) assumes that the neural network 420 can only predict the depth up to an unknown affine transformation. In the case where the neural network 420 is trained on (I, G) and (I, G′), the loss function based on equation (3) can be set to zero for both (I, G) and (I, G′) with the same prediction f(I). Although there are limitations to using equation (3); for example, the exact depth value cannot be determined, a neural network trained using a loss function based on equation (3) can also provide useful depth information; for example, the relative order of depth is preserved. Likewise, since the minimization problem represented by equation (3) has a closed form solution, equation (3) can be computed efficiently.

[0096] Furthermore, Equation (3) can be modified to incorporate the per-pixel confidence for the ground truth W, resulting in Equation (4):

[0097]

[0098] Among them, in equation (4) Represents per-element multiplication.

[0099] Figure 5 A neural network 420 of the system 402 is shown according to an example embodiment. The neural network 420 may be a convolutional encoder-decoder neural network. Figure 5 , as shown in the legend shown in the lower left, the encoder layers of the neural network 420 are shown in white, the decoder layers are shown in gray, and the two-dimensional (2D) convolution (Conv2D) layers are shown as filled with diagonal lines. In some examples, part or all of the neural network 420 can be implemented using specialized hardware and / or software designed for neural networks; for example, specialized neural network circuits, software libraries with software supporting neural networks.

[0100] As described above, neural network 420 may receive raw dual pixel image 410 as input and responsively generate depth map 430 for raw dual pixel image 410 as output. Figure 5 In the context of , the original dual pixel image 410 may be input as a three-dimensional (3D) array of pixels (the 3D array of pixels is in Figure 5 The upper left corner of the image is represented as "32H×32W×D" pixels), and the value of parameter N' is set equal to "4D". In the context of neural network 420, depth map 430 is a 3D array of depth values, which is a 3D array of depth values ​​in Figure 5 The upper right portion of FIG. 4 is represented as “16H×16W×4D” for the depth map 430 .

[0101] In the description of neural network 420, H is a positive integer associated with the number of horizontal pixels (or pixel rows) in original dual-pixel image 410, W is a positive integer associated with the number of vertical pixels (or pixel columns) in original dual-pixel image 410, D is a positive integer representing the number of pixels used for a pixel depicted in original dual-pixel image 410, and N' and N" are both positive integer parameters. In one specific example, H=32, W=42, and D=2, i.e., original dual-pixel image 410 is an array of 1024 rows×1344 columns×2 pixels (i.e., dual pixels), and the corresponding depth map 430 has 512 rows×672 columns×8 depth values. In some examples, neural network 420 is trained only on dual-pixel images, so the input to neural network 420 (such as original dual-pixel image 410) includes images in which D=2. In other examples, other values ​​of H, W, and / or D are possible.

[0102] During operation of the neural network 420, the original dual pixel image 410 can be received at the Conv2D layer 510. The Conv2D layer 510 can perform a 2D convolution on the original dual pixel image 410 to generate a 16H×16W×4D value array 570a. The 16H×16W×4D value array 570a can be provided as an input to a stride 1 encoder function (S1EF) 512. Figure 5 Indicates that S1EF 512 is called with an input parameter of 4D and a parameter N' equal to 4D to perform a stride 1 encoder operation on the 16H x 16W x 4D array of values ​​570a.

[0103] Fig. 6A A stride 1 encoder function 600 is shown that can be used by the neural network 420 to perform a stride 1 encoder operation according to an example embodiment. For example, the stride 1 encoder function 600 can be used to implement some or all of S1EF 512, S1EF 522, S1EF 524, S1EF 532, S1EF 534, S1EF 542, S1EF 544, S1EF 552, and S1EF 554.

[0104] The stride 1 encoder function 600 may be called with an input parameter “Int” to perform a stride 1 encoder operation on an input H×W×N value array 630. Upon calling, the stride 1 encoder function 600 may provide the H×W×N value array 630 to both the Conv2D layer 620 and the additive + parameter rectifier linear unit (APReLU) 610. The Conv2D layer 620 may perform a 2D convolution on the H×W×N value array 630 with stride 1 to generate a H×W×Int value array 632. The H×W×Int value array 632 may be provided to a separable 2D convolution layer 622, which may perform a separable 2D convolution on the H×W×Int value array 632 to generate a H×W×Int value array 634. The H×W×Int value array 634 may be provided to the Conv2D layer 624, which may perform a 2D convolution on the H×W×Int value array 634 to generate an H×W×N′ value array 636. The APReLU 610 may then perform addition and parametric linear unit operations on both the H×W×N value array 630 and the H×W×N′ value array 636 to generate an H×W×N′ value array 640. The H×W×N′ value array 640 may then be output by the stride 1 encoder function 600.

[0105] Back to Figure 5 , the output of S1EF 512 may be a 16H×16W×4D array of values ​​570b, which may be provided to both the stride 2 encoder function (S2EF) 520 and the decoder function (DF) 514 (ie, as a skip connection). Figure 5It is shown that S2EF 520 may be called with an input parameter of 8D and a parameter N' equal to 16D to perform a stride 2 encoder operation on a 16H x 16W x 4D array of values ​​570b.

[0106] Figure 6B A stride 2 encoder function S2EF 650 is shown that can be used by the neural network 420 to perform stride 2 encoder operations according to an example embodiment. For example, the stride 2 encoder function 650 can be used to implement some or all of S2EF 520, S2EF 530, S2EF 540, and S2EF 550.

[0107] The stride 2 encoder function 650 may be called with one input parameter "Int" to operate on an input 2H×2W×N value array 680. When called, the stride 2 encoder function 650 may provide the 2H×2W×N value array 680 to both a max pooling (MaxPool) layer 660 and a Conv2D layer 670. The MaxPool layer 660 may perform a max pooling operation on the 2H×2W×N value array 680 to generate a H×W×N value array 682. The Conv2D layer 620 may perform a 2D convolution with stride 2 on the H×W×N value array 680 to generate a H×W×Int value array 684. The H×W×Int value array 684 may be provided to a separable 2D convolution layer 672, which may perform a separable 2D convolution on the H×W×Int value array 684 to generate a H×W×Int value array 686. The H×W×Int value array 686 may then be provided to the Conv2D layer 674, which may perform a 2D convolution on the H×W×Int value array 686 to generate an H×W×N′ value array 688. Both the H×W×N value array 682 and the H×W×N′ value array 688 may be provided to the APReLU 662. The APReLU 662 may perform addition and parametric linear unit operations on both the H×W×N value array 682 and the H×W×N′ value array 688 to generate an H×W×N′ value array 690. The H×W×N′ value array 690 may then be output by the stride 2 encoder function 650.

[0108] Back to Figure 5, the output of S2EF 520 may be an 8H×8W×16D value array 572a. S1EF 522 may then be called with an input parameter of 8D and a parameter N′ equal to 16D to perform a stride 1 encoder operation on the value array 572a to generate an 8H×8W×16D value array 572b. S1EF 524 may be called with an input parameter of 8D and a parameter N′ equal to 16D to perform a stride 1 encoder operation on the value array 572b to generate an 8H×8W×16D value array 572c. The value array 572c may be provided to both S2EF 530 and DF 526 (i.e., as a skip connection).

[0109] S2EF 530 may be called with an input parameter of 8D and a parameter N' equal to 32D to perform a stride 2 encoder operation on an 8H×8W×16D value array 572c. The output of S2EF 530 is a 4H×4W×32D value array 574a. S1EF 532 may then be called with an input parameter of 8D and a parameter N' equal to 32D to perform a stride 1 encoder operation on the value array 574a to generate a 4H×4W×32D value array 574b. S1EF 534 may be called with an input parameter of 8D and a parameter N' equal to 32D to perform a stride 1 encoder operation on the value array 574b to generate a 4H×4W×32D value array 574c. The value array 574c may be provided to S2EF 540 and DF 536 (i.e., as a skip connection).

[0110] S2EF 540 may be called with an input parameter of 16D and a parameter N' equal to 64D to perform a stride 2 encoder operation on a 4H×4W×32D value array 574c. The output of S2EF 540 may be a 2H×2W×64D value array 576a. S1EF 542 may then be called with an input parameter of 16D and a parameter N' equal to 64D to perform a stride 1 encoder operation on the value array 576a to generate a 2H×2W×64D value array 576b. S1EF 544 may be called with an input parameter of 16D and a parameter N' equal to 64D to perform a stride 1 encoder operation on the value array 576b to generate a 2H×2W×64D value array 576c. The value array 576c may be provided to S2EF 550 and DF 546 (i.e., as a skip connection).

[0111] S2EF 550 may be called with an input parameter of 16D and parameter N' equal to 64D to perform a stride 2 encoder operation on 2H×2W×64D value array 576c. The output of S2EF 550 may be an H×W×64D value array 578a. S1EF 552 may then be called with an input parameter of 16D and parameter N' equal to 64D to perform a stride 1 encoder operation on value array 578a to generate H×W×64D value array 578b. S1EF 554 may be called with an input parameter of 16D and parameter N' equal to 64D to perform a stride 1 encoder operation on value array 578b to generate H×W×64D value array 578c.

[0112] The value array 578c may be provided to the DF 546. Figure 5 Indicates that DF 546 may be called with an input parameter of 16D, a parameter N' equal to 64D, and a parameter N" equal to 64D to perform decoder operations on value array 576c and value array 578c.

[0113] Figure 7 A decoder function 700 is shown that can be used by neural network 420 to perform decoder operations according to an example embodiment. For example, decoder function 700 can be used to implement some or all of DF 514, DF 526, DF 536, and DF 546.

[0114] The decoder function 700 may be called with one input parameter "Int" to perform a decoder on two input value arrays (H×W×N value array 730 and H / 2×W / 2×N' value array 734). Upon calling, the decoder function 700 may provide the H×W×N value array 730 to the separable 2D convolution layer 710, which may perform a separable 2D convolution on the value array 730 to generate an H×W×Int value array 732. The decoder function 700 may provide the H / 2×W / 2×N' value array 734 to the transposed 2D convolution layer 720, which may perform a transposed 2D convolution on the value array 734 to generate an H×W×Int value array 736. The separable 2D convolution layer 722 may perform a separable 2D convolution on the H×W×Int value array 736 to generate an H×W×Int value array 738. APReLU 610 may perform addition and parametric linear unit operations on both H×W×Int value array 732 and H×W×Int value array 738 to generate H×W×N″ value array 740 . H×W×N″ value array 740 may then be output by decoder function 700 .

[0115] Back to Figure 5, the output of DF 546 can be a 2W×2W×64D value array 576d, which can be provided to DF 536. DF 536 can be called with an input parameter of 8D, a parameter N' equal to 32D, and a parameter N" equal to 32D to perform decoder operations on value array 574c and value array 576d. The output of DF 536 can be a 4W×4W×32D value array 574d, which can be provided to DF 526.

[0116] DF 526 may be called with an input parameter of 8D, a parameter N′ equal to 16D, and a parameter N″ equal to 16D to perform decoder operations on value array 572 c and value array 574 d. The output of DF 526 may be 8W×8W×16D value array 572 d, which may be provided to DF 514.

[0117] DF 514 may be called with an input parameter of 4D, a parameter N′ equal to 4D, and a parameter N″ equal to 4D to perform decoder operations on value array 570 b and value array 572 d . The output of DF 514 may be a 16W×16W×4D value array 570 c , which may be provided to Conv2D layer 516 . Conv2D layer 516 may perform a 2D convolution of the 16W×16W×4D value array 570 c to generate a 16W×16W×4D value array that constitutes depth map 430 . Depth map 430 may then be output from neural network 420 ; e.g., for use by depth rendering software 440 , as discussed above.

[0118] Example multi-camera rig

[0119] Fig. 8A A multi-camera device 800 is shown that can obtain multiple dual-pixel images of an object at different viewpoints; for example, to provide ground truth depth data to train neural network 420. Multi-camera device 800 can include multiple mobile computing devices, such as mobile computing device 810 and mobile computing device 820, each of which is equipped with a camera capable of capturing dual-pixel images, such as mobile camera 812 and mobile camera 822. The resulting dual-pixel images that can be used as training data for neural network 420 are similar to dual-pixel images captured using the same or similar type of camera on other mobile computing devices (e.g., a user's mobile computing device).

[0120] The mobile computing devices in the multi-camera device 800 can be physically joined together as part of a single structure using various methods (such as screws or adhesives). Some or all of the mobile computing devices in the multi-camera device 800 can be spaced apart in a systematic manner to ensure that each mobile computing device captures a different viewpoint of an object. For example, the multi-camera device 800 can be arranged so that the four cameras are spaced apart around the central camera relative to the XY plane (i.e., the cameras are spaced apart in the up / down / left / right directions, and each camera contains a small baseline to the central camera). Thus, the mobile computing device 810 can capture the left viewpoint, and the mobile computing device 820 can capture the right viewpoint. As described above, because there is parallax in multiple directions, the use of different viewpoints eliminates the aperture problem. Capturing images from (slightly) different viewpoints also ensures that points in the image can be visible in at least one other image, resulting in fewer points without corresponding relationships. When multiple dual-pixel images of an object are taken by the multi-camera device 800, structure from motion and multi-view stereo techniques can be used to calculate a high-quality depth map from the captured dual-pixel images. Therefore, the use of the multi-camera device 800 can obtain a high-quality depth map, which can be used as a ground truth depth map for training the neural network 420.

[0121] Acquiring high-quality depth maps using structure from motion and multi-view stereo techniques may involve using a multi-camera device 800 to simultaneously capture images of an object from each of the viewpoints (i.e., each of the cameras on the multi-camera device 800). Without tight temporal synchronization between the cameras, objects in the images may move and / or the lighting may vary between the captured images, which may degrade the quality of the resulting ground truth depth map. For example, mobile camera 812 may capture a first viewpoint of object A at time T, and mobile camera 822 may capture a second viewpoint of object A at time T+100ms, based on camera latency. Even with a latency as small as 100ms, capturing the second viewpoint at T+100ms may present a completely different image context for object A than at time T.

[0122] One approach to taking synchronized photos with multiple cameras is to use specialized equipment configured with fixed-position wired hardware triggers. However, this approach may require specialized wiring and other heavy equipment that is not easily moved around, thereby limiting the types of objects that can be captured by the multi-camera device 800 and degrading the quality of the data set. More importantly, in order for the neural network 420 to effectively predict depth maps, it is useful for the multi-camera device 800 to collect training data from mobile computing devices that represent the devices that will use the neural network 420. For example, the mobile computing devices 810 and 820 of the multi-camera device 800 can be mobile computing devices that typically utilize the neural network 420 (e.g., off-the-shelf smartphones).

[0123] However, approaches utilizing representative mobile computing devices may face hardware / software delays that are difficult to address on each of the devices. For example, a synchronization approach using the audio port of a mobile computing device (such as a selfie stick trigger) may face variable delays of around 100ms. Other approaches include using audio waveforms, Bluetooth TM Methods of communication as well as GPS can show similarly high variable delays.

[0124] A method for capturing synchronized data using a representative mobile computing device without additional hardware and without hardware / software latency is described herein. For example, the method may include a multi-camera architecture to account for network latency between each of the mobile computing devices and to account for operating system latency within each mobile computing device.

[0125] Figure 8B An arrangement of mobile computing devices in a multi-camera device 800 is shown to facilitate a multi-camera architecture 830. The multi-camera architecture 830 may include a single master device 832 and multiple client devices 834, such as client device 834A, client device 834B, and client device 834C. Rather than interacting with each device on the multi-camera device 800, the master device 832 may broadcast commands to the client devices 834 to obtain dual pixel images.

[0126] The master device 832 may be designated from among the devices of the multi-camera device 800 based on the physical location of each of the devices on the multi-camera device 800 (e.g., the device at the top left of the multi-camera device 800 may be designated as the master device) and / or by user selection. For example, the user selection may include designating the master device by interacting with an interface of the device on the multi-camera device 800. When designated, the master device 832 may be configured to act as a WiFi connection to the client device 834. TM Alternatively, the multi-camera device 800 may be connected to a separate WiFi hotspot. TM router connection, which can automatically designate a master device 832. In some examples, a master device for the multi-camera architecture 830 is selected only once per image collection session.

[0127] To initiate communication, the host device 832 and the client device 834 may use, for example, the Transmission Control Protocol (TCP) or WiFi. TMThe communication link between the master device 832 and each of the client devices 834 is established using the standard network protocol of Wi-Fi Protected Access 2 (WPA2). Once connected, the master device 832 and the client device 834 can communicate through a modified Network Time Protocol (NTP) synchronization to estimate network and operating system delays. These delays can determine the correct time offset to be added by the client device 834 in order to synchronize with the clock of the master device 832.

[0128] Fig. 9 9 is a message flow diagram illustrating a protocol for synchronizing the clocks of a multi-camera device 830 according to an example embodiment. The protocol begins at block 902. At block 902, a master device 832 is selected and a communication mechanism (e.g., WiFi) is established. TM Hotspot, WiFi TM router). In block 906, the master device 832 determines the time, records the time in timestamp T1, and sends timestamp T1 to the client device 900, which arrives at the client device 900 at the time recorded in timestamp T2. Therefore, the delay from the master device 832 to the client device 900 can be calculated as the difference in time recorded in timestamps T1 and T2; for example, T2-T1. Let O represent the offset between the master device 832 and the client device 900 and let D represent the transmission time of the message between the master device 832 and the client device 900, T2-T1 can be determined as O+D.

[0129] At 908, the client device 900 determines the timestamp T3 and sends the timestamp T3 to the master device 832, which arrives at the master device 832 at the timestamp T4. At 910, the master device 824 sends the timestamp T4 to the client device 900. Therefore, the delay from the client device 834 to the master device 832 can be calculated as T4-T3, which can be determined as O+D. At box 912, the client device 900 determines the correct offset value O using the above relationship. The client device 900 then uses the offset value O to synchronize its clock with the master device 832. At 914, the client device 900 transmits the offset value O to the master device 832.

[0130] At block 916, a broadcast trigger is provided by the master device 832 (e.g., in response to a button click to provide a broadcast trigger to capture a dual pixel image). At 918, the master device 832 broadcasts a future trigger time T to the client device 900 (e.g., to perform a trigger 500 ms in the future). The client device 900 then waits until time T to invoke the trigger.

[0131] It is worth noting that client device 900 is for the purpose of example. Client device 900 can be embodied by any device of client device 834. Moreover, frame 902-914 can be implemented simultaneously on all client devices 834, so that master device 832 can receive the calculated offset of all client devices 834. Therefore, as a result of receiving the calculated offset from all client devices 834, master device 832 can set future trigger time T to be greater than maximum client offset, so as to allow network delay. As an example calculation, it is assumed that maximum client offset is determined to be 550ms. Thus, the value of future trigger time T can be set to T>550ms. Therefore, since the clock of client device 834 is synchronized with master device 832, all triggers can be executed at the same future time T, and all client devices 834 can capture scenes at the same time, as well as a group of synchronized images of the scenes.

[0132] In some instances, the devices of the multi-camera device 800 may use hardware timestamping; however, hardware timestamping may require the use of specialized hardware and operating systems on the client devices. Therefore, in order to maintain representative devices on the multi-camera device 800, in some examples, the above process may utilize software timestamping. For example, the client device 834 may periodically send heartbeat messages to the master device 832 to update the offset as clock drift and network latency change as part of the software timestamping.

[0133] However, even if Fig. 9 8 shows clock synchronization in a multi-camera device 800, a variable delay may exist between a request to capture a dual-pixel image and the camera's response to the request. The delay may occur due to the operating system of the client device; for example, when processing concurrent tasks.

[0134] Repeated requests can be used to address operating system delays. With repeated requests, the client device's operating system can transmit a set of request parameters (e.g., parameters for requesting a dual-pixel image) to the camera at one time, and the camera's hardware can save these parameters. The camera can then periodically capture dual-pixel images with a small change in capture frequency until the request is canceled. This prevents the operating system from always delaying each new dual-pixel image request.

[0135] In some examples, there may be variable latency introduced by the operating system when the request parameters are first delivered to the camera's hardware. However, this variable latency can be addressed by pushing the periodic image sequence to align with a specific phase through sampling: when the phase of the image sequence is greater than a desired threshold, issue another repeat request and try again.

[0136] Fig.101 is a message flow diagram illustrating a process of using a repeat request by a multi-camera device 830 according to an example embodiment. The process starts at 1004. At 1004, the master device 832 of the multi-camera device 830 may issue a trigger request to the operating system of the client device to capture an image. The trigger request may include a desired exposure time E, a desired phase of the image sequence D, a desired synchronization accuracy P, and / or other parameters. At block 1006, the operating system 1000 may generate a repeat request and send the repeat request to the camera 1002. For example, the repeat request may include the desired exposure time E, and possibly other parameters. At block 1008, the repeat request may be delayed by the operating system 1000 (e.g., when the operating system 1000 is performing other tasks). At step 1010, the repeat request may be received by the camera 1002, which acts on the repeat request and begins to capture images periodically. At 1010, the camera 1002 may send back a first image timestamp T_1 (indicating the time when the first image in the sequence was captured) to the operating system 1000. At block 1012, the operating system 1000 may calculate a sequence phase to determine a relative offset from which images are periodically captured. For example, an exposure time of E = 50 ms, capturing images at A_1 50 ms, A_2 = 100 ms, A_3 = 150 ms, may have a sequence phase of S = 0. In another example, an exposure time of E = 50 ms, capturing images at B_1 75 ms, B_2 = 125 ms, B_3 = 175 ms, may have a sequence phase of S = 25. Thus, assuming that an example image is captured at time A with an exposure time of E, the sequence phase S may be determined as S = A - E*k, where k is an integer for which |AE*k| is minimal.

[0137] As an example calculation of the sequence phase at block 1012, assume that the operating system 1000 issues a repeat request at time T_0=0, with an exposure time E=50 ms, a desired synchronization accuracy P=10 ms, and assume that the first time timestamp has a time T_1=75 ms (thus, the second image arrives at T_2=125 ms, and the third image arrives at T_3=175 ms). Therefore, the delay of the operating system 1000 can be calculated as T_1-E=75 ms-50 ms=25 ms. If the desired phase D=0 and the exposure time E=50 ms, then the repeat request can have a sequence phase of 75 ms-50 ms=25 ms, which is greater than the desired synchronization accuracy P=10 ms. That is, a suitable sequence phase will be in the range of DP to D+P.

[0138] At block 1014, upon determining that the sequence phase is greater than the desired accuracy, the operating system 1000 may re-initiate the repeat request with the camera 1002. At block 1016, the re-initiated repeat request may again be delayed by the operating system 1000. At block 1018, the repeat request may be received by the camera 1002, which acts upon the repeat request and begins capturing images periodically. The camera 1002 sends back to the operating system 1000 a first image timestamp X_1 (indicating the time when the first image in the new sequence was captured). At block 1020, the operating system calculates the sequence phase.

[0139] As an example calculation of the sequence phase at box 1020, assume that the operating system 1000 issues a repeat request at time X_0=345ms, where the exposure time E=50ms, the desired synchronization accuracy P=10ms, and assume that the first timestamp X_1=397ms (thus, the second image arrives at X_2=447ms, and the third image arrives at X_3=497ms). Therefore, the delay of the operating system 1000 can be calculated at X_0-E=397-345=52ms. If the desired phase D=0 and the exposure time E=50ms, then the repeat request can have a sequence phase of 397ms-400ms=-3ms. This sequence phase is less than the desired synchronization accuracy P=10ms (e.g., in the range of DP to D+P), which indicates that the sequence phases are fully aligned. Therefore, at box 1022, the operating system 1000 notifies the master device 832 that the phases are now aligned.

[0140] In some examples, Fig.10 The process may be adapted to temporarily interrupt a repetitive request with a high priority single request for a given exposure time E. For example, after a repetitive request with a calculated sequence phase of 25 ms, a single frame may be injected to delay the subsequent images in the sequence by approximately a predetermined amount of time, rather than re-initiating the repetitive request. Injection of additional single frames may continue until the repetitive request sequence phase reaches the desired accuracy.

[0141] Fig.11A message flow diagram of a process using a single frame request by a multi-camera device 830 according to an example embodiment is shown. The process starts at 1102. At 1102, the master device 832 of the multi-camera device 830 may issue a trigger request to the operating system 1000 of the client device to capture an image. The trigger request may include a desired exposure time E, a desired phase of the image sequence D, a desired synchronization accuracy P, and / or other parameters. At box 1104, the operating system 1000 may generate a repeat request and send the request to the camera 1002. For example, the repeat request may include the desired exposure time E, and possibly other parameters. At box 1106, the repeat request may be delayed by the operating system 1000 (e.g., when the operating system 1000 is performing other tasks). At 1108, the repeat request may be received by the camera 1002, which acts on the repeat request and begins to capture images periodically. The camera 1002 sends back a first image timestamp Y_1 (indicating the time when the first image in the sequence was captured) to the operating system 1000. At step 1110, operating system 1000 calculates the sequence phase.

[0142] To illustrate the calculation at 1110, for example, assume that the operating system 1000 issues a repeat request at time Y_0=0, where the exposure time E=50ms, the desired synchronization accuracy P=10ms, and assume that the first time timestamp Y_1=75ms (thus, the second image arrives at Y_2=125ms, and the third image arrives at Y_3=175ms). Therefore, the delay of the operating system 1000 can be calculated as Y_1-E=75ms-50ms=25ms. If the desired phase D=0 and the exposure time E=50ms, then the repeat request has a sequence phase of 75ms-50ms=25ms, which is greater than the desired synchronization accuracy P=10ms. That is, the appropriate sequence phase will be in the range of DP to D+P.

[0143] At block 1112, upon determining that the sequence phase is greater than the desired accuracy, the operating system 1000 may initiate a high priority request to capture a single frame to offset the sequence phase with a single frame. Based on the example at block 1110 described above, the high priority request may be a request to capture a single frame with an exposure time of 25ms+50ms=75ms. In this example, the 75ms single frame may be offset from the example sequence phase of 25 milliseconds at 1110 so that the new sequence phase is 0; for example, 75ms+25ms=100ms. When the exposure time E=50ms, the new sequence phase is now 0ms. At block 1114, the high priority request may be delayed by the operating system 1000. At 1116, the high priority request is received by the camera 1002, which may responsively capture a single frame to insert the single frame into the image sequence of the repeated request, thereby delaying the image sequence by 75ms. Once the single frame is completed, the camera 1002 sends back to the operating system 1000 the first image timestamp Z (indicating the time when the first image in the shifted sequence was captured). At step 1118, the operating system 1000 may calculate the sequence phase in a similar manner as described above. At step 1120, upon indicating that the sequence phases are sufficiently aligned, the operating system 1000 notifies the master device 832 that the phases are now aligned.

[0144] In some examples, operating system 1000 and camera 1002 can be embodied in any client device 834. Also, Fig.10 and Fig.11 Some or all of can be implemented on all client devices 834 simultaneously, so that the master device 832 can receive notification of the phase alignment of all client devices 834. Thus, once the master device 832 is notified that all clients 834 are phase aligned, the master device 832 can schedule a trigger as before, after which the client device 834 can send the captured image to its destination (e.g., saved to disk, transmitted over the Internet, sent to a downstream application, etc.). In addition, since the clock of the client device 836 can be synchronized with the master device 832 (e.g., Fig. 9 As described above), the phase target for each of the client devices 836 will be equal to the sum of the client device's clock offset and the desired exposure time E. Therefore, phase alignment is only necessary each time the user changes the desired exposure time E.

[0145] Sample Data Network

[0146] Fig.12A distributed computing architecture 1200 is depicted according to an example embodiment. The distributed computing architecture 1200 includes server devices 1208, 1210 configured to communicate with programmable devices 1204a, 1204b, 1204c, 1204d, 1204e via a network 1206. The network 1206 may correspond to a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a wireless wide area network (WWAN), a corporate intranet, the public Internet, or other types of networks configured to provide a communication path between networked computing devices. The network 1206 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public Internet.

[0147] Although Fig.12 Only five programmable devices are shown, and the distributed application architecture can serve dozens, hundreds, or thousands of programmable devices. Moreover, programmable devices 1204a, 1204b, 1204c, 1204d, 1204e (or other additional programmable devices) can be any computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mounted device (HMD), a network terminal, a mobile computing device, etc. In some examples, such as shown in programmable devices 1204a, 1204b, 1204c, 1204d, 1204e, the programmable device can be directly connected to the network 1206. In other examples, such as shown in programmable device 1204d, the programmable device can be indirectly connected to the network 1206 via an associated computing device (such as programmable device 1204c). In this example, programmable device 1204c can act as an associated computing device to pass electronic communications between programmable device 1204d and network 1206. In other examples, such as shown by programmable device 1204e, the computing device can be part of and / or inside a vehicle, such as a car, truck, bus, boat or ship, airplane, etc. Fig.12 In other examples not shown, programmable devices can be connected to network 1206 directly or indirectly.

[0148] Server devices 1208, 1210 can be configured to perform one or more services as requested by programmable devices 1204a-1204e. For example, server devices 1208 and / or 1210 can provide content to programmable devices 1204a-1204e. The content can include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. The content can include compressed and / or uncompressed content. The content can be encrypted and / or unencrypted. Other types of content are also possible.

[0149] As another example, server devices 1208 and / or 1210 may provide programmable devices 1204a-1204e with access to software for database, search, computing, graphics, audio, video, web / internet utilization, and / or other functions. Many other examples of server devices are possible.

[0150] Computing device architecture

[0151] Fig.13 is a block diagram of a computing device 1300 according to an example embodiment. Specifically, Fig.13 The computing device 1300 shown in the figure can be configured to perform at least one function of and / or related to the computing device 100, the camera 110, the dual pixel image 200, the system 300, 402 (including but not limited to the neural network 420), the stride 1 encoder function 600, the stride 2 encoder function 650, the decoder function 700, the multi-camera device 800, 830, the distributed computing architecture 1200 and / or the method 1500, 1600.

[0152] The computing device 1300 may include a user interface module 1301, a network communication module 1302, one or more processors 1303, data storage 1304, one or more cameras 1318, one or more sensors 1320, and a power system 1322, all of which may be linked together via a system bus, network, or other connection mechanism 1305.

[0153] User interface module 1301 can be used to send data to external user input / output device and / or receive data from external user input / output device.For example, user interface module 1301 can be configured to send data to user input device and / or receive data from user input device, and this user input device is such as touch screen, computer mouse, keyboard, auxiliary keyboard, touch pad, tracking ball, joystick, speech recognition module, and / or other similar equipment.User interface module 1301 can also be configured to provide output to user display device, and this user display device is such as one or more screens (comprising touch screen), cathode ray tube (CRT), liquid crystal display, light emitting diode (LED), display, printer, electric bulb, and / or other similar equipment using digital light processing (DLP) technology.User interface module 1301 can also be configured to utilize the equipment such as loudspeaker, speaker jack, audio output port, audio output device, earphone and / or other similar equipment to generate audible output. User interface module 1301 can be further configured with one or more haptic devices that can generate tactile outputs, such as vibrations and / or other outputs detectable by touch and / or physical contact with computing device 1300. In some examples, user interface module 1301 can be used to provide a graphical user interface (GUI) for utilizing computing device 1300.

[0154] The network communication module 1302 may include one or more devices that provide one or more wireless interfaces 1307 and / or one or more wired interfaces 1308 that can be configured to communicate via a network. The wireless interface(s) 1307 may include one or more wireless transmitters, receivers, and / or transceivers, such as Bluetooth TM Transceiver, Transceiver, Wi-Fi TM Transceiver, WiMAX TM The wired interface(s) 1308 may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a universal serial bus (USB) transceiver, or a similar transceiver that is configurable to communicate with a wired network via a twisted pair, a coaxial cable, a fiber optic link, or a similar physical connection.

[0155] In some examples, network communication module 1302 can be configured to provide reliable, secure, and / or authenticated communication. For each communication described herein, information for facilitating reliable communication (e.g., guaranteed message delivery) can be provided, which may be part of a message header and / or trailer (e.g., packet / message sequence information, encapsulation header and / or trailer, size / time information, and transmission verification information such as cyclic redundancy check (CRC) and / or parity check value). One or more cryptographic protocols and / or algorithms can be used to protect (e.g., encode or encrypt) and / or decrypt / decode communications, such as but not limited to data encryption standard (DES), advanced encryption standard (AES), Rivest-Shamir-Adelman (RSA) algorithm, Diffie-Hellman algorithm, secure socket protocols such as secure socket layer (SSL) or transport layer security (TLS), and / or digital signature algorithm (DSA). Other cryptographic protocols and / or algorithms can also be used, or other cryptographic protocols and / or algorithms other than those listed herein can be used to protect (then decrypt / decode) communications.

[0156] The one or more processors 1303 may include one or more general purpose processors and / or one or more special purpose processors (e.g., digital signal processors (DSPs), tensor processing units (TPUs), graphics processing units (GPUs), and application specific integrated circuits (ASICs), etc.). The one or more processors 1303 may be configured to execute computer readable instructions 1306 contained in the data storage 1304 and / or other instructions as described herein.

[0157] The data store 1304 may include one or more non-transitory computer-readable storage media that may be read and / or accessed by at least one of the one or more processors 1303. The one or more non-transitory computer-readable storage media may include volatile and / or non-volatile storage components such as optical, magnetic, organic, or other memory or optical disk storage, which may be fully or partially integrated with at least one of the one or more processors 1303. In some examples, the data store 1304 may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or optical disk storage device), while in other examples, the data store 1304 may be implemented using two or more physical devices.

[0158] Data store 1304 may include computer-readable instructions, and possibly additional data. In some examples, data store 1304 may include storage required to perform at least a portion of the methods, scenarios, and techniques described herein and / or at least a portion of the functionality of the devices and networks described herein. In some examples, data store 1304 may include storage for one or more machine learning systems and / or one or more trained machine learning models 1312 (e.g., one or more untrained, partially trained, and / or fully trained machine learning systems, such as one or more instances of neural network 420). In particular in these examples, computer-readable instructions 1306 may include instructions that, when executed by processor(s) 1303, enable computing device 1300 to provide some or all of the functionality of one or more machine learning systems and / or one or more trained machine learning models 1312.

[0159] In some examples, computing device 1300 may include camera(s) 1318. Camera(s) 1318 may include one or more image capture devices, such as still cameras and / or video cameras, that are equipped to capture light and record the captured light in one or more images; that is, camera(s) 1318 may generate image(s) of the captured light. The one or more images may be one or more still images and / or one or more images used in a video image. Camera(s) 1318 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light. In some examples, one or more of camera(s) 1318 may capture dual-pixel images and / or include one or more dual-pixel light sensors, such as dual-pixel light sensor 120. In some examples, computing device 1300 and / or camera(s) 1318 may be part or all of a multi-camera device, such as multi-camera device 800.

[0160] In some examples, computing device 1300 may include one or more sensors 1320. Sensors 1320 may be configured to measure conditions within computing device 1300 and / or conditions in the environment of computing device 1300 and provide data regarding these conditions. For example, sensors 1320 may include one or more of: (i) sensors for obtaining data about computing device 1300, such as, but not limited to, a thermometer for measuring the temperature of computing device 1300, a battery sensor for measuring the power of one or more batteries of power system 1322, and / or other sensors measuring the condition of computing device 1300; (ii) identification sensors for identifying other objects and / or devices, such as, but not limited to, radio frequency identification (RFID) readers, proximity sensors, one-dimensional barcode readers, two-dimensional barcode (e.g., quick response (QR) code) readers, and laser trackers, wherein these identification sensors may be configured to read identifiers, such as RFID tags, barcodes, QR codes, and / or other devices and / or objects configured to be read and provide at least identification information; (iii) sensors for measuring the position and / or movement of computing device 1300, such as, but not limited to, (iv) an environmental sensor for obtaining data indicative of the environment of computing device 1300, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a dual-pixel light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a motion sensor, a microphone, a sound sensor, an ultrasonic sensor, and / or a smoke sensor; and / or (v) a force sensor for measuring one or more forces (e.g., inertial forces and / or G-forces) acting around computing device 1300, such as, but not limited to, one or more sensors measuring: forces in one or more dimensions, torques, ground forces, frictional forces, and / or a zero moment point (ZMP) sensor that identifies the ZMP and / or the location of the ZMP. Many other examples of sensor 1320 are also possible.

[0161] The power system 1322 may include one or more batteries 1324 and / or one or more external power interfaces 1326 for providing power to the computing device 1300. When electrically coupled to the computing device 1300, each of the one or more batteries 1324 may be used as a source of stored power for the computing device 1300. The one or more batteries 1324 of the power system 1322 may be configured to be portable. Some or all of the one or more batteries 1324 may be easily removed from the computing device 1300. In other examples, some or all of the one or more batteries 1324 may be inside the computing device 1300 and may therefore not be easily removed from the computing device 1300. Some or all of the one or more batteries 1324 may be rechargeable. For example, a rechargeable battery may be recharged via a wired connection between the battery and another power source, such as by being outside the computing device 1300 and connected to one or more power sources of the computing device 1300 via one or more external power interfaces. In other examples, some or all of the one or more batteries 1324 may be non-rechargeable batteries.

[0162] The one or more external power interfaces 1326 of the power system 1322 may include one or more wired power interfaces, such as a USB cable and / or a power cord, that implement a wired power connection to one or more power sources external to the computing device 1300. The one or more external power interfaces 1326 may include one or more wireless power interfaces, such as a Qi wireless charger, that implement a wireless power connection to one or more external power sources, such as via a Qi wireless charger. Once a power connection with an external power source is established using the one or more external power interfaces 1326, the computing device 1300 may draw power from the external power source of the established power connection. In some examples, the power system 1322 may include associated sensors, such as a battery sensor associated with one or more batteries or other types of power sensors.

[0163] Cloud-based servers

[0164] Fig.14 A network of computing clusters arranged as a cloud-based server system according to example embodiments is described. Fig.14A network 1206 of computing clusters 1409a, 1409b, 1409c arranged as a cloud-based server system is depicted according to an example embodiment. Computing clusters 1409a, 1409b, 1409c can be cloud-based devices that store program logic and / or data for cloud-based applications and / or services; for example, executing at least one function of computing device 100, camera 110, dual-pixel image 200, system 300, 402 (including but not limited to neural network 420), stride 1 encoder function 600, stride 2 encoder function 650, decoder function 700, multi-camera device 800, 830, distributed computing architecture 1200 and / or method 1500, 1600 and / or at least one function related to computing device 100, camera 110, dual-pixel image 200, system 300, 402 (including but not limited to neural network 420), stride 1 encoder function 600, stride 2 encoder function 650, decoder function 700, multi-camera device 800, 830, distributed computing architecture 1200 and / or method 1500, 1600.

[0165] In some embodiments, computing clusters 1409a, 1409b, 1409c may be a single computing device residing in a single computing center. In other embodiments, computing clusters 1409a, 1409b, 1409c may include multiple computing devices in a single computing center, or even multiple computing devices in multiple computing centers located in different geographic locations. For example, Fig.14 Each of the computing clusters 1409a, 1409b, 1409c is depicted as residing at a different physical location.

[0166] In some embodiments, the data and services at the computing clusters 1409a, 1409b, 1409c may be encoded as computer-readable information stored in a non-transitory, tangible computer-readable medium (or computer-readable storage medium) and accessible by other computing devices. In some embodiments, the computing clusters 1409a, 1409b, 1409c may be stored on a single disk drive or other non-transitory, tangible storage medium, or may be implemented on multiple disk drives or other tangible storage media located at one or more different geographic locations.

[0167] Fig.14 A cloud-based server system is depicted according to an example embodiment. Fig.14, the functionality of the system 402, the multi-camera devices 800, 830, and / or computing devices may be distributed among computing clusters 1409a, 1409b, 1409c. Computing cluster 1409a may include one or more computing devices 1400a, a cluster storage array 1410a, and a cluster router 1411a connected by a local cluster network 1412a. Similarly, computing cluster 1409b may include one or more computing devices 1400b, a cluster storage array 1410b, and a cluster router 1411b connected by a local cluster network 1412b. Likewise, computing cluster 1409c may include one or more computing devices 1400c, a cluster storage array 1410c, and a cluster router 1411c connected by a local cluster network 1412c.

[0168] In some embodiments, each of computing clusters 1409a, 1409b, 1409c may have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. However, in other embodiments, each computing cluster may have a different number of computing devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster may depend on the computing tasks or tasks assigned to each computing cluster.

[0169] In computing cluster 1409a, for example, computing device 1400a can be configured to perform various computing tasks of system 402, multi-camera device 800, 830, and / or computing devices. In one embodiment, various functions of system 402, multi-camera device 800, 830, and / or computing devices can be distributed between one or more computing devices 1400a, 1400b, 1400c. Computing devices 1400b and 1400c in respective computing clusters 1409b and 1409c can be configured similarly to computing device 1400a in computing cluster 1409a. On the other hand, in some embodiments, computing devices 1400a, 1400b, and 1400c can be configured to perform different functions.

[0170] In some embodiments, computing tasks and stored data associated with the system 402, multi-camera devices 800, 830 and / or computing devices may be distributed across computing devices 1400a, 1400b, 1400c based at least in part on processing requirements of the system 402, multi-camera devices 800, 830 and / or computing devices, processing capabilities of the computing devices 1400a, 1400b, 1400c, latency of network links between computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that may affect cost, speed, fault tolerance, resilience, efficiency and / or other design goals of the overall system architecture.

[0171] The cluster storage arrays 1410a, 1410b, 1410c of the computing clusters 1409a, 1409b, 1409c may be storage arrays that include disk array controllers configured to manage read and write access to the hard disk drive groups. The disk array controllers may also be configured, alone or in conjunction with their respective computing devices, to manage backup or redundant copies of data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0172] Similar to the way that the functionality of the system 402, multi-camera device 800, 830, and / or computing device can be distributed across the computing devices 1400a, 1400b, 1400c of the computing clusters 1409a, 1409b, 1409c, various active portions and / or backup portions of these components can be distributed across the cluster storage arrays 1410a, 1410b, 1410c. For example, some cluster storage arrays can be configured to store a portion of the data of the system 402, multi-camera device 800, 830, and / or computing device, while other cluster storage arrays can store other (multiple) portions of the data of the system 402, multi-camera device 800, 830, and / or computing device. In addition, some cluster storage arrays can be configured to store backup versions of data stored in other cluster storage arrays.

[0173] The cluster routers 1411a, 1411b, 1411c in the computing clusters 1409a, 1409b, 1409c may include network devices configured to provide internal and external communications for the computing clusters. For example, the cluster router 1411a in the computing cluster 1409a may include one or more Internet switching and routing devices configured to (i) provide local area network communications between the computing device 1400a and the cluster storage array 1410a via a local cluster network 1412a, and (ii) provide wide area network communications between the computing cluster 1409a and the computing clusters 1409b and 1409c via a wide area network link 1413a to the network 1206. Cluster routers 1411b and 1411c may include similar network devices as cluster router 1411a, and cluster routers 1411b and 1411c may perform similar networking functions for computing clusters 1409b and 1409b as cluster router 1411a performs for computing cluster 1409a.

[0174] In some embodiments, the configuration of the cluster routers 1411a, 1411b, 1411c may be based at least in part on the data communication requirements of the computing devices and the cluster storage array, the data communication capabilities of the network devices in the cluster routers 1411a, 1411b, 1411c, the latency and throughput of the local cluster network 1412a, 1412b, 1412c, the latency, throughput, and cost of the wide area network links 1413a, 1413b, 1413c, and / or other factors that may affect cost, speed, fault tolerance, resilience, efficiency, and / or other design criteria of a reasonable system architecture.

[0175] Example methods of operation

[0176] Fig.15 15 is a flow chart of a method 1500 according to an embodiment. The method 1500 may be performed by a computing device, such as computing device 1300. The method may begin at block 1510, where the computing device may receive a dual-pixel image of at least a foreground object, the dual-pixel image comprising a plurality of dual-pixels, wherein a dual-pixel of the plurality of dual-pixels may comprise a left pixel and a right pixel, the left pixel and the right pixel both representing light incident on a single dual-pixel element for capturing the dual-pixel image, such as at least the foreground object described above. Figure 1 , 2 , 4 and 5 discussed in the context of.

[0177] At block 1520, the computing device may be used to train a machine learning system to determine a depth map associated with a dual pixel image, such as described above at least in Figure 3 and Figure 4 discussed in the context of .

[0178] At block 1530, the computing device may be used to provide a trained machine learning system, such as described above at least in Figure 3 and Figure 4 discussed in the context of .

[0179] In some examples, training the machine learning system to determine the depth map may include training the machine learning system to determine the depth map based on a loss function, which loss function may include an estimated affine mapping of the depth map, such as described above at least in Figure 4 discussed in the context of .

[0180] In some examples, training the machine learning system to determine the depth map based on a loss function may include training the machine learning system to determine the depth map based on a loss function, where the loss function may include a difference between an estimated affine mapping of the depth map and a reference depth map, such as described above at least in Figure 4 discussed in the context of .

[0181] In some examples, the foreground object may have a first object type, and training the machine learning system to determine the depth map may include training the machine learning system to determine the depth map using multiple images of objects having the first object type, such as described above at least in Figure 4 discussed in the context of .

[0182] In some examples, the first object type can be associated with a human face, such as at least in Figure 4 discussed in the context of .

[0183] In some examples, receiving at least a dual-pixel image of a foreground object at a computing device may include: obtaining a plurality of dual-pixel images of at least the foreground object within a threshold time period using a multi-camera device, each of the plurality of dual-pixel images may include a plurality of dual pixels, wherein the multi-camera device may include a plurality of cameras; and receiving at least the plurality of dual-pixel images of the foreground object at the computing device, such as at least Fig. 8A , 8B , 9, 10 and 11 discussed in the context of.

[0184] In some examples, each of multiple cameras in a multi-camera device may include a system clock, and obtaining multiple dual-pixel images of a foreground object using the multi-camera device may include: determining a main camera among the multiple cameras of the multi-camera device, the main camera having a main clock; at each camera of the multiple cameras of the multi-camera device except the main camera, determining an offset between the system clock of the camera and the main clock; determining a maximum offset among the offsets of the multiple cameras; at the main camera, sending a message to each of the multiple cameras of the multi-camera device except the main camera to capture a dual-pixel image at a future trigger time, wherein the future trigger time may be greater than the maximum offset; and capturing a dual-pixel image of the foreground object at the future trigger time by each of the multiple cameras of the multi-camera device, such as at least the above-mentioned Fig. 8A , 8B , 9, 10 and 11 discussed in the context of.

[0185] In some examples, sending a message to each of the multiple cameras of the multi-camera device except the main camera to capture a dual-pixel image at a future trigger time may include: sending a repetitive request to each of the multiple cameras of the multi-camera device except the main camera to periodically capture multiple dual-pixel images starting at a future trigger time; and capturing a dual-pixel image of a foreground object by each of the multiple cameras of the multi-camera device at a future trigger time may include capturing multiple dual-pixel images by each of the multiple cameras of the multi-camera device starting at a future trigger time, such as at least in the above Fig. 8A , 8B , 9, 10 and 11 discussed in the context of.

[0186] In some examples, sending a repetitive request to each of a plurality of cameras of a multi-camera device except a main camera to start periodically capturing a plurality of dual-pixel images at a future trigger time may further include: after sending the repetitive request, sending a high priority request to one or more delayed cameras of the multi-camera device, wherein the high priority request may interrupt the repetitive request and may reset the time period for periodically capturing a plurality of dual-pixel images, such as at least in the above-mentioned Fig. 8A , 8B , 9, 10 and 11 discussed in the context of.

[0187] In some examples, training a machine learning system to determine a depth map may include training a neural network to determine a depth map, such as described above at least in Figure 3 and Figure 4 discussed in the context of .

[0188] Fig.16 16 is a flow chart of method 1600 according to an embodiment. Method 1600 may be performed by a computing device, such as computing device 1300. Method 1600 may begin at block 1610, where the computing device may receive a dual-pixel image of at least a foreground object, where the dual-pixel image may include a plurality of dual-pixels, where a dual-pixel in the plurality of dual-pixels may include a left pixel and a right pixel, where the left pixel and the right pixel both represent light incident on a single dual-pixel element for capturing the dual-pixel image, such as at least the foreground object described above. Figure 3 and Figure 4 discussed in the context of .

[0189] At block 1620, the computing device may provide the dual-pixel image to the trained machine learning system to determine a depth map for a plurality of dual pixels of the dual-pixel image, such as described above at least in Figure 3 , 4 , 5, 6A, 6B and 7 discussed in the context of.

[0190] At block 1630, the computing device may be configured to determine an output image based on the dual pixel image and the depth map, such as described above at least in Figure 4 discussed in the context of .

[0191] In some examples, determining an output image based on a dual-pixel image and a depth map may include: receiving a request at a computing device to blur a background of the dual-pixel image, wherein the background of the dual-pixel image does not include a foreground object; and determining the output image by using the computing device to blur at least a portion of the background of the dual-pixel image based on the depth map, such as at least in the above-described Figure 4 discussed in the context of .

[0192] In some examples, method 160 may further include: providing an output image using a computing device, such as at least Figure 4 discussed in the context of .

[0193] In some examples, the computing device may include a camera; and receiving a dual-pixel image of at least a foreground object may include: generating a dual-pixel image of at least the foreground object using the camera; and receiving the generated dual-pixel image from the camera at the computing device, such as at least Figure 4 discussed in the context of .

[0194] In some examples, a trained machine learning system can be trained using a computing device, such as at least Figure 3 and Figure 4 discussed in the context of .

[0195] In some examples, the trained machine learning system can be trained using a second computing device different from the computing device; and providing the dual-pixel image to the trained machine learning system to determine a depth map of multiple dual pixels for the dual-pixel image can include: receiving the trained machine learning system at the computing device; and after receiving the trained machine learning system at the computing device, providing the dual-pixel image to the trained machine learning system of the computing device to determine a depth map of multiple dual pixels for the dual-pixel image, such as at least in the above Figure 3 and Figure 4 discussed in the context of .

[0196] In some examples, the trained machine learning system may reside on a second computing device different from the computing device; and providing the dual-pixel image to the trained machine learning system to determine a depth map of multiple dual pixels for the dual-pixel image may include: receiving the dual-pixel image from the second computing device at the second computing device; after receiving the dual-pixel image, the second computing device provides the dual-pixel image to the trained machine learning system of the second computing device to determine a depth map of multiple dual pixels for the dual-pixel image; and sending the depth map from the second computing device to the computing device, such as at least the above-mentioned Figure 3 and Figure 4 discussed in the context of .

[0197] In some examples, the trained machine learning system may reside on a second computing device different from the computing device; and providing a dual-pixel image to the trained machine learning system to determine a depth map of multiple dual-pixels for the dual-pixel image may include: receiving a dual-pixel image from a second computing device at the second computing device; after receiving the dual-pixel image, the second computing device provides the dual-pixel image to the trained machine learning system of the second computing device to determine a depth map of multiple dual-pixels for the dual-pixel image, and wherein determining, by the computing device, an output image based on the dual-pixel image and the depth map may include: determining the output image based on the dual-pixel image and the depth map using the second computing device, and sending the output image from the second computing device to the computing device, such as at least in the above-mentioned Figure 3 and Figure 4 discussed in the context of .

[0198] The present disclosure is not limited to the specific embodiments described in the present application, which are intended to be explanations of various aspects. Without departing from the scope and spirit of the present invention, many modifications and variations will be apparent to those skilled in the art. According to the foregoing description, in addition to those listed herein, methods and compositions that are functionally equivalent within the scope of the present disclosure are apparent to those skilled in the art. Such modifications and variations are intended to fall within the scope of the appended claims.

[0199] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the drawings, similar symbols generally identify similar components unless the context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized and other changes may be made without departing from the spirit or scope of the subject matter presented herein. It should be readily understood that the various aspects of the present disclosure generally described herein and illustrated in the accompanying drawings may be arranged, replaced, combined, segmented, and designed in accordance with a variety of different configurations expressly disclosed herein.

[0200] With respect to any or all of the ladder diagrams, scenarios, and flow charts in the accompanying drawings and as discussed herein, each block and / or communication may represent the processing of information and / or the transmission of information according to an example embodiment. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, depending on the functionality involved, the functions described as blocks, transmissions, communications, requests, responses, and / or messages may not be performed in the order shown or discussed, including substantially concurrently or in reverse order. In addition, more or fewer blocks and / or functions may be used in conjunction with the ladder diagrams, scenarios, and flow charts discussed herein. Figure 1 The ladder diagrams, scenarios and flow charts can be used together, and these ladder diagrams, scenarios and flow charts can be combined with each other in part or in whole.

[0201] The frame representing the processing of information can correspond to a circuit that can be configured to perform the specific logical functions of the method or technology described herein. Alternatively or additionally, the frame representing the processing of information can correspond to a module, fragment, or part (including relevant data) of the program code. The program code may include one or more instructions executable by the processor for implementing the specific logical functions or actions in the method or technology. The program code and / or relevant data may be stored on any type of computer-readable medium, such as a storage device or other storage medium including a disk or hard drive.

[0202] Computer readable media may also include non-transitory computer readable media, such as non-transitory computer readable media that can store data for a short period of time, such as register memory, processor cache, and random access memory (RAM). Computer readable media may also include non-transitory computer readable media that store program code and / or data for a longer period of time, such as secondary or persistent long-term storage, such as read-only memory (ROM), optical or magnetic disks, compact disk read-only memory (CD-ROM). Computer readable media may also be any other volatile or non-volatile storage system. Computer readable media may be considered to be computer readable storage media, for example, or a tangible storage device.

[0203] Moreover, the boxes representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device. However, other information transfers may be between software modules and / or hardware modules in different physical devices.

[0204] Although various aspects and embodiments are disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. Therefore, the various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the appended claims.

Claims

1. A computer-implemented method, characterized in that the method comprises: Receiving, at a computing device, a dual-pixel image of at least a foreground object, the dual-pixel image comprising a plurality of dual-pixels, wherein a dual-pixel in the plurality of dual-pixels comprises a left pixel and a right pixel, the left pixel and the right pixel each representing light incident on a single dual-pixel element used to capture the dual-pixel image; training a machine learning system to determine a depth map associated with the dual pixel image using the computing device; and providing the trained machine learning system using the computing device, Wherein training the machine learning system to determine the depth map comprises using a loss function that calculates the sum of squared differences between an affine mapping of the estimated depth map and a ground truth depth map associated with the dual pixel image.

2. The computer-implemented method of claim 1 , wherein: The foreground object has a first object type, and wherein training the machine learning system to determine the depth map includes training the machine learning system to determine the depth map using a plurality of images of objects having the first object type.

3. The computer-implemented method of claim 2, wherein: The first object type is associated with a human face.

4. The computer-implemented method of claim 1 , wherein: Receiving, at the computing device, the dual pixel image of at least the foreground object comprises: obtaining, using a multi-camera device, a plurality of dual-pixel images of at least the foreground object within a threshold time period, each of the plurality of dual-pixel images comprising a plurality of dual pixels, wherein the multi-camera device comprises a plurality of cameras; And receiving, at the computing device, the plurality of dual-pixel images of at least the foreground object.

5. The computer-implemented method of claim 4, wherein: Each of the plurality of cameras of the multi-camera device comprises a system clock, and wherein obtaining the plurality of dual-pixel images of the foreground object using the multi-camera device comprises: determining a master camera of the plurality of cameras of the multi-camera device, the master camera having a master clock; At each camera of the plurality of cameras of the multi-camera device other than the master camera, determining an offset between a system clock of the camera and the master clock; determining a maximum offset of the offsets of the plurality of cameras; At the master camera, sending a message to each of the plurality of cameras of the multi-camera device except the master camera to capture a dual-pixel image at a future trigger time, wherein the future trigger time is greater than the maximum offset; and A dual-pixel image of the foreground object is captured by each of the multiple cameras of the multi-camera device at the future trigger time.

6. The computer-implemented method of claim 5, wherein: Sending a message to each of the plurality of cameras of the multi-camera device except the main camera to capture a dual-pixel image at the future trigger time includes: sending a repetitive request to each of the plurality of cameras of the multi-camera device except the main camera to periodically capture a plurality of dual-pixel images starting at the future trigger time; and Wherein capturing a dual-pixel image of the foreground object at the future trigger time by each of the multiple cameras of the multi-camera device includes periodically capturing multiple dual-pixel images by each of the multiple cameras of the multi-camera device starting at the future trigger time.

7. The computer-implemented method of claim 6, wherein: Sending the repetitive request to each of the plurality of cameras of the multi-camera device except the main camera to start periodically capturing a plurality of dual-pixel images at the future trigger time further comprises: After sending the repeat request, a high priority request is sent to one or more delayed cameras of the multi-camera device, wherein the high priority request interrupts the repeat request and resets a time period for periodically capturing the plurality of dual-pixel images.

8. The computer-implemented method of claim 1, wherein: Training the machine learning system to determine the depth map includes training a neural network to determine the depth map.

9. A computer-implemented method, characterized in that: The method comprises: Receiving, at a computing device, a dual-pixel image of at least a foreground object, the dual-pixel image comprising a plurality of dual-pixels, wherein a dual-pixel in the plurality of dual-pixels comprises a left pixel and a right pixel, the left pixel and the right pixel each representing light incident on a single dual-pixel element used to capture the dual-pixel image; providing the dual pixel image to a trained machine learning system to determine a depth map for the plurality of dual pixels of the dual pixel image; and determining, using the computing device, an output image to be output by a computing device based on the dual pixel image and the depth map, Wherein training the machine learning system to determine the depth map comprises using a loss function that calculates the sum of squared differences between an affine mapping of the estimated depth map and a ground truth depth map associated with the dual pixel image.

10. The computer-implemented method of claim 9, wherein: Determining the output image based on the dual-pixel image and the depth map includes: receiving, at the computing device, a request to blur a background of the dual pixel image, wherein the background of the dual pixel image does not include the foreground object; and The output image is determined by blurring at least a portion of the background of the dual pixel image based on the depth map using the computing device.

11. The computer-implemented method of claim 9, further comprising: The output image is provided using the computing device.

12. The computer-implemented method of claim 9, wherein: The computing device comprises a camera, and wherein receiving the dual pixel image of at least the foreground object comprises: generating, using the camera, the dual pixel image of at least the foreground object; and At the computing device, a generated dual pixel image is received from the camera.

13. The computer-implemented method of claim 9, wherein: The trained machine learning system is trained using the computing device.

14. The computer-implemented method of claim 9, wherein: The trained machine learning system is trained using a second computing device that is different from the computing device; as well as Wherein, providing the dual pixel image to the trained machine learning system to determine the depth map for the plurality of dual pixels of the dual pixel image comprises: receiving, at the computing device, the trained machine learning system; and After receiving the trained machine learning system at the computing device, the dual pixel image is provided to the trained machine learning system of the computing device to determine a depth map for the plurality of dual pixels of the dual pixel image.

15. The computer-implemented method of claim 9, wherein: The trained machine learning system resides on a second computing device different from the computing device; as well as Wherein, providing the dual pixel image to the trained machine learning system to determine the depth map for the plurality of dual pixels of the dual pixel image comprises: receiving, at the second computing device, the dual pixel image from the computing device; After receiving the dual pixel image, the second computing device provides the dual pixel image to the trained machine learning system of the second computing device to determine the depth map for the plurality of dual pixels of the dual pixel image; and The depth map is sent from the second computing device to the computing device.

16. The computer-implemented method of claim 9, wherein: The trained machine learning system resides on a second computing device different from the computing device; Wherein, providing the dual pixel image to the trained machine learning system to determine the depth map for the plurality of dual pixels of the dual pixel image comprises: receiving, at the second computing device, the dual pixel image from the computing device; and After receiving the dual pixel image, the second computing device provides the dual pixel image to the trained machine learning system of the second computing device to determine the depth map for the plurality of dual pixels of the dual pixel image, and Wherein, determining the output image based on the dual-pixel image and the depth map by the computing device comprises: determining, using the second computing device, the output image based on the dual pixel image and the depth map; and The output image is sent from the second computing device to the computing device.

17. A computing device, characterized in that The computing device comprises: one or more processors; and A data store having stored thereon computer-executable instructions, the computer-executable instructions, when executed by the one or more processors, causing the computing device to perform functions, the functions comprising: receiving a dual-pixel image of at least a foreground object, the dual-pixel image comprising a plurality of dual-pixels, wherein a dual-pixel in the plurality of dual-pixels comprises a left pixel and a right pixel, the left pixel and the right pixel each representing light incident on a single dual-pixel element used to capture the dual-pixel image; providing the dual pixel image to a trained machine learning system to determine a depth map for the plurality of dual pixels of the dual pixel image; and determining an output image to be output by a computing device based on the dual pixel image and the depth map, Wherein training the machine learning system to determine the depth map comprises using a loss function that calculates the sum of squared differences between an affine mapping of the estimated depth map and a ground truth depth map associated with the dual pixel image.

18. The computing device of claim 17, wherein: Determining the output image based on the dual-pixel image and the depth map includes: receiving a request to blur a background of the dual pixel image, wherein the background of the dual pixel image does not include the foreground object; and The output image is determined by blurring at least a portion of the background of the dual pixel image based on the depth map.

Images