Dynamic camera selection

Abstract

The present disclosure provides more efficient and / or efficient techniques for determining information about a physical environment. Such techniques optionally complement or replace other techniques for determining information about a physical environment. Some techniques described herein encompass switching between which cameras are used to calculate the depth of a location in a physical environment. The switching may occur when the current image does not have sufficient feature correlation for calculating the depth of the location. Other techniques described herein encompass switching which cameras to use to obtain sufficient data for locations within a representation (e.g., a three-dimensional representation) of a physical environment. The handover may occur in response to determining that sufficient data for the location is not present.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 350,595, filed on June 9, 2022, entitled “DYNAMIC CAMERA SELECTION,” which is hereby incorporated by reference in its entirety for all purposes. Background Art

[0003] Some devices include cameras for capturing images of a physical environment to determine information about the physical environment (such as calculating the depth of a location). Such determinations are limited by the images used and / or captured. Therefore, there is a need to provide more effective and / or efficient techniques for determining information about a physical environment. Summary of the invention

[0004] The present disclosure provides more effective and / or efficient techniques for determining information about a physical environment. Such techniques may optionally supplement or replace other techniques for determining information about a physical environment.

[0005] Some techniques described herein encompass switching which cameras are used to calculate the depth of a location in a physical environment. The switching may occur when a current image does not have sufficient feature correlations for calculating the depth of the location. Other techniques described herein encompass switching which cameras are used to obtain sufficient data for a location within a representation (e.g., a three-dimensional representation) of a physical environment. The switching may occur in response to determining that there is not enough data for the location.

[0006] In the technology described herein, the cameras can be arranged in different configurations on the device. For example, three cameras can be positioned in a triangular pattern, with two cameras located on a horizontal axis and the third camera above or below the horizontal axis. For another example, four cameras can be positioned in a rectangular pattern. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0008] Figure 1 is a block diagram of an exemplary computing system.

[0009] Figure 2 is a block diagram illustrating a device having interconnected subsystems.

[0010] Figure 3 is a block diagram illustrating an apparatus for determining information about a physical environment.

[0011] Figure 4Ais a block diagram illustrating a camera array of three cameras.

[0012] Figure 4B is a block diagram illustrating a camera array of four cameras.

[0013] Figure 5 is a flow chart illustrating a method for calculating the depth of a location.

[0014] Figure 6 is a flow chart illustrating a method for obtaining sufficient data about a physical environment. DETAILED DESCRIPTION

[0015] The following description sets forth exemplary methods, parameters, etc. However, it should be appreciated that such description is not intended to limit the scope of the present disclosure, but is provided as a description of exemplary embodiments.

[0016] Some of the techniques described herein encompass switching which cameras are used to calculate the depth of a location in a physical environment. The switch may occur when the current image does not have sufficient feature correlation to calculate the depth of the location. In one example, a device includes multiple cameras with at least partially overlapping fields of view of a physical environment. The device causes a group of cameras to capture images of the physical environment. Then, the images are used to try to calculate the depth of different locations within the image. When the image does not have sufficient feature correlation to calculate the depth of the location, the device causes different groups of cameras to capture images of the physical environment. Then, the images captured by the different groups are used to calculate the depth of the location. The different groups may or may not include cameras from the original group of cameras. In some examples, different groups of cameras are selected from multiple possible different groups of cameras. In some examples, in response to the image not having sufficient feature correlation, the device causes a single camera to capture multiple images to be used to calculate the depth of the location.

[0017] Other techniques described herein encompass switching which cameras are used to obtain sufficient data for a location within a representation of a physical environment. The switching may occur in response to determining that there is not enough data for the location. In one example, a device includes multiple cameras having at least partially overlapping fields of view of a physical environment. The device causes a set of cameras to capture images of the physical environment. The images are used to generate a depth map of the physical environment, the depth map including distances to different locations in the physical environment. In some examples, the depth map and the images are used to generate a representation of the physical environment, the representation including the locations of the identified objects within the physical environment. The device then determines that the representation does not include sufficient data for a particular location. For example, the set of cameras may not be able to capture images of a particular location. After determining a shortcoming of the representation, the device causes a different set of one or more cameras to capture images of the physical environment.

[0018] In the techniques described herein, multiple cameras can be in different configurations on a device. For example, the multiple cameras can include three cameras positioned in a triangular pattern, where two cameras are located on a horizontal axis and the third camera is above or below the horizontal axis. This configuration can allow one camera to be switched with another camera when a problem occurs. For another example, the multiple cameras can include four cameras positioned in a rectangular pattern. This configuration can allow a current pair of cameras with a specific distance between the current pair to be switched to a new pair of cameras with a different distance between the new pair.

[0019] In the method described herein where one or more steps depend on one or more conditions being met, it should be understood that the method can be repeated in multiple repetitions so that in the process of repetition, all conditions for determining the steps in the method have been met in different repetitions of the method. For example, if the method needs to perform the first step (if the condition is met), and perform the second step (if the condition is not met), then the ordinary technician will know that the steps stated are repeated until both the condition is met and the condition is not met (in no particular order). Therefore, the method described as having one or more steps depending on one or more conditions being met can be rewritten as a method of repeating until each condition described in the method is met. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing a contingent operation based on the satisfaction of the corresponding one or more conditions, and is therefore able to determine whether the possible situation has been met without explicitly repeating the steps of the method until all conditions for determining the steps in the method have been met. It will also be understood by ordinary technicians in the art that, similar to the method with the contingent step, the system or computer-readable storage medium can repeat the steps of the method as many times as needed to ensure that all the contingent steps have been performed.

[0020] Although the following description uses the term "first", "second" etc. to describe various elements, these elements should not be limited by the term. In some examples, these terms are used to distinguish an element from another element. For example, without departing from the scope of the various described embodiments, the first device may be referred to as the second device, and similarly, the second device may be referred to as the first device. In some examples, the first device and the second device are two separate references to the same device. In some embodiments, the first device and the second device are both devices, but they are not the same device or the same type of device.

[0021] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limiting. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "including" and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0022] The term "if" is optionally interpreted to mean "when," "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that ..." or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]," or "in response to detecting [a stated condition or event]," depending on the context.

[0023] Now turn to Figure 1 , depicts a block diagram of a computing system 100. Computing system 100 is a non-limiting example of a computing system that may be used to perform the functions described herein. It should be appreciated that other computer architectures of computing systems may be used to perform the functions described herein.

[0024] In the illustrated example, computing system 100 includes a processor subsystem 110 coupled (e.g., wired or wirelessly) to memory 120 (e.g., system memory) and I / O interface 130 via an interconnect 150 (e.g., a system bus, one or more memory locations, or other communication channels for connecting multiple components of computing system 100). In addition, I / O interface 130 is coupled (e.g., wired or wirelessly) to I / O device 140. In some examples, I / O interface 130 is included with I / O device 140 so that the two are a single component. It should be appreciated that there may be one or more I / O interfaces, each of which is coupled to one or more I / O devices. In some examples, multiple instances of processor subsystem 110 may be coupled to interconnect 150.

[0025] The computing system 100 can be any of various types of devices, including but not limited to a system on a chip, a server system, a personal computer system (e.g., an iPhone, iPad, or MacBook), a sensor, and the like. In some examples, the computing system 100 is included with or coupled to a physical component for the purpose of modifying the physical component in response to an instruction (e.g., the computing system 100 receives an instruction to modify the physical component, and in response to the instruction, causes the physical component to be modified (e.g., by an actuator)). Examples of such physical components include accelerometers, brakes, gearboxes, motors, pumps, refrigeration systems, suspension systems, steering controls, vacuum systems, valves, and the like. As used herein, a sensor includes one or more hardware components that detect information about a physical environment near (e.g., surrounding) the sensor. In some examples, the hardware components of the sensor include a sensing component (e.g., an image sensor or a temperature sensor), a transmitting component (e.g., a laser or radio transmitter), a receiving component (e.g., a laser or radio receiver), or any combination thereof. Examples of sensors include angle sensors, chemical sensors, brake pressure sensors, contact sensors, contactless sensors, electrical sensors, flow sensors, force sensors, gas sensors, humidity sensors, cameras, inertial measurement units, leak sensors, level sensors, light detection and ranging systems, metal sensors, motion sensors, particle sensors, photoelectric sensors, position sensors (e.g., global positioning systems), precipitation sensors, pressure sensors, proximity sensors, radio detection and ranging systems, radiation sensors, speed sensors (e.g., measuring the speed of an object), temperature sensors, time of flight sensors, torque sensors, and ultrasonic sensors. Although in Figure 1 Although a single computing system is shown in FIG. 1 , computing system 100 may also be implemented as two or more computing systems operating together.

[0026] In some examples, processor subsystem 110 includes one or more processors or processing units configured to execute program instructions to perform the functions described herein. For example, processor subsystem 110 may execute an operating system, a middleware system, one or more application programs, or any combination thereof.

[0027] In some examples, the operating system manages the resources of the computing system 100. Examples of the types of operating systems covered herein include batch operating systems (e.g., multiple virtual storage (MVS)), time-sharing operating systems (e.g., Unix), distributed operating systems (e.g., Advanced Interactive Executive (AIX)), network operating systems (e.g., Microsoft Windows Server), real-time operating systems (e.g., QNX). In some examples, the operating system includes various programs, instruction sets, software components, and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and for facilitating communication between various hardware and software components. In some examples, the operating system uses a priority-based scheduler that assigns priorities to different tasks to be executed by the processor subsystem 110. In such examples, the priorities assigned to the tasks are used to identify the next task to be executed. In some examples, the priority-based scheduler identifies the next task to be executed when the previous task completes execution (e.g., the highest priority task runs to completion unless another higher priority task is ready).

[0028] In some examples, the middleware system provides one or more services and / or capabilities to applications (e.g., one or more applications running on the processor subsystem 110) outside of the services provided by the operating system (e.g., data management, application services, messaging, authentication, API management, etc.). In some examples, the middleware system is designed for heterogeneous computer clusters to provide hardware abstraction, low-level device control, implementation of common functions, messaging between processes, package management, or any combination thereof. Examples of middleware systems include lightweight communications and marshaling (LCM), PX4, Robot Operating System (ROS), ZeroMQ. In some examples, the middleware system uses a graph architecture to represent processes and / or operations, where processing occurs in nodes that can receive, publish, and reuse sensor data, control, state, planning, actuators, and other messages. In such examples, applications (e.g., applications executed on the processor subsystem 110 as described above) can be defined using a graph architecture so that different operations of the application are included with different nodes in the graph architecture.

[0029] In some examples, a publish-subscribe model is used to send a message from a first node in a graph architecture to a second node in the graph architecture, where the first node publishes data on a channel to which the second node can subscribe. In such examples, the first node may store the data in a memory (e.g., a local memory of the memory 120 or the processor subsystem 110), and notify the second node that the data has been stored in the memory. In some examples, the first node notifies the second node that the data has been stored in the memory by sending a pointer (e.g., a memory pointer, such as an identifier of a memory location) to the second node so that the second node can access the data from the location where the first node stores the data. In some examples, the first node will send the data directly to the second node so that the second node will not need to access the memory based on the data received from the first node.

[0030] The memory 120 may include a computer-readable medium (e.g., a non-transitory or transitory computer-readable medium) that may be used to store program instructions that may be executed by the processor subsystem 110 to cause the computing system 100 to perform various operations described herein. For example, the memory 120 may store program instructions to implement the various operations described in FIG. 4 and / or Figure 5 The functions associated with the described process.

[0031] The memory 120 may be implemented using different physical non-transitory memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM—SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read-only memory (PROM, EEPROM, etc.), etc. The memory in the computing system 100 is not limited to a main storage device, such as the memory 120. Instead, the computing system 100 may also include other forms of storage devices, such as cache memory in the processor subsystem 110 and auxiliary storage devices (e.g., hard disk drives, storage arrays, etc.) on the I / O devices 140. In some examples, these other forms of storage devices may also store program instructions that can be executed by the processor subsystem 110 to perform the operations described herein. In some examples, the processor subsystem 110 (or each processor within the processor subsystem 110) includes a cache or other form of on-board memory.

[0032] The I / O interface 130 may be any of various types of interfaces configured to couple to and communicate with other devices. In some examples, the I / O interface 130 includes a bridge chip (e.g., a south bridge) from a front-end bus to one or more back-end buses. The I / O interface 130 may be coupled to one or more I / O devices (e.g., an I / O device 140) via one or more corresponding buses or other interfaces. Examples of I / O devices include storage devices (hard drives, optical drives, removable flash drives, storage arrays, SANs, or their associated controllers), network interface devices (e.g., to a local area network or a wide area network), sensor devices (e.g., cameras, radars, lidars, ultrasonic sensors, GPS, inertial measurement devices, etc.), and auditory or visual output devices (e.g., speakers, lights, screens, projectors, etc.). In some examples, the computing system 100 is coupled to a network via a network interface device (e.g., configured to communicate via Wi-Fi, Bluetooth, Ethernet, etc.).

[0033] Figure 2 A block diagram of a device 200 having interconnected subsystems is depicted. In the illustrated example, the device 200 includes three different subsystems (i.e., a first subsystem 210, a second subsystem 220, and a third subsystem 230) that are coupled to each other (e.g., wired or wirelessly). Figure 1 Described in Figure 2 Examples of possible computer architectures for the subsystems included in (i.e., computing system 100). Figure 2 Three subsystems are shown in FIG. 2 , but device 200 may include more or fewer subsystems.

[0034] In some examples, some subsystems are not connected to another subsystem (e.g., the first subsystem 210 may be connected to the second subsystem 220 and the third subsystem 230, but the second subsystem 220 may not be connected to the third subsystem 230). In some examples, some subsystems are connected via one or more wires, while other subsystems are connected wirelessly. In some examples, one or more subsystems are wirelessly connected to one or more computing systems external to the device 200, such as a server system. In such examples, the subsystems may be configured to communicate wirelessly with one or more computing systems external to the device 200.

[0035] In some examples, device 200 includes a housing that fully or partially encloses subsystems 210-230. Examples of device 200 include household appliances (e.g., a refrigerator or air conditioning system), robots (e.g., a robotic arm or a robotic vacuum cleaner), vehicles, etc. In some examples, device 200 is configured to navigate device 200 in a physical environment (with or without direct user input).

[0036] In some examples, one or more subsystems of the device 200 are used to control, manage, and / or receive data from one or more other subsystems of the device 200 and / or one or more computing systems remote from the device 200. For example, the first subsystem 210 and the second subsystem 220 may each be a camera that is capturing images for use by the third subsystem 230 to make decisions. In some examples, at least a portion of the device 200 is used as a distributed computing system. For example, a task may be divided into different parts, with the first part being performed by the first subsystem 210 and the second part being performed by the second subsystem 220.

[0037] Attention is now directed to techniques for determining information about a physical environment using an example of a device having a camera that captures images of the physical environment. In some examples, the device determines that there is a lack of feature correlation between images from a current set of cameras, and selects a different set of cameras for use in achieving sufficient feature correlation. In other examples, the device determines that there is not enough information in the representation of the physical environment, and selects a new set of cameras for use in capturing enough information to add to the representation. It should be understood that more or fewer cameras (including a single camera) and other types of sensors are within the scope of the present disclosure and may benefit from the techniques described herein.

[0038] According to the techniques described herein, the device uses sensors to locate objects within a physical environment. Such positioning may include estimating (e.g., calculating) the depth (e.g., distance from the device) of an object (e.g., substantially the object or a portion (e.g., not all) of the object). Different sensors or combinations of sensors may be used to estimate depth, including cameras and activity range sensors (e.g., light or radio detection and ranging systems).

[0039] Estimating depth with a camera can use monocular images (e.g., a single camera sensor that captures still or sequential images) or stereo images (e.g., multiple camera sensors that capture still or sequential images). The following techniques for estimating depth can be used in any combination with each other.

[0040] A technique for estimating depth uses depth cues to identify the relative positions of different objects in a physical environment. Examples of depth cues include comparing the sizes of different objects (e.g., objects appear smaller when they are farther away), textures (e.g., the texture of an object is less recognizable (e.g., lower quality) when it is farther away), shadows (e.g., the shadow of an object may indicate that one part of the object is farther away than another part), linear perspective (e.g., objects converge to the horizon as they are farther away), motion parallax (e.g., farther objects appear to move slower than closer objects), binocular parallax (e.g., closer objects have greater parallax between two images than farther objects), and the apparent size of known objects (e.g., when identifying a type of object, the size of an object may be constrained by the typical size of objects of that type). Using one or more of these depth cues, the technique determines that an object is closer or farther away than another object.

[0041] Another technique for estimating depth identifies correspondences between two images (e.g., captured at two different times using two different image sensors (such as two different cameras) or a single image sensor (such as a single camera)). The technique then uses geometry (e.g., epipolar geometry) to estimate the depth of an area within the image. In one example, using correspondence to estimate depth includes identifying a feature in a first image (e.g., one or more pixels in an image, such as a corner, edge, or any distinguishing portion of an image) and identifying a corresponding feature in a second image (e.g., a feature in the second image that is determined to match a feature in the first image). In some examples, such features are identified independently and compared to each other. In other examples, features in the first image are used to identify corresponding features in the second image. After identifying the features, the difference (sometimes referred to as shift or disparity) between the locations of the features in their respective images is calculated. The depth of the feature is determined based on the disparity, the focal length of the camera that captured the image, and the distance between the cameras that captured the images. In another example, using correspondence to estimate depth includes dividing the image into a plurality of regions and identifying a plurality of features in each region. Different images are then compared to the features in each region to find corresponding features. If enough features are identified in enough regions (e.g., above a threshold), depth is calculated for the region based on a calibrated model of the relative geometric positions of the cameras using the calculated disparity as described above. Thus, in such examples, a different threshold number of corresponding features is required than for the region for which depth is being calculated.

[0042] Another technique for estimating depth uses a neural network to estimate depth. For example, a neural network takes an image as input and outputs a depth value based on a depth cue (such as the depth cue described above). In some examples, the neural network learns to regress depth from depth cues in an image via supervised learning by minimizing a loss function (e.g., a regression loss).

[0043] In the techniques described herein, one or more cameras may be calibrated before, during, or after performing certain steps. In some examples, calibration includes the process of determining specific camera parameters to determine the accurate relationship between a three-dimensional point in the physical environment and its corresponding two-dimensional projection (e.g., pixel) in the image. Such parameters eliminate distortion in the image, thereby establishing a relationship between image pixels and physical environment dimensions. Distortion can be captured by a distortion coefficient, the value of which reflects the amount of radial distortion (e.g., when light bends more at the edge of a lens than at the optical center of the lens) and tangential distortion (e.g., when the lens is not parallel to the image plane) in the image. The distortion coefficients include intrinsic parameters (e.g., focal length and optical center) and extrinsic parameters (e.g., rotation and translation of the camera) of each camera. In some examples, extrinsic parameters are used to transfer between world (e.g., physical environment) coordinates and camera coordinates, and intrinsic parameters are used to transfer between camera coordinates and pixel coordinates.

[0044] The techniques described herein use images captured by a camera to perform calibration. For example, calibration may include comparing an image captured by a first camera with an image captured by a second camera to identify differences in order to determine distortion coefficients. In such an example, the distortion coefficients are determined by comparing image features captured by cameras in a controlled environment for which the ground truth geometric locations of the features are known. In some examples, the camera used for calibration may or may not be included in a set of cameras used to determine information about a physical environment. For example, a camera not included in the set of cameras may be used to calibrate a camera in the set of cameras, such as to determine distortion coefficients for a camera in the set of cameras.

[0045] The above techniques rely on capturing one or more images of an object to determine the depth of the object. Therefore, some techniques described herein switch from using a first set of images to another set of images, such as images from different cameras that are oriented differently and / or located in a different position than at least some of the cameras used to capture the first set.

[0046] Figure 3 is a block diagram illustrating a device 300 for determining information about a physical environment according to some techniques described herein. Although primarily described with respect to calculating depth Figure 3, but it should be understood that similar techniques can be used for other determinations (e.g., identifying, classifying, or determining information about objects or other elements in a physical environment).

[0047] Device 300 includes multiple subsystems that are at least partially interconnected. For example, device 300 includes multiple cameras (i.e., camera 310) that are each connected (e.g., wired or wirelessly) to camera selector 320. It should be appreciated that one or more subsystems of device 300 may be combined or further decomposed into more subsystems. For example, a camera subsystem may include a camera selector and a depth processor, or a depth processor may include a camera selector.

[0048] In some examples, each of the cameras 310 is configured to capture an image and send the image to the camera selector 320 or store the image in a specific location specific to each camera. In some examples where the camera stores the image at a specific location, the camera notifies the camera selector 320 that the image has been stored (and optionally includes the location where the image has been stored, such as through a pointer to a memory location), and the camera selector 320 is configured to access the location where the image has been stored. In other examples where the camera stores the image at a specific location, the camera does not notify the camera selector 320 that the image has been stored, but rather the camera selector 320 requests access to the known location of one or more stored images. As used herein, "sending" an image from one subsystem to another subsystem may refer to actually sending the image to another subsystem or storing the image so that the other subsystem can access the image.

[0049] In some examples, cameras 310 correspond to two or more cameras configured to capture images of at least partially overlapping areas of a physical environment. Examples of configurations of cameras 310 are described in Figure 4A and Figure 4B This is exemplified in , which will be discussed further below.

[0050] As described above, device 300 includes camera selector 320. Camera selector 320 may be configured to identify one or more cameras from camera 310 for further processing, such as depth processing by depth processor 330. In some examples, camera selector 320 receives images from each of cameras 310 and determines which images to send to depth processor 330. In other examples, camera selector 320 receives images from only a subset (e.g., less than all) of cameras in camera 310 (where the subset is selected by camera selector 320 before or after camera selector 320 receives any images from cameras in camera 310), and sends all received images to depth processor 330. In some examples, camera selector 320 causes a camera to capture an image and then sends the captured image to depth processor 330. In some examples, camera selector 320 causes one or more cameras to stop capturing images and / or send images to camera selector 320.

[0051] like Figure 3 As shown, device 300 also includes a depth processor 330. Depth processor 330 may be configured to calculate the depth of a location within the physical environment based on images captured by one or more cameras selected by camera selector 320. In other examples, depth processor 330 is included in a device separate from device 300 (such as a remote server). Depth processor 330 may calculate depth using any of the techniques described herein (or any combination thereof). For example, depth processor 330 may (1) receive a first image and a second image, (2) identify a feature in the first image and a corresponding feature in the second image, and (3) calculate the depth of the feature using epipolar geometry.

[0052] In some examples, different computing systems (e.g., different systems on a chip executing camera selector 320 and / or depth processor 330) are assigned to receive images from different cameras. For example, a first computing system may be configured to receive images from a first camera and a second camera to determine depth information for a location using the images from the first camera and the second camera, and another computing system may be configured to receive images from other cameras to determine depth information using the images from the other cameras. In such an example, the images may be stored on a memory local to the respective computing systems to reduce the time to access pixel information and / or the requirement to send pixel information between computing systems. In another example, a first computing system is configured to receive images from a first camera and store the received images in a memory local to the first computing system, and a second computing system is configured to receive images from a second camera and store the received images in a memory local to the second computing system, wherein one of the two computing systems is configured to determine depth information using the images from the two cameras. In such an example, a computing system that does not determine depth information may send only a portion (e.g., less than all) of the image to a computing system that determines depth information to reduce the amount of data moved to different computing systems. The portion may correspond to a lower resolution image and / or to only a subset of the images required for the computation (e.g., some images are sent and some are not sent). In other examples, feature correlation is performed in object space such that an object is identified in a first image and an object is identified in a second image, and then the objects are compared to each other to determine correspondence between the images. Such a comparison would not require comparing individual pixels.

[0053] In some examples, the depth processor 330 uses the depth calculated for the location to generate a depth map for the physical environment. In such examples, the depth map includes the depth of different locations within the physical environment. The depth map can then be used by the device 300 to make decisions, such as how to navigate the physical environment. As used herein, a depth map is sometimes referred to as a representation of the physical environment. In some examples, the depth processor 330 uses other data detected by other types of sensors in conjunction with the image to generate a depth map. For example, an optical or radio detection and ranging system can be used to identify the depth of a featureless area and / or provide calibration for depth calculations. In such examples, the optical or radio detection and ranging system can capture data only for a specific area and / or at a specific resolution (e.g., a resolution lower than an image captured by a camera (such as, as described herein, a camera in ready mode)).

[0054] In some examples, device 300 generates a representation of the physical environment (e.g., a three-dimensional representation or object view) using the depth map and one or more images captured by camera 310. In such examples, the representation includes additional information about the physical environment (i.e., in addition to the depth), such as identification of objects within the physical environment, and any other information that will help device 300 make decisions about the physical environment. In some examples, other data detected by other types of sensors (e.g., light or radio detection and ranging systems) is used in combination with the images to generate the representation.

[0055] In some examples, the depth processor 330 cannot match a particular feature in the first image with a particular feature in the second image. In such examples, the depth of the particular feature cannot be calculated due to lack of feature correlation. In other examples, the depth processor 330 determines that there is a lack of feature correlation with respect to a subset of the images other than the particular feature. In such examples, the particular location cannot be located within the physical environment.

[0056] In some examples, the depth processor 330 sends a message to the camera selector 320 in response to determining the lack of feature correlation. The message may include an identification of the location that the lack of feature correlation is affecting, a confidence level that the lack of feature correlation has occurred, a depth map (or an update to the depth map if, for example, the depth map is managed by or stored in a memory local to the camera selector 320), a representation generated from the depth map (or an update to the representation if, for example, the representation is managed by or stored in a memory local to the camera selector 320), an indication that the depth map or the representation generated from the depth map has been updated, or any combination thereof. In other examples, the lack of feature correlation is reflected in the depth map and / or the representation generated from the depth map, which is accessible to the camera selector 320. In some examples, the camera selector 320 may be configured to operate at a fixed rate such that the camera selector 320 identifies one or more cameras from the camera 310 for further processing according to a fixed rate (e.g., every 100 milliseconds).

[0057] In response to receiving a message, determining that information related to the physical environment has been updated, or initiating an operation, the camera selector 320 may determine whether to switch and which one or more cameras to switch to in order to obtain sufficient information. This determination may be based on: (1) whether there is a lack of feature correlation, (2) whether the representation of the physical environment lacks depth information for the location, (3) whether there is no data about the location, (4) whether there is sufficient information to classify the object located at the location, (5) whether there is sufficient depth calculation (e.g., the current depth calculation has been determined to be incorrect or no depth calculation exists), (6) whether the object is determined to be in the line of sight of a particular camera such that objects behind the object are hidden, or (7) any combination thereof. In some examples, the camera selector 320 uses the representation to perform a geospatial search to determine an area in which information is needed. The geospatial search may include identifying a portion of the physical environment in which the device 300 is moving and determining which portions of the representation are relevant to the area. In some examples, the geospatial search may include a spatial decomposition of one or more portions of the physical environment and ranking them in terms of importance to the device 300 based on semantic knowledge of the physical environment, such as the location, velocity, and heading of the device 300 within the physical environment, the location and classification of other objects in the physical environment, and specific targets. In some examples, selection based on such criteria prioritizes high resolution in a small window in one direction at one time and prioritizes low resolution in a larger window at another time. In other examples, selection based on such criteria prioritizes depth perception between twenty and one hundred meters at one time and prioritizes depth resolution between five and twenty-five meters at another time.

[0058] In some examples, camera selector 320 identifies a portion (e.g., less than all) of the physical environment for which device 300 requires additional information, and sends an image corresponding to the portion to depth processor 330. In such examples, a partial image (e.g., not the entire image) may be sent to depth processor 330 to reduce the amount of data that depth processor 330 needs to process.

[0059] In some examples, camera selector 320 determines that depth information for the location is not needed for the current determination that needs to be made by device 300. For example, device 300 may be traveling at a speed that does not require the location for determination at this time. For another example, device 300 may use information other than images captured by a camera (e.g., a map received by device 300 or an optical or radio detection and ranging system) to identify sufficient information about the location.

[0060] In some examples, camera selector 320 determines to select a new camera group in response to identifying a problem with the current camera group. For example, the problem may include lens flare, stray light, occlusion, hardware failure, software failure, a location where the depth calculation is determined to be incorrect, an area of ​​the physical environment that is not adequately covered by the current camera group, etc. In some examples, device 300 continues to navigate the physical environment while attempting to resolve the problem using a different group of one or more cameras.

[0061] With more than two cameras, camera selector 320 can form a group of cameras between different cameras (e.g., a stereo pair consisting of two cameras) to capture a portion of the physical environment, and / or have redundant groups of cameras to capture the same portion of the physical environment (e.g., multiple stereo pairs covering at least partially overlapping areas). In some examples, having multiple cameras increases availability (e.g., in the event of failure or occlusion of individual cameras) and / or information about the physical environment.

[0062] In some examples, a first group of cameras in cameras 310 is established by default (e.g., predefined before device 300 begins executing an application). In such examples, the first group may be changed to a second group of cameras (e.g., the second group may or may not include cameras in the first group) to alleviate the problem determined by camera selector 320. In some examples, the first group and / or the second group are established based on the likelihood that the cameras included in the respective groups are able to capture images of a particular location in the physical environment.

[0063] In some examples, the camera selector 320 determines the best camera group from a plurality of different groups of cameras. In some examples, the best group consists of one or more cameras. In one example, the best camera group is selected based on a lookup table indicating the next group of cameras to be used (e.g., in a particular situation). In some examples, the lookup table may be populated based on the distance of the location from the device 300. For example, when the location is within a specific distance from the device 300, a first order of possible camera groups that prioritize short distances between cameras (e.g., small baselines) may be used, and when the location exceeds a specific distance from the device 300, a second order of possible camera groups that prioritize long distances between cameras (e.g., large baselines) may be used. In some examples, the lookup table is generated (e.g., established) before the camera 310 captures any images. In other examples, the lookup table is generated (e.g., established) based on images received by the camera 310, such as by learning how to best respond to problems encountered by a specific configuration of the camera on the device 300.

[0064] In some examples, the best camera group is selected based on information identified by camera selector 320, such as information indicating the context of device 300 (e.g., speed, acceleration, path, weather, time of day, etc.). In such examples, images used to select the best group can be captured while device 300 is moving.

[0065] In some examples, feature comparison operations occur simultaneously with respect to multiple different sets of cameras to determine which set of cameras to use by the depth processor 330. In some examples, different sets of cameras capture images at different rates and / or resolutions to reduce the computational cost of performing multiple feature comparison operations simultaneously. In such examples, some feature comparison operations are used for diagnostic operations while other feature comparison operations are used by the depth processor 330 to calculate depth, where the feature comparison operations used for the diagnostic operations are performed at a lower rate than the rate used by the depth processor 330.

[0066] In some examples, different groups of cameras alternate at a faster rate than required for the determination purpose. In such examples, a particular group of cameras captures images at a rate required for the determination purpose, and another group of cameras captures images between the times that the particular group of cameras captures images for use in determining when to switch to the other group of cameras.

[0067] In some examples, camera selector 320 determines whether the location for which device 300 needs depth information is close to or far from device 300. In such examples, camera selector 320 may cause one or more cameras to capture images at a lower resolution when the location is closer and at a higher resolution when the location is farther away.

[0068] After using the new group of one or more cameras, the camera selector 320 may determine whether the new group is able to determine sufficient information for the location. In some examples, when the new group is able to be used to determine sufficient information for the location, the device 300 may perform one or more operations based on the information for the location, such as navigating the device 300 in the physical environment. In some examples, when the new group cannot be used to determine sufficient information for the location, the camera selector 320 may select a different group of one or more cameras based on one or more techniques discussed above. In such examples, the camera selector 320 may continue to select a different group of one or more cameras until sufficient information is determined for the location or a threshold number of groups of cameras are tried. In some examples, when a threshold number of groups of one or more cameras are tried and information is still needed to make a determination, the device 300 may perform operations other than changing to a different group of one or more cameras to try to resolve the problem. Examples of operations include changing a navigation characteristic of the device 300 (e.g., changing from a first path to a second path) or an operating characteristic of the device 300 (e.g., reducing the speed of the device 300).

[0069] In some examples, camera selector 320 is configured to change the mode of a camera from camera 310, such as changing the camera from standby (e.g., off or in a lower power mode) to ready (e.g., on or in a mode capable of capturing images at a particular rate and / or resolution). In some examples, camera selector 320 predicts that the camera will be needed and changes the camera from standby to ready. In such examples, camera selector 320 may cause the camera to change modes so that the camera can be used by depth processor 330 without having to change modes when it is determined to be necessary. For example, camera selector 320 may cause the camera to transition to ready mode before determining a lack of feature correlation between current images or a lack of information in the representation of the physical environment.

[0070] In some examples, camera selector 320 determines to transition from the current set of cameras to a previous set of cameras, such as a default camera set. In such examples, the determination may be based on determining that sufficient information has been determined for the location or that the reason for the previous set of cameras' lack of feature relevance has been resolved (e.g., a predetermined amount of time has passed, it has been determined that images from the previous set of cameras have sufficient feature relevance, or the operating state of device 300 has changed (such as an operating state that may have caused the previous set of cameras to now have sufficient feature relevance)).

[0071] Figure 4A 4 is a block diagram of a camera array 400 illustrating three cameras. The camera array 400 includes a first camera 410, a second camera 420, and a third camera 430. In some examples, the camera array 400 is attached to a device (e.g., device 300) and is configured to capture images of a physical environment. In such examples, the field of view of each camera in the camera array 400 may at least partially overlap, so that all cameras can capture images of a specific area of ​​the physical environment. In some examples, the first camera 410, the second camera 420, and the third camera 430 are each oriented in a different direction. In other examples, at least two of the first camera 410, the second camera 420, and the third camera 430 are each oriented in the same direction.

[0072] In some examples, first camera 410 and second camera 420 are on a first axis (e.g., a horizontal axis) with each camera separated by a first distance. In such examples, third camera 430 may be offset from the other cameras and on a second axis different from the first axis. Third camera 430 may be below the other cameras (e.g., Figure 4A shown) or above ( Figure 4A4. (not illustrated in the figure). Placing the cameras at different axes may allow different cameras to capture the field of view at different angles. In some examples, the third camera 430 is at a second distance from the first camera 410 and at a third distance from the second camera 420. In such examples, the second distance and the third distance may be the same or different. In some examples, the second distance and / or the third distance may be the same or different from the first distance. Placing the cameras at different distances from each other may allow different groups of cameras to have different baselines to change how calculations are performed when processing images for information about the physical environment. In some examples, the third camera 430 is offset from the vertical axis associated with the first camera 410 and the vertical axis associated with the second camera 420, so that the third camera 430 is between the vertical axis associated with the first camera 410 and the vertical axis associated with the second camera 420. As described above, placing the cameras at different axes may allow different cameras to capture the field of view at different angles.

[0073] Figure 4B 4 is a block diagram of a camera array 440 illustrating four cameras. The camera array includes a first camera 450, a second camera 460, a third camera 470, and a fourth camera 480. In some examples, the camera array 440 is attached to a device (e.g., device 300) and is configured to capture images of a physical environment. In such examples, the field of view of each camera in the camera array 440 may overlap at least partially, so that all cameras can capture images of a specific area of ​​the physical environment. In some examples, the first camera 450, the second camera 460, the third camera 470, and the fourth camera 480 are each oriented in a different direction. In other examples, at least two of the first camera 450, the second camera 460, the third camera 470, and the fourth camera 480 are each oriented in the same direction (e.g., the first camera 450 and the second camera 460; the first camera 450 and the third camera 470; or the first camera 450 and the fourth camera 480).

[0074] In some examples, first camera 450 and second camera 460 are on a first axis (e.g., a horizontal axis) with each camera separated by a first distance. In such examples, third camera 470 and fourth camera 480 may be offset from the other cameras and on a second axis different from the first axis (e.g., a different horizontal axis that is parallel to another horizontal axis). Third camera 470 and / or fourth camera 480 may be below the other cameras ( Figure 4B exemplified below) or above ( Figure 4B4. As described above, having the cameras at different axes may allow different cameras to capture the field of view at different angles. In some examples, the third camera 470 and the fourth camera 480 have a second distance separating each camera. In such examples, the second distance may be the same as or different from the first distance. As described above, having the cameras at different distances from each other may allow different groups of cameras to have different baselines to change how calculations are performed when processing images for information about the physical environment. Having the camera groups at the same distance from each other may allow different groups of cameras to be easily switched between without changing the group's ability to capture objects at a specific distance that is not subject to orientation.

[0075] In some examples, the third camera 470 is a second distance from the first camera 450 and a third distance from the second camera 460. In such examples, the second distance and the third distance may be the same or different. In some examples, the second distance and / or the third distance may be the same or different from the first distance. In some examples, the third camera 470 is along a vertical axis associated with the first camera 450 and offset from but parallel to the vertical axis associated with the second camera 460, such that the third camera 470 is below the first camera 450 and diagonally to the second camera 460. Having the camera groups along the same axis as one another can allow different groups of cameras to be easily switched between without changing the group's ability to capture an object at a particular viewpoint.

[0076] In some examples, the fourth camera 480 is at a fourth distance from the second camera 460 and at a fifth distance from the third camera 470. In such examples, the fourth distance and the fifth distance may be the same or different. In some examples, the fourth distance and / or the fifth distance may be the same or different from the first distance, the second distance, or the third distance. In some examples, the fourth camera 480 is offset along a vertical axis associated with the second camera 460 and from a vertical axis associated with the first camera 450, so that the fourth camera 480 is below the second camera 460 and diagonally to the first camera 450. In some examples, small angular differences in the mounting positions of the cameras result in different effects on the cameras. In such examples, when one camera experiences a particular effect, the other camera is less likely to suffer the same problem, thereby making the system more robust.

[0077] Figure 5 is a flow chart illustrating a method 500 for calculating a depth of a location. Some operations in the method 500 may optionally be combined, the order of some operations may optionally be changed, and some operations may optionally be omitted.

[0078] In some examples, method 500 is performed at a computing system (e.g., computing system 100) that communicates with a camera (e.g., a camera in camera 310, first camera 410, or first camera 450). In some examples, the computing system and the camera are included in a device (e.g., device 200 or device 300). In some examples, the device includes one or more actuators and / or one or more sensors in addition to the camera. In some examples, the camera is connected to one or more processors of the device via at least one or more wires; in some examples, the camera is wirelessly connected to one or more processors of the device; in some examples, one or more processors are included in a component of the device that is separate from the camera; in some examples, one or more processors are included in the camera; in some examples, multiple processors of the device perform the method, wherein at least one step is performed by one or more processors on a first system on chip (i.e., SoC) and a second step is performed by a second SoC, and wherein the first SoC and the second SoC are distributed in different locations on the device, wherein the different locations are separated by at least 12 inches. In some examples, method 500 is performed while the device including the camera performs an operation (such as navigating in a physical environment).

[0079] At 510, method 500 includes receiving a first image (e.g., a representation of a physical environment having one or more color channels (e.g., a red channel, a green channel, and a blue channel) captured by a first camera (e.g., computing system 100, first subsystem 210, a camera in camera 310, first camera 410, or first camera 450).

[0080] At 520, method 500 includes receiving a second image captured by a second camera (e.g., computing system 100, first subsystem 210, a camera in camera 310, second camera 420, or second camera 460), wherein the second camera is different from the first camera, and wherein the first camera and the second camera are established as a first set of cameras (e.g., a camera pair) for calculating a depth of a location (e.g., a point location) (in some examples, the device includes the first camera and the second camera; in some examples, the first image is captured before or after the first set of cameras is established; in some examples, receiving the first image includes accessing a memory location using an expected location or a location identified in a message received from the first camera; in some examples, before receiving the first image, the device establishes the first set of cameras for calculating a depth of the location; in a In some examples, the first group of cameras does not include a third camera; in some examples, the first group of cameras includes one or more cameras in addition to the first camera and the second camera; in some examples, the first group of cameras is established by default (e.g., predefined before the device starts executing an application that requires depth calculations); in some examples, the first group of cameras is established based on the likelihood that the cameras included in the first group of cameras can capture an image of the location; in some examples, the second image is captured before or after the first group of cameras is established; in some examples, the second image is captured simultaneously with the first image (e.g., captured approximately simultaneously, such as indicating that the first camera and the second camera captured images simultaneously); in some examples, receiving the second image includes accessing a memory location using an expected location or a location identified in a message received from the second camera).

[0081] At 530, method 500 includes calculating a first depth of the location based on the first image and the second image in accordance with a determination that the first image and the second image have sufficient feature correlation for calculating the depth of the location (in some examples, when calculating the first depth based on the first image and the second image, there is no determination that the third image and the fourth image have sufficient feature correlation; in some examples, the device determines that the first image and the second image have sufficient feature correlation for calculating the depth of the location; in some examples, a second device different from the device determines that the first image and the second image have sufficient feature correlation for calculating the depth of the location; in some examples, determining that the first image and the second image have sufficient feature correlation The feature correlation includes determining that a threshold number of features can be identified in the two images; in some examples, determining that the first image and the second image have sufficient feature correlation includes determining that the depth calculated for the location is similar to an expected depth, such as compared to surrounding depth or depth calculated for the location at a different (e.g., previous) time; in some examples, calculating the depth based on the first image and the second image is also based on one or more images other than the first image and the second image; in some examples, calculating the depth includes identifying a feature correlation between at least two images (e.g., a corner, an edge, or a uniquely identifiable portion in at least two images) that exceeds a threshold, and then measuring a disparity between the at least two images). In some examples, method 500 also includes calculating the first depth of the location not based on an image captured by a third camera (in some examples, the feature correlation of the first depth is not based on an image captured by the third camera, although calibration of the first image and / or the second image is performed based on an image captured by the third camera).

[0082] At 540, method 500 includes determining that the first image and the second image do not have sufficient feature correlation (e.g., lack of feature correlation) based on a determination (in some examples, the device determines that the first image and the second image do not have sufficient feature correlation to calculate the depth of the location; in some examples, a second device different from the device determines that the first image and the second image do not have sufficient feature correlation to calculate the depth of the location; in some examples, determining that the first image and the second image do not have sufficient feature correlation includes determining that a threshold number of features cannot be identified in the two images; in some examples, determining that the first image and the second image do not have sufficient feature correlation includes determining that the depth calculated for the location is different from an expected depth, such as compared to surrounding depth or to a depth calculated for the location at a different (e.g., previous) time). In some examples, determining that the first image and the second image do not have sufficient feature correlation to calculate the depth of the location includes identifying a feature in the first image that is not included in the second image (e.g., an object or a portion of an object, such as an edge). In some examples, determining that the first image and the second image do not have sufficient feature correlation to calculate the depth of the location includes identifying a fault in the first image (in some examples, the fault is a lens flare or an electrical / software fault). In some examples, the determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location includes a determination that a threshold number of features in the first image are not included in the second image, and wherein the threshold number is at least two (in some examples, the determination includes a determination that one or more features in the first image are not included in the second image and one or more features in the second image are not included in the second image). In some examples, the determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location includes: dividing the first image into a plurality of portions; and based on the determination that a first portion of the plurality of portions does not have sufficient feature correlation: based on the determination that a second portion of the plurality of portions does not have sufficient feature correlation, determining that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location based on determining that a threshold number of the plurality of portions do not have sufficient feature correlation, wherein the first portion is different from the second portion (in some examples, the second portion does not overlap with the first portion).

[0083] At 550, method 500 includes calculating a second depth of the location based on the third image and the fourth image in accordance with a determination that the third image and the fourth image have sufficient feature correlation for calculating the depth of the location (in some examples, the device determines that the third image and the fourth image have sufficient feature correlation for calculating the depth of the location; in some examples, a second device different from the device determines that the third image and the fourth image have sufficient feature correlation for calculating the depth of the location; in some examples, determining that the third image and the fourth image have sufficient feature correlation includes determining that a threshold number of features can be identified in the two images; in some examples, determining that the third image and the fourth image have sufficient feature correlation includes determining that the depth calculated for the location similar to an expected depth, such as compared to ambient depth or a depth calculated for the location at a different (e.g., previous) time, wherein: the third image is captured by a third camera (in some examples, the third camera is inactive (e.g., low power mode, turned off, capturing images less frequently, or capturing images at a lower resolution) while the first camera pair is active; in some examples, the third camera is active (e.g., similar to the first camera and the second camera), but is not used to calculate the depth of the location), the third camera is different from the first camera, the third camera is different from the second camera, and the fourth image is captured by a fourth camera (in some examples, the fourth camera is the first camera or the second camera; in some examples, the fourth camera is different from the first camera The device includes a third camera and a fourth camera; in some examples, the third image is captured before or after the second group of cameras is established; in some examples, the third image is received by the device by accessing a memory location using an expected location or a location identified in a message received from the third camera; in some examples, before receiving the third image, the device establishes a second group of cameras for calculating the depth of the location; in some examples, the second group of cameras does not include the first camera and / or the second camera; in some examples, in addition to the third camera and the fourth camera, the second group of cameras also includes one or more cameras; in some examples, the second set of cameras is established based on the likelihood that the cameras included in the second set of cameras can capture images of the location; in some examples, the fourth image is captured before or after the second set of cameras is established; in some examples, the fourth image is captured simultaneously with the third image (e.g., captured approximately simultaneously, such as indicating that the third camera and the fourth camera captured the image simultaneously); in some examples, receiving the fourth image includes accessing a memory location using an expected location or a location identified in a message received from the fourth camera; in some examples, the second set of cameras is established after determining that the first image and the second image do not have sufficient feature correlation; in some examples, the second set of cameras is established in response to determining that a depth at the location is required;In some examples, the third image is captured before or after establishing the second group of cameras; in some examples, the third image is captured simultaneously with the first image and / or the second image (e.g., captured approximately simultaneously, such as indicating that all three cameras captured images simultaneously); in some examples, receiving the third image includes accessing a memory location using an expected location or a location identified in a message received from the third camera). In some examples, method 500 also includes calculating a second depth of the location further based on a determination to select a second camera group from a plurality of different groups of cameras (in some examples, the plurality of different groups of cameras do not include the first camera group). In some examples, the determination to select the second group of cameras is based on an image captured by the third camera when the third camera is in a lower power mode than the cameras in the first group of cameras (in some examples, the lower power mode is an off mode). In some examples, the determination to select the second group of cameras is based on a priority order of the camera groups, and wherein the priority order is established before receiving the first image. In some examples, when a determination is made that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location, the third camera is in a standby mode (e.g., a low power or off mode). In some examples, the image captured by the third camera is used to calibrate the feature correlation between the first image and the second image. In some examples, the third camera is the first camera. In some examples, calculating the second depth based on the third image and the fourth image is based on a determination that the first image and the second image do not have sufficient feature correlation (e.g., lack feature correlation) for calculating the depth of the location. ;

[0084] In some examples, method 500 also includes calculating a third depth of the location based on the fifth image and the sixth image based on a determination that the first image and the second image do not have sufficient feature correlation (e.g., lack feature correlation) for calculating the depth of the location, based on a determination that the third image and the fourth image do not have sufficient feature correlation (e.g., lack feature correlation) for calculating the depth of the location, and based on a determination that the fifth image and the sixth image have sufficient feature correlation for calculating the depth of the location (in some examples, the device determines that the fifth image and the sixth image have sufficient feature correlation for calculating the depth of the location; in some examples, a second device different from the device determines that the fifth image and the sixth image have sufficient feature correlation for calculating the depth of the location), wherein: the fifth image is captured by a fifth camera (in some examples, The fifth camera is inactive (e.g., low power mode, turned off, capturing images less frequently, or capturing images at a lower resolution) while the first group of cameras and / or the second group of cameras are active; in some examples, the fifth camera is active (e.g., similar to the first and second cameras, but not used to calculate the depth of the location), the fifth camera is different from each of the first, second, third, and fourth cameras, the sixth image is captured by the sixth camera (in some examples, the fourth camera is the first camera or the second camera; in some examples, the fourth camera is different from the first and second cameras), the sixth image is not captured by the fifth camera, and the fifth and sixth cameras are established as a third group of cameras for calculating the depth of the location (in some examples, the third group of cameras is different from the first and second groups of cameras).

[0085] In some examples, method 500 also includes calculating a fourth depth of the location based on the seventh image and the eighth image based on a determination that the first image and the second image do not have sufficient feature correlation (e.g., lack feature correlation) for calculating the depth of the location, and based on a determination that the seventh image and the eighth image have sufficient feature correlation for calculating the depth of the location (in some examples, the device determines that the seventh image and the eighth image have sufficient feature correlation for calculating the depth of the location; in some examples, a second device different from the device determines that the seventh image and the eighth image have sufficient feature correlation for calculating the depth of the location), wherein: the seventh image is captured by a seventh camera (in some examples, the seventh camera is inactive (e.g., low power mode, turned off, capturing images less frequently, or capturing images at a lower resolution) while the first set of camera pairs is active; in some examples, the seventh camera is active (e.g., similar to the first camera and the second camera), but is not used to calculate the depth of the location), and the eighth image is captured by the seventh camera (in some examples, the fourth camera is the first camera or the second camera; in some examples, the fourth camera is different from the first camera and the second camera).

[0086] In some examples, method 500 also includes calculating a depth of a location based on an image captured by the first camera and an image captured by the second camera based on a determination that a reason for the lack of feature correlation for the first group of cameras has been resolved (in some examples, the determination includes an amount of time that has passed; in some examples, the determination includes a determination based on an image (such as an image from a camera of the first group of cameras or an image from a camera not included in the first group of cameras) (in some examples, the first group of cameras is not used until it is determined that the reason for the lack of feature correlation for the first group of cameras has been resolved; in some examples, the calculation is for a new location; in some examples, the calculation is for the same location as before).

[0087] In some examples, the above techniques are performed in a system for calculating the depth of a location. In such examples, the system includes: a first camera, a second camera, a third camera, and a fourth camera, wherein the first camera and the second camera are on a first axis, and wherein the third camera and the fourth camera are on a second axis different from the first axis (in some examples, the first axis and the second axis are parallel; in some examples, the first camera and the third camera are on a third axis, the second camera and the fourth camera are on a fourth axis, and the third axis is parallel to the fourth axis; in some examples, the first axis and the third axis are perpendicular).

[0088] Note that the following description of method 600 (ie, Figure 6 The details of the process described in ) also apply in a similar manner to Figure 5 For example, method 500 may optionally include one or more of the characteristics of the various methods described below with reference to method 600. For example, calculating the second depth of a location according to method 500 may further include determining sufficient data for the location within the representation according to a representation (e.g., a three-dimensional representation).

[0089] Figure 6 is a flow chart illustrating a method 600 for obtaining sufficient data about a physical environment. Some operations in the method 600 may optionally be combined, the order of some operations may optionally be changed, and some operations may optionally be omitted.

[0090] In some examples, method 600 is performed at a computing system (e.g., computing system 100) that communicates with a camera (e.g., a camera in camera 310, first camera 410, or second camera 460). In some examples, the computing system and the camera are included in a device (e.g., device 200 or device 300). In some examples, the device includes one or more actuators and / or one or more sensors in addition to the camera. In some examples, the camera is connected to one or more processors of the device via at least one or more wires; in some examples, the camera is wirelessly connected to one or more processors of the device; in some examples, one or more processors are included in a component of the device that is separate from the camera; in some examples, one or more processors are included in the camera; in some examples, multiple processors of the device perform the method, wherein at least one step is performed by one or more processors on a first system on chip (i.e., SoC) and a second step is performed by a second SoC, and wherein the first SoC and the second SoC are distributed in different locations on the device, wherein the different locations are separated by at least 12 inches. In some examples, method 600 is performed while the device including the camera performs an operation (such as navigating in a physical environment).

[0091] At 610, method 600 includes receiving a representation (e.g., a world representation, a virtual representation, an object view representation, a three-dimensional representation) of a physical environment (in some examples, the representation is not an image), wherein the representation is generated based on a first set of one or more images (e.g., a representation of the physical environment having one or more color channels (e.g., a red channel, a green channel, and a blue channel)) captured by a first set of one or more cameras (e.g., camera 310, first camera 410, or first camera 450) (in some examples, the representation is generated based on one or more depth calculations, one or more lidar images, one or more depth maps, or any combination thereof; in some examples, the representation includes a representation of an object that has been identified as including one or more characteristics associated with a particular type of object; in some examples, the device includes the first set of one or more cameras). In some examples, the representation is generated based on a depth map (in some examples, the depth map includes information related to distances to surfaces on objects in the physical environment). In some examples, the depth map is generated using images captured by a set of one or more cameras (in some examples, the depth map is generated using lidar data; in some examples, the depth map is generated by feature correlation between images).

[0092] At 620 and 630, method 600 includes, based on a determination that the representation does not include sufficient data for a location within the representation (in some examples, the determination that the representation does not include sufficient data for a location within the representation includes the representation not including (1) any data about the location, (2) sufficient information to classify an object located at the location, (3) a depth calculation for the location, (4) a sufficient depth calculation (e.g., a current depth calculation has been determined to be incorrect), or any combination thereof), and based on a determination that the second set of one or more cameras is capable of capturing images (e.g., one or more images from each of the one or more cameras) to obtain sufficient data for the location, sending instructions to use the second set of one or more cameras (in some examples, the instructions cause the use of the second set of one or more cameras relative to the location of the representation) one or more images in the corresponding field of view; in some examples, based on the determination that the second group of one or more cameras is able to capture images to obtain sufficient data for the location, abandoning initiating an operation (e.g., the operation mentioned in paragraph

[0089] , such as navigating the physical environment) before updating the representation based on the image from the second group of one or more cameras; in some examples, the second group of one or more cameras includes multiple cameras, wherein the instruction to capture an image is sent to each camera), wherein the second group of one or more cameras is different from the first group of one or more cameras (in some examples, the second group of one or more cameras includes at least one camera not included in the first group of one or more cameras; in some examples, the second group of one or more cameras includes at least one camera included in the first group of one or more cameras). In some examples, the determination that the representation does not include sufficient data for the location within the representation includes performing a geospatial search of the representation (in some examples, the geospatial search includes identifying an area of ​​the physical environment that the device is moving toward; in some examples, the determination that the representation does not include sufficient data for the location within the representation includes a determination that the representation does not include depth information for the location). In some examples, the second group of one or more cameras consists of two cameras (in some examples, the first group of one or more cameras consists of two cameras). In some examples, when it is determined that the representation does not include sufficient data for a location within the representation, a camera in the second set of one or more cameras is placed in a standby mode (in some examples, the standby mode is a low power mode or is turned off).

[0093] At 640, method 600 includes, based on a determination that the representation includes sufficient data for a location within the representation, forgoing sending instructions to use a second set of one or more cameras (and in some examples, initiating navigation based on the representation in addition to forgoing sending the instructions).

[0094] In some examples, method 600 also includes sending navigation instructions after sending instructions to use the second set of one or more cameras (in some examples, the navigation instructions identify a path for the device to take; in some examples, the navigation instructions identify driving characteristics (e.g., speed) of the device based on an updated representation of the physical environment, wherein the updated representation is generated based on one or more images captured by the second set of one or more cameras (in some examples, the updated representation is generated based on images from the first set of one or more cameras); and, based on a determination that the representation includes sufficient data for a location within the representation, sending instructions for navigation based on the representation of the physical environment (i.e., in this branch, instructions will not be sent to the second set of one or more cameras to capture one or more images).

[0095] In some examples, method 600 also includes, before making a determination that the representation does not include sufficient data for a location within the representation: based on a determination that a camera in the second set of one or more cameras is in a standby mode (in some examples, the standby mode is a low power mode or off); based on a determination that the second set of one or more cameras will need to update the representation (in some examples, the determination that the second set of one or more cameras will need to update the representation is based on a determination that the physical environment is (1) crowded, (2) having particular weather conditions, or (3) about to change due to navigation of the device), sending instructions to change the cameras to a second mode different from the standby mode (e.g., ready, active, or higher power mode).

[0096] In some examples, method 600 also includes, based on a determination that the representation does not include sufficient data for a location within the representation, forgoing sending instructions to use a second set of one or more cameras based on a determination based on a current navigation context (in some examples, the current navigation context is travel speed or direction of travel).

[0097] In some examples, the instruction includes a request to capture one or more images in a first mode (e.g., a higher power mode or an active mode), and wherein method 600 further includes: at the device, prior to sending the instruction, sending a second instruction to capture one or more images using a second set of one or more cameras in a second mode different from the first mode (e.g., a lower power mode, such as capturing images at a lower resolution and / or capturing images less frequently), wherein the one or more images captured in the second mode are used to determine to send the instruction.

[0098] In some examples, method 600 also includes sending instructions to change movement characteristics (e.g., reduce speed) based on a determination that an updated representation of the physical environment does not include sufficient data for a location within the updated representation, wherein the updated representation is generated based on one or more images captured by a second set of one or more cameras, and wherein the location within the updated representation corresponds to a location within the representation (in some examples, the location within the representation is the same as the location within the updated representation).

[0099] Note that the above description of method 500 (ie, Figure 5 The details of the process described in ) also apply in a similar manner to Figure 6 For example, the method 600 may optionally include one or more features of the various methods described above with reference to the method 500. For example, the depth calculated in the method 500 may be data included in the representation.

[0100] For the purpose of explanation, the preceding description is described by reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the invention to the disclosed precise form. According to the above teachings, many modifications and variations are possible. These embodiments are selected and described in order to best explain the principles of these technologies and their practical applications. Others skilled in the art can thus best utilize these technologies and various embodiments with various modifications suitable for the intended specific use.

[0101] Although the disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of the disclosure and examples defined by the claims.

[0102] As described above, one aspect of the present technology is to collect and use data available from various sources to improve the determination of information about the physical environment. The present disclosure contemplates that, in some examples, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person and / or a specific location. Such personal information data may include an image of a person, an image of data related to a person, an image of a location, or any other identifying or personal information.

[0103] The disclosure contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other uses of such personal information data will comply with sound privacy policies and / or privacy measures. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easy for users to access and should be updated as changes in the collection and / or use of data occur. Personal information from users should be collected for legal and reasonable entity purposes and should not be shared or sold outside of these legal purposes. In addition, policies and practices should be adapted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including considerations of special jurisdictions. Therefore, different privacy measures may be maintained for different types of personal data in each country.

[0104] Furthermore, the intention of this disclosure is that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting the collection of data and deleting data once it is no longer needed.

Claims

1. A method for obtaining sufficient data to make a decision about a physical environment, the method comprising: At the device: receiving a representation of a physical environment, wherein the representation is generated based on a first set of one or more images captured by a first set of one or more cameras; in response to a determination that the representation does not include sufficient data for a location within the representation and a second set of one or more cameras is capable of capturing images to obtain sufficient data for the location, sending instructions to use the second set of one or more cameras, wherein the second set of one or more cameras is different from the first set of one or more cameras; as well as Based on a determination that the representation includes sufficient data for the location within the representation, sending instructions to use the second set of one or more cameras is foregone.

2. The method according to claim 1, further comprising: At the device: After sending the instructions to use the second set of one or more cameras, sending instructions to navigate based on an updated representation of the physical environment, wherein the updated representation is generated based on one or more images captured by the second set of one or more cameras; and Based on a determination that the representation includes sufficient data for the location within the representation, instructions are sent to navigate based on the representation of the physical environment.

3. The method according to any one of claims 1 to 2, wherein the representation is generated based on a depth map.

4. The method of claim 3, wherein the depth map is generated using images captured by a set of one or more cameras.

5. The method according to any one of claims 1 to 4, wherein the second set of one or more cameras consists of two cameras.

6. The method of any one of claims 1 to 5, wherein the determination that the representation does not include sufficient data for a location within the representation comprises performing a geospatial search of the representation.

7. The method of any one of claims 1 to 6, wherein when the determination is made that the representation does not include sufficient data for a location within the representation, a camera in the second set of one or more cameras is in a standby mode.

8. The method according to any one of claims 1 to 7, further comprising: At the device: Prior to making the determination that the representation does not include sufficient data for a location within the representation: Based on a determination that a camera in the second set of one or more cameras is in a standby mode and that the second set of one or more cameras will be needed to update the representation, an instruction is sent to change the camera to a second mode different from the standby mode.

9. The method according to any one of claims 1 to 8, further comprising: At the device: Based on the determination that the representation does not include sufficient data for the location within the representation, and based on a determination based on a current navigation context, sending an instruction to use the second set of one or more cameras is foregone.

10. The method of any one of claims 1 to 9, wherein the instruction comprises a request to capture the one or more images in a first mode, and wherein the method further comprises: At the device: Before sending the instruction, a second instruction is sent to capture one or more images in a second mode different from the first mode using the second group of one or more cameras, wherein the one or more images captured in the second mode are used to determine sending the instruction.

11. The method according to any one of claims 1 to 10, further comprising: At the device: Based on a determination that an updated representation of the physical environment does not include sufficient data for a location within the updated representation, sending an instruction to change a movement characteristic, wherein the updated representation is generated based on the one or more images captured by the second set of one or more cameras, and wherein the location within the updated representation corresponds to the location within the representation.

12. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a device, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 11.

13. A device, comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 11.

14. A device, comprising: Device for carrying out the method according to any one of claims 1 to 11.

15. A computer program product, the computer program product comprising one or more programs configured to be executed by one or more processors of a device, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 11.

16. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a device, the one or more programs comprising instructions for: receiving a representation of a physical environment, wherein the representation is generated based on a first set of one or more images captured by a first set of one or more cameras; Based on a determination that the representation does not include sufficient data for a location within the representation and a second set of one or more cameras is capable of capturing images to obtain sufficient data for the location, sending instructions to use the second set of one or more cameras, wherein the second set of one or more cameras is different from the first set of one or more cameras; and Based on a determination that the representation includes sufficient data for the location within the representation, sending instructions to use the second set of one or more cameras is foregone.

17. A device, comprising: First Camera; a second camera, the second camera being different from the first camera; a third camera, the third camera being different from the first camera and the second camera; a fourth camera different from the third camera, the second camera, and the first camera, wherein the first camera and the second camera are on a first axis, and wherein the third camera and the fourth camera are on a second axis different from the first axis; one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: receiving a representation of a physical environment, wherein the representation is generated based on a first set of one or more images captured by a first set of one or more cameras, and wherein the first set includes the first camera and the second camera; Based on a determination that the representation does not include sufficient data for a location within the representation and that a second set of one or more cameras is capable of capturing images to obtain sufficient data for the location, sending instructions to use the second set of one or more cameras, wherein the second set of one or more cameras is different than the first set of one or more cameras, and wherein the second set includes the third camera and the fourth camera; and Based on a determination that the representation includes sufficient data for the location within the representation, sending instructions to use the second set of one or more cameras is foregone.

18. A device, comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: receiving a representation of a physical environment, wherein the representation is generated based on a first set of one or more images captured by a first set of one or more cameras; in response to a determination that the representation does not include sufficient data for a location within the representation and a second set of one or more cameras is capable of capturing images to obtain sufficient data for the location, sending instructions to use the second set of one or more cameras, wherein the second set of one or more cameras is different from the first set of one or more cameras; as well as Based on a determination that the representation includes sufficient data for the location within the representation, sending instructions to use the second set of one or more cameras is foregone.

19. A device, comprising: receiving a representation of a physical environment, wherein the representation is generated based on a first set of one or more images captured by a first set of one or more cameras; in response to a determination that the representation does not include sufficient data for a location within the representation and a second set of one or more cameras is capable of capturing images to obtain sufficient data for the location, sending instructions to use the second set of one or more cameras, wherein the second set of one or more cameras is different from the first set of one or more cameras; as well as Based on a determination that the representation includes sufficient data for the location within the representation, sending instructions to use the second set of one or more cameras is foregone.

20. A computer program product, the computer program product comprising one or more programs configured to be executed by one or more processors of a device, the one or more programs comprising instructions for: receiving a representation of a physical environment, wherein the representation is generated based on a first set of one or more images captured by a first set of one or more cameras; Based on a determination that the representation does not include sufficient data for a location within the representation and a second set of one or more cameras is capable of capturing images to obtain sufficient data for the location, sending instructions to use the second set of one or more cameras, wherein the second set of one or more cameras is different from the first set of one or more cameras; and Based on a determination that the representation includes sufficient data for the location within the representation, sending instructions to use the second set of one or more cameras is foregone.

21. A method for obtaining sufficient data to make decisions about a physical environment, the method comprising: At the device: receiving a first image captured by a first camera; receiving a second image captured by a second camera, wherein the second camera is different from the first camera, and wherein the first camera and the second camera are established as a first set of cameras for calculating a depth of a location; calculating a first depth for the location based on the first image and the second image based on a determination that the first image and the second image have sufficient feature correlation for calculating the depth for the location; as well as Based on a determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location and that the third image and the fourth image have sufficient feature correlation for calculating the depth of the location, calculating a second depth for the location based on the third image and the fourth image, wherein: The third image is captured by a third camera, The third camera is different from the first camera, The third camera is different from the second camera, The fourth image is captured by a fourth camera, The fourth camera is different from the third camera, and The third camera and the fourth camera are established as a second group of cameras for calculating the depth of the location.

22. The method of claim 21, wherein calculating the first depth of the location is not based on images captured by the third camera.

23. The method according to any one of claims 21 to 22, further comprising: At the device: Based on a determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location, the third image and the fourth image do not have sufficient feature correlation for calculating the depth of the location, and the fifth image and the sixth image have sufficient feature correlation for calculating the depth of the location, a third depth of the location is calculated based on the fifth image and the sixth image, wherein: The fifth image is captured by a fifth camera, the fifth camera is different from each of the first camera, the second camera, the third camera, and the fourth camera, The sixth image is captured by a sixth camera, The sixth image is not captured by the fifth camera, and The fifth camera and the sixth camera are established as a third group of cameras for calculating the depth of the location.

24. The method according to any one of claims 21 to 23, further comprising: At the device: Based on a determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location and that a seventh image and an eighth image have sufficient feature correlation for calculating the depth of the location, a fourth depth of the location is calculated based on the seventh image and the eighth image, wherein the seventh image is captured by a seventh camera, and wherein the eighth image is captured by the seventh camera.

25. The method of any one of claims 21 to 24, wherein calculating the second depth of the location is further based on a determination of selecting the second group of cameras from a plurality of different groups of cameras.

26. The method of claim 25, wherein the determination to select the second set of cameras is based on images captured by the third camera while the third camera is in a lower power mode than cameras in the first set of cameras.

27. The method of any one of claims 25 to 26, wherein the determination to select the second group of cameras is based on a priority order of camera groups, and wherein the priority order is established prior to receiving the first image.

28. The method according to any one of claims 21 to 27, further comprising: At the device: Based on a determination that a cause for the lack of feature correlation of the first set of cameras has been resolved, a depth of a location is calculated based on an image captured by the first camera and an image captured by the second camera.

29. The method of any one of claims 21 to 28, wherein the determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location comprises identifying features in the first image that are not included in the second image.

30. The method of any one of claims 21 to 29, wherein the determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location comprises identifying a fault in the first image.

31. A method according to any one of claims 21 to 30, wherein the determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location includes a determination that a threshold number of features in the first image are not included in the second image, and wherein the threshold number is at least two.

32. The method of any one of claims 21 to 31, wherein the determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location comprises: dividing the first image into a plurality of parts; as well as Based on a determination that a first portion of the multiple portions does not have sufficient feature correlation and a second portion of the multiple portions does not have sufficient feature correlation, it is determined that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location based on determining that a threshold number of the multiple portions do not have sufficient feature correlation, wherein the first portion is different from the second portion.

33. The method of any one of claims 21 to 32, wherein when the determination is made that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location, the third camera is in a standby mode.

34. The method of any one of claims 21 to 33, wherein images captured by the third camera are used to calibrate feature correlations between the first image and the second image.

35. The method of any one of claims 21 to 34, wherein the third camera is the first camera.

36. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a device, the one or more programs comprising instructions for performing the method according to any one of claims 21 to 35.

37. A device, comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 21 to 35.

38. A device, comprising: Device for carrying out the method according to any one of claims 21 to 35.

39. A computer program product comprising one or more programs configured to be executed by one or more processors of a device, the one or more programs comprising instructions for performing the method according to any one of claims 21 to 35.

40. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a device, the one or more programs comprising instructions for: receiving a first image captured by a first camera; receiving a second image captured by a second camera, wherein the second camera is different from the first camera, and wherein the first camera and the second camera are established as a first set of cameras for calculating a depth of a location; calculating a first depth for the location based on the first image and the second image based on a determination that the first image and the second image have sufficient feature correlation for calculating the depth for the location; as well as Based on a determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location and that the third image and the fourth image have sufficient feature correlation for calculating the depth of the location, calculating a second depth for the location based on the third image and the fourth image, wherein: The third image is captured by a third camera, The third camera is different from the first camera, The third camera is different from the second camera, The fourth image is captured by a fourth camera, The fourth camera is different from the third camera, and The third camera and the fourth camera are established as a second group of cameras for calculating the depth of the location.

41. A device, comprising: First Camera; a second camera, the second camera being different from the first camera; a third camera, the third camera being different from the first camera and the second camera; a fourth camera different from the third camera, the second camera, and the first camera, wherein the first camera and the second camera are on a first axis, and wherein the third camera and the fourth camera are on a second axis different from the first axis; one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: receiving a first image captured by the first camera; receiving a second image captured by the second camera, and wherein the first camera and the second camera are established as a first set of cameras for calculating a depth of a location; calculating a first depth for the location based on the first image and the second image based on a determination that the first image and the second image have sufficient feature correlation for calculating the depth for the location; as well as Based on a determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location and that the third image and the fourth image have sufficient feature correlation for calculating the depth of the location, calculating a second depth for the location based on the third image and the fourth image, wherein: The third image is captured by the third camera, The fourth image is captured by the fourth camera, and The third camera and the fourth camera are established as a second group of cameras for calculating the depth of the location.

42. A device, comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: receiving a first image captured by a first camera; receiving a second image captured by a second camera, wherein the second camera is different from the first camera, and wherein the first camera and the second camera are established as a first set of cameras for calculating a depth of a location; calculating a first depth for the location based on the first image and the second image based on a determination that the first image and the second image have sufficient feature correlation for calculating the depth for the location; as well as Based on a determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location and that the third image and the fourth image have sufficient feature correlation for calculating the depth of the location, calculating a second depth for the location based on the third image and the fourth image, wherein: The third image is captured by a third camera, The third camera is different from the first camera, The third camera is different from the second camera, The fourth image is captured by a fourth camera, The fourth camera is different from the third camera, and The third camera and the fourth camera are established as a second group of cameras for calculating the depth of the location.

43. A device, comprising: means for receiving a first image captured by a first camera; means for receiving a second image captured by a second camera, wherein the second camera is different from the first camera, and wherein the first camera and the second camera are established as a first set of cameras for calculating a depth of a location; means for calculating a first depth of the location based on the first image and the second image based on a determination that the first image and the second image have sufficient feature correlation for calculating the depth of the location; as well as Based on a determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location and the third image and the fourth image have sufficient feature correlation for calculating the depth of the location, means for calculating a second depth of the location based on the third image and the fourth image, wherein: The third image is captured by a third camera, The third camera is different from the first camera, The third camera is different from the second camera, The fourth image is captured by a fourth camera, The fourth camera is different from the third camera, and The third camera and the fourth camera are established as a second group of cameras for calculating the depth of the location.

44. A computer program product, the computer program product comprising one or more programs configured to be executed by one or more processors of a device, the one or more programs comprising instructions for: receiving a first image captured by a first camera; receiving a second image captured by a second camera, wherein the second camera is different from the first camera, and wherein the first camera and the second camera are established as a first set of cameras for calculating a depth of a location; calculating a first depth for the location based on the first image and the second image based on a determination that the first image and the second image have sufficient feature correlation for calculating the depth for the location; as well as Based on a determination that the first image and the second image do not have sufficient feature correlation for calculating the depth of the location and that the third image and the fourth image have sufficient feature correlation for calculating the depth of the location, calculating a second depth for the location based on the third image and the fourth image, wherein: The third image is captured by a third camera, The third camera is different from the first camera, The third camera is different from the second camera, The fourth image is captured by a fourth camera, The fourth camera is different from the third camera, and The third camera and the fourth camera are established as a second group of cameras for calculating the depth of the location.

Images