Managing representations of environments

Abstract

The present disclosure generally relates to managing the display of content among electronic devices in an environment.

Description

MANAGING REPRESENTATIONS OF ENVIRONMENTSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims priority to U.S. Provisional Patent Application Serial No. 63 / 634,261, entitled “MANAGING REPRESENTATIONS OF ENVIRONMENTS,” filed April 15, 2024, and to U.S. Provisional Patent Application Serial No. 63 / 540,034, entitled “MANAGING REPRESENTATIONS OF ENVIRONMENTS,” filed September 22, 2023, which are hereby incorporated by reference in their entireties for all purposes.BACKGROUND

[0002] Electronic devices are becoming more and more ubiquitous in the home, but configuration and automation of electronic devices across devices, device types, manufacturers, and / or platforms typically require individual and / or manual setup and control.SUMMARY

[0003] Improved techniques for managing a representation of an environment are desired. Current techniques for managing a representation of an environment may require users to assign a position to an electronic device, the position maintained until a user manually changes the position. More effective and / or efficient techniques for managing a representation of an environment using an example of an area model and different accessory devices are possible.

[0004] Some techniques are described herein for updating an area model of an environment using images from a sensor that has been newly added to the environment. In some examples, images from the newly added sensor is used to update portions of the area model that were previously updated by images from other sensors. For example, the newly added sensor might have visibility to new locations of the environment that were not previously included in the area model. In such an example, images from the newly added sensor can be used to update the area model to include information corresponding to the new locations. In some examples, the newly added sensor updates portions of the area model that were last updated by another sensor based on whether the other sensor is still activelyupdating the area model and / or whether the newly added sensor is determined to have a better view than the other sensor.

[0005] Other techniques are described herein for correlating multiple images from a single sensor to an area model of an environment to selectively update the area model. In some examples, the single sensor can be a mobile camera that captures images to update portions of the area model that were previously updated using images from a stationary sensor (e.g., that may no longer be active in the environment). The correlation can provide a position of the multiple images relative to the area model so as to identify whether a portion of the area model needs to be updated. In some examples, no correlation is required for images from the stationary sensor. In such examples, the mobile sensor, though requiring correlation, is able to provide different perspectives of portions of the environment.

[0006] In some examples, a method that is performed at an electronic device is described. In some examples, the method comprises: receiving, via a first sensor, a first image of a physical environment; updating, using the received first image, a first portion of a representation of the physical environment; receiving, via a second sensor different from the first sensor, a second image of the physical environment; after updating, using the received first image, the first portion of the representation of the physical environment and in accordance with a determination that a first set of one or more criteria is satisfied, updating, using the second image, the first portion of the representation of the physical environment; and causing performance of an operation on an electronic device located in the physical environment, using the representation of the physical environment, responsive to a sensed event in the first portion of the physical environment.

[0007] In some examples, a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device is described. In some examples, the one or more programs includes instructions for: receiving, via a first sensor, a first image of a physical environment; updating, using the received first image, a first portion of a representation of the physical environment; receiving, via a second sensor different from the first sensor, a second image of the physical environment; after updating, using the received first image, the first portion of the representation of the physical environment and in accordance with a determination that a first set of one or more criteria is satisfied, updating, using the second image, the first portion of the representation of the physical environment; and causing performance of an operation onan electronic device located in the physical environment, using the representation of the physical environment, responsive to a sensed event in the first portion of the physical environment.

[0008] In some examples, a transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device is described. In some examples, the one or more programs includes instructions for: receiving, via a first sensor, a first image of a physical environment; updating, using the received first image, a first portion of a representation of the physical environment; receiving, via a second sensor different from the first sensor, a second image of the physical environment; after updating, using the received first image, the first portion of the representation of the physical environment and in accordance with a determination that a first set of one or more criteria is satisfied, updating, using the second image, the first portion of the representation of the physical environment; and causing performance of an operation on an electronic device located in the physical environment, using the representation of the physical environment, responsive to a sensed event in the first portion of the physical environment.

[0009] In some examples, an electronic device is described. In some examples, the electronic device comprises one or more processors and memory storing one or more programs configured to be executed by the one or more processors. In some examples, the one or more programs includes instructions for: receiving, via a first sensor, a first image of a physical environment; updating, using the received first image, a first portion of a representation of the physical environment; receiving, via a second sensor different from the first sensor, a second image of the physical environment; after updating, using the received first image, the first portion of the representation of the physical environment and in accordance with a determination that a first set of one or more criteria is satisfied, updating, using the second image, the first portion of the representation of the physical environment; and causing performance of an operation on an electronic device located in the physical environment, using the representation of the physical environment, responsive to a sensed event in the first portion of the physical environment.

[0010] In some examples, an electronic device is described. In some examples, the electronic device comprises means for performing each of the following steps: receiving, via a first sensor, a first image of a physical environment; updating, using the received firstimage, a first portion of a representation of the physical environment; receiving, via a second sensor different from the first sensor, a second image of the physical environment; after updating, using the received first image, the first portion of the representation of the physical environment and in accordance with a determination that a first set of one or more criteria is satisfied, updating, using the second image, the first portion of the representation of the physical environment; and causing performance of an operation on an electronic device located in the physical environment, using the representation of the physical environment, responsive to a sensed event in the first portion of the physical environment.

[0011] In some examples, a computer program product is described. In some examples, the computer program product comprises one or more programs configured to be executed by one or more processors of an electronic device. In some examples, the one or more programs include instructions for: receiving, via a first sensor, a first image of a physical environment; updating, using the received first image, a first portion of a representation of the physical environment; receiving, via a second sensor different from the first sensor, a second image of the physical environment; after updating, using the received first image, the first portion of the representation of the physical environment and in accordance with a determination that a first set of one or more criteria is satisfied, updating, using the second image, the first portion of the representation of the physical environment; and causing performance of an operation on an electronic device located in the physical environment, using the representation of the physical environment, responsive to a sensed event in the first portion of the physical environment..

[0012] In some examples, a method that is performed at an electronic device in communication with a first sensor is described. In some examples, the method comprises: receiving, via the first sensor, a first image of a physical environment; receiving, via the first sensor, a second image of the physical environment, wherein the second image is different from the first image; after receiving the first image and the second image, updating a first portion of a representation of the physical environment by correlating the first image and the second image, wherein, before updating the first portion, the first portion is based on an image received from a second sensor different from the first sensor, and wherein the representation of the physical environment includes an identification of an object within the physical environment.

[0013] In some examples, a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device in communication with a first sensor is described. In some examples, the one or more programs includes instructions for: receiving, via the first sensor, a first image of a physical environment; receiving, via the first sensor, a second image of the physical environment, wherein the second image is different from the first image; after receiving the first image and the second image, updating a first portion of a representation of the physical environment by correlating the first image and the second image, wherein, before updating the first portion, the first portion is based on an image received from a second sensor different from the first sensor, and wherein the representation of the physical environment includes an identification of an object within the physical environment.

[0014] In some examples, a transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device in communication with a first sensor is described. In some examples, the one or more programs includes instructions for: receiving, via the first sensor, a first image of a physical environment; receiving, via the first sensor, a second image of the physical environment, wherein the second image is different from the first image; after receiving the first image and the second image, updating a first portion of a representation of the physical environment by correlating the first image and the second image, wherein, before updating the first portion, the first portion is based on an image received from a second sensor different from the first sensor, and wherein the representation of the physical environment includes an identification of an object within the physical environment.

[0015] In some examples, an electronic device in communication with a first sensor is described. In some examples, the electronic device comprises one or more processors and memory storing one or more programs configured to be executed by the one or more processors. In some examples, the one or more programs includes instructions for: receiving, via the first sensor, a first image of a physical environment; receiving, via the first sensor, a second image of the physical environment, wherein the second image is different from the first image; after receiving the first image and the second image, updating a first portion of a representation of the physical environment by correlating the first image and the second image, wherein, before updating the first portion, the first portion is based on an image received from a second sensor different from the first sensor, and wherein the representationof the physical environment includes an identification of an object within the physical environment.

[0016] In some examples, an electronic device in communication with a first sensor is described. In some examples, the electronic device comprises means for performing each of the following steps: receiving, via the first sensor, a first image of a physical environment; receiving, via the first sensor, a second image of the physical environment, wherein the second image is different from the first image; after receiving the first image and the second image, updating a first portion of a representation of the physical environment by correlating the first image and the second image, wherein, before updating the first portion, the first portion is based on an image received from a second sensor different from the first sensor, and wherein the representation of the physical environment includes an identification of an object within the physical environment.

[0017] In some examples, a computer program product is described. In some examples, the computer program product comprises one or more programs configured to be executed by one or more processors of an electronic device in communication with a first sensor. In some examples, the one or more programs include instructions for: receiving, via the first sensor, a first image of a physical environment; receiving, via the first sensor, a second image of the physical environment, wherein the second image is different from the first image; after receiving the first image and the second image, updating a first portion of a representation of the physical environment by correlating the first image and the second image, wherein, before updating the first portion, the first portion is based on an image received from a second sensor different from the first sensor, and wherein the representation of the physical environment includes a mapping of an object within the physical environment.

[0018] In some examples, a method that is performed at a first device that is in communication with a display generation component is described. In some examples, the method comprises: receiving, from a second device and a third device, media data, wherein the third device is different from the first device and the second device; and after receiving the media data: in accordance with a determination that a common object within the media data has a first orientation relative to the second device and the third device, displaying, via the display generation component, content in a first manner; and in accordance with a determination that the common object within the media data has a second orientation, different from the first orientation, relative to the second device and the third device,displaying, via the display generation component, the content in a second manner different from the first manner.

[0019] In some examples, a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first device that is in communication with a display generation component is described. In some examples, the one or more programs includes instructions for: receiving, from a second device and a third device, media data, wherein the third device is different from the first device and the second device; and after receiving the media data: in accordance with a determination that a common object within the media data has a first orientation relative to the second device and the third device, displaying, via the display generation component, content in a first manner; and in accordance with a determination that the common object within the media data has a second orientation, different from the first orientation, relative to the second device and the third device, displaying, via the display generation component, the content in a second manner different from the first manner.

[0020] In some examples, a transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first device that is in communication with a display generation component is described. In some examples, the one or more programs includes instructions for: receiving, from a second device and a third device, media data, wherein the third device is different from the first device and the second device; and after receiving the media data: in accordance with a determination that a common object within the media data has a first orientation relative to the second device and the third device, displaying, via the display generation component, content in a first manner; and in accordance with a determination that the common object within the media data has a second orientation, different from the first orientation, relative to the second device and the third device, displaying, via the display generation component, the content in a second manner different from the first manner.

[0021] In some examples, a first device configured to communicate with a display generation component is described. In some examples, the first device comprises one or more processors and memory storing one or more programs configured to be executed by the one or more processors. In some examples, the one or more programs includes instructions for: receiving, from a second device and a third device, media data, wherein the third device is different from the first device and the second device; and after receiving the media data: inaccordance with a determination that a common object within the media data has a first orientation relative to the second device and the third device, displaying, via the display generation component, content in a first manner; and in accordance with a determination that the common object within the media data has a second orientation, different from the first orientation, relative to the second device and the third device, displaying, via the display generation component, the content in a second manner different from the first manner.

[0022] In some examples, a first device configured to communicate with a display generation component is described. In some examples, the first device comprises means for performing each of the following steps: receiving, from a second device and a third device, media data, wherein the third device is different from the first device and the second device; and after receiving the media data: in accordance with a determination that a common object within the media data has a first orientation relative to the second device and the third device, displaying, via the display generation component, content in a first manner; and in accordance with a determination that the common object within the media data has a second orientation, different from the first orientation, relative to the second device and the third device, displaying, via the display generation component, the content in a second manner different from the first manner.

[0023] In some examples, a computer program product is described. In some examples, the computer program product comprises one or more programs configured to be executed by one or more processors of a first device that is in communication with a display generation component. In some examples, the one or more programs include instructions for: receiving, from a second device and a third device, media data, wherein the third device is different from the first device and the second device; and after receiving the media data: in accordance with a determination that a common object within the media data has a first orientation relative to the second device and the third device, displaying, via the display generation component, content in a first manner; and in accordance with a determination that the common object within the media data has a second orientation, different from the first orientation, relative to the second device and the third device, displaying, via the display generation component, the content in a second manner different from the first manner.

[0024] One or more computer-readable storage medium can include executable instructions for performing these techniques. One or more computer program product can include executable instructions for performing these techniques. Some examples of suchcomputer-readable storage mediums are non-transitory. Some examples of such computer- readable storage mediums are transitory. Some examples of such computer program products are non-transitory. Some examples of such computer program products are transitory. Such computer-readable storage medium and / or other computer program product are configured for execution by one or more processors.DESCRIPTION OF THE FIGURES

[0025] Embodiments described below in the Detailed Description may reference the following drawings.

[0026] FIG. 1 A is a block diagram illustrating an electronic device in accordance with some examples.

[0027] FIG. IB illustrates an exemplary electronic device in accordance with some examples.

[0028] FIGS. 2A-2D are block diagrams illustrating positions of accessory devices in an environment as sensors are added and removed from the environment in accordance with some examples.

[0029] FIG. 3 is a flow diagram illustrating a method for managing a model of an environment in accordance with some examples.

[0030] FIGS. 4A-4C are block diagrams illustrating correlation of images from a mobile device to update an area model of an environment in accordance with some examples.

[0031] FIG. 5 is a flow diagram illustrating a method for updating a model of an environment in accordance with some examples.

[0032] FIGS. 6A-6C illustrate exemplary user interfaces for displaying content in accordance with some examples.

[0033] FIG. 7 is a flow diagram illustrating a method for displaying content in accordance with some examples.DETAILED DESCRIPTION

[0034] The following detailed description describes one or more different techniques that can be implemented as methods, devices, systems, computer readable mediums, and the like. This detailed description is intended to assist in understanding the scope of the present disclosure. It is not, however, intended to limit the scope of the present disclosure. For example, some of the phrases and terminology used below are for the purpose of providing examples rather than to limit the scope of the present disclosure.

[0035] Turning to FIGS. 1 A-1B, an example electronic device 100 is illustrated. Electronic device 100 is a non-limiting example of an electronic device that can be used to perform functionality described here. With reference to the block diagram of FIG. 1 A, electronic device 100 includes processor subsystem 110 communicating with memory 120 (e.g., a system memory) and VO interface 130 via interconnect 150 (e.g., a system bus, one or more memory locations, or other communication channel for connecting multiple components of electronic device 100). In addition, VO interface 130 is communicating with (e.g., wired or wirelessly) VO device 140. In some examples, VO interface 130 is included with I / O device 140 such that the two are a single component. It should be recognized that there can be one or more VO interfaces, with each VO interface communicating with one or more VO devices. In some examples, multiple instances of processor subsystem 110 can be communicating via interconnect 150.

[0036] Electronic device 100 can be any of various types of devices, including, but not limited to, a system on a chip, a server system, a personal electronic device, a smart phone, a smart watch, a wearable device, a tablet, a laptop computer, a fitness tracking device, a headmounted display (HMD) device, a desktop computer, an accessory (e.g., switch, light, speaker, air conditioner, heater, window cover, fan, lock, media playback device, television, and so forth), a controller, a hub, and / or a sensor. In some examples, a sensor includes one or more hardware components that detect information about a physical environment in proximity of (e.g., surrounding) the sensor. In some examples, a hardware component of a sensor includes a sensing component (e.g., an image sensor or temperature sensor), a transmitting component (e.g., a laser or radio transmitter), and / or a receiving component (e.g., a laser or radio receiver). Examples of sensors include an angle sensor, a breakage sensor such as a glass breakage sensor, a chemical sensor, a contact sensor, a non-contact sensor, a flow sensor, a force sensor, a gas sensor, a humidity or moisture sensor, an imagesensor (e.g., a RGB camera and / or an infrared sensor), an inertial measurement unit, a leak sensor, a level sensor, a metal sensor, a microphone, a motion sensor, a particle sensor, a photoelectric sensor (e.g., ambient light and / or solar), a position sensor (e.g., a global positioning system), a precipitation sensor, a pressure sensor, a proximity sensor, a radiation sensor, a range or depth sensor (e.g., RADAR, LiDAR), a speed sensor, a temperature sensor, a time-of-flight sensor, a torque sensor, and an ultrasonic sensor, a vacancy sensor, an voltage and / or current sensor, and / or a water sensor. In some examples, sensor data is captured by fusing data from one sensor with data from one or more other sensors. Although a single electronic device is shown in FIGS. 1A-1B, electronic device 100 can also be implemented as two or more electronic devices operating together.

[0037] In some examples, processor subsystem 110 includes one or more processors or processing units configured to execute program instructions to perform functionality described herein. For example, processor subsystem 110 can execute an operating system and / or one or more applications.

[0038] Memory 120 can include a computer readable medium (e.g., non-transitory or transitory computer readable medium) usable to store (e.g., configured to store, assigned to store, and / or that stores) program instructions executable by processor subsystem 110 to cause electronic device 100 to perform various operations described herein. For example, memory 120 can store program instructions to implement the functionality associated with method 300 described below.

[0039] Memory 120 can be implemented using different physical, non-transitory memory media, such as hard disk storage, optical drive storage, floppy disk storage, removable disk storage, removable flash drive, storage array, a storage area network (e.g., SAN), flash memory, random access memory (e g., RAM-SRAM, EDO RAM, SDRAM, DDR SDRAM, and / or RAMBUS RAM), and / or read only memory (e.g., PROM and / or EEPROM).

[0040] VO interface 130 can be any of various types of interfaces configured to communicate with other devices. In some examples, VO interface 130 includes a bridge chip (e.g., Southbridge) from a front-side bus to one or more back-side buses. VO interface 130 can communicate with one or more VO devices (e.g., VO device 140) via one or more corresponding buses or other interfaces. Examples of VO devices include storage devices (e.g., as described above with respect to memory 120), network interface devices (e.g., to alocal or wide-area network), sensor devices (e.g., as described above with respect to sensors), a physical user-interface device (e.g., a physical keyboard, a mouse, and / or a joystick), and an auditory and / or visual output device (e.g., speaker, light, screen, and / or projector). In some examples, the visual output device is referred to as a display generation component. The display generation component is configured to provide visual output, such as display via an LED display or image projection. As used herein, “displaying” content includes causing to display the content (e.g., video data rendered or decoded by a display controller) by transmitting, via a wired or wireless connection, data (e.g., image data or video data) to an integrated or external display generation component to visually produce the content.

[0041] In some examples, EO device 140 includes one or more camera sensors (e.g., one or more optical sensors and / or one or more depth camera sensors), such as for recognizing a subject and / or a subject’s gestures (e.g., hand gestures and / or air gestures) as input. In some examples, an air gesture is a gesture that is detected without the user touching an input element that is part of the device (or independently of an input element that is a part of the device) and is based on detected motion of a portion of the user’s body through the air including motion of the user’s body relative to an absolute reference (e.g., an angle of the user’s arm relative to the ground or a distance of the user’s hand relative to the ground), relative to another portion of the user’s body (e.g., movement of a hand of the user relative to a shoulder of the user, movement of one hand of the user relative to another hand of the user, and / or movement of a finger of the user relative to another finger or portion of a hand of the user), and / or absolute motion of a portion of the user’s body (e.g., a tap gesture that includes movement of a hand in a predetermined pose by a predetermined amount and / or speed, or a shake gesture that includes a predetermined speed or amount of rotation of a portion of the user’s body).

[0042] In some examples, EO device 140 is integrated with other components of electronic device 100. In some examples, EO device 140 is separate from other components of electronic device 100. In some examples, EO device 140 includes a network interface device that permits electronic device 100 to communicate with a network or other electronic devices, in a wired or wireless manner. Exemplary network interface devices include Wi-Fi, Bluetooth, NFC, USB, Thunderbolt, Ethernet, Thread, UWB, and so forth. With reference to FIG. IB, example electronic device 100 can house one or more processors 150, memory 152, input devices 154, sensors 156, and display 158.

[0043] Attention is now directed towards techniques for managing a representation (e.g., an area model) of an environment. The representation can be used to direct processing of information for electronic devices in the environment. In some examples, the representation is used by an ecosystem of software and hardware components to perform operations with respect to the environment. For example, one or more applications can manage hardware devices in the environment based on an understanding of the environment as provided by the representation. In some examples, the representation can help identify what electronic devices are located where in the environment, so that states of the electronic devices can be identified and / or changed. Such techniques are described below in the context of an area model and accessory devices in a home, but it is noted that other types of objects, environments, and / or sensors can be used with techniques described herein. For example, electronic devices in an outdoor area can also employ representations of environments to coordinate their functionalities. In addition, techniques optionally complement or replace other techniques for managing a representation of an environment.

[0044] FIGS. 2A-2D are block diagrams illustrating positions of accessory devices in an environment as sensors are added and removed from the environment in accordance with some examples. In each figure of FIGS. 2A-2D, environment 200 is intended to reflect positions of different devices in environment 200 at particular times according to an area model. It should be recognized that the area model can be a list, a tree, a graph, an array, a visual representation, an image, a data object, a two-dimensional image, a two-dimensional video, a two-dimensional model, a two-dimensional animation, a two-dimensional representation, a three-dimensional image, a three-dimensional video, a three-dimensional model, a three-dimensional animation, and / or a three-dimensional representation. It should also be recognized that the area model can include more, less, and / or different information than as described further below.

[0045] As illustrated in FIG. 2 A, environment 200 includes three different portions: left space 200A, space 200B, and right space 200C. In the example of FIG. 2A, left space 200A and right 200C are rooms and space 200B is a hallway. Left space 200A includes three accessory devices (e.g., first lamp 212 at position 204, TV 214 at position 206, and speaker 216 at position 208) and a single sensor (e.g., a first camera at position 202). It should be recognized that environment 200 can include more, less, and / or different objects than illustrated in FIG. 2A. As illustrated in FIG. 2A, space 200B and right space 200C do notinclude any accessory devices or cameras according to the area model. At FIG. 2 A, the area model includes identifiers for first lamp 212, TV 214, and speaker 216. In some examples, the area model includes other data in addition to and / or instead of identifiers of each device. For example, the area model can include location, orientation, functionality, connection information, permissions, and / or security for different devices in the area model.

[0046] In some examples, the area model is at least partially configured by a user. For example, the user can manually add different accessory devices (e.g., lamp 212, TV 214, and / or speaker 216) to the area model through an application (e.g., of the ecosystem and / or a home application executing on a personal device of the user). In some examples, adding the different accessory devices can include defining identifications, locations, orientations, functionality, connection information, permissions, and / or security of the different accessory devices. After being added, information corresponding to the different accessory devices can be automatically added to and / or updated in the area model. For example, images can be captured of environment 200 that are used to determine positions of the different accessory devices within environment 200. In some examples, the area model is not configured by a user. Instead, the area model is automatically generated and updated based on images captured of environment 200.

[0047] In some examples, while the area model for FIG. 2A is established for environment 200, an event can occur within environment 200. For example, a subject (e.g., a person, a user, and / or an animal) can walk into left space 200A, triggering lamps located in left space 200 A to turn on. In such an example, the area model can be used in (1) identifying that first lamp 212 is in left space 200 A and / or (2) turning on first lamp 212 in response to the subject entering left space 200 A. In some examples, the area model does not include an automation to identify a position of first lamp 212 and / or to turn on first lamp 212. Instead, the area model includes a position of first lamp 212, and the area model is used to identify the position when executing the automation. In some examples, the area model includes the automation.

[0048] As illustrated in FIG. 2B, left space 200A still includes three accessory devices, first lamp 212, TV 214, and speaker 216. To note, while first lamp 212 and speaker 216 have maintained their position in FIG. 2B relative to FIG. 2A, TV 214 has shifted left — from position 206 to position 218 in FIG. 2B as compared to FIG. 2 A. In some examples, the change in position of TV 214 is determined using an image captured by the first sensor atposition 202. For example, a location of the first sensor can be known and so, as a result of TV 214 being moved to the left in the image captured by the first sensor, the area model can be updated to include position 218 instead of position 206 for TV 214.

[0049] As illustrated in FIG. 2B, right space 200C includes an additional sensor (e.g., a second sensor at position 220). In some examples, the second sensor is added to environment 200 after FIG. 2A. In such examples, adding the second sensor can include providing power to and / or installing the second sensor in environment 200. In some examples, the second sensor is added to an application (e.g., of the ecosystem and / or a home application) configured for environment 200. In some examples, the second sensor can be added to position 220 in environment 200. In such examples, a user can define position 220 for the second sensor. In other examples, the second sensor can determine its position using the area model, such as by identifying objects within an image captured by the second sensor and calibrating the image using the area model to determine that the second sensor is at position 220. For example, the second sensor can capture the image that includes a portion of environment 200 captured by the first sensor (e.g., position 208) and, based on a predefined field of view of the second sensor and an angle of speaker 216 at position 208 in the image, determine that the second sensor is located at position 220.

[0050] As mentioned above, the application can already include the first sensor at position 202 and already be updating the area model based on images captured by the first sensor. In some examples, after adding the second sensor, images captured by the second sensor can be used to update the area model. For example, the second sensor can have a field of view that includes both position 208 and position 222. In such an example, speaker 216 at position 208 in the area model can be updated using images from the first sensor at position 202 and / or the second sensor at position 220. In some examples, sensors closer, with higher- resolution, and / or determined to have a better view of a position are used to update information about the position. In such examples, the first sensor at position 202 can be configured to update position 208 within the area model while the second sensor at position 220 is configured to update position 222. In some examples, the second sensor captures an image of environment 200 and a second lamp is identified at position 222. In such examples, the second lamp can be added to the area model, as illustrated by second lamp 224 being included in environment 200.

[0051] In some examples, while the area model for FIG. 2B is established for environment 200, an event can occur within environment 200. For example, a subject can sit down to the right of position 204 and to the left of position 202, triggering TV 214 to be turned on because the subject is determined to be looking in the direction of TV 214. In such an example, the area model can be used in (1) identifying that TV 214 is located at position 218 and / or (2) turning on TV 214 in response to the subject looking toward position 218 (whereas if TV 214 was still determined to be located at position 208, TV 214 would not be turned on in response to the subject looking toward position 218). In some examples, the area model does not include an automation to identify a position of TV 214 and / or turn on TV 214. Instead, the area model includes a position of TV 214 (e.g., via TV 214), and the area model is used to identify the position when executing the automation. In some examples, the area model includes the automation.

[0052] At FIG. 2C, the first sensor (formerly at position 202 in FIG. 2B) has been removed from environment 200 (e.g., has been powered down and / or unregistered from environment 200). In some examples, removing the first sensor from environment 200 does not cause the area model to immediately disappear. For example, first lamp 212 at position 204 and TV 214 at position 218 can continue to be represented in the area model (e.g., as first lamp 212 and TV 214) even though the first sensor is no longer used to update position 204, position 218, first lamp 212, and / or TV 214. As illustrated in FIG. 2C, speaker 216 at position 208 and the lamp at 222 have maintained their position in FIG. 2C relative to FIG. 2B.

[0053] In some examples, while the area model for FIG. 2C remains established for environment 200, an event can occur within environment 200. For example, a subject can walk from space 200B (e.g., detected by the second sensor at position 220) upward such that the second sensor loses visibility of the subject, triggering first lamp 212 to be turned on because the subject is determined to be heading toward first lamp 212. In such an example, the area model can be used in identifying that first lamp 212 is located at position 204 even though the first sensor is no longer established for that portion of environment 200 and the second sensor does not have visibility of that portion of environment 200. For another example, as the subject walks upward such that the second sensor loses visibility of the subject, speaker 216 at position 208 can reduce volume because the subject is walking away from speaker 216. In such an example, the area model can be used in identifying that speaker216 is located at position 208 while the second sensor has visibility toward second position208.

[0054] As illustrated in FIG. 2D, first lamp 212 at 204, TV 214 at position 218, and the second lamp at 222 have maintained their position in FIG. 2D relative to FIG. 2C. At FIG. 2D, speaker 216 (formerly at position 208 in FIG. 2C) has been moved to position 226.

[0055] In some examples, after the first sensor has been removed from environment 200, the second sensor at position 220 captures an image used to determine that speaker 216 has moved. In such examples, the area model is updated to reflect that speaker 216 is located at position 226 instead of position 208. Such updates can occur even when the second sensor has a smaller field of view, a different field of view, and / or a lower resolution than the first sensor.

[0056] In some examples, the area model is updated because of a time since position 208 had last been confirmed (e.g., by an image captured by the first sensor). In such examples, if less than a predefined period of time has passed, the area model is not updated based on images from the second sensor because the first sensor is determined to be more likely to accurately capture the current position of speaker 216. In other examples, the area model is updated because the first sensor is no longer in communication with (and / or has not been in communication within a predefined period of time) a device updating the area model. In such examples, if the first sensor is still in communication with the device updating the area model (even if the first sensor has not captured an image since the second sensor), the area model is not updated based on images from the second sensor because the first sensor is determined to be more likely to accurately capture the current position of speaker 216. In other examples, the area model is updated based on whichever image is more recently captured, irrespective of whether a sensor capturing the image is still registered to environment 200 and / or in communication with the device updating the area model.

[0057] In some examples, while the area model for FIG. 2D is established for environment 200, an event can occur within environment 200. For example, a subject can walk toward position 226, triggering speaker 216 to be turned up. In such an example, the area model can be used in (1) identifying that speaker 216 is located at position 226 and / or (2) to turn on speaker 216 in response to the subject heading toward position 218 (whereas if speaker 216 was still determined to be located at position 208, speaker 216 would not beturned on in response to the subject moving toward position 226). In some examples, the area model does not include an automation to identify a position of speaker 216 and / or turn on speaker 216. Instead, the area model includes a position of speaker 216, and the area model is used to identify the position when executing the automation. In some examples, the area model includes the automation.

[0058] The operations described above (e.g., description corresponding to FIGS. 2A-2C) and below (e.g., description corresponding to FIGS. 3, 4A-4C, and / or 5) can be performed using various ecosystems of devices. Conceptually, a source device obtains and delivers data representing the environment to a decision controller. In the foregoing examples, for instance, an accessory device in the form of a sensor acts as a source device by providing sensor output about environment 200. The sensor output can be provided to a controller device with sufficient computation power to process the incoming information and update an area model for environment 200. Examples of electronic devices having sufficient computational power to act as controllers include a smart phone, a tablet, a laptop, and / or a desktop computer. Controller functionality may also be integrated into devices that have other primary functionalities, such as a media playback device, a smart speaker, a tabletop dock or smart screen, a television, and so forth. Source devices, such as the sensor, can in some instances have sufficient computational power to act as controller devices. It should be appreciated that computational power generally represents a design choice that is balanced with power consumption, packaging, and cost. For example, a source device that is wired to main electricity may be more likely to take on controller device functionality than a battery- powered device, even though both are possible. The controller device, upon determining a decision based on the obtained sensor output, updates an area model for use by the controller device and / or another device. In the foregoing examples, the controller device updates the position of a television, in addition to other accessories.

[0059] The various ecosystem of devices described above can connect and communicate with one another using various communication configurations. Some exemplary configurations involve direct communications such as device-to-device connections. For example, a source device (e.g., camera) can capture images of environment 200, determine a position of a subject and, acting as a controller device, update an area model corresponding to environment 200 for use by the controller device and / or other devices to send an instruction to an accessory device to change states. The connection with the accessory device can bewired or wireless. The connection can be a direct device-to-device connection such as Bluetooth. Some exemplary configurations involve mesh connections. For example, an accessory device may use a mesh connection such as Thread to connect with other devices in environment 200. Some exemplary configurations involve local and / or wide area networks and may employ a combination of wired (e.g., Ethernet) and wireless (e.g., Wi-Fi, Bluetooth, Thread, and / or UWB) connections. For example, a camera may connect locally with a controller hub in the form of a smart speaker, and the smart speaker may relay instructions remotely with a smart phone, over a cellular or Internet connection.

[0060] The description herein provides for the use of data from various possible sensing sources, including RGB image, event, and / or proximity sensors to improve interactions with connected devices within an environment. In some instances, these sources may include electronic devices situated in an enclosed space, working in coordination to offer potential benefit to users. For example, security systems often incorporate cameras and other sensors. Implementers should consider and adhere to prevailing privacy standards and policies in maintaining user privacy. These standards and policies should meet industry, government, and / or other legal requirements.

[0061] As an example, implementers can take steps to anonymize sensor data. For instance, sensors may operate at low resolution for automatic object detection, and capture at higher resolutions upon explicit user instruction. Additionally or alternatively, sensor data that are captured in the enclosed space can also be restricted to devices that are located in the enclosed space, and not be transmitted and stored in the cloud. Sensor data may be limited to devices that are in close proximity to one another, such that sensor data from one environment is not physically transmitted beyond the environment. Peer to peer communication protocols of limited range may be used to limit communication to trusted and / or local devices. Additionally or alternatively, identification information can be removed from (or otherwise not become associated with) the data. During set up, users may be allowed to selectively block the use of sensor information, such that users are allowed to, for example, opt-in or opt-out of certain features. For example, users may be allowed to enable ambient light detection while disabling high resolution camera imaging.

[0062] FIG. 3 is a flow diagram illustrating a method (e.g., method 300) for managing a model of an environment in accordance with some examples, including those discussed with reference to FIGS. 2A-2D. In some examples, one or more operations of method 300 isperformed by a controller device as described above with reference to FIGS. 2A-2B. In some examples, the controller device is or is integrated with an electronic device such as device 100 described above with reference to FIGS. 1 A-1B. Some operations in method 300 are, optionally, combined, the orders of some operations are, optionally, changed, and some operations are, optionally, omitted. In some examples, operations of method 300 are shared between a first electronic device and a different second electronic device.

[0063] In some examples, method 300 is performed at an electronic device (e.g., electronic device 100, the first lamp at position 204, TV 214 at position 206, speaker 216 at position 208, the second lamp at position 220, the first sensor at position 202, the second sensor at position 220, and / or smart phone 404). In some examples, the electronic device is an accessory, a controller, a speaker, a smart light, a laptop, a smart phone, a smart watch, a tablet, a head-mounted display (HMD) device, a fitness tracking device, and / or an electronic device. In some examples, the electronic device includes and / or is in communication with one or more input devices (e.g., a touch-sensitive display, a rotatable input mechanism, a camera (e.g., a telephoto, wide angle, and / or ultra-wide angle camera), and / or a sensor, such as a gyroscope and / or a heart rate sensor) and / or one or more output devices (e.g., a speaker, a haptic output device, a display screen, a projector, and / or a touch-sensitive display). In some examples, the one or more output devices include a display generation component (e.g., a display screen, a projector, and / or a touch-sensitive display).

[0064] The electronic device receives (302), via a first sensor (e.g., a telephoto, wide- angle, and / or ultra-wide angle camera) (e.g., a mobile and / or movable camera) (e.g., 202), a first image of a physical environment (e.g., an area, a predefined area, and / or a portion of the physical environment) (e.g., while the electronic device is in communication with the first sensor) (e.g., 200). In some examples, the first sensor is in communication with the electronic device when, before, and / or after the electronic device receives the first image.

[0065] After (and / or in response to) receiving the first image of the physical environment, the electronic device updates (304) (and / or generates), using the first image, a first portion (e.g., a portion including 204, 206, and / or 208) of a representation (e.g., an area model) (e.g., a model of the physical environment, a representation that is processed by one or more electronic devices to allows one or more electronic devices to perform one or more operations, a map of the physical environmental, and / or a three-dimensional map of one ormore objects in the physical environment) (e.g., the area model as described above with respect to FIGS. 2A-2C) of the physical environment.

[0066] The electronic device receives (306) (e.g., after updating the first portion using the first image), via a second sensor e.g., 220) different from the first sensor, a second image of the physical environment (e.g., while the electronic device is in communication with the second sensor). In some examples, the first sensor is not in communication with the electronic device when the electronic device receives the second image.

[0067] After (and / or in response to) receiving the second image of the physical environment, after updating (and / or generating), using the first image, the first portion of the representation of the physical environment, in accordance with a determination that a first set of one or more criteria is satisfied (e.g., that the electronic device is no longer in communication with the first sensor and / or that the electronic device has not received a third image from the first sensor within a predefined amount of time), the electronic device updates (308) (and / or changes), using the second image, the first portion of the representation (e.g., the area model as described above with respect to FIGS. 2B-2D) of the physical environment. In some examples, updating the first portion using the second image causes the first portion to no longer be based on the first image.

[0068] In some examples, after (and / or in response to) receiving the first image of the physical environment, the electronic device updates (and / or generates), using the first image, a second portion (e.g., a portion including 204, 206, 208, and / or 222) of the representation, wherein the second portion is different from the first portion. In some examples, after (and / or in response to) receiving the second image of the physical environment and after updating, using the first image, the second portion of the representation of the physical environment, the electronic device maintains (e.g., does not update and / or keeps) the second portion (e.g., portion including 204 and 206) of the representation of the physical environment (e.g., does not update, using the second image, the second portion of the representation of the physical environment). In some examples, an area corresponding to the second portion of the representation of the physical environment is not included in the second image. In some examples, an area corresponding to the second portion of the representation of the physical environment is included in the second image.

[0069] In some examples, after (and / or in response to) receiving the second image of the physical environment and after updating, using the first image, the first portion of the representation of the physical environment (and / or in conjunction with updating, using the second image, the first portion of the representation of the physical environment), the electronic device updates (and / or modifies, changes, and / or generates), using the second image, a third portion (e.g., portion including 222) of the representation of the physical environment, wherein the third portion of the representation was not updated (and / or modified, changed, and / or generated) using the first image (and / or while updating the first portion of the representation). In some examples, after (and / or in response to) receiving the first image of the physical environment, the electronic device does not update, using the first image, the third portion of the representation of the physical environment.

[0070] In some examples, the first set of one or more criteria includes a criterion that is satisfied when a fourth image was not received from (e.g., obtained by, captured by, and / or transmitted from) the first sensor within a first threshold period of time (e.g., 1 day, 1 week, or 1 year). In some examples, the criterion, which is satisfied when the fourth image was not received from the first sensor within the first threshold period of time, is not satisfied when the fourth image was received from the first sensor within the first threshold period of time.

[0071] In some examples, the first set of one or more criteria includes a criterion that is satisfied when the electronic device is not in communication with (e.g., connected to, transmitting data to, and / or receiving data from) the first sensor (e.g., currently not, such as at the time of and / or in conjunction with updating, using the second image, the first portion of the representation of the physical environment, and / or has not been in communication with the first sensor within a second threshold period of time, such as 1 day, 1 week, or 1 year). In some examples, the criterion, which is satisfied when the electronic device is not in communication with the first sensor (and / or has not been in communication with the first sensor within a second threshold period of time, such as 1 day, 1 week, or 1 year), is not satisfied when the electronic device is in communication with the first sensor (and / or has been in communication with the first sensor within the second threshold period of time).

[0072] In some examples, the first set of one or more criteria includes a criterion that is satisfied when a fifth image (e.g., of the first portion and / or another portion different from the first portion) was more recently received from (and / or captured by) the second sensor than the first sensor. In some examples, the criterion, which is satisfied when the fifth image wasmore recently received from the second sensor than the first sensor, is not satisfied when the fifth image was more recently received from the first sensor than the camera.

[0073] In some examples, the first sensor has a larger field of view than the second sensor. In some examples, the second sensor has a larger field of view than the first sensor. In some examples, the first sensor has a field of view of a first size. In some examples, the second sensor has a field of view of a second size different from (e.g., larger or smaller than) the first size. In some examples, the first sensor has a different field of view than the second sensor. In some examples, the second sensor has a larger field of view than the first sensor.

[0074] In some examples, the representation is based on (e.g., generated from, captured in, and / or taken from) a sixth image received from (and / or captured by) a third camera different from the first sensor and the second sensor. In some examples, before updating, using the second image, the first portion of the representation of the physical environment and / or before updating, using the first image, the first portion of the representation of the physical environment, the electronic device updates (and / or generates), using the sixth image, the representation (e.g., the first portion, the second portion, and / or another portion different from the first portion and the second portion) of the physical environment.

[0075] In some examples, a fourth portion of the representation is updated (and / or generated) using (and / or based on) the sixth image. In some examples, the fourth portion of the representation is not updated using (and / or based on) the second image (e.g., because the fourth portion is not included in the second image). In some examples, before updating, using the second image, the first portion of the representation of the physical environment and / or before updating, using the first image, the first portion of the representation of the physical environment, the electronic device updates (and / or generates), using the sixth image, the fourth portion of the physical environment. In some examples, after (and / or in response to) receiving the first image of the physical environment, the electronic device maintains the fourth portion of the representation (e.g., does not update the fourth portion using the first image, such as the electronic device does not update, using the first image, the fourth portion of the representation of the physical environment). In some examples, after (and / or in response to) receiving the second image of the physical environment, the electronic device maintains the fourth portion of the representation (e.g., the electronic device does not update, using the second image, the fourth portion of the representation of the physical environment).

[0076] In some examples, after updating, using the second image, the representation of the physical environment (e.g., a respective portion, the first portion, and / or the second portion of the representation of the physical environment), the electronic device receives, from the second sensor, a seventh image of the physical environment. In some examples, after (and / or in response to) receiving the seventh image of the physical environment (and / or in accordance with a determination that the first set of one or more criteria is satisfied), the electronic device updates, using the seventh image, the first portion of the representation of the physical environment.

[0077] In some examples, after updating, using the seventh image, the first portion of the representation of the physical environment, the electronic device receives, from the second sensor, an eighth image of the physical environment. In some examples, after (and / or in response to) receiving the eighth image of the physical environment (and / or in accordance with a determination that the first set of one or more criteria is satisfied), the electronic device maintains the first portion of the representation of the physical environment (e.g., does not update, using the eighth image, the first portion of the representation of the physical environment). In some examples, after (and / or in response to) receiving the eighth image of the physical environment (and / or in accordance with a determination that the first set of one or more criteria is satisfied) (and / or in accordance with a determination that another set of one or more criteria different from the first set of one or more criteria is satisfied), the electronic device updates, using a ninth image from a fourth camera (e.g., the first sensor or another camera different from the first sensor) different from the second sensor, the first portion of the representation of the physical environment.

[0078] In some examples, after updating, using the seventh image, the first portion of the representation of the physical environment, the electronic device receives, from the first sensor, a tenth image of the physical environment. In some examples, after (and / or in response to) receiving the tenth image of the physical environment (and / or in accordance with a determination that another set of one or more criteria different from the first set of one or more criteria is satisfied), the electronic device updates, using the tenth image, the first portion of the representation of the physical environment.

[0079] In some examples, the representation of the physical environment indicates (and / or includes) a location of one or more objects in the physical environment. In some examples, the one or more objects includes an accessory (e.g., a smart speaker, a mediadevice, a smart light, a smart window shade, and / or a switch). In some examples, the representation of the physical environment includes and / or indicates an identification of the one or more objects in the physical environment. In some examples, the representation of the physical environment includes and / or indicates a control for the one or more objects in the physical environment. In some examples, in response to detecting an input corresponding to the control, the electronic device performs an operation with respect to the one or more objects (e.g., turns a light on or off). In some examples, the representation of the physical environment is a map and / or include a map of one or more objects (e.g., physical objects) in the physical environment.

[0080] In some examples, after updating, using the second image, the first portion of the representation of the physical environment, the electronic device (310) performs, using the first portion of the representation of the physical environment, a first operation (e.g., causing an object (e.g., an accessory and / or other type of electronic device) to perform an operation, such as turn on or off, change a volume, change a color, change a brightness, and / or change a temperature).

[0081] In some examples, after updating, using the second image, the first portion of the representation of the physical environment (and, in some examples, one or more other portions), the electronic device performs, using a fifth portion of the representation of the physical environment, a second operation (e.g., causing an object (e.g., an accessory and / or other type of electronic device) to perform an operation, such as turn on or off, change a volume, change a color, change a brightness, and / or change a temperature), wherein the fifth portion is not updated using an eleventh image received from the second sensor, and wherein the fifth portion is different from the first portion. In some examples, the fifth portion is updated using a twelfth image received from the first sensor and / or another camera different from the first sensor and the second sensor. In some examples, the fifth portion is updated before receiving a thirteenth image and / or being in communication with the second sensor. In some examples, the fifth portion of the representation is an old and / or less recently updated portion of the representation.

[0082] In some examples, while the electronic device is no longer in communication with the first sensor (and continues to be in communication with the second sensor), after updating, using the first image, the first portion of the representation of the physical environment, the electronic device detects, via the second sensor (e.g., using a fourteenthimage captured by the second sensor), an event (e.g., a subject performing an action and / or task, a temperature change, a time change, an object performing an operation, and / or a notification being received) occurring in an area of the physical environment corresponding to the first portion. In some examples, while the electronic device is no longer in communication with the first sensor, in response to detecting the event occurring in the area of the physical environment corresponding to the first portion, the electronic device performs a third operation (e.g., turn on or off, change a volume, change a color, change a brightness, and / or change a temperature).

[0083] In some examples, performing the first operation includes changing a state of a first accessory in (e.g., located in) the area of the physical environment. In some examples, the first accessory is not in a field of view of the second sensor. In some examples, the first accessory is (and / or was in a field of view of the first sensor).

[0084] In some examples, the first image has higher resolution that the second image. In some examples, the second image is used to update the first portion of the representation of the physical environment when the first portion was previously updated using a higher resolution image because the second image is captured more recently than the first image.

[0085] In some examples, the representation of the physical environment indicates a location of one or more accessories in the physical environment. In some examples, a position (e.g., a location and / or orientation) of the second sensor within the physical environment is determined (e.g., before updating, using the second image, the representation of the physical environment) using the representation of the physical environment.

[0086] In some examples, an object (e.g., subject (e.g., person and / or animal), an inanimate object, and / or a mechanical and / or electronic object) in a respective image received from (and / or captured by) the second sensor is identified using the representation (e.g., identified using the respective image and data not included, determined, and / or identified in an image (e.g., the respective image and / or another image) received from the second sensor).

[0087] In some examples, the second sensor is not in communication with the electronic device before (and / or in conjunction with) updating, using the first image, the first portion of the representation of the physical environment. In some examples, the second sensor is in communication with the electronic device after (e.g., directly after and / or at some point after) updating, using the first image, the first portion of the representation.

[0088] In some examples, the electronic device is in communication with one or more input devices (e.g., a camera, a depth sensor, a microphone, a hardware input mechanism, a rotatable input mechanism, a heart monitor, a temperature sensor, and / or a touch-sensitive surface). In some examples, after updating, using the first image, the first portion of the representation of the physical environment, the electronic device detects, via the one or more input devices, an input (e.g., a tap input and / or a non -tap input, such as a verbal input, an audible request, an audible command, an audible statement, a swipe input, a hold-and-drag input, a gaze input, an air gesture, and / or a mouse click) corresponding to a request to add the second sensor to a set of one or more cameras established (e.g., defined, set, and / or programmed, such as by a user) for the physical environment. In some examples, the second sensor is not included in the set of one or more cameras established for the physical environment before detecting the input corresponding to a request to add the second sensor to the set of one or more cameras established for the physical environment.

[0089] In some examples, in response to detecting the input corresponding to the request to add the second sensor to the set of one or more cameras established for the physical environment, the electronic device adds the second sensor to the set of one or more cameras established for the physical environment. In some examples, the second sensor is included in the set of one or more cameras established for the physical environment in conjunction with (e.g., before, during, and / or after) updating, using the second image, the first portion of the representation of the physical environment. In some examples, a fifteenth image received from a camera not included in the set of one or more cameras established for the physical environment is not used to update the representation of the physical environment. In some examples, in conjunction with updating, using the first image, the first portion of the representation of the physical environment, the first sensor is included in the set of one or more cameras established for the physical environment. In some examples, in conjunction with updating, using the second image, the first portion of the representation of the physical environment, the first sensor is not included in the set of one or more cameras established for the physical environment (e.g., the first sensor was removed for the set of one or more cameras established for the physical environment before updating, using the second image, the first portion of the representation of the physical environment).

[0090] In some examples, before updating, using the second image, the first portion of the representation of the physical environment, the electronic device detects that the firstsensor is no longer included in the set of one or more cameras established for the physical environment, wherein the first set of one or more criteria includes a criterion that is satisfied when a sixteenth image being used to update the representation is received from a camera in the set of one or more cameras established for the physical environment.

[0091] In some examples, the representation of the physical environment includes a nonvisual model (e.g., one or more indications, variables, separate pieces of data, and / or separate pieces of content) of one or more objects (e.g., an accessory device, a subject, a piece of furniture, a wall, a roof, a floor, and / or an electronic device) in the physical environment. In some examples, the representation of the physical environment includes identification of a capability (e.g., ability to perform an operation, such as turning on and off or outputting a particular color of light) of a first object (e.g., an accessory device, a subject, a piece of furniture, a wall, a roof, a floor, and / or an electronic device) in the physical environment. In some examples, the capability is not visual and / or identified in an image (e.g., the first image, the second image, and / or another image different from the first image and the second image). In some examples, the capability is identified (and / or obtained) via a first database (e.g., of the electronic device or of another electronic device in communication with the electronic device) including identification of the first object. In some examples, the representation of the physical environment includes a non-visual identification of a state of a second object (e.g., the first object or another object different from the first object) (e.g., that the second object is performing a function and / or instructed to perform a function, such as turning on and / or off) in the physical environment. In some examples, the state is identified in the first image and / or the second image. In some examples, the state is not identified in an image (e.g., the first image, the second image, and / or another image different from the first image and the second image). In some examples, the state is identified (and / or obtained) via a second database (e.g., the first database and / or another database different from the first database) (e.g., of the electronic device or of another electronic device in communication with the electronic device) including identification of the second object. In some examples, the representation of the physical environment includes an identification (e.g., a textual and / or unique identification) of (e.g., a reference to) a third object (e.g., the first object, the second object, or another object different from the first object and the second object) in the physical environment. In some examples, the representation of the physical environment includes a mapping of a fourth object (e.g., the first object, the second object, the third object, or another object different from the first object, the second object, and the third object) in the physicalenvironment. In some examples, the mapping maps and / or links the fourth object to a label (e.g., of a room, a home, the environment, and / or another physical area). In some examples, the mapping maps and / or links the fourth object to a location of the fourth object in the physical environment.

[0092] Note that details of the processes described above with respect to method 300 (e.g., FIG. 3) are also applicable in an analogous manner to other methods described herein. For example, method 500 optionally includes one or more of the characteristics of the various methods described above with reference to method 300. For example, the representation of the physical environment of method 500 can be the representation of the physical environment of method 300. For brevity, these details are not repeated below.

[0093] FIGS. 4A-4C are block diagrams illustrating correlation of images from a mobile device to update an area model of an environment in accordance with some examples. In each figure of FIGS. 4A-4C, environment 200 is illustrated on left and smart phone 404 is illustrated on right.

[0094] Consistent with discussion above, it should be recognized that environment 200 in FIGS. 4A-4C is intended to reflect positions of different devices in environment 200 at particular times according to an area model. It should be recognized that the area model can be a list, a tree, a graph, an array, a visual representation, an image, a data object, a two- dimensional image, a two-dimensional video, a two-dimensional model, a two-dimensional animation, a two-dimensional representation, a three-dimensional image, a three-dimensional video, a three-dimensional model, a three-dimensional animation, and / or a three-dimensional representation. It should also be recognized that the area model can include more, less, and / or different information than as described further below. At FIGS. 4A-4C, smart phone 402 includes display generation component 406 (e.g., a screen, a touch-sensitive display, and / or a projector) to display content. While illustrated as a smart phone, smart phone 404 can (1) be another type of electronic device, such as a smart watch, a fitness tracking device, a head mounted display device, and / or a camera and / or (2) include one or more components and / or functionality described herein, including those described above with respect to electronic device 100.

[0095] As illustrated in FIG. 4 A, environment 200 includes same portions as in FIGS. 2A-2C (e.g., left space 200A, space 200B, and right space 200C). In some examples, FIG. 4Ais a continuation of the example described above with respect to FIG. 2C. It should be recognized that environment 200 can include more, less, and / or different objects than illustrated in FIG. 4A. As illustrated in FIG. 4A, environment 200 includes a position of smart phone 404 in environment 200 (labeled as position 402). In some examples, a subject (e.g., a person and / or a user) is carrying smart phone 404.

[0096] At FIG. 4A, the area model includes information for four devices, such as lamp 212 at position 204, TV 214 at position 218, speaker 216 at position 226, and second lamp 224 at position 222). As mentioned above, the information in the area model can be determined by images captured by the first sensor (e.g., formerly at position 202) and the second sensor at position 220. For example, the first sensor can have captured images including positions 204 and 218. In such an example, the area model can include first lamp 212 at position 204 and TV 214 at position 218 based on the images captured by the first sensor. For another example, the second sensor at position 220 can have captured images including position 222. In such an example, the area model can include second lamp 224 at position 222 based on the images captured by the second sensor.

[0097] As illustrated in FIG. 4A, smart phone 404 is displaying camera user interface 408, with a live preview of a field of view of one or more cameras of smart phone 404. In some examples, the area model is being updated by images captured by second sensor at position 220 (e.g., as described above with respect to FIGS. 2A-2D) and / or the one or more cameras of smart phone 404. In such examples, the second sensor can be a stationary and / or a resident camera that has an established position within environment 200 such that images captured by the second sensor can be of a specific area of environment 200 and that a position of the second sensor and / or a view of the images do not need to be determined using the images. In some examples, smart phone 404 can be a mobile camera that does not have an established position within environment 200 such that a position and / or a view of images captured by smart phone 404 needs to be determined to correlate to the area model. The position and / or the view of images captured by smart phone 404 can be determined by identifying objects within images captured by smart phone 404. The objects can be compared to objects in the area model to identify the position of smart phone 404. The position and / or the view of images captured by smart phone 404 can also be identified using a non-image sensor of smart phone 404, such as a global positioning system (e.g., GPS), an inertial measurement unit (e.g., IMU), and / or an ultra-wideband (e.g., UWB) radio.

[0098] At FIG. 4B, smartphone 404 has moved from right space 200C to left space 200A, to position 414. In some examples, the one or more cameras of smart phone 404 capture images while in left space 200A (e.g., labeled as live preview 410). As illustrated in FIG. 4B, live preview 410 includes a portion of environment 200 with a corner and speaker 412. In some examples, speaker 412 corresponds to speaker 216 at position 226.

[0099] At FIG. 4C, smartphone 404 has moved within left space 200A, to position 416. In some examples, the one or more cameras of smart phone 404 capture one or more images while at position 416 (e.g., illustrated in live preview 410). As illustrated in FIG. 4C, live preview 310 includes another portion of environment 200 different from the portion illustrated in FIG. 4B. In some examples, the portion in FIG. 4C is a different orientation, location, and / or view as compared to FIG. 4B though at least partially overlaps. In such examples, the portion illustrated in FIG. 4C includes at least part of the portion illustrated in FIG. 4B. For example, the portion in live preview 310 at FIG. 4C includes the same corner as in FIG. 4B and speaker 412 at a different orientation.

[0100] In some examples, by correlating and / or comparing an image at position 414 and / or an image at position 416 with information of environment 200 in the area model (e.g., an image captured by another camera, such as the first sensor and / or the second sensor described above), a position of the image at position 416 relative to the area model can be determined so as to locate objects within the image at position 416. For example, speaker 412 in live preview 410 of FIGS. 4B-4C can be placed within the area model and determined that a previous position of speaker 412 (e.g., as illustrated in FIGS. 4A-4B) requires update and that speaker 216 is now placed to the left at position 228.

[0101] In some examples, the previous position is determined to require update based on one or more criteria described above, such as the one or more cameras of smart phone 404 being closer to position 228 than the second sensor, higher resolution than the second sensor, and / or determined more likely to accurately capture a current position of speaker 216 than the second sensor, a time since a position corresponding to speaker 216 had last been confirmed (e.g., by an image captured by the second sensor), the second sensor is no longer in communication with (and / or has not been in communication within a predefined period of time) a device (e.g., smart phone 404 or another device different from smart phone 404) updating the area model, and / or an image captured by smart phone 404 is more recent than a most-recent image captured by the second sensor.

[0102] Consistent with discussion above with respect to FIGS. 2A-2D, the area model can be used in changing states of accessory devices in environment 200. For example, a subject can be detected in left space 200 A by either an image from smart phone 404 or second sensor 220. The subject can be determined to be looking in the direction of position 228, resulting in speaker 216 at position 228 turning on. In contrast, if speaker 216 was still determined to be located at position 226, speaker 216 would not be turned on in response to the subject looking in the direction of position 228.

[0103] FIG. 5 is a flow diagram illustrating a method (e.g., method 500) for updating a model of an environment in accordance with some examples, including those discussed with reference to FIGS. 4A-4C. In some examples, one or more operations of method 500 is performed by a controller device as described above with reference to FIGS. 2A-2B. In some examples, the controller device is or is integrated with an electronic device such as device 100 described above with reference to FIGS. 1 A-1B. Some operations in method 500 are, optionally, combined, the orders of some operations are, optionally, changed, and some operations are, optionally, omitted. In some examples, operations of method 500 are shared between a first electronic device and a different second electronic device.

[0104] In some examples, method 500 is performed at an electronic device (e.g., electronic device 100, the first lamp at position 204, TV 214 at position 206, speaker 216 at position 208, the second lamp at position 222, the first sensor at position 202, the second sensor at position 220, and / or smart phone 404) in communication with a first camera (e.g., camera of and / or in communication with smart phone 404) (e.g., as described above in relation to method 300). In some examples, the electronic device is an accessory, a controller, a speaker, a smart light, a laptop, a smart phone, a smart watch, a tablet, a head-mounted display (HMD) device, a fitness tracking device, and / or an electronic device. In some examples, the electronic device includes and / or is in communication with one or more input devices (e.g., a touch-sensitive display, a rotatable input mechanism, a camera (e.g., a telephoto, wide angle, and / or ultra-wide angle camera), and / or a sensor, such as a gyroscope and / or a heart rate sensor) and / or one or more output devices (e.g., a speaker, a haptic output device, a display screen, a projector, and / or a touch-sensitive display). In some examples, the one or more output devices include a display generation component (e.g., a display screen, a projector, and / or a touch-sensitive display) (e.g., 406).

[0105] The electronic device receives (502), via the first sensor, a first image (e.g., 410 in FIG. 4B) of a physical environment (e.g., an area, a predefined area, and / or a portion of the physical environment) (e.g., 200). In some examples, the first image is captured at a first location within the physical environment. In some examples, the first image includes a first perspective of the physical environment.

[0106] The electronic device receives (504), via the first sensor, a second image (e.g., 410 in FIG. 4C) of the physical environment, wherein the second image is different from the first image. In some examples, the second image is captured at a second location within the physical environment. In some examples, the second location is different from the first location. In some examples, the second image includes a second perspective of the physical environment. In some examples, the second perspective is different from the first perspective.

[0107] After receiving the first image and the second image (and / or in response to receiving the first image and / or the second image), the electronic device updates (506) (and / or changes) a first portion (e.g., portion including 208 in FIG. 4C) of a representation (e.g., the area model as described above with respect to FIGS. 4A-4C) (e.g., an area model) of the physical environment by correlating the first image and the second image, wherein, before updating the first portion, the first portion is based on an image received from a second sensor (e.g., 202 and / or 220) (e.g., a stationary camera and / or a camera for which the electronic device is able to determine a position and / or orientation without correlating multiple images received from the second sensor) different from the first sensor, and wherein the representation of the physical environment includes a mapping of an object within the physical environment. In some examples, the mapping maps and / or links the object to a label (e.g., of a room, a home, the environment, and / or another physical area). In some examples, the mapping maps and / or links the object to a location of the object in the physical environment. In some examples, the second sensor is in communication with the electronic device. In some examples, the second sensor is in communication with the electronic device when and / or while the electronic device receives the first image and / or the second image. In some examples, the second sensor is in communication with the electronic device when and / or while the electronic device updates the first portion. In some examples, the first portion is updated based on the second sensor without correlating multiple images.

[0108] In some examples, after receiving the first image and the second image, the electronic device (508) maintains a second portion (e.g., portion including 204, 206, and / or222) of the representation of the physical environment (e.g., does not update and / or change the second portion of the representation of the physical environment by correlating the first image and the second image), wherein the second portion is different from the first portion. In some examples, before, during, and / or after maintaining the second portion, the second portion is based on a third image received from the second sensor.

[0109] In some examples, correlating the first image and the second image provides a position (e.g., a location, orientation, a view point, and / or a field of view) of (and / or corresponding to) the first image, the second image, or a combination thereof. In some examples, updating the first portion of the representation of the physical environment uses (and / or is based on) the position (e.g., to identify what and / or where to update).

[0110] In some examples, before receiving the first image and the second image, the electronic device (510) receives, via the second sensor, a fourth image of the physical environment. In some examples, after (and / or in response to) receiving the fourth image of the physical environment, the electronic device (512) updates (and / or generates) the first portion of the representation of the physical environment without correlating the fourth image with another image (e.g., of the physical environment) (e.g., from the second sensor and / or the first sensor) (e.g., the first image and / or the second image).[OHl] In some examples, the electronic device is in communication with the second sensor while updating the first portion of the representation of the physical environment by correlating the first image and the second image. In some examples, the first sensor is in an enclosure of the electronic device. In some examples, the second sensor and / or the third camera is in the enclosure or another enclosure, different from the enclosure, of the electronic device. In some examples, the second sensor is not in the enclosure of the electronic device. In some examples, the second computer is outside of the enclosure of the electronic device.

[0112] In some examples, the electronic device receives, via a fourth camera, a fifth image of the physical environment. In some examples, after (and / or in response to) receiving the fifth image of the physical environment, the electronic device updates, using the fifth image, a third portion of the representation of the physical environment (e.g., by correlating the fifth image with another image (e.g., the first image, the second image, and / or another image different from the first image and the second image) or without correlating the fourth with another image) (e.g., without updating the first portion of the representation of thephysical environment), wherein the third portion is different from the first portion (and / or the second portion).

[0113] In some examples, the electronic device receives (e.g., via the first sensor or another camera different from the first sensor ) a sixth image of the physical environment, wherein the sixth image is different from the first image and the second image. In some examples, the sixth image is captured at a third location within the physical environment. In some examples, the third location is different from the first location and / or the second location. In some examples, the fifth image includes a third perspective of the physical environment. In some examples, the third perspective is different from the first perspective and / or the second perspective. In some examples, after receiving the first image, the second image, and the sixth image (and / or in response to receiving the sixth image), the electronic device updates (and / or changes) the representation of the physical environment (e.g., the first portion of the representation of the physical environment) by correlating the sixth image and the second image without correlating the sixth image and the first image.

[0114] In some examples, the representation of the physical environment includes (and / or indicates) a location of one or more objects in the physical environment. In some examples, the one or more objects includes an accessory (e.g., a smart speaker, a media device, a smart light, a smart window shade, and / or a switch). In some examples, the representation of the physical environment includes and / or indicates an identification of the one or more objects in the physical environment. In some examples, the representation of the physical environment includes and / or indicates a control for the one or more objects in the physical environment. In some examples, in response to detecting an input corresponding to the control, the electronic device performs an operation with respect to the one or more objects (e.g., turns a light on or off).

[0115] In some examples, the representation of the physical environment includes a nonvisual model (e.g., one or more indications, variables, separate pieces of data, and / or separate pieces of content) of one or more objects (e.g., an accessory device, a subject, a piece of furniture, a wall, a roof, a floor, and / or an electronic device) (e.g., the object and / or one or more other objects different from the object) in the physical environment. In some examples, the representation of the physical environment includes identification of a capability (e.g., ability to perform an operation, such as turning on and off or outputting a particular color of light) of the object (e.g., an accessory device, a subject, a piece of furniture, a wall, a roof, afloor, and / or an electronic device) in the physical environment. In some examples, the capability is not visual and / or identified in an image (e.g., the first image, the second image, and / or another image different from the first image and the second image). In some examples, the capability is identified (and / or obtained) via a first database (e.g., of the electronic device or of another electronic device in communication with the electronic device) including identification of the first object. In some examples, the representation of the physical environment includes a non-visual identification of a state of the object (e.g., that the object is performing a function and / or instructed to perform a function, such as turning on and / or off) in the physical environment. In some examples, the state is identified in the first image and / or the second image. In some examples, the state is not identified in an image (e.g., the first image, the second image, and / or another image different from the first image and the second image). In some examples, the state is identified (and / or obtained) via a second database (e.g., the first database and / or another database different from the first database) (e.g., of the electronic device or of another electronic device in communication with the electronic device) including identification of the object. In some examples, the representation of the physical environment includes an identification (e.g., a textual and / or unique identification) of (e.g., a reference to) the object in the physical environment.

[0116] Note that details of the processes described above with respect to method 500 (e.g., FIG. 5) are also applicable in an analogous manner to the methods described herein. For example, method 300 optionally includes one or more of the characteristics of the various methods described above with reference to method 500. For example, the first sensor and / or the second sensor of method 300 can be the second sensor of method 500. For brevity, these details are not repeated below.

[0117] Attention is now directed towards techniques for displaying content. FIG. 6A-6C illustrate the different manner in which content can be displayed based on a spatial relationship of two or more devices.

[0118] FIGS. 6A-6C illustrate device 600 and device 602 displaying and arranging content based on a determined spatial relationship of device 600 and device 602. As discussed in greater detail below, the spatial relationship between device 600 and device 602 is determined based on captured media data. It should be recognized that device 600 and device 602 can be a number of different types of devices, such as a smart phone, a smart watch, a tablet, a laptop, a monitor, a communal device, a smart speaker, an accessory, apersonal gaming system, a desktop computer, a fitness tracking device, and / or a headmounted display (HMD) device. In some examples, device 600 and device 602 include and / or are in communication with one or more input devices and / or sensors (e.g., a camera, a lidar detector, a motion sensor, an infrared sensor, a touch-sensitive surface, a physical input mechanism (such as a button or a slider), and / or a microphone). Such sensors can be used to detect presence of, attention of, statements from, inputs corresponding to, requests from, and / or instructions from a user in an environment. In some examples, the field-of-view of device 600 and / or device 602 corresponds to the field-of-view of one or more cameras of device 600 and / or device 602. In some examples, device 600 and device 602 include and / or are in communication with one or more output devices (e.g., a display screen, a projector, a touch-sensitive display, speaker, and / or a movement component). Such output devices can be used to present information and / or cause different visual changes of device 600 and device 602.

[0119] For the purposes of the following discussion, context is helpful with respect to the term media data. In some examples, media data includes data (e.g., photo data, video data, depth data, and / or sound data) captured via one or more media capturing devices and / or sensors. In some examples, media data includes anonymized metadata derived from data captured via one or more media capturing devices and / or sensors. In some examples, the anonymized metadata includes data with respect to the detected location of a common object relative to device 600 and / or device 602 (e.g., the common object is to the left of device 600 and / or 10 degrees from center of device 600). In some examples, the one or more media capturing devices and / or sensors include a red-green-blue (RGB) color sensor, ultrasonic sensor, inertial measurement unit (IMU) sensor, ultra-wideband (UWB) sensor, near field communication (NFC) sensor, and / or 60 gigahertz radar sensor. Device 600 and device 602 use media data to recognize, identify, observe, and / or understand the positioning of both device 600 and device 602 relative to a common object (e.g., a user, furniture, and / or another device).

[0120] At FIG. 6A, media data is captured by and / or received from device 600 and device 602. Based on the media data, a determination is made (e.g., by device 600, device 602, and / or another device different from device 600 and device 602 as described further below) that no common object is positioned within a field-of-view of device 600 and / or device 602. At FIG. 6A, content is not displayed (e.g., via device 600 and / or device 602). In someexamples, device 600 and / or device 602 do not display content based on the determination that a common object is not detected within the media data. In some examples, when a common object is not detected in the media data, content is displayed (e.g., via device 600 and / or device 602). In such examples, content can be displayed in a default manner. In some examples, the orientation of the display of content in the default manner is not determined by user 610. In some examples, both device 600 and device 602 display content while content is displayed in the default manner.

[0121] In between FIG. 6 A and FIG. 6B, user 610 moves to be within the field-of-view of device 600 and / or device 602. It should be noted that as an example for these figures that user 610 has a preference that content be displayed with a left justification. Both device 600 and device 602 are programmed to display content based on the preference of user 610.

[0122] At FIG. 6B, media data is captured by and / or received from device 600 and device 602. At FIG. 6B, based on the media data, a determination is made that device 600 is positioned to the left of user 610 (e.g., from the perspective of user 610) and device 602 is positioned to the right of user 610 (e.g., from the perspective of user 610). For example, in some examples, when the media data is captured via a RGB color sensor, device 602 can capture an image of user 610 and device 600 can capture an image of user 610. The perspectives of both images are analyzed (e.g., compared) to determine the positioning of device 600 and / or device 602 relative to user 610 and / or each other. As an additional example, when the media data is captured via a NFC sensor, device 600 and / or device 602 can be equipped with a NFC sensor. The NFC sensor can exchange data such as GPS data, micro-location data, and / or position data with the other device (e.g., a NFC sensor of device 600 can communicate with device 602) or a personal device of user 610 (e.g., a smartwatch of user 610 and / or a smartphone of user 610) that allow a determination to be made regarding the positioning of device 600 and / or device 602 relative to each other (e.g., and / or user 610). As an additional example, when the media data is captured via a UWB sensor, device 600 and / or device 602 can be equipped with a UWB sensor. The UWB sensor can exchange signal with the other device (e.g., a UWB sensor of device 600 can communicate with device 602) and / or a personal device of user 610 (e.g., a smartphone and / or smartwatch) and an angle of arrival of the signals are determined to determine the positioning of device 600 and device 602 relative to each other and / or user 610. As an additional example, when media data is captured via an ultrasonic sensor and / or 60 gigahertz radar sensor, both device 600 anddevice 602 are equipped with ultrasonic sensors and / or 60 gigahertz radar sensor. Both ultrasonic sensors and / or 60 gigahertz radar sensors emit signals (e.g., ultrasonic waves and / or radio waves) towards user 610 and receive reflected waves. The positioning of both device 600 and device 602 relative to each other (and / or user 610) is determined based on characteristics of the reflected waves.

[0123] Based on these two determinations, it is determined that device 600 is to the left of device 602 (e.g., from the perspective of user 610). As illustrated in FIG. 6B, based on user 610 having a preference that content be displayed with a left justification and because the determination is made that device 600 is positioned to the left of device 602 (e.g., and / or because a determination is made that device 600 is to the left of user 610), content 604 (e.g., represented by four squares) is displayed by device 600 and content 604 is not displayed by device 602. As illustrated in FIG. 6B, because device 600 has a vertical orientation, content 604 is displayed in a portrait orientation (e.g., a first manner). In some examples, content 604 is displayed when a determination is made that user 610 is not within the media data. In some examples, additional content is displayed (e.g., by device 600 and / or device 604) while content 604 is displayed. In some examples, the display of content 604 is based on the spatial relationship of device 600 and device 602 while the display of content different from content 604 is not based on the spatial relationship of device 600 and device 602. In some examples, when device 600 has a vertical orientation, content 604 is displayed in an orientation different from the portrait orientation. In some examples, based on the determination that device 600 is to the left of device 602, device 602 displays content (e.g., content different from content 604). In some examples, based on the determination that device 600 is to the left of device 602 content 604 is moved from device 602 to device 600 and content different from 604 remains displayed by device 602. In some examples, when device 600 and device 602 concurrently display content and the determination is made that that device 600 is to the left of device 602, computer system 600 displays primary content (e.g., content of greater importance to user 610) and device 602 displays secondary content (e.g., content of less importance to user 610).

[0124] In some examples, based on the spatial positioning of device 600 and / or device 602 relative to user 610, a display setting is enabled. For example, at FIG. 6B, based on a determination that device 600 is to the left of device 602 (e.g., and / or based on a determination that device 600 is positioned to the left of user 610), a display setting (e.g.,brightness setting, resolution setting, frequency setting and / or inactive setting) of device 600 is enabled. More specifically, in some examples, based on the determination that device 600 is to the left of user 610 (e.g., and / or device 602), a display setting is enabled on device 600 that designates device 600 as a primary display (e.g., the primary display is responsible for displaying primary user interface objects) and a display setting is enabled on device 602 that designates device 602 as a secondary display (e.g., the secondary device is responsible for displaying secondary user interface objects). As an additional example, in some examples, based on the determination that device 600 is to the left of user 610 and device 602 is to the right of user 610, the right edge of device 600 and the left edge of device 602 are configured such that content 604 can be moved from the left edge of device 602 to the right edge of device 600 or vice versa and the left edge of device 600 and the right edge of device 602 are configured such that content cannot be moved passed the left edge of device 600 and content cannot be moved passed the right edge of device 602. In some examples, based on the spatial positioning of device 600 and / or device 602 relative to user 610, a display setting is disabled. For example, at FIG. 6B, based on a determination that device 600 is to the left of device 602 (e.g., and / or based on a determination that device 600 is positioned to the left of user 610), a display setting (e.g., brightness setting, resolution setting, frequency setting and / or inactive setting) of device 602 is disabled.

[0125] It should be noted that FIG. 6C illustrates an alternate scenario in which device 600 and device 600 are in different positions (e.g., device 600 is located to the right of device 602 and device 602 is located on the left of device 600). Either. FIG. 6B or FIG. 6C can follow FIG. 6A.

[0126] At FIG. 6C, the preference of user 610 continues to be that content 604 is displayed with a left justification. At FIG. 6C, media data is captured by and / or received from device 600 and device 602. At FIG. 6C, based on the media data, a determination is made that device 602 is positioned to the left of user 610 (e.g., from the perspective of user 610) and device 600 is positioned to the right of to user 610. Based on these two determinations, it is determined that device 602 is to the left of device 600 (e.g., from the perspective of user 610). As illustrated in FIG. 6C, based on the determination that device 602 is positioned to the left of device 600, (e.g., and / or based on the determination that device 602 is positioned to the left of user 610), content 604 is displayed by device 602 (e.g., and device 600 does not display content 604). As illustrated in FIG. 6C, because device 602 has a horizontalorientation, content 604 is displayed in a landscape orientation (e.g., a second manner different from the first manner illustrated in FIG. 6B). In some examples, when device 602 has a horizontal orientation, content 604 is displayed in an orientation different from the landscape orientation.

[0127] It should be noted that displaying content 604 in different manners results in content 604 being displayed in different locations. For example, as illustrated in FIG. 6B, content 604 is displayed in a portrait orientation (e.g., a first manner). To contrast, as illustrated in FIG. 6C, content 604 is displayed in a landscape orientation (e.g., a second manner). Based on the difference orientations between FIG. 6B and FIG. 6C, content 604 is displayed in different positions and / or different locations in FIG. 6B and FIG. 6C. As another example, in some examples, based on a determination that device 600 is positioned above device 602, content 604 is displayed via both device 600 and device 602 since neither device is positioned further leftward than the other (e.g., a different manner than the first manner and the second manner).

[0128] In some examples, when user 610 is not detected within the media data, content 604 ceases to be displayed. For example, in some examples, when user 610 is within the field of view of device 600 and / or device 602 and the sensor responsible for capturing the media data is covered, content 604 ceases to be displayed. For another example, in some examples, based on a determination that user 610 moves outside of the field-of-view of device 600 and / or device 602, content 604 ceases to be displayed.

[0129] In some examples, based on a determination that device 600 and device 602 have a new spatial positioning relative to user 610, content 604 is rearranged. For example, while content 604 is displayed via device 602 (e.g., as illustrated in FIG. 6C) and as a result of user 610 moving to the right, a determination is made that the distance between device 602 and user 610 is greater than a threshold. Based on the determination that the distance between user 610 and device 602 is greater than the threshold, content 604 is displayed via device 600 and not device 602.

[0130] In some examples, based on a determination that device 600 and device 602 have a new spatial positioning relative to user 610, content 604 is not rearranged. For example, while content 604 is displayed via device 602 (e.g., as illustrated in FIG. 6C) and as a result of user 610 moving to the right, a determination is made that the distance between user 610and device 602 is greater than the threshold. Content 604 continues to be displayed via device 602 though a determination is made that the distance between user 610 and device 602 is greater than the threshold, content 604 continues to be displayed via device 602.

[0131] In some examples, content 604 is moved between device 602 and device 600 based on a determination that the spatial relationship of device 600 and device 602 changes relative to user 610. For example, in a scenario where a determination is made that the relative positioning of device 600 and device 602 is changed (e.g., device 600 is moved relative to device 602, device 602 is moved relative to device 600 and / or device 600 and device 602 are moved relative to each other), the display of content 604 is moved from device 600 to device 602 and / or content 604 is moved from device 602 to device 600.

[0132] In some examples, the above discussion of FIGS. 6A-6C has device 600 and / or device 602 as display devices (e.g., computer displays, televisions, and / or head-mounted displays) whose display operations are controlled by an external device (e.g., a laptop, personal computing device, smartphone, and / or head mounted device). That is, instructions are transmitted (e.g., by the external device) to device 600 and / or device 602 that control the display operations of device 600 and / or device 602. For example, at FIG. 6B, when device 600 displays content 604 and device 602 does not display content 604, the transmitted instructions instruct device 600 to display content 604 and the transmitted instructions instruct device 602 to not display content 604. In other examples, the above discussion of FIGS. 6A-6C has device 600 or device 602 as display devices (e.g., computer displays, televisions, and / or head-mounted displays) whose display operations are controlled by the other device. For example, device 600 can transmit instructions to device 602 that instruct (e.g., and / or cause) device 602 to display a set of content or vice versa.

[0133] FIG. 7 is a flow diagram illustrating a method (e.g., method 700) for displaying content in accordance with some examples. Some operations in method 700 are, optionally, combined, the orders of some operations are, optionally, changed, and some operations are, optionally, omitted. Method 700 provides an intuitive way for displaying content. In some examples, method 700 is performed at a first device (e.g., a watch, a phone, a tablet, a fitness tracking device, a processor, a head-mounted display (HMD) device, a communal device, a media device, a speaker, a television, a display device, and / or a personal computing device) that is in communication with (and / or including) a display generation component (e.g., a display screen, a projector, and / or a touch-sensitive display).

[0134] The first device receives (702), from a second device (e.g., a watch, a phone, a tablet, a fitness tracking device, a processor, a head-mounted display (HMD) device, a communal device, a media device, a speaker, a television, a display device, and / or a personal computing device) and a third device (e.g., a watch, a phone, a tablet, a fitness tracking device, a processor, a head-mounted display (HMD) device, a communal device, a media device, a speaker, a television, a display device, and / or a personal computing device), media data (e.g., image, video, audio data, and / or anonymized metadata that is derived from image, video, audio, and / or media data) (e.g., the first device receives first media data from the second device and second media data from the third device), wherein the third device is different from the first device and the second device. In some examples, the third device is a different type of device than the second device and / or the first device. In some examples, the second device and the third device are the same types of devices. In some examples, the first device concurrently receives media data from the second device and the third device. In some examples, the first device serially receives media data from the second device and the third device. In some examples, the first device, the second device, and / or the third device includes the display generation component (e.g., each device can include a separate display generation component). In some examples, the media data consists of data from a single frame (e.g., a single frame of a still photo or a single frame from a video). In some examples, the media data includes data from multiple frames (e.g., multiple frames of a two or more still photo or a single frame from a video). In some examples, the multiple frames are captured by different cameras. In some examples, the multiple frames are captured by the same camera. In some examples, the media data is captured via a depth sensor (e.g., a LiDAR, stereo sensor, time- of-flight sensor, and / or structured light sensors).

[0135] After (704) receiving the media data (e.g., in response to receiving the media data or in conjunction with receiving the media data) and in accordance with a determination that a common object (e.g., a user, an animate object such as an individual and / or animal, an inanimate object such as a chair, keyboard, television, and / or desk) within the media data (e.g., a representation of the common object is included in the media data, the common object is within a field of view of one or more cameras of the second device and the third device, the common object is visible in resulting media) has a first orientation relative to (e.g., with respect to) the second device and the third device (e.g., and / or the first device) (e.g., the common object is to the side of both the second device and the third device, the common object is above the second device and below the third device or vice versa, the commonobject is above or below the second device and the third device, the common object is in front of the second device and / or the third device and / or the common object is in front of the second device and behind the third device or vice versa), the first device displays (706), via the display generation component, content (e.g., one or more user interface objects such as graphical object and / or textual object) in a first manner (e.g., the content is displayed by the second device and not the third device or vice versa, the content is displayed centered with respect to the second device and not the third device or vice versa, the content is displayed by both the second device and the third device, the content is displayed off center with respect to the second device or the third device, the content is displayed with respect to the positioning of the common object). In some examples, the display generation component includes a first display generation component and a second display generation component different from the second display generation component. In some examples, the second device includes the first display generation component (e.g., the first device and / or the third device does not include the first display generation component). In some examples, the third device includes the second display generation component (e.g., the first device and / or the second device does not include the second display generation component).

[0136] After (704) receiving the media data and in accordance with a determination that the common object within the media data has a second orientation (e.g., the common object is to the side of both the second device and the third device, the common object is above the second device and below the third device or vice versa, the common object is above or below the second device and the third device, the common object is in front of the second device and / or the third device and / or the common object is in front of the second device and behind the third device or vice versa), different from the first orientation, relative to (e.g., with respect to) the second device and the third device (e.g., and / or the first device) (e.g., the common object is to the side of both the second device and the third device, the common object is above the second device and below the third device or vice versa, the common object is above or below the second device and the third device, the common object is in front of the second device and / or the third device and / or the common object is in front of the second device and behind the third device or vice versa), the first device displays (708), via the display generation component, the content in a second manner (e.g., the content is displayed by the second device and not the third device or vice versa, the content is displayed centered with respect to the second device and not the third device or vice versa, the content is displayed by both the second device and the third device, the content is displayed off centerwith respect to the second device or the third device, the content is displayed with respect to the positioning of the common object) different from the first manner. In some examples, the second device includes the display generation component. In some examples, the third device includes the display generation component. In some examples, none of the first device, the second device, or the third device includes the display generation component. In some examples, the content is displayed while and / or before the first device receives the media data. In some examples, the media data is captured by one or more input devices such as a camera and / or a microphone that is included in the second device and / or the third device. In some examples, the second device is the first device. In some examples, the second device is different and / or distinct from the first device. In some examples, the first device transitions the display of the content from the first manner to the second manner based on a determination that the common object moves.

[0137] In some examples, displaying the content in the first manner includes displaying (e.g., via the display generation component) a set of one or more user interface objects (e.g., a set of one or more textual user interface objects and / or a set of one or more graphical user interface objects) at a first position (e.g., at the top, at the bottom, near an edge, and / or centered) (e.g., of a user interface and / or of the display generation component). In some examples, displaying the content in the second manner includes displaying the set of one or more user interface objects at a second position (e.g., at the top, at the bottom, near an edge, and / or centered) (e.g., of a user interface and / or of the display generation component) different from the first position. In some examples, the first device moves the set of one or more user interface objects from the first position to the second position based on a determination that the common object in the media data transitions from the first orientation to the second orientation. In some examples, the first device moves the set of one or more user interface objects from the first position to the second position based on a determination that the second device and / or the third move relative to each other. In some examples, the first device does not move the set of one or more user interface objects from the first position to the second position based on a determination that the common object in the media data transitions from the first orientation to the second orientation. In some examples, the appearance of the set of one or more user interface objects is different when the set of one or more user interface objects is displayed at the first position in contrast to when the set of one or more user interface objects is displayed at the second position.

[0138] In some examples, displaying the content in the first manner includes displaying a set of one or more user interface objects on a display screen of the second device. In some examples, displaying the content in the second manner includes displaying the set of one or more user interface objects includes display the set of one or more user interface objects on a display screen of the third device.

[0139] In some examples, displaying the content in the first manner includes displaying, via the display generation component (e.g., the first display generation component and / or the second display generation component), a set of one or more user interface objects (e.g., a set of one or more graphical user interface objects and / or a set of one or more textual user interface objects). In some examples, displaying the content in the second manner includes forgoing display of, via the display generation component (e.g., the first display generation component and / or the second display generation component), the set of one or more user interface objects (e.g., or any user interface object). In some examples, displaying the content in the second manner includes displaying the set of one or more user interface objects via a different display generation component than when displaying the content in the first manner (e.g., (1) displaying the content in the first manner includes displaying, via the first display generation component, the set of one or more user interface objects and (2) displaying the content in the second manner includes displaying, via the second display generation component, the set of one or more user interface objects). In some examples, when the set of one or more user interface objects is displayed as part of displaying the content in the second manner, the set of one or more user interface objects has a different appearance than when the set of one or more user interface objects is displayed as part of displaying the content in the first manner. In some examples, displaying the content in the first manner includes displaying the set of one or more user interface objects for a predetermined period of time (e.g., 1-60 seconds).

[0140] In some examples, after receiving the media data (e.g., or before receiving the media data or while receiving the media data) and in accordance with a determination, based on the positioning of the common object in the media data (e.g., the determination is based on the media data and / or the determination is made based on the positioning of the common object in the media data), that the second device and the third device have a first spatial relationship (e.g., the second device is above the third device, the second device is to the side of the third device, the second device is in front of the third device, and / or the second deviceis behind the third device), the first device enables a first setting (e.g., a display setting, such as a brightness setting, resolution setting, and / or frequency setting) of the first device (e.g., and / or of the display generation component, such as the first display generation component). In some examples, enabling the first setting of the first device includes configuring the first device to have the first setting. In some examples, enabling the first setting of the first device includes causing the first device to enable the first setting. In some examples, enabling the first setting of the first device includes displaying, via the first display generation component, content with the first setting active. In some examples, after receiving the media data and in accordance with a determination, based on the positioning of the common object in the media data (e.g., the determination is based on the media data and / or the determination is made based on the positioning of the common object in the media data), that the second device and the third device have a second spatial relationship (e.g., the second device is above the third device, the second device is to the side of the third device, the second device is in front of the third device, and / or the second device is behind the third device) different from the first spatial relationship, the first device enables a second setting (e.g., a brightness setting, resolution setting, and / or frequency setting) of the first device (e.g., and / or of the display generation component, such as the first display generation component) different from the first setting of the first device. In some examples, enabling the second setting of the first device includes configuring the second device to have the second setting. In some examples, enabling the second setting of the first device includes causing the first device to enable the second setting. In some examples, enabling the second setting of the first device includes displaying, via the first display generation component, content with the second setting active. In some examples, after receiving the media data (e.g., or before receiving the media data or while receiving the media data) and in accordance with a determination that the second device and the third device have the second spatial relationship, the first device disables (or forgoes enabling) the first setting of the first device. In some examples, after receiving the media data (e.g., or before receiving the media data or while receiving the media data) and in accordance with a determination that the second device and the third device have the second spatial relationship, the first device forgoes disabling the first setting of the first device. In some examples, the first setting and / or the second setting correspond to a display setting. In some examples, the first setting and / or the second setting correspond to a common external accessory of the first device. In some examples, after receiving the media data (e.g., or before receiving the media data or while receiving the media data) and in accordance with adetermination that the second device and the third device have the first spatial relationship or the second spatial relationship, the first device enables the first setting and the second setting.

[0141] In some examples, after receiving the media data (e.g., or before receiving the media data), in accordance with a determination, based on the positioning of the common object in the media data (e.g., the determination is based on the media data and / or the determination is made based on the positioning of the common object in the media data), that the second device and the third device have a first spatial relationship (e.g., the second device is above the third device, the second device is to the side of the third device, the second device is in front of the third device, and / or the second device is behind the third device), the first device disables a first setting (e.g., a display setting, such as a brightness setting, resolution setting, and / or frequency setting) of the first device (e.g., and / or of the display generation component, such as the first display generation component). In some examples, after receiving the media data, in accordance with a determination, based on the positioning of the common object in the media data (e.g., the determination is based on the media data and / or the determination is made based on the positioning of the common object in the media data), that the second device and the third device have a second spatial relationship (e.g., the second device is above the third device, the second device is to the side of the third device, the second device is in front of the third device, and / or the second device is behind the third device) different from the first spatial relationship, the first device disables a second setting (e.g., a display setting, such as a brightness setting, resolution setting, and / or frequency setting) of the first device (e.g., and / or of the display generation component) different from the first setting. In some examples, after receiving the media data (e.g., or before receiving the media data) and in accordance with a determination that the second device and the third device have the second spatial relationship, the first device disables and / or forgoes disabling the first setting of the first device. In some examples, the first setting and / or the second setting correspond to a display setting. In some examples, the first setting and / or the second setting correspond to a common external accessory of the first device. In some examples, after receiving the media data (e.g., or before receiving the media data) and in accordance with a determination that the second device and the third device have the first spatial relationship or the second spatial relationship, the first device disables the first setting and the second setting.

[0142] In some examples, the first device identifies (e.g., via a set of one or more cameras and / or via one or more sensors) a user of the first device as the common object (e.g., of the first device, second device, and / or third device) (e.g., a primary user and / or nonprimary user) (e.g., a user that is being tracked or a user that is not being tracked) (e.g., a face of the user). In some examples, a portion of the user is being tracked such as the head, hand, foot, torso, and / or lower torso of the user.

[0143] In some examples, the media data is captured via a red-green-blue (RGB) sensor (light to photocurrent sensor, light to analog voltage sensor, and / or light to digital voltage sensor).

[0144] In some examples, the media data is captured via an inertial measurement unit, ultra-wideband sensor, near field communication sensor, 60 gigahertz radar sensor, or any combination thereof.

[0145] In some examples, the first device is in communication with (e.g., and / or includes) (e.g., wireless communication and / or wired communication) an input device (e.g., a camera, a depth sensor, a microphone, a hardware input mechanism, a rotatable input mechanism, a heart monitor, a temperature sensor, and / or a touch-sensitive surface). In some examples, displaying the content in the first manner includes displaying a set of one or more user interface objects (e.g., a set of one or more graphical user interfaces objects and / or a set of one or more textual user interface objects). In some examples, while displaying the content in the first manner (e.g., or the second manner), the first device detects, via the input device, a change in a position of the common object (e.g., a change in the positioning of the common object relative to the first device, the second device, and / or the third device) (e.g., the common object moves to the right, to the left, above and / or beneath the first device, second device, and / or the third device). In some examples, the change in the position of the common object happens as a result of the first device, the second device, and / or the third device being moved. In some examples, in response to detecting the change in the position of the common object, the first device rearranges (e.g., reposition, arrange differently, regroup, and / or readjust) (e.g., via the display generation component, such as the first display generation component and / or the second display generation component) the set of one or more user interface objects (e.g., the set of one or more user interface objects is moved to the left, moved to the right, moved higher and / or moved lower). In some examples, rearranging the set of one or more user interface object is based on the change in the position of the commonobject. In some examples, the set of one or more user interface objects is rearranged for a period of time (e.g., 1-60 seconds). In some examples, the set of one or more user interface objects is rearranged to the initial configuration after the period of time has expired. In some examples, rearranging the set of one or more user interface objects includes moving the set of one or more user interface objects between different displays.

[0146] In some examples, the first device is in communication with (e.g., and / or includes) (e.g., wireless communication and / or wired communication) an input device (e.g., a camera, a depth sensor, a microphone, a hardware input mechanism, a rotatable input mechanism, a heart monitor, a temperature sensor, and / or a touch-sensitive surface). In some examples, displaying the content in the first manner includes displaying a set of one or more user interface objects (e.g., a set of one or more graphical user interfaces objects and / or a set of one or more textual user interface objects). In some examples, while displaying the content in the first manner (e.g., or the second manner), the first device detects, via the input device, a change in a position of the common object (e.g., a change in the positioning of the common object relative to the first device, second device, and / or third device) (e.g., the common object moves to the right, to the left, higher and / or lower). In some examples, the change in the position of the common object happens as a result of the first device, the second device, and / or the third device being moved. In some examples, in response to detecting the change in the position of the common object, the first device forgoes rearranging (e.g., via the display generation component) the set of one or more user interface objects.

[0147] In some examples, while displaying the content in the first manner (e.g., after receiving the first set of media data) (e.g., or the second manner), the first device detects (e.g., via a set of one or more cameras that the first device is in communication with and / or one or more depth sensors that the first device is in communication with) that a user is not positioned at a first location (e.g., the user has moved outside of the field-of-view of the first device and the third device, the user has left the first location, the user is not within the field- of-view of the first device and the third device for a predetermined period of time (e.g., 1-60 seconds, the user is not within a range of a microphone of the first device, second device and / or third device, and / or the user is not within a Bluetooth range of the first device, the second device, and / or the third device)) (e.g., the first location corresponds to the first device, second device, and / or the third device). In some examples, in response to detecting that the user is not at the first location, the first device ceases to display, via the display generationcomponent, the content (e.g., in the first manner and / or in any manner). In some examples, in response to detecting that the user is not at the first location, the first device continues to display the content in the first manner. In some examples, in response to detecting that the user is not at the first location, the first device displays the content in the second manner or a third manner different from the first manner and the second manner. In some examples, in response to detecting that the user is not at the first location, the first device displays different content (e.g., content that is different from the content that is displayed when the first user is not detected within the field-of-view of the first device and the third device).

[0148] In some examples, the media data is a first set of media data. In some examples, while displaying the content in the first manner (e.g., after receiving the first set of media data) (e.g., or the second manner), the first device receives, from the second device and the third device, a second set of media data (e.g., image, video, and / or audio data) (e.g., the first device receives media data from the second device and media data from the third device) different from the first set of media data (e.g., the second set of media data includes more data than the first set of media data, the second set of media data corresponds to a different point of view than the first set of media data, and / or the second set of media data is of a different quality than the first set of media data). In some examples, in response to receiving the second set of media data and in accordance with a determination that no common object is within (e.g., not included and / or the second media data does not have data that corresponds to the common object) the second set of media data, the first device ceases to display, via the display generation component, the content (e.g., in the first manner and / or in any manner). In some examples, in response to receiving the second set of media data and in accordance with a determination that a common object is within the second set of media data, the first device continues to display the content in the first manner. In some examples, in response to receiving the second set of media data and in accordance with a determination that no common object is within the second set of media data, the first device displays different content (e.g., content that is different than the content that is displayed when the second set of media data is received)

[0149] In some examples, after receiving the media data (e.g., in response to receiving the media data or in conjunction with receiving the media data) and in accordance with a determination that the media data does not have a common object, the first device displays, via the display generation component, the content in a default manner (e.g., the defaultmanner is selected by the user or the default manner is selected by the first device and / or the second device). In some examples, the first device and / or the second device displays the content and the third device does not display the content when the content is displayed in the default manner. In some examples, the third device displays the content and the first device and / or the second device do not display the content when the content is displayed in the default manner. In some examples, the first device, the second device, and / or the third device display the content when the content is displayed in the default manner.

[0150] In some examples, the first device is in communication with (e.g., and / or includes) an input device (e.g., a camera, a depth sensor, a microphone, a hardware input mechanism, a rotatable input mechanism, a heart monitor, a temperature sensor, and / or a touch-sensitive surface). In some examples, before receiving the media data, the first device detects, via the input device, a user (e.g., a primary user and / or non-primary user) (e.g., a user that is being tracked or a user that is not being tracked). In some examples, in response to detecting the user, the first device displays, via the display generation component, the content. In some examples, in response to detecting the user, the first device displays the content in a third manner that is different from and / or distinct from the first manner and / or the second manner.

[0151] In some examples, the display generation component includes a first display generation component of the second device and a second display generation component of the third device. In some examples, the second display generation component is different from the first display generation component. In some examples, the first display generation component and the second display generation component are the same or different types of display generation components. In some examples, displaying the content in the first manner (and / or second manner) includes moving, via the first display generation component and (e.g., or) the second display generation component, the content between being displayed via the first display generation component and being displayed via the second display generation component (e.g., moving the content from being displayed via the first display generation to being displayed via the second display generation component and / or vice versa). In some examples, displaying the content in the second manner includes moving, via the first display generation component and (e.g., or) the second display generation component, the content between being displayed via the first display generation and being displayed via the second display generation component. In some examples, displaying content in the first manner includes moving content from being displayed via the first display generation component tobeing displayed via the second display generation component and displaying the content in the second manner includes moving content from being displayed via the second display generation component to being displayed via the first display generation component and / or vice versa. In some examples, displaying the content in the first manner and / or the second manner includes displaying the content via the first display generation component and the second display generation component. In some examples, displaying the content in the first manner and / or second manner includes displaying the content via the first display generation component and not via the second display generation component or vice versa.

Claims

CLAIMSWhat is claimed is:

1. A method, comprising: at an electronic device: receiving, via a first sensor, a first image of a physical environment; updating, using the received first image, a first portion of a representation of the physical environment; receiving, via a second sensor different from the first sensor, a second image of the physical environment; and after updating, using the received first image, the first portion of the representation of the physical environment, and in accordance with a determination that a first set of one or more criteria is satisfied: updating, using the received second image, the first portion of the representation of the physical environment; and causing performance of an operation on an electronic device located in the physical environment, using the representation of the physical environment, responsive to a sensed event in the first portion of the physical environment.

2. The method of claim 1, further comprising: updating, using the received first image, a second portion of the representation, wherein the second portion is different from the first portion; and forego updating the second portion of the representation using the received second image.

3. The method of any one of claims 1-2, wherein the first set of one or more criteria includes a criterion that is satisfied when: an additional image was not received from the first sensor within a threshold duration, or the electronic device is not in communication with the first sensor.

4. The method of any one of claims 1-3, wherein the representation of the physical environment indicates a location of one or more objects in the physical environment.

5. The method of claim 4, wherein the representation of the physical environment includes an identification of respective capabilities of the one or more objects.

6. The method of any one of claims 1-5, further comprising: sensing, using a sensor, an event in the physical environment wherein the sensed event is based on one or more of: a presence of a subject in the physical environment, a position of the subject, an orientation of the subject, an identification of the subject, and an interaction of the subject with one or more objects in the physical environment.

7. The method of any one of claims 1-6, further comprising: sensing the sensed event while the electronic device is no longer in communication with the first sensor, wherein the event occurred in an area of the physical environment corresponding to the first portion.

8. The method of any one of claims 1-7, further comprising: sensing the sensed event after movement of the field of view of the first sensor, wherein the event occurred in an area of the physical environment corresponding to the first portion, and wherein sensed event involves the one or more objects in the physical environment.

9. The method of any one of claims 1-8, wherein performing the operation on the device includes changing a state of an electronic accessory located in the physical environment, and wherein the electronic accessory is not in a field of view of at least one or more of the first sensor and the second sensor.

10. The method of any one of claims 1-9, wherein the first sensor has a field of view of the environment different than the field of view of the second sensor.

11. The method of any one of claim 1-10, wherein the sensor has a field of view of the physical environment different than the fields of view of the first sensor and the second sensor.

12. A non-transitory computer-readable medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the method of any one of claims 1-11.

13. An electronic device, comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 1-11.

14. An electronic device, comprising: means for performing the method of any one of claims 1-11.

15. A computer program product, comprising one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the method of any one of claims 1-11.

Images