Image-based environmental control for electronic devices

Abstract

The subject disclosure provides systems and methods for understanding and facilitating activities of enrolled people within a home or other physical environment. A system may use images captured by one or more cameras located within and / or around the home or other physical environment to identify and track the locations of enrolled people within the home, and to control one or more controllable accessories around the home based on the locations of the enrolled people.

Description

IMAGE-BASED ENVIRONMENTAL CONTROL FOR ELECTRONIC DEVICESTECHNICAL FIELD

[0001] The present description relates generally to electronic devices including, for example, image-based environmental control for electronic devices.BACKGROUND

[0002] Homes are often provided with individual control components for individual electronic devices, such as light switches for turning on and off lights and thermostats for controlling a heater or an air conditioner. Other electronic devices within a home, such as televisions and speakers, can also be individually controlled using buttons, switches, or other control elements on the electronic devices themselves, or a dedicated remote control for an electronic device.BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Certain features of the subject technology are set forth in the appended claims. However, for purpose of explanation, several embodiments of the subject technology are set forth in the following figures.

[0004] FIG. 1 illustrates an example environment in accordance with one or more implementations.

[0005] FIG. 2 illustrates an example device that may implement a system for image-based environmental control for electronic devices in accordance with one or more implementations.

[0006] FIG. 3 illustrates an example of a portion of an enrollment process for image-based control in accordance with one or more implementations.

[0007] FIG. 4 illustrates an example of a map of a physical environment in accordance with one or more implementations.

[0008] FIG. 5 illustrates an example use case in which a location of a person’s feet is inferred based other detected portions of the person in accordance with one or more implementations.

[0009] FIG. 6 illustrates an example of tracking a location of a person with respect to a two-dimensional map in accordance with one or more implementations.

[0010] FIG. 7A illustrates an example of an unenrolled person in an authorized zone of a home in accordance with one or more implementations.

[0011] FIG. 7B illustrates an example of an unenrolled person in an unauthorized zone of a home in accordance with one or more implementations.

[0012] FIG. 8 illustrates an example of user-location-based control of a mobile device in accordance with one or more implementations.

[0013] FIG. 9 illustrates an example of image-based object location detection in accordance with one or more implementations.

[0014] FIG. 10 illustrates an example of image and model based control in accordance with one or more implementations.

[0015] FIG. 11 illustrates an example process flow for image-based control in accordance with one or more implementations.

[0016] FIG. 12 illustrates an example of user-location-based media output in accordance with one or more implementations.

[0017] FIG. 13 illustrates an example of image-based gesture control in accordance with one or more implementations.

[0018] FIG. 14 illustrates an example of a projected control element for image-based gesture control in accordance with one or more implementations.

[0019] FIG. 15 illustrates an example of an image-based intercom operation in accordance with one or more implementations.

[0020] FIGS. 16A-16C illustrate various poses that can be detected using images from one or more cameras in accordance with one or more implementations.

[0021] FIG. 17 illustrates another example process flow for image-based control in accordance with one or more implementations.

[0022] FIG. 18 illustrates aspects of a calibration operation in accordance with one or more implementations.

[0023] FIG. 19 illustrates a flow diagram of an example process for image-based control in accordance with one or more implementations.

[0024] FIG. 20 illustrates a flow diagram of an example process for image and model based control of devices within a home in accordance with one or more implementations.

[0025] FIG. 21 illustrates a flow diagram of another example process for location -based control in accordance with one or more implementations.

[0026] FIG. 22 illustrates a flow diagram of an example process for user-location-based control of a mobile device in accordance with one or more implementations.

[0027] FIG. 23 illustrates a flow diagram of an example process for user-location-based media output in accordance with one or more implementations.

[0028] FIG. 24 illustrates a flow diagram of an example process for image-based intercom operations in accordance with one or more implementations.

[0029] FIG. 25 illustrates a flow diagram of an example process for pose-based control in accordance with one or more implementations.

[0030] FIG. 26 illustrates an example electronic system with which aspects of the subject technology may be implemented in accordance with one or more implementations.DETAILED DESCRIPTION

[0031] The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology can be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein and can be practiced using one or more other implementations. In one or moreimplementations, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology.

[0032] Aspects of the subject technology provide systems, devices, and methods for understanding and facilitating activities of enrolled people (e.g., registered users) within a home or other physical environment. A system may use images captured by one or more cameras located within and / or around the home or other physical environment, to identify and track the locations of enrolled people within the home, and to control one or more controllable accessories around the home based on the locations of the enrolled people.

[0033] In accordance with one or more implementations, this control can include turning on and / or off lights, speakers, media devices, etc. as an enrolled person moves around the home, surfacing controllers on a mobile device of the enrolled person for controllable accessories at or near the location of the enrolled person, providing gesture-based control of the controllable accessories (e.g., on a wall or other surface of the home), locating objects (e.g., sunglasses, keys, wallet, phone, etc.) for an enrolled person using computer vision, providing intercom services within and / or among homes or other physical environments, and / or other control features for controlling aspects of the environment of the enrolled people.

[0034] FIG. 1 illustrates an example physical environment 100 that includes various devices in accordance with one or more implementations. Not all of the depicted components may be used in all implementations, however, and one or more implementations may include additional or different components than those shown in the figure. Variations in the arrangement and type of the components may be made without departing from the spirit or scope of the claims as set forth herein. Additional components, different components, or fewer components may be provided.

[0035] The physical environment 100 may include a house, a home, an office space, or any other indoor or outdoor physical environment. In the example of FIG. 1, the physical environment 100 may include and / or overlap with a network environment that includes electronic devices such as a smart speaker 102, a smart speaker 103, a media player 104 (e.g., a set top box), a media player 105 (e.g., a set top box), a mobile device 106, one or more cameras 107 (e.g., fixed-location cameras), one or more home hub devices 109, and and / or one or more wearable devices 137 (e.g., a smart watch, a head mountable device, a pendant device, etc.). In various implementations, the mobile device 106 may be implemented as, for example, a portable computing device such as a laptop computer, a smartphone, a tablet device, or the like.

[0036] As shown, the physical environment 100 (and / or network environment) may also include one or more other electronic devices (e.g., controllable accessories, such as fixed- location controllable accessories) that are controllable by one or more of the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, and / or the one or more wearable devices 137. As examples the other electronic devices may include one or more display devices 117 (e.g., televisions, monitors, or the like), one or more appliances 121 (e.g., washing machines, refrigerators, dryers, toasters, coffee makers, waffle makers, countertop cookers, dishwashers, stoves, ovens, heaters, air conditioners, and / or the like), one or more light sources 111 (e.g., light bulbs, lamps, and / or the like), one or more security devices 123 (e.g., one or more motion sensors, door sensors, window sensors, doorbell cameras, alarms, etc.), one or more climate control devices 113 (e.g., a thermostat or the like), one or more smart outlets, one or more smart switches, and / or any other controllable electronic devices and / or components that include and / or are communicatively coupled to, for example, one or more wired or wireless interfaces, such as WLAN radios, cellular radios, Bluetooth radios, Zigbee radios, near field communication (NFC) radios, and / or other wireless radios. In one or more implementations, one or more of the media players 104 and 105 may be integrated into, or may be separate from, the corresponding display device 117 or 119.

[0037] As shown, the cameras 107 may each have a field-of-view that includes at least a portion of the physical environment 100. The cameras 107 may be communicatively coupled to the home hub device 109. Cameras 107 may include visible light cameras, infrared cameras, and / or other image sensors. In one or more implementations, the home hub device 109 may be integrally formed and / or packaged with one of the cameras 107. In one or more other implementations, the home hub device 109 may be integrally formed and / or packaged with the smart speaker 102, the smart speaker 103, the media player 104, or the media player 105.

[0038] One or more of the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more appliances 121, the one or more light sources 111, the one or more display devices 117, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components in the physical environment 100 may be, and / or may include all or part of, the device discussed below with respect to FIG. 2, and / or the electronic system discussed below with respect to FIG. 26.

[0039] In one or more implementations, the home hub device 109 may also be configured to control, as one of the controllable accessories such as fixed-location controllable accessories in the physical environment, one or more of the smart speaker 102, the smart speaker 103, the media player 104, and / or the media player 105. In one or more implementations, control of a device (e.g., by the home hub device 109) may include turning a device on, turning a device off, waking a device from a sleep state, placing a device in a sleep state, initiating, stopping, pausing, and / or otherwise controlling output of media content (e.g., sound, video, or the like) from the device, increasing or decreasing an amount of an output (e.g., a volume of sound or a brightness of a display) from the device, switching between device modes of operation, scheduling start and / or stop times for device operations, and / or otherwise controlling (e.g., by providing instructions from the home hub device to the device) operation of the device.

[0040] As shown, the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more appliances 121, the one or more light sources 111, the one or more display devices 117, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components in the physical environment 100 may be connected to a local area network (“LAN”) 108, and / or may be connected (e.g., directly or via the LAN 108) to any of others of the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more display devices 117, the one or more wearable devices 137, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components. In one or more implementations, any or all of the devices of FIG. 1 may be connected (e.g., via the LAN 108), to a network 110, and / or one or more servers, such as server 114.

[0041] In one or more implementations, one, two, or more than two (e.g., all) of the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more display devices 117, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components may be associated with (e.g., registered to and / or signed into) a common account, such as an account (e.g., user account) with the server 114. As examples, the account may be an account of anindividual user (e.g., an enrolled person 101 that has enrolled, as described herein in connection with FIG. 3, with the home hub device 109 and / or the server 114 for participation in home control operations) or a group account (e.g., a family account or other group account for an enrolled person 101). In one or more implementations, the devices can be registered to different user accounts and the user accounts themselves may be grouped or otherwise associated with one another (e.g., user accounts of a family). In one or more implementations, any or all of the electronic devices of FIG. 1 may include a microphone for capturing sound input to that device and / or a speaker for outputting sound from that device.

[0042] In one or more implementations, the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more display devices 117, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components may form part of a connected home environment 116, and the LAN 108 may communicatively (directly or indirectly) couple any two or more of the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more display devices 117, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components within the connected home environment 116.

[0043] The network 110 may communicatively (directly or indirectly) couple any two or more of the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more display devices 117, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components with the server 114, for example, in conjunction with the LAN 108. In one or more implementations, some pairs or groups of the electronic devices of FIG. 1 (e.g., the mobile device 106 and the media player 105, the wearable device 137 and the media player 105, the media player 105 and the display device 117, the media player 104 and the display device 117, the mobile device 106 and the media player 104, or the mobile device 106 and the home hub device 109, etc.) may communicate directly over a secure direct connection in somescenarios, such as when the pairs or groups of the electronic devices are in proximity to each other.

[0044] Although the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more display devices 117, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components are depicted in FIG. 1 as forming a part of a connected home environment in which all of the devices are connected to the LAN 108, one or more of the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more display devices 117, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components may not be a part of the connected home environment and / or may not be connected to the LAN 108 at one or more times.

[0045] In one or more implementations, the LAN 108 may include one or more different network devices / network medium and / or may utilize one or more different wireless and / or wired network technologies, such as Ethernet, optical, Wi-Fi, Bluetooth, Zigbee, Powerline over Ethernet, coaxial, Ethernet, Z-Wave, cellular, or generally any wireless and / or wired network technology that may communicatively couple two or more devices.

[0046] In one or more implementations, the network 110 may be an interconnected network of devices that may include, and / or may be communicatively coupled to, the Internet. For explanatory purposes, the physical environment 100 is illustrated in FIG. 1 as including the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more display devices 117, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components, and the server 114; however, the physical environment 100 may include any number of electronic devices, cameras, accessories, and / or any number of servers.

[0047] In one or more implementations, one or more of the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more display devices 117, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components may provide a system for training a machine learning model using training data, where the trained machine learning model is subsequently deployed to that electronic device and / or other ones of the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more display devices 117, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components.

[0048] Further, the mobile device 106 and / or the home hub device 109 may provide one or more machine learning frameworks for training machine learning models and / or developing applications using such machine learning models. In an example, such machine learning frameworks can provide various machine learning algorithms and models for different problem domains in machine learning. In an example, one or more of the cameras 107, the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more display devices 117, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components may include a deployed machine learning model that provides an output of data corresponding to a prediction or transformation or some other type of machine learning input or output.

[0049] As shown in FIG. 1, at any of various times, the physical environment 100 may be occupied by one or more people and / or animals. For example, an enrolled person 101 may be located in one portion of the physical environment 100, an unenrolled person 133 may be located in another portion of the physical environment 100, and / or one or more animals, such as a pet 125, may be located in another portion of the physical environment 100. The physical environment 100 may also include other objects that are inanimate and / or not controllable, such as furniture 122. In one or more implementations, images of the physical environment 100, captured by the cameras 107, may be provided to the home hub device 109. The home hub device 109 may determine, based on the images from the cameras 107, the locations of the enrolled person 101, the unenrolled person 133, the pet 125, and / or any other people or animals that are located within the physical environment 100. As discussed infurther detail hereinafter in connection with various examples, based on the determined locations, the home hub device 109 may control one or more of the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more display devices 117, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components that are in proximity to the enrolled person 101, the unenrolled person 133, and / or the pet 125 (e.g., to provide image-based environmental control for the physical environment 100 and / or the enrolled person 101, the unenrolled person 133, and / or the pet 125).

[0050] In one or more implementations, the server 114 may be configured to perform operations in association with user accounts such as: storing data (e.g., user settings / preferences, files such as documents and / or photos, etc.) with respect to user accounts, sharing and / or sending data with other users with respect to user accounts, backing up device data with respect to user accounts, and / or associating devices and / or groups of devices with user accounts.

[0051] One or more of the servers such as the server 114 may be, and / or may include all or part of the device discussed below with respect to FIG. 2, and / or the electronic system discussed below with respect to FIG. 26. For explanatory purposes, a single server 114 is shown and discussed herein. However, one or more servers may be provided, and each different operation may be performed by the same or different servers.

[0052] FIG. 2 illustrates an example device that may implement a system for image-based control (e.g., image-based environmental control) for electronic devices in accordance with one or more implementations. For example, the device 200 of FIG. 2 can correspond to any of the smart speaker 102, the smart speaker 103, the media player 104, the media player 105, the mobile device 106, the one or more home hub devices 109, the one or more wearable devices 137, the one or more display devices 117, the one or more appliances 121, the one or more light sources 111, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components, and / or the server 114 of FIG. 1. Not all of the depicted components may be used in all implementations, however, and one or more implementations may include additional or different components than those shown in the figure. Variations in the arrangement and type of the components may be made withoutdeparting from the spirit or scope of the claims as set forth herein. Additional components, different components, or fewer components may be provided.

[0053] The device 200 may include a processor 202, a memory 204, a communication interface 206, an input device 208, and an output device 210. The processor 202 may include suitable logic, circuitry, and / or code that enable processing data and / or controlling operations of the device 200. In this regard, the processor 202 may be enabled to provide control signals to various other components of the device 200. The processor 202 may also control transfers of data between various portions of the device 200. Additionally, the processor 202 may enable implementation of an operating system or otherwise execute code to manage operations of the device 200.

[0054] The memory 204 may include suitable logic, circuitry, and / or code that enable storage of various types of information such as received data, generated data, code, and / or configuration information. The memory 204 may include, for example, random access memory (RAM), read-only memory (ROM), flash, and / or magnetic storage.

[0055] In one or more implementations, in a case where the device 200 corresponds to the home hub device 109, the memory 204 may store one or more detection models (e.g., trained machine learning models that have been trained to detect, based on images, a person, an animal, a portion of a person or animal, an identity of a person, an object, a gesture, a pose of a person, etc.) and / or information associated with one or more user accounts for one or more applications and / or services, using data stored locally in memory 204. Moreover, the input device 208 may include suitable logic, circuitry, and / or code for capturing input, such as audio input, sound input remote control input, touchscreen input, keyboard input, camera input, etc. The output device 210 may include suitable logic, circuitry, and / or code for generating notifications, alerts and / or other output, such as audio output, display output, light output, and / or haptic and / or other tactile output (e.g., vibrations, taps, etc.).

[0056] The communication interface 206 may include suitable logic, circuitry, and / or code that enables wired or wireless communication, such as between any of the electronic devices 102-107 and / or the server 114 over the network 110 (e.g., in conjunction with the LAN 108). The communication interface 206 may include, for example, one or more of a Bluetooth communication interface, a cellular interface, an NFC interface, a Zigbee communication interface, a WLAN communication interface, a USB communication interface, or generally any communication interface.

[0057] In one or more implementations, one or more of the processor 202, the memory 204, the communication interface 206, the input device 208, and / or one or more portions thereof, may be implemented in software (e.g., subroutines and code), may be implemented in hardware (e.g., an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a state machine, gated logic, discrete hardware components, or any other suitable devices) and / or a combination of both.

[0058] FIG. 3 illustrates an example enrollment operation that may be performed for enrolling a person with the connected home environment 116, the home hub device 109, and / or the server 114. As shown in FIG. 3, in one or more implementations, the device 200 may include one or more cameras 302 (e.g., one of the cameras 107 or another camera), one or more sensors 304 (e.g., depth sensors, LIDAR sensors, radar sensors, or the like), and / or processing circuitry 306 (e.g., memory 204 and / or processor 202). As illustrated by FIG. 3, an enrollment operation using the device 200 may include capturing one or more images of the person to be enrolled. For example, the device 200 may capture one or more images of the person’s face 310 and torso 312 in some implementations. As indicated in FIG. 3, the processing circuitry 306 may receive the images of the face 310 and the torso 312 (e.g., and / or sensor data from the sensor(s) 304), and generate (307) enrollment information for the enrolled person 101. In one or more implementations, the enrollment information may include stored images of the face 310 and / or torso 312 and / or encoded and / or encrypted information corresponding to the face 310 and / or torso 312. In one or more implementations, the images of the face 310 and / or encoded and / or encrypted information corresponding to the person’s face 310 may be persistently stored (with explicit permission of the person). In one or more implementations, the images of the torso 312 and / or encoded and / or encrypted information corresponding to the torso 312 may be updated (e.g., occasionally or periodically based on updated images captured by the camera(s) 107 in the physical environment 100) during operation of the home hub device 109, the cameras 107, and / or the connected home environment 116, such as to account for changes in the clothing of the enrolled person 101 over time. In one or more implementations, an identifier (e.g., a name or other text and / or numbers such as an email address, a telephone number, a username, or other identifier) may be obtained and stored with the enrollment information for an enrolled person 101.

[0059] Once a person has been enrolled (e.g., as the enrolled person 101), the home hub device 109 may determine and / or track the location of the enrolled person 101 while theenrolled person 101 is within the physical environment 100 and / or within the field-of-view of one or more of the cameras 107. For example, in one or more implementations, a two- dimensional map of the physical environment 100 may be generated (e.g., by the home hub device 109 and / or the server 114), and the location of the enrolled person 101 may be determined with respect to the two-dimensional map.

[0060] For example, FIG. 4 illustrates an example of a map 401 (e.g., a two-dimensional map or a three-dimensional map) of the physical environment 100. In the example of FIG. 4, the map 401 is a two-dimensional map that represents at least a portion of the physical environment 100 (e.g., one floor of a home) that includes a living room 400, a dining room 402, a kitchen 404, a bedroom 406, and a bathroom 408. This is merely illustrative, and it is understood that a physical environment, such as a home, may include any number and or types of rooms and or other spaces. Furthermore, a two-dimensional map may be a single layer two-dimensional map, or may include multiple two-dimensional layers (e.g., a two- dimensional layer for each of multiple floors or stories of a home or other building). In one or more implementations, a two-dimensional map may include the two-dimensional locations and / or boundaries of rooms and / or objects, without including the heights or other third dimensional information for the objects. A three-dimensional map may additionally include the heights or other third dimensional information for the objects.

[0061] In the example of FIG. 4, the map 401 indicates that the physical environment 100 can be subdivided into various zones, such as operational zones. In the example of FIG. 4, the map 401 indicates that the physical environment 100 includes a zone I, corresponding to an entryway, a zone II, corresponding to the living room 400, a zone III, corresponding to the kitchen 404, a zone IV, corresponding to the dining room 402, a zone V, corresponding to the bedroom 406, a zone VI, corresponding to the bathroom 408, and a zone VII corresponding to a hallway between the bedroom 406 and the living room 400, the dining room 402, and the kitchen 404.

[0062] In the example of FIG. 4, the home hub device 109 and a camera 107 are located in the living room 400. In this example, a display device 117 is disposed on or near a wall 440 of the living room 400, and a couch 420, a coffee table 422, and an end table 424 (e.g., corresponding to furniture 122 of FIG. 1) are located in the living room 400. In this example, controllable accessories in the living room include a light source 111 (e.g., a table lamp) located on the end table 424, and a climate control device 113 (e.g., a thermostat) provided on another wall of the living room 400. In this example, a dining room table 426 is located inthe dining room 402, and an island 428 and a counter 430 are located in the kitchen 404. In this example, controllable accessories in the kitchen include a smart speaker 102, an overhead light source 111, and one or more controllable appliances, such as a refrigerator 436 and a stove 438. In this example, a bed 432 and two end tables 434 are located in the bedroom 406. In this example, controllable accessories in the bedroom 406 include light sources 111 at or near each of the end tables 434. In this example, controllable accessories in the bathroom 408 include a smart speaker 103.

[0063] As shown in the example of FIG. 4, one or more additional cameras may be located in the physical environment 100, such as a camera 107 in the dining room 402 and a camera 107 in the bedroom 406. It is appreciated that the cameras 107 that are positioned within the physical environment 100, and with a field-of-view that includes portions of the physical environment 100, are placed and authorized by the person or people who occupy the physical environment 100, and with the permission of the person or people who occupy the physical environment 100.

[0064] As indicated in FIG. 4, the home hub device 109 may determine a location 450 of a person within the physical environment 100. For example, the location 450 may be a location of the enrolled person 101. Determining the location 450 of the enrolled person 101 may include detecting the enrolled person by detecting a person and / or a portion thereof in one or more images from one or more cameras 107, matching the detected person and / or portion thereof to an enrolled person, and obtaining a previously stored identifier of the enrolled person (e.g., a name, a number, or other unique identifier), previously stored during an enrollment operation for the enrolled person. As shown, the home hub device 109 may determine the location 450 of the enrolled person 101 relative to (e.g., on) the map 401. In one more implementations, the map 401, including an indicator of the location 450, may be visible by a user, such as the enrolled person 101, such as by displaying the map 401 on the display device 117 and / or the mobile device 106. As indicated in FIG. 4, the home hub device 109 may determine a location 452 of another person within the physical environment 100. For example, the location 452 may be the location of another enrolled person, or an unenrolled person 133, within the physical environment 100.

[0065] In one or more implementations, the home hub device 109 may be programmed to, and or may learn to (e.g., by adjusting weights and / or other parameters of one or more machine learning models), control one or more of the control accessories within the physical environment 100 based on the location 450 of an enrolled user (e.g., enrolled person 101) andor a location 452 of an unenrolled user (e.g., unenrolled person 133). For example, based on the location 450 of the enrolled person 101 in the living room 400, the home hub device 109 may turn on the light source 111 on the end table 424, may output media content associated with the enrolled user (e.g., enrolled person 101) (e.g., previously watched or favorited, or associated with one or more user-provided or learned preferences) on the display device 117, and / or may set one or more climate settings at the climate control device 113 (e.g., a thermostat) based on one or more user-provided and / or learned climate preferences of the enrolled user (e.g., enrolled person 101). As another example, based on the location 452 of the unenrolled person 133, the home hub device 109 may turn on the light source 111 over the island 428, may output general media content (e.g., unassociated with the unenrolled person 133) from the smart speaker 102, and / or may operate the refrigerator 436 or the stove 438 based in part on the presence of a person in the kitchen 404. As another example, based on the location 450 of the enrolled person 101 in the living room 400 and the location 452 of the unenrolled person 133 in the kitchen 404, the home hub device 109 may turn off or power down the light sources 111 at or near the end tables 434 in the bedroom 406, and or turn off or power down the smart speaker 103 in the bathroom 408.

[0066] In one or more implementations, determining a location of a person, such as the location 450 of the enrolled person 101, may include detecting at least a portion of the enrolled person 101 in one or more images from one or more of the cameras 107, and determining, based on at least the detected portion of the enrolled person in the one or more images, a location of the person’s feet with respect to the map 401. For example, determining the location of the person’s feet with respect to the map 401 may include detecting one or more of the person’s feet in one or more of the images, and transforming or projecting the location of the person’s feet in the one or more images to a location in the map 401. However, as illustrated in FIG. 5, in one or more use cases, a location 500 of the feet of the enrolled person 101 may not be visible to any of the cameras 107. In the example of FIG. 5, a piece of furniture 122, such as a couch or a chair, blocks a view of the feet of the enrolled person 101 by the camera 107. In one or more implementations, the home hub device 109 may infer the location 500 of the feet of the enrolled person 101, based on a detection of one or more other portions of the user, such as the user’s face 310 and or torso 312. Inferring the location of the feet of the enrolled person may be performed, in an image space, prior to transformation of the location of the feet to a location in the map 401, or the location of the feet of the enrolled person may be inferred directly in the physical space corresponding to the map 401.

[0067] In addition to determining the location of one or more people and / or pets within the physical environment 100, the home hub device 109 may track the location of one or more people and / or pets within the physical environment 100 overtime. For example, FIG. 6 illustrates a use case in which the enrolled person 101 has moved, along a path 602, from a prior location 600 to a current location 450. In this example, the prior location 600 is in the living room 400 (e.g., zone II), as in the example of FIG. 4, and the current location 450 is in the bathroom 408 (e.g., zone VI). In this example, the physical environment 100 does not include any cameras in the bathroom 408 (e.g., for privacy purposes). However, by tracking the path 602 of the enrolled person 101 from the prior location 600 to the current location 450, the home hub device 109 may be able to infer the location 450 of the enrolled person 101 in the bathroom 408 (e.g., by determining that the path 602 leads into a space, corresponding to the bathroom, within the physical environment 100 in which no cameras 107 are available). In the example of FIG. 6, the home hub device 109 may control one or more of the controllable accessories in the living room 400 and or the bathroom 408 based on a new location 450 of the enrolled person 101 in the bathroom 408. For example, based on the new location 450 of the enrolled person 101 within the bathroom 408, the home hub device 109 may turn on one or more light sources within the bathroom 408, may turn off one or more light sources 111 in the living room 400, and / or may move the output of media content associated with the enrolled person 101 from the living room to the bathroom. For example, in the use case in which the enrolled person 101 was watching a show or listening to music using the display device 117 in the living room 400, and then walked along the path 602 into the bathroom 408, the home hub device may, based on the location 450 of the enrolled person 101 in the bathroom 408, initiate output of at least an audio portion of the show or music using the smart speaker 103 in the bathroom 408. In this way, the enrolled person 101 can continue to listen to the media content that they were previously listening to in the living room 400, while in the bathroom 408 or any other location within the physical environment (e.g., without having to provide any user input to an electronic device to cause the media content to follow them through the physical environment). In one or more implementations, (e.g., if the location 450 of the enrolled person 101 is away from the living room 400 for a predetermined amount of time), the home hub device 109 may cease outputting the media content by the display device 117 in the living room 400.

[0068] FIGS. 7A and 7B illustrate another example of location-based control of the physical environment 100. In the example of FIG. 7A, the home hub device 109 determines, using one or more images from the camera 107 in the living room 400, that an unenrolledperson (e.g., unenrolled person 133) is at a location 700 within zone 1 of the physical environment 100. For example, zone 1 may correspond to an entryway to the physical environment 100, such as at or near a front door 701 of a home. In one or more implementations, the home hub device 109 may have been programmed, and or may have learned, that unenrolled persons are permitted to be within zone I of the physical environment 100. For example, this may allow delivery persons to enter partway into a home to leave packages at a secure location inside the front door 701.

[0069] As illustrated in figure 7B, if the unenrolled person is detected at a location 703 that is outside of zone I, and within the physical environment 100, the home hub device 109 may cause one or more devices within and / or associated with the physical environment 100 to generate an alert. For example, the home hub device 109 may cause the display device 117 to generate an alert 704. For example, the alert 704 may be an audible alert or alarm, or a visual alert, such as including text indicating to the unenrolled person that they are in an unauthorized portion of the physical environment and should immediately leave. As another example, the home hub device 109 may transmit an alert to the mobile device 106 of an enrolled person 101. As shown an alert 702 may be displayed and or otherwise output by the mobile device 106, indicating that an unenrolled person 133 is in an unauthorized portion of the physical environment 100. The alert 702 can be displayed by the mobile device 106 while the mobile device 106 is within the physical environment 100 or outside the physical environment 100. In various implementations, the zones of the physical environment 100 may be set at the home hub device 109 by a user, or may be learned by the home hub device 109 based on images of the physical environment 100 from the cameras 107, the map 401, and / or behaviors or activities of people and / or pets within the physical environment, as determined using the images from the cameras 107.

[0070] As discussed herein, in one or more implementations, one or more of the controllable accessories within the physical environment 100 may be controllable by a device other than the home hub device 109, such as the mobile device 106 or the wearable device 137. However, it can be inconvenient and or inefficient for a user of the mobile device 106 to scroll through all of the controllable accessories in the entire physical environment 100 in order to find one or more nearby control the accessories for control. Although mobile devices, such as smartphones, have their own location detection technology, such as Global Positioning System technology, such technology may not be able to provide the location of the mobile device 106 with an accuracy that is sufficient to distinguish different locations within a physical environment such as a home. In one or more implementations, the homehub device 109 may provide (e.g., transmit) a location 450 of an enrolled user (e.g., enrolled person 101) that is associated with the mobile device 106, to that mobile device 106. The mobile device 106 may be provided with the ability to estimate its own location within the physical environment 100, based on the provided location of the user of that device.

[0071] FIG. 8 illustrates an example in which home hub device 109 provides the location of the enrolled person 101 to the mobile device 106 while the mobile device 106 is within the physical environment 100. In this example, while the enrolled person 101 is at a location 450 within the bedroom 406, the home hub device 109 provides an indication 804 of the location 450 of the enrolled person 101 to the mobile device 106. As shown, based on the location 450 within zone V corresponding to the bedroom 406, the mobile device 106 may surface one or more control elements 806 for controlling the controllable accessories within zone V. In this example, the control elements 806 include a control element 816 for a first one of the light sources 111 (e.g., “Lamp B”) in the bedroom 408, and a control element 818 for a second one of the light sources 111 (e.g., “Lamp C”) in the bedroom 408. In this way, the enrolled person 101 may be provided with the ability to use the mobile device 106 to turn on and / or off light sources 111 in the bedroom 406 (e.g., by transmitting instructions to the light sources 111 directly or via the home hub device 109) while the enrolled person 101 and the mobile device 106 are in the bedroom 406.

[0072] FIG. 8 also illustrates how, at an earlier time, when the enrolled person 101 was at a prior location 600 in the living room 400, the home hub device 109 provided an indication 800 of the prior location 600 in zone II, corresponding to the living room 400. As shown, based on the prior location 600 within zone II, the mobile device 106 surfaced control elements 802 for controlling one or more controllable accessories in zone II. In this example, the control elements 802 include a control element 810 for controlling the display device 117 (e.g., a television, or TV), a control element 812 for controlling the light source 111 on the end table 424, and a control element 814 for controlling the thermostat such as climate control device 113 (e.g., to raise or lower the temperature within the physical environment or a portion thereof). In this way, even if the mobile device 106 itself is unable to determine its precise location within the physical environment 100, the mobile device 106 may be provided with the ability to surface a relevant set of control elements for the controllable accessories that are nearby the current location of the enrolled person 101.

[0073] In one or more implementations, the mobile device 106 may perform a confirmation operation prior to surfacing the control elements 802 or 806 based on the location of theenrolled person 101. For example, the mobile device 106 may use one or more sensors of the mobile device 106 itself, such as one or more accelerometers and / or other inertial measurement unit sensors within the mobile device 106, to determine whether the mobile device 106 is moving with the location of the enrolled person 101, prior to surfacing the control elements 806 or 802. In this way, the mobile device 106 may avoid surfacing the control elements 806 for the controllable accessories in the bedroom 406 in a use case in which the enrolled person 101 is at the location 450 within zone V, but the mobile device 106 has been left behind in zone II.

[0074] FIG. 9 illustrates an example use case in which the home hub device 109 controls one or more other devices in the physical environment 100 to indicate a location of an object within the physical environment 100. For example, the home hub device 109 may be provided with one or more object detection models 900 that are trained for detection of one or more particular objects. In the example of FIG. 9, the home hub device 109 has detected, using the object detection model 900, a pair of sunglasses 902 on a piece of furniture 122, using one or more images from one or more cameras 107. In this example the home hub device 109 may operate a light source 111 near the location of the sunglasses 902, such as a table lamp on an end table 424, to indicate the location of the detected sunglasses 902. For example, the home hub device 109 may transmit an instruction 905 to the light source 111 to turn on to indicate the (e.g., approximate) location of the sunglasses 902. As shown in FIG. 9, the home hub device 109 may also, or alternatively, provide an indication 903 of the location of the sunglasses 902 to another device such as the mobile device 106. As shown, the mobile device 106 may display an image of the map 401, with an indicator 904 of the location of the sunglasses 902, to allow a user of the mobile device 106 (e.g., the enrolled person 101) to navigate to the location of the sunglasses 902. In this way, the home hub device 109 and the cameras 107 may be used to help a person keep track of and / or locate important and / or lost items, such as keys, wallets, sunglasses, paperwork, purses, bags, jewelry, eyeglasses, remote controls, or the like.

[0075] In one or more implementations, the home hub device 109 may control one or more other devices (e.g., controllable accessories) in the physical environment 100 based on one or more models of the physical environment 100. For example, as shown in FIG. 10, the home hub device 109 may be configured to generate control instructions for one or more other devices within the home based at least in part on a video view of the home (e.g., based on one or more images from one or more cameras 107 captured over time), and a set 1000 of models of the home. For example, the home hub device 109 may be configured to generate controlinstructions for one or more other devices within the home based at least in part on a video view of the home (e.g., based on one or more images from one or more cameras 107) based on a topological model 1002 of the physical environment 100, an operational model 1004 of the physical environment 100, and a behavioral model 1006 of the physical environment 100. Any or all of the topological model 1002, the operational model 1004, and / or the behavioral model 1006 may be trained based on one or more images (e.g., a video view) of the physical environment 100 from the camera(s) 107. Any or all of the topological model 1002, the operational model 1004, and / or the behavioral model 1006 may be configured to, after training, receive one or more images (e.g., a video view, such as a sequence of images over time) as input, and to generate an output responsive to receiving the one or more images.

[0076] For example, the topological model 1002 may include a map (e.g., the map 401) of the physical environment 100. For example, the operational model 1004 may include a model of one or more operational combinations of the other devices. For example, the one or more operational combinations may include one or more learned sets of devices that are commonly (e.g., frequently or routinely) operated together (e.g., a display device 117 in the living room 400 may be commonly turned on at the same or similar time that the lamp on the end table 424 is turned on, or the light sources in the bedroom 406 may be commonly turned on at or near the same time that the lamp on the end table 424 is turned off). For example, the behavioral model 1006 may include a model of one or more learned activities of one or more enrolled people for the home. For example, the one or more learned activities may be associated with a location within the home. For example, the one or more learned activities may include activities associated with a time. For example, the one or more learned activities may include a learned behavior of an enrolled person 101 in which the enrolled person 101 commonly turns on a particular set of light sources 111 upon arriving home after work (e.g., at or near 5pm). In this example, based at least in part on the behavioral model 1006 (e.g., and factoring in the current location of the enrolled person 101 relative to the topological model 1002 and / or device interoperability information in the operational model 1004), the home hub device 109 may turn on that particular set of light sources 111 when (or just before) the enrolled person 101 arrives home from work (e.g., at or near 4:55pm).

[0077] As another example, the one or more learned activities may include a learned behavior of an enrolled person 101, in which the enrolled person 101 commonly turns off the display device 117 (e.g., a television) and turns on a reading light (e.g., on the end table 424) at a particular time (e.g., at or near a bedtime of the enrolled person 101). In this example, based at least in part on the behavioral model 1006 (e.g., and factoring in the current locationof the enrolled person 101 relative to the topological model 1002 and / or device interoperability information in the operational model 1004), the home hub device 109 may turn on the reading light when the display device 117 is turned off , and / or or may turn off the display device 117 at a particular time. As another example, the one or more learned activities may include a learned behavior of an enrolled person 101 in which the enrolled person 101 commonly plays a particular podcast or a particular show during a morning period of time on certain days of the week. In this example, based at least in part on the behavioral model 1006 (e.g., and factoring in the current location of the enrolled person 101 relative to the topological model 1002 and / or device interoperability information in the operational model 1004), the home hub device 109 may (e.g., automatically and without user input) operate the display device 117 and / or a smart speaker that is near the current location of the enrolled person to play the particular podcast or show during the morning period of time.

[0078] In the example of FIG. 10, a control application 1008 receives the outputs of the topological model 1002, the operational model 1004, and / or the behavioral model 1006, and, responsively, generates instructions (e.g., control instructions) for one or more other devices (e.g., light sources, media players, display devices, smart speakers, etc.) based on the outputs of the models. In one or more other implementations, the control application may receive the images and the topological model 1002, the operational model 1004, and / or the behavioral model 1006 themselves, and generate the instructions based on the images and the received models. A control application 1008 may be a general control application for controlling multiple controllable accessories, or may be a control application specifically for controlling one particular controllable accessory (e.g., a television-related application for operating a television) or one particular type or category of controllable accessory (e.g., a smart switch operation or smart outlet application for controlling any device plugged into a smart switch or outlet, or an appliance-related application for controlling appliances from a particular manufacturer).

[0079] FIG. 11 illustrates an example process flow for image-based control in accordance with one or more implementations. As shown in FIG. 11, the home hub device 109 may receive one or more images from the camera(s) 107, and perform one or more detection operations 1100 using the images. In one or more implementations, the images from the cameras 107 may be reduced in resolution before performing the detection operations 1100. For example, the detection operations 1100 may detect a person (e.g., an enrolled person), a part of a person (e.g., a face, a torso, etc.), an animal, and / or an object in an image space of the images.

[0080] In one or more implementations, the home hub device 109 may then perform a transformation operation 1102 one or more locations, in image space, of one or more detected persons, portions thereof, animals, and / or objects. For example, the transformation operation 1102 may project (e.g., using a homography or plane-to-plane transformation) the one or more locations, in image space, to one or more corresponding locations in physical space (e.g., with respect to the map 401) of the physical environment 100. As shown, the transformation operation 1102 may be performed using calibration information. For example, the calibration information may have been obtained during a prior calibration operation (e.g., as discussed in further detail hereinafter in connection with FIG. 18).

[0081] As shown in FIG. 11, the home hub device 109 may generate, based on the one or more locations in physical space as received from the transformation operation 1102, one or more control instructions for controlling an electronic component (e.g., one of the controllable accessories) in the physical environment 100. For example, an instruction generation operation 1104 may generate instructions for controlling one or more components (e.g., controllable accessories) in the physical environment 100 based on the location(s) of one or more people in the physical environment, and based on component information for the physical environment 100. For example, the component information may include the locations of the various electronic components within the physical environment, component types, connection information, pairing information, and / or other information for connecting to and / or controlling the various electronic components. In one or more implementations, the component information may be included as part of the map 401, or may be provided separately from the map 401. In one or more implementations, the instruction generation operation 1104 may be performed by the control application 1008 of FIG. 10.

[0082] FIG. 12 illustrates an example of controlling an electronic component based on a determined location of a person (e.g., the enrolled person 101). In the example of FIG. 12, at a first time, the home hub device 109 determines that a location 450 of an enrolled person 101 is in the living room 400 (e.g., in zone II), and media content 1200 (e.g., video content and / or audio content) is being output by a display device 117 in the living room 400. As shown, the home hub device 109 may later determine that the enrolled person 101 has moved from a prior location 600 in the living room 400 to a location 450 in the bathroom 408. In this example, the home hub device 109 moves (e.g., at least a portion, such as an audio portion of) the output of the media content 1200 from the display device 117 to the smart speaker 103 in the bathroom 408. In this way, the home hub device 109 can control theoutput of media content to follow the location of the enrolled person 101 as they move around the physical environment 100.

[0083] In one or more implementations, the home hub device 109 may be configured to detect gestures performed by person (e.g., an enrolled person 101) for controlling one or more of the controllable accessories in the physical environment 100. For example, FIG. 13 illustrates an example use case in which an enrolled user (e.g., enrolled person 101) performs a gesture of 1300 on a surface in the physical environment 100. In this example, the gesture 1300 is performed on an arm 1304 of a couch or sofa. In other examples, the gesture 1300 may be performed on another surface of the sofa or couch, on a wall 1306, on the floor, on the end table 424, or on any other surface of the physical environment 100 that is within the field-of-view of the camera 107. In this example, the home hub device 109 may, using a gesture detection model 1301, detect the gesture 1300 and control the light source 111 on the end table 424, such as to increase or decrease the brightness of the light source 111, to turn on the light source 111, or to turn off the light source 111. By performing gesture detection at the home hub device 109 using the images from the cameras 107, the entire physical environment 100 can become a set of available surfaces for performing gesture controls for controlling controllable accessories in the physical environment 100 (e.g., which can reduce the cost and complexity of the home hub device 109 by reducing or eliminating the inclusion of user interface components such as touchscreens or touch surfaces).

[0084] In one or more implementations, the home hub device 109 may also control one or more additional light sources 1400, such as one or more projectors, to project an image of a control element onto a surface in the physical environment 100. For example, in the example of FIG. 14, a light source 1400 is controlled, by the home hub device 109, to project an image 1402 of a control element for the light source 111 onto the surface 1404 of the end table 424. It is appreciated that the image 1402 of the control element can be projected on to the arm 1304 of the furniture 122, or any other surface in the physical environment 100. In this way, in addition to providing gesture-based control using images from the cameras 107, the home hub device 109 can project images to facilitate a user’s understanding of what gestures are available and / or where those gestures can be performed.

[0085] In one or more implementations, one or more home hub devices 109 may provide image-based intercom services. For example, FIG. 15 illustrates a use case in which an enrolled person 101 in the living room 400 of the physical environment 100 provides a request 1501 to a home hub device 109 in the physical environment 100 (e.g., via voice inputto a smart speaker 102 having a microphone) to speak to another enrolled person 1506 in another physical environment 1500. For example, the other physical environment 1500 may be another room within the same home as the living room 400, or may be an entirely separate physical environment, such as another home remote from the physical environment 100. In this example, the request 1501 from the enrolled person 101 may include a name of the other enrolled person 1506.

[0086] For example, the enrolled person 101 may simply say “ Hey Jim”. A microphone signal generated by one or more microphones at the smart speaker 102 may be provided to the home hub device 109. The home hub device 109 may process the microphone signals and determine that the name “Jim” corresponds to the other enrolled person 1506. The home hub device 109 may determine (e.g., based on images from one or more cameras 107 at the other physical environment 1500 and / or based on communication with another home hub device 109 at the other physical environment 1500) the location of the other enrolled person 1506. Based on the location of the other enrolled person 1506, and the locations (e.g., as known from a two-dimensional map of the other physical environment) of one or more controllable accessories, such as a smart speaker 1502 at the other physical environment 1500, the home hub device 109 may cause output of the voice input received at the smart speaker 102 by the smart speaker 1502 (e.g., to be heard by the other enrolled person 1506). The other enrolled person 1506 may then speak a reply to the enrolled person 101, which may be received by a microphone of the smart speaker 1502 and transmitted (e.g., by the home hub device at the physical environment 1500) to the home hub device 109 at the physical environment 100, for output to the enrolled person 101 using a controllable accessory (e.g., the smart speaker 102 or another device with a speaker if the enrolled person 101 has moved since speaking the request 1501) near the enrolled person 101. In this way, the home hub device(s) 109 in one or more physical environments can use images from one or more cameras to allow enrolled people in remote locations to speak to naturally each other without, for example, initiating a call or a video conference using a mobile device, essentially as if they were in the same room (e.g., even without knowing each other’s locations).

[0087] In one or more implementations, the home hub device 109 may also be configured to detect a pose or position of an enrolled person 101 or an unenrolled person 133, based on one or more images from one or more cameras 107. For example, FIG. 16A illustrates a use case in which an enrolled person 101 is in a standing position within a field-of-view of a camera 107. In this example use case, the home hub device 109 may determine, based on one or more images from one or more cameras 107, that the enrolled person 101 is standing. Asanother example, FIG. 16B illustrates a use case in which an enrolled person 101 is in a sitting position within a field-of-view of a camera 107. In this example use case, the home hub device 109 may determine, based on one or more images from one or more cameras 107, that the enrolled person 101 is sitting. In this example use case, the home hub device 109 may control one or more controllable accessories near the seated location of the enrolled person 101 based on the location and the seated pose of the enrolled person 101 (e.g., which may indicate that the enrolled person 101 may remain at their current location for long enough to turn on a nearby light or play music from a nearby speaker).

[0088] As another example, FIG. 16C illustrates a use case in which an enrolled person 101 is in a laying position within a field-of-view of a camera 107. In this example use case, the home hub device 109 may determine, based on one or more images from one or more cameras 107, that the enrolled person 101 is laying down. In this example use case, the home hub device 109 may determine, responsive to detecting the enrolled person 101 in the laying position, whether an emergency condition is present. For example, the home hub device 109 may determine that the enrolled person 101 is laying down in their bed during the night, and determine that no emergency condition is present. As another example, the home hub device 109 may determine that the enrolled person is laying down on the kitchen floor, for at least a predetermined amount of time (e.g., more than five seconds, more than ten seconds, more than thirty seconds, more than one minute, or more than five minutes, such as without changing position), and that an emergency condition may thus be present. In one or more implementations, when the home hub device 109 determines that an emergency condition is present, the home hub device may cause one or more other devices, within the physical environment 100 and / or outside the physical environment 100, to generate an alert indicating the emergency condition. For example, the home hub device 109 may transmit an alert to a mobile device 106 of the enrolled person 101 or to a device of another person, such as an emergency contact of the enrolled person 101.

[0089] FIG. 17 illustrates additional details of an example process flow for image-based control, in accordance with one or more implementations. As shown in FIG. 17, in one or more implementations, the detection operation 1100 may be performed by one or more detection models. For example, the detection models may each be trained to detect one or more types of elements in an image (or sets and / or series of image). In the example of FIG. 17, the detection operations 1100 are performed by a face detector 1606, a torso detector1608, a person detector 1609, and an object detector 1610. For example, the face detector1606 may be a machine learning model that has been trained to detect human faces in images.For example, the torso detector 1608 may be a machine learning model that has been trained to detect human torsos in images. For example, the person detector 1609 may be a machine learning model that has been trained to detect a person in an image. For example, the object detector 1610 may be a machine learning model that has been trained to detect one or more particular objects (e.g., sunglasses, keys, wallets, etc.) in one or more images.

[0090] As illustrated in FIG. 17, in one or more implementations, enrollment information (e.g., obtained during one or more enrollment operations of the type shown in FIG. 3) may also be provided to any or all of the detectors in the detection operations 1100. For example, providing the enrollment information to the face detector 1606 may allow the face detector 1606 to detect and identify a face of a particular enrolled person. However, in one or more other implementations the detectors of the detection operations 1100 may perform detection operations without using enrollment information. For example, in one or more implementations, enrollment information may be provided to a merging block 1600 that receives outputs from the detectors of the detection operations 1100.

[0091] For example, the merging block 1600 may receive, from the detection operations 1100, one or more indications of one or more detected faces, one or more indications of one or more detected torsos, one or more indications of one or more detected persons, and / or one or more indications of one or more detected objects. The merging block 1600 may determine (e.g., using a Hungarian algorithm or other suitable matching algorithm) which of the detected faces, torsos, persons, and / or objects (if any) correspond which others of the detected faces, torsos, persons, and / or objects, and / or to an identifier of an enrolled person. For example, the merging block 1600 may determine that a detected face and a detected torso belong to a single person detected by the person detector, and / or may determine (e.g., based on the enrollment information) that the single person to which the detected face and the detected torso correspond is an enrolled person having a known or stored identifier. In various use cases, only portions of a person may be visible within the field-of-view of the cameras 107, and the merging block may determine that a detected face is not associated with any detected torso, or that a detected torso is not associated with any detected face, in a particular image. In one or more implementations, the merging block 1600 may determine that the detected face to which no detected torso corresponds in a particular image corresponds to a torso detected in another (e.g., prior) image. The merging block 1600 may also determine that a detected torso to which no detected face corresponds in a particular image corresponds to a face detected in another (e.g., prior image).

[0092] In one or more implementations, the output of the merging block 1600 may include a set of data for each detected person, animal, and / or object. For example, the set of data for a particular detected person animal, and / or object may include an identifier (e.g., a name or other external identifier if the detected person is an enrolled person, or a unique internal identifier if the detected person cannot be identified as an enrolled person), a location, in image space, of the detected person, animal, and / or object, and one or more locations, in image space, of one or more detected portions (e.g., a face and / or a torso) of the detected person, animal, and / or object.

[0093] As illustrated in FIG. 17, the output of the merging block 1600 may be provided to the transformation operations 1102. As discussed herein, the transformation operations 1102 may project the locations of the detected person, animal, object, and / or portion thereof in the image space to locations of each detected person and / or object in a physical space, such as relative to the map 401 of the physical environment 100. As shown, the locations in physical space, such as with respect to the map 401, that are generated by the transformation operations 1102 may be provided (e.g., combined with, or without, the output of the merging block 1600) to an instruction generation operation 1104, at which one or more location-based operations 1618 may generate instructions for operation of one or more controllable accessories. For example, the location-based operations 1618 may include a location based operation A for controlling one or more light sources, a location based operation B for controlling one or more media players, a location based operation C for controlling one or more smart speakers, and / or any other control operations that are configured to control one or more controllable accessories based on the location of a person in a physical environment. As illustrated in FIG.17, in one or more implementations, component information, as described herein in connection with FIG. 11, may be provided to the location-based operations 1618. The location-based operations 1618 may generate the instructions for one or more controllable accessories based on the locations received from the transformation operations 1102 and / or based on the component information.

[0094] As shown in FIG. 17, in one or more implementations, the output of the merging block 1600, and / or the images from the cameras 107, may also be provided to one or more other detectors 1602. For example, the other detectors 1602 may include a pose detector 1612, a gesture detector 1614, and / or a body part detector 1616. For example, the pose detector 1612 may be a machine learning model that has been trained to detect the pose (e.g., standing, seated, laying down, etc.) of a person in an image, as described herein in connection with FIGS. 16A-16C. For example, the gesture detector 1614 may be a machine learningmodel that has been trained to detect one or more gestures performed on one or more surfaces of a physical environment, as described herein in connection with FIGS. 13 and 14. For example, the gesture detector 1614 may be a machine learning model that has been trained to detect one or more other portions of a person’s body (e.g., joints, such as elbow, knees, or shoulders, hands, feed, eyes, other facial features, or the like). In one or more other implementations, the other detectors 1602 may perform detection operations using the images from the cameras 107 as part of the detection operation 1100. In one or more implementations, the output(s) of the other detector(s) 1602 may also be provided to the transformation operations 1102, and the transformation operations 1102 may determine locations in the physical space of the physical environment 100 of a detected pose, gesture, or body part.

[0095] As shown in FIG. 17, an output of the merging block 1600 may be provided to a tracking operation 1604. The tracking operation 1604 may track (e.g., based on the detection information from the detection operations 1100, and / or the output of the merging block 1600) the location of each detected person in the physical environment 100 (e.g., including enrolled persons and unenrolled persons), and / or may track the locations of one or more individually detected elements from the face detector 1606, the torso detector 1608, the person detector 1609, and / or the object detector 1610, over time. As shown, tracking information from the tracking operation 1604 may be provided to the transformation operation 1102, for transformation of tracked locations in image space, to tracked locations in physical space (e.g., relative to the map 401). In one or more other implementations, the tracking operation 1604 may be performed after the transformation operation 1102. Tracking information (e.g., tracked locations over time) for an identified person may be used, for example, to infer the location of the person when the person is not within a field-of-view of any of the cameras 107 (e.g., as in the example of FIG. 6 or FIG. 12). In one or more implementations, tracking feedback information for one or more elements (e.g., a face, a torso, etc.) of a person may be provided to the merging block 1600, which may allow the merging block 1600 to merge a detected face and a detected torso to the same detected person, even when the detected face or the detected torso are not identifiable, from a current image frame alone, as belonging to the same person (e.g., by determining that an unidentified torso has been moving together with an identified face through the physical environment, to indicate a merging the unidentified torso with the identified face).

[0096] FIG. 18 illustrates aspects of a calibration operation that may be formed for calibrating the transformation operations 1102. In one or more implementations, once themap 401 has been generated (e.g., based on an imaging and / or depth scan of the physical environment 100, such as using a mobile device 106 having cameras and / or depth sensors), and once the camera(s) 107 is / are installed, a user (e.g., enrolled person 101 or another person) may be instructed (e.g., by the home hub device 109) to move to several locations in the physical environment 100. For example, the user may be provided with an image of the map 401 with locations 1800, 1802, 1804, and 1806 displayed thereon. The user may move to the physical locations in the physical environment that are indicated by the locations 1800, 1802, 1804, and 1806 displayed on the map 401. While the user is at each of the locations indicated by the locations 1800, 1802, 1804, and 1806 displayed on the map 401, the cameras 107 may capture images of the physical environment. Using these images of the user and the known locations 1800, 1802, 1804, and 1806 of the user in the physical environment at the time the images were captured, the home hub device 109 (or another device or server) may determine the transformation between the image space of the cameras 107 that capture the images, and the physical environment 100 to which the map 401 corresponds.

[0097] FIG. 19 illustrates a flow diagram of an example process 1900 for image-based control (e.g., image-based environmental control), in accordance with one or more implementations. For explanatory purposes, the process 1900 is primarily described herein with reference to the devices and / or components described in connection with FIGS. 1, 2, and 4. However, the process 1900 is not limited to the devices and / or components described in connection with FIGS. 1, 2, and 4, and one or more blocks (or operations) of the process 1900 may be performed by one or more other suitable devices. Further for explanatory purposes, the blocks of the process 1900 are described herein as occurring in serial, or linearly. However, multiple blocks of the process 1900 may occur in parallel. In addition, the blocks of the process 1900 need not be performed in the order shown and / or one or more blocks of the process 1900 need not be performed and / or can be replaced by other operations.

[0098] At block 1902, an enrolled person (e.g., enrolled person 101) for a home (e.g., physical environment 100) may be identified (e.g., by a home hub device 109, such as using detection operations 1100), using one or more cameras (e.g., camera(s) 107, such as fixed- location cameras). For example, the cameras may be disposed at one or more (e.g., fixed) first locations within the home, and each may have a field-of-view that covers a portion of an interior environment of the home. In one or more implementations, the home hub device may be configured to identify the enrolled person by detecting a torso (e.g., torso 312) of the enrolled person in an image captured by the one or more cameras, and identifying the enrolled person based on (e.g., the detected torso and) a previously stored image of the torsoof the enrolled person, the previously stored image captured during an enrollment operation (e.g., as described herein in connection with FIG. 3) for the home (e.g., by matching the detected torso to the previously stored image of the torso).

[0099] At block 1904, a location (e.g., location 450) of the enrolled person may be tracked (e.g., by the home hub device 109, such as by performing tracking operations 1604), using the one or more cameras, with respect to a map (e.g., the map 401) of the home. For example, the map may include a two-dimensional map of the home. In one or more implementations, the home hub device may be further configured to calibrate the two- dimensional map to the interior environment of the home using the one or more cameras (e.g., as described herein in connection with FIG. 18).

[0100] In one or more implementations, the home hub device may be configured to track the location of the person by identifying multiple locations of the enrolled person in an image space of multiple respective time-separated image frames, the multiple time-separated image frames obtained from the one or more cameras; transforming (e.g., at transformation operation 1102) the multiple locations of the enrolled person in the image space to multiple respective locations in the map; and tracking (e.g., at tracking operation 1604) the location of the enrolled person using the multiple locations in the map. Tracking the location of the enrolled person may include storing a location history of the enrolled person, inferring a current location of the enrolled person based on the location history, and / or predicting one or more upcoming locations based on the location history and the current location.

[0101] At block 1906, one or more fixed-location controllable accessories may be controlled (e.g., by the home hub device 109 and / or control application 1008, such as by performing instruction generation operations 1104) based on the location of the enrolled person with respect to the map. For example, the one or more fixed-location controllable accessories may be disposed at one or more fixed second locations within the home (e.g., as described herein in connection with FIG. 4). As examples, the one or more fixed-location controllable accessories may include one or more of: a light (e.g., a light source 111, such as a ceiling light, as wall-mounted light, a lamp, a chandelier, or the like), a speaker (e.g., smart speaker 102 or smart speaker 103), an electronic display (e.g., a display device 117 or display device 119), a thermostat (e.g., climate control device 113), a security component (e.g., security device 123), or an appliance (121). In one or more implementations, the home hub device may include at least one of the one or more cameras (e.g., the home hub device may include one or more cameras). In one or more implementations, the home hub device mayinclude at least one of the fixed-location controllable accessories (e.g., the home hub device may be implemented as part of a smart speaker or a media player device).

[0102] In one or more implementations, the map of the home may identify a plurality of zones (e.g., zones I, II, III, IV, V, VI, and VII of FIG. 4) within the home. The home hub device may control the one or more fixed-location controllable accessories based on the location of the enrolled person with respect to the map by activating at least one of the one or more fixed-location controllable accessories (e.g., the display device 117 in zone II in FIG. 12) that is located within a first zone of the plurality of zones, when the enrolled person enters the first zone. The home hub device may also control the one or more fixed-location controllable accessories based on the location of the enrolled person with respect to the map by deactivating the at least one of the one or more fixed-location controllable accessories that is located within the first zone when the enrolled person leaves the first zone (e.g., as shown in FIG. 12).

[0103] In one or more implementations, the home hub device may also determine, based on one or more images from the one or more cameras, that an unenrolled person (e.g., unenrolled person 133) is in a first zone (e.g., zone I in FIG. 7A), refrain from generating any alert with respect to the unenrolled person while the unenrolled person in in the first zone, and determine, based on one or more additional images from the one or more cameras, that the unenrolled person is in a second zone (e.g., zone II of FIG. 7B) of the plurality of zones, and generate, responsive to determining that the unenrolled person is in the second zone, an alert (e.g., alert 702 or alert 704) that the unenrolled person is in the second zone. For example, the home hub device may operate one or more of the one or more fixed-location controllable accessories (e.g., the display device 117, as in the example of FIG. 7B) to output the alert. As another example, the home hub device may provide the alert to a device (e.g., mobile device 106) of the enrolled person. Determining that an unenrolled person in in a particular zone may include detecting a person and / or a portion thereof (e.g., face and / or a torso of the person) in one or more images, and determining that the detected person and / or portion thereof does not have any match among enrolled persons (and / or portions thereof).

[0104] In one or more implementations, the home hub device may provide the tracked location of the enrolled person to a mobile device (e.g., mobile device 106) of the enrolled person, to facilitate control of the one or more fixed-location controllable accessories with the mobile device based on the tracked location (e.g., as discussed herein in connection with FIG. 8). In one or more implementations, the home hub device may include one or more machinelearning models (e.g., object detector(s) 1610) that have been trained to detect an object in one or more images. The home hub device may receive, from the enrolled person, a request to locate the object in the home; detect, using the machine learning model and one or more images from the one or more cameras (e.g., by providing the one or more images to the machine learning model as input(s) and obtaining a resulting output of the machine learning model), the object at a location in the home; and generate an output (e.g., indicator 904, a sound, and / or light from a light source 111) indicating the location in the home (e.g., as described herein in connection with FIG. 9, such as after determining the location of the object by transforming an image-space location into a physical-space location). In one or more implementations, the home hub device may be configured to operate one or more of the one or more fixed-location controllable accessories (e.g., the light source 111) to provide the output indicating the location (e.g., as described herein in connection with FIG. 9). In one or more implementations, the home hub device may be configured to provide the output (e.g., indicator 904) indicating the location to a device of the enrolled person, for display on the map by the device (e.g., as described herein in connection with FIG. 9).

[0105] FIG. 20 illustrates a flow diagram of an example process 2000 for image based control, in accordance with one or more implementations. For explanatory purposes, the process 2000 is primarily described herein with reference to the devices and / or components described in connection with FIGS. 1, 2, and 4. However, the process 2000 is not limited to the devices and / or components described in connection with FIGS. 1, 2, and 4, and one or more blocks (or operations) of the process 2000 may be performed by one or more other suitable devices. Further for explanatory purposes, the blocks of the process 2000 are described herein as occurring in serial, or linearly. However, multiple blocks of the process 2000 may occur in parallel. In addition, the blocks of the process 2000 need not be performed in the order shown and / or one or more blocks of the process 2000 need not be performed and / or can be replaced by other operations.

[0106] At block 2002, a home hub device (e.g., home hub device 109) for a home (e.g., physical environment 100) may control multiple other devices (e.g., any or all of the one or more appliances 121, the one or more light sources 111, the one or more display devices 117, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components in the physical environment 100) within the home based at least in part on a video view of the home (e.g., a video view formed from a series of images captured over time by one or more cameras 107 within the home ), and based on a topological model (e.g., topological model 1002) of the home, an operational model (e.g.,operational model 1004) of the home, and a behavioral model (e.g., behavioral model 1006) of the home. For example, the topological model may include a two-dimensional map (e.g., map 401) of the home. As described herein, the two-dimensional map may indicate the locations of one or more rooms, zones, furniture items, electronic devices and / or accessories, doors, windows, and / or any of the topological features of the home. The operational model of the home may include a model of multiple operational combinations of the plurality of other devices. The behavioral model of the home may include a model of one or more learned activities of one or more enrolled people (e.g., enrolled person 101) for the home. For example, each of the one or more learned activities may be associated with a location within the home (e.g., one or more activities that the home hub device has learned that an enrolled person typically performs at or near the location). For example, each of the one or more learned activities may be associated with a time.

[0107] FIG. 21 illustrates a flow diagram of an example process 2100 for location-based control, in accordance with one or more implementations. For explanatory purposes, the process 2100 is primarily described herein with reference to the devices and / or components described in connection with FIGS. 1, 2, and 4. However, the process 2100 is not limited to the devices and / or components described in connection with FIGS. 1, 2, and 4, and one or more blocks (or operations) of the process 2100 may be performed by one or more other suitable devices. Further for explanatory purposes, the blocks of the process 2100 are described herein as occurring in serial, or linearly. However, multiple blocks of the process 2100 may occur in parallel. In addition, the blocks of the process 2100 need not be performed in the order shown and / or one or more blocks of the process 2100 need not be performed and / or can be replaced by other operations.

[0108] At block 2102, an electronic device (e.g., home hub device 109) may detect (e.g., in a detection operation 1100) an enrolled person (e.g., enrolled person 101) at a location in an image (e.g., an image captured by a camera 107) of a physical environment. For example, the physical environment may include a home, and the two-dimensional map may be calibrated to an interior of the home based on one or more additional images of the home (e.g., additional images captured during a calibration operation, as described herein in connection with FIG. 18). In one or more implementations, the electronic device may be located at a fixed location in the home, and the process 2100 may also include generating the map based on an imaging scan of the home as performed by a mobile device (e.g., mobile device 106) separate from the electronic device, (e.g., by carrying the mobile device aroundthe home while capturing images and / or sensor data using camera(s) and / or sensor(s) of the mobile device).

[0109] At block 2104, the electronic device may project (e.g., in a transformation operation 1102) the location of the enrolled person in the image to a two-dimensional location (e.g., location 450) in a two-dimensional map of the physical environment. In one or more implementations, projecting the location of the enrolled person in the image to a two- dimensional location in a two-dimensional map of the physical environment may include identifying a location of a foot of the enrolled person in the image, and projecting the location of the foot in the image to the two-dimensional location in the two-dimensional map. In one or more use cases, the foot may be directly detected in the image. In one or more other use cases, the image may not include an image of a foot of the enrolled person, and projecting the location of the foot may include determining an inferred location of the foot of the enrolled person in the two-dimensional map from the location of the enrolled person in the image (e.g., from the detected location of another portion of the person, as discussed herein in connection with FIG. 5).

[0110] At block 2106, an electronic component (e.g., the smart speaker 102, the smart speaker 103, any or all of the one or more appliances 121, the one or more light sources 111, the media player 105, the media player 104, the one or more display devices 117 and / or 119, the one or more security devices 123, the one or more climate control devices 113, and / or other electronic devices and / or components in the physical environment 100) in the physical environment may be controlled (e.g., based on instructions generated in an instruction generation operation 1104, and / or by a control application 1008) based on the two- dimensional location of the enrolled person in the two-dimensional map. In one or more use cases, the electronic component may include a component (e.g., a speaker or a display) of the electronic device (e.g., in an implementation in which the home hub device is incorporated into another device, such as a smart speaker, a display device, or the like). In various other use cases, the electronic component may include a light source (e.g., a light source 111) at a fixed location in the physical environment, a speaker (e.g., smart speaker 102 or 103), an alarm (e.g., an alarm component of a security device 123), or a media output device (e.g., media player 105 or 104, and / or display device 117 and / or 119).[OHl] In one or more implementations, detecting the enrolled person may include detecting at least one of a previously enrolled face (e.g., face 310) or a previously enrolled torso (e.g., torso 312) of the enrolled person in the image (e.g., by matching a face or a torsoin the mage to the previously enrolled face or torso). For example, detecting the enrolled person may include detecting the previously enrolled face and the previously enrolled torso of the enrolled person in the image, and merging (e.g., in a merging block 1600) the detections of the previously enrolled face and the previously enrolled torso to identify the enrolled person. Detecting the enrolled person may also include obtaining a previously stored identifier of the enrolled person (e.g., a name, a number, or other unique identifier), previously stored during an enrollment operation for the enrolled person.

[0112] FIG. 22 illustrates a flow diagram of an example process 2200 user-location-based control of a mobile device, in accordance with one or more implementations. For explanatory purposes, the process 2200 is primarily described herein with reference to the devices and / or components described in connection with FIGS. 1, 2, and 4. However, the process 2200 is not limited to the devices and / or components described in connection with FIGS. 1, 2, and 4, and one or more blocks (or operations) of the process 2200 may be performed by one or more other suitable devices. Further for explanatory purposes, the blocks of the process 2200 are described herein as occurring in serial, or linearly. However, multiple blocks of the process 2200 may occur in parallel. In addition, the blocks of the process 2200 need not be performed in the order shown and / or one or more blocks of the process 2200 need not be performed and / or can be replaced by other operations.

[0113] At block 2202, a mobile device (e.g., mobile device 106) may receive, from a home hub device (e.g., home hub device 109) of a home (e.g., physical environment 100) while the mobile device is located within the home, a location of a user (e.g., enrolled person 101) of the mobile device. In one or more implementations, the location of the user may be determined by the home hub device and provided to the mobile device without determining the location of the mobile device at the home hub device, and / or without the home hub device being aware of the mobile device. The location of the user may have been determined by the home hub device based on one or more images from one or more cameras within the home, and without communication with the mobile device.

[0114] At block 2204, the mobile device may surface, based on the location, a set of controls (e.g., control elements 806) for controlling one or more controllable accessories (e.g., light sources 111 in the bedroom 406 in example use case) in the home, the controllable accessories separate from the mobile device. For example, the set of controls may be subset of controls that are stored at the mobile device for the home.

[0115] In one or more implementations, the mobile device may receive a user input to a control (e.g., control element 816 or 818) of the set of controls, and control, by the mobile device, at least one of the one or more controllable accessories in the home based on the user input (e.g., by turning a controllable accessory on, turning a controllable accessory off, waking a controllable accessory from a sleep state, placing a controllable accessory in a sleep state, initiating, stopping, pausing, and / or otherwise controlling output of media content from the controllable accessory, increasing or decreasing an amount of the output from the controllable accessory, switching between modes of operation of the controllable accessory, scheduling start and / or stop times for device operations for the controllable accessory, and / or otherwise controlling operation of the controllable accessory). For example, the one or more controllable accessories may include one or more of: a light (e.g., light source 111), a speaker (e.g., smart speaker 102 or 103), an electronic display (e.g., display device 117 or display device 119), a thermostat (e.g., climate control device 113), a security component (e.g., security device 123), a smart speaker (e.g., smart speaker 102 or 103), a media player device (e.g., media player 104 or 105), or an appliance (e.g., appliance 121).

[0116] In one or more implementations, the mobile device may also receive, from the home hub device, an updated location of the user of the mobile device (e.g., after movement of the user from the location to the updated location), and surface, and based on the updated location, an updated set of controls (e.g., control elements 802, such as another subset of the controls stored at the mobile device) for controlling one or more other controllable accessories (e.g., display device 117, light source 111 on the end table 424, or the climate control device 113) in the home, the other controllable accessories separate from the mobile device. For example, the location and the one or more controllable accessories may be located within a first zone (e.g., zone II corresponding to a living room 400) within the home, and the updated location and the one or more other controllable accessories may be located in a second zone (e.g., zone V, corresponding to the bedroom 406) of the home.

[0117] In one or more implementations, the mobile device may also determine, independent of the location of the user received from the home hub device, a location or a motion of the mobile device. For example, the location or a motion of the mobile device may be determined based on GPS signals, WiFi signals, and / or other signals sent and / or received by the mobile device. In one or more implementations, prior to surfacing the set of controls, the mobile device may determine, based on the location of the user from the home hub device and the location or the motion of the mobile device, that the mobile device is moving with the user (e.g., to confirm that the mobile device is with the user prior to surfacing the controls).In one or more implementations, the mobile device may determine that the mobile device is not located or moving with the user, and abstain from surfacing controls based on the location of the user.

[0118] FIG. 23 illustrates a flow diagram of an example process 2300 user-location-based media output, in accordance with one or more implementations. For explanatory purposes, the process 2300 is primarily described herein with reference to the devices and / or components described in connection with FIGS. 1, 2, and 4. However, the process 2300 is not limited to the devices and / or components described in connection with FIGS. 1, 2, and 4, and one or more blocks (or operations) of the process 2300 may be performed by one or more other suitable devices. Further for explanatory purposes, the blocks of the process 2300 are described herein as occurring in serial, or linearly. However, multiple blocks of the process 2300 may occur in parallel. In addition, the blocks of the process 2300 need not be performed in the order shown and / or one or more blocks of the process 2300 need not be performed and / or can be replaced by other operations.

[0119] At block 2302, media content (e.g., media content 1200) may be output by a first media output device (e.g., display device 117) that is located within a first zone (e.g., zone II) within a home. In various use cases, the first media output device may include a first smart speaker or a first electronic display. The media content may include video content, audio content, or the like.

[0120] At block 2304, a home hub device (e.g., home hub device 109) within the home may determine, based on one or more images captured by at least one camera (e.g., one or more cameras 107) within the home, that an enrolled person (e.g., enrolled person 101) is within a second zone (e.g., zone VI) within the home. In one or more use cases, determining that the enrolled person is within the second zone within the home may include detecting at least an enrolled torso (e.g., torso 312) of the enrolled person within the second zone in an image of at least a portion of the second zone. In one or more other use cases, the second zone may be outside of a field-of-view of the at least one camera, and determining that the enrolled person is within the second zone within the home may include inferring that the enrolled person is within the second zone based on a plurality of prior locations (e.g., path 602) of the enrolled person within the home, the plurality of prior locations of the enrolled person based on a plurality of images of at least a portion of the enrolled person from the at least one camera (e.g., based on image-based tracking of the location of the enrolled person within the home, such as is discussed herein in connection with FIG. 6 and / or FIG. 12).

[0121] At block 2306, the home hub device may cause output of the media content by a second media output device (e.g., smart speaker 103) that is located within the second zone within the home (e.g., as discussed herein in connection with FIG. 12). For example, the home hub device may cause the output of the media content by the second media output device by generating an instruction to output the media content, and providing the instruction to the second media output device. In one or more implementations, the home hub device may also cause ceasing of the outputting of the media content by the first media output device while the enrolled person is outside the first zone (e.g., as described herein in connection with FIG. 12). For example, the home hub device may cause the ceasing output of the media content by the first media output device by generating an instruction to cease outputting the media content, and providing the instruction to the first media output device. In various use cases, the second media output device may include a second media output device or a second electronic display.

[0122] In one or more use cases, the first zone may correspond to a first room within the home, and the second zone may correspond to a second room within the home. In one or more other use cases, the first zone may correspond to a first portion of a room within the home, and the second zone may correspond to a second portion of the room within the home.

[0123] In one or more implementations, the process 2300 may also include identifying, by the home hub device within the home based on one or more additional images captured by the at least one camera within the home, a gesture performed by the enrolled person. The process 2300 may also include causing, by the home hub device, modification of the outputting of media content by the second media output device based on the gesture. For example, the gesture may include a gesture performed on a surface within the home (e.g., as discussed herein in connection with FIG. 13). In one or more implementations, the process 2300 may also include controlling, by the home hub device, at least one light source (e.g., a light source 1400, such as a projector) within the home to project an indicator (e.g., image 1402) of the surface within the home as a surface for performing gestures (e.g., as discussed herein in connection with FIG. 14).

[0124] FIG. 24 illustrates a flow diagram of an example process 2400 for image-based intercom operations, in accordance with one or more implementations. For explanatory purposes, the process 2400 is primarily described herein with reference to the devices and / or components described in connection with FIGS. 1, 2, and 4. However, the process 2400 is not limited to the devices and / or components described in connection with FIGS. 1, 2, and 4,and one or more blocks (or operations) of the process 2400 may be performed by one or more other suitable devices. Further for explanatory purposes, the blocks of the process 2400 are described herein as occurring in serial, or linearly. However, multiple blocks of the process 2400 may occur in parallel. In addition, the blocks of the process 2400 need not be performed in the order shown and / or one or more blocks of the process 2400 need not be performed and / or can be replaced by other operations.

[0125] At block 2402, a home hub device (e.g., home hub device 109) within a home (e.g., physical environment 100) may receive a request (e.g., a request 1501) from a first enrolled user (e.g., enrolled person 101) at a first location (e.g., location 450) within the home to communicate with a second enrolled user. For example, the request may be received as a spoken request. For example, the spoken request may include the name of the second enrolled user. For example, the spoken request may include only the name of the second enrolled user (e.g., “Jim”), or may include a phrase including the name of the second enrolled user (e.g., “Hey Jim”).

[0126] At block 2404, the home hub device may determine, based on one or more images of the home or another home, a second location of the second enrolled user. For example, the home hub device may determine the location of the second enrolled user based on one or more images received from one or more cameras in the home if the second enrolled user is within the home, or may receive the location of the second enrolled user from another home hub device at the other home if the second enrolled user is within the other home or another location.

[0127] At block 2406, a speaker may be identified at or near the second location of the second enrolled user. For example, the speaker may be a standalone speaker (e.g., a smart speaker 1502) or may be included in another device, such as a display device or a mobile device. The speaker may be identified based on a known location of the speaker (e.g., in a map, such as a map 401), and the second location of the second enrolled user (e.g., the speaker may be the nearest known speaker to the second location).

[0128] At block 2408, a voice of the first enrolled user may be received with a microphone of a device (e.g., a microphone of a smart speaker 102 or another device having a microphone) at or near the first location of the first enrolled user. The voice may be received as part of the request, and / or following the request. In one or more use cases, the first enrolled user and the device may be located within a first zone (e.g., zone II) within the home, and the second enrolled user and the speaker are located within a second zone (e.g.,zone I, III, IV, V, VI, or VII) within the home. For example, the first zone may correspond to a first room within the home, and the second zone may correspond a second room within the home. In one or more other use cases, the first enrolled user and the device may be located within the home, and the second enrolled user and the speaker may be located within a second home (e.g., physical environment 1500), remote from the home.

[0129] At block 2410, the home hub device may provide, to the speaker, the voice of the first enrolled user for output by the speaker (e.g., by transmitting a signal including an encoded version of the voice to the speaker). In this way, the first enrolled user and the second enrolled user may be provided with an intercom system for communicating while the first enrolled user and the second enrolled user are in remote locations, in which the intercom system automatically determines and directs the voice of the first enrolled user to the speaker that is nearest the second enrolled user (e.g., without receiving any selection of the nearest speaker by either of the users).

[0130] In one or more implementations, prior to providing the voice of the first enrolled user for output by the speaker, the home hub device may determine whether the second enrolled user is available to receive the output of the speaker. For example, determining whether the second enrolled user is available to receive the output of the speaker may include determining, based on one or more images of the second enrolled user, whether the second enrolled user is asleep (e.g., based on a detected pose and / or the detected location of the second enrolled user). As another example, determining whether the second enrolled user is available to receive the output of the speaker may include determining, based on one or more images of the second enrolled user, whether the second enrolled user is busy (e.g., based on a detected activity being performed by the second enrolled user).

[0131] In one or more implementations, the process 2400 may also include providing, by the home hub device to another speaker at or near the first location of the first enrolled user, a voice of the second enrolled user received from another microphone of another device at or near the second location. For example, the second enrolled user may respond to the words spoken by the first enrolled user, and the response from the second enrolled user may be automatically directed to the nearest speaker to the location of the first enrolled user.

[0132] FIG. 25 illustrates a flow diagram of an example process 2500 for pose-based control, in accordance with one or more implementations. For explanatory purposes, the process 2500 is primarily described herein with reference to the devices and / or components described in connection with FIGS. 1, 2, and 4. However, the process 2500 is not limited tothe devices and / or components described in connection with FIGS. 1, 2, and 4, and one or more blocks (or operations) of the process 2500 may be performed by one or more other suitable devices. Further for explanatory purposes, the blocks of the process 2500 are described herein as occurring in serial, or linearly. However, multiple blocks of the process 2500 may occur in parallel. In addition, the blocks of the process 2500 need not be performed in the order shown and / or one or more blocks of the process 2500 need not be performed and / or can be replaced by other operations.

[0133] At block 2502, a home hub device (e.g., home hub device 109) for a home (e.g., physical environment 100) may identify, based on one or more images from one or more cameras (e.g., cameras 107) disposed at one or more fixed locations within the home, an enrolled person (e.g., enrolled person 101) within the home. Identifying the enrolled person may include performing any or all of the operations described herein in connection with FIGS. 11, 17, and / or 21.

[0134] At block 2504, the home hub device may determine, based on the one or more images, a pose of the enrolled person. For example, determining the pose of the enrolled person may include providing one or more of the one or more images to a pose detector, such as the pose detector 1612 of FIG. 16, and obtaining the pose as an output of the pose detector.

[0135] At block 2506, the home hub device may generate, based at least in part on the pose, a control signal for at least one of: an output component (e.g., a speaker or a display) of the home hub device or another device (e.g., mobile device 106, wearable device 137, or another device). In one or more implementations, the process 2500 may also include determining, based at least in part on the pose, an emergency condition of the enrolled person. For example, the control signal may include a control signal for generating an emergency alert. For example, the control signal may include an emergency alert signal for the other device, in which the other device includes an electronic device of a person other than the enrolled person. As another example, the control signal may include a control signal for a speaker (e.g., smart speaker 102 or 103) within the home to output the emergency alert.

[0136] In one or more implementations, determining the pose may include determining the pose at a first time, and generating the control signal based at least in part on the pose may include generating the control signal based on the pose and one or more previously determined poses at one or more second times prior to the first time (e.g., to confirm the user has maintained the pose for at least a predetermined period of time). For example, generating the control signal may include generating the control signal responsive to detecting a fall ofthe enrolled person based on the pose and the one or more previously determined poses (e.g., by determining that the enrolled person is laying down and has been laying down for at least a predetermined period of time, such as on a floor of the home)

[0137] In one or more other examples, determining the pose may include determining that the enrolled person is laying down, and generating the control signal may include generating the control signal to turn down or deactivate a light at or near a location of the enrolled person. In one or more other examples, determining the pose comprises determining that the enrolled person is sitting in a chair, and generating the control signal may include generating the control signal to turn on or activate a reading light at or near a location of the enrolled person.

[0138] As described above, one aspect of the present technology is the gathering and use of data available from specific and legitimate sources for image-based control, such as for a home environment. The present disclosure contemplates that in some instances, this gathered data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data can include audio data, voice data, demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses, encryption information, data or records relating to a user’s health or level of fitness (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other personal information.

[0139] The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, the personal information data can be used for image-based control, such as for a home environment. Accordingly, use of such personal information data may facilitate authentication operations. Further, other uses for personal information data that benefit the user are also contemplated by the present disclosure. For instance, health and fitness data may be used, in accordance with the user’s preferences to provide insights into their general wellness, or may be used as positive feedback to individuals using technology to pursue wellness goals.

[0140] The present disclosure contemplates that those entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with well-established privacy policies and / or privacy practices. In particular, such entities would be expected to implement and consistently apply privacy practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy of users. Such information regarding the use of personal data shouldbe prominently and easily accessible by users, and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate uses only. Further, such collection / sharing should occur only after receiving the consent of the users or other legitimate basis specified in applicable law. Additionally, such entities should consider taking any needed steps for safeguarding and securing access to such personal information data and ensuring that others with access to the personal information data adhere to their privacy policies and procedures. Further, such entities can subject themselves to evaluation by third parties to certify their adherence to widely accepted privacy policies and practices. In addition, policies and practices should be adapted for the particular types of personal information data being collected and / or accessed and adapted to applicable laws and standards, including jurisdiction-specific considerations which may serve to impose a higher standard. For instance, in the US, collection of or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); whereas health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0141] Despite the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and / or software elements can be provided to prevent or block access to such personal information data. For example, in the case of imagebased control, such as for a home environment, the present technology can be configured to allow users to select to “opt in” or “opt out” of participation in the collection of personal information data during registration for services or anytime thereafter. In addition to providing “opt in” and “opt out” options, the present disclosure contemplates providing notifications relating to the access or use of personal information. For instance, a user may be notified upon downloading an app that their personal information data will be accessed and then reminded again just before personal information data is accessed by the app.

[0142] Moreover, it is the intent of the present disclosure that personal information data should be managed and handled in a way to minimize risks of unintentional or unauthorized access or use. Risk can be minimized by limiting the collection of data and deleting data once it is no longer needed. In addition, and when applicable, including in certain health related applications, data de-identification can be used to protect a user’s privacy. Deidentification may be facilitated, when appropriate, by removing identifiers, controlling the amount or specificity of data stored (e.g., collecting location data at city level rather than atan address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods such as differential privacy.

[0143] Therefore, although the present disclosure broadly covers use of personal information data to implement one or more various disclosed embodiments, the present disclosure also contemplates that the various embodiments can also be implemented without the need for accessing such personal information data. That is, the various embodiments of the present technology are not rendered inoperable due to the lack of all or a portion of such personal information data.

[0144] FIG. 26 illustrates an electronic system 2600 with which one or more implementations of the subject technology may be implemented. The electronic system 2600 can be, and / or can be a part of, one or more of the electronic devices and / or the server shown in FIG. 1. The electronic system 2600 may include various types of computer readable media and interfaces for various other types of computer readable media. The electronic system 2600 includes a bus 2608, one or more processing unit(s) 2612, a system memory 2604 (and / or buffer), a ROM 2610, a permanent storage device 2602, an input device interface 2614, an output device interface 2606, and one or more network interfaces 2616, or subsets and variations thereof.

[0145] The bus 2608 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system 2600. In one or more implementations, the bus 2608 communicatively connects the one or more processing unit(s) 2612 with the ROM 2610, the system memory 2604, and the permanent storage device 2602. From these various memory units, the one or more processing unit(s) 2612 retrieves instructions to execute and data to process in order to execute the processes of the subject disclosure. The one or more processing unit(s) 2612 can be a single processor or a multi-core processor in different implementations.

[0146] The ROM 2610 stores static data and instructions that are needed by the one or more processing unit(s) 2612 and other modules of the electronic system 2600. The permanent storage device 2602, on the other hand, may be a read-and-write memory device. The permanent storage device 2602 may be a non-volatile memory unit that stores instructions and data even when the electronic system 2600 is off. In one or more implementations, a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) may be used as the permanent storage device 2602.

[0147] In one or more implementations, a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) may be used as the permanent storage device 2602. Like the permanent storage device 2602, the system memory 2604 may be a read-and- write memory device. However, unlike the permanent storage device 2602, the system memory 2604 may be a volatile read-and-write memory, such as random access memory. The system memory 2604 may store any of the instructions and data that one or more processing unit(s) 2612 may need at runtime. In one or more implementations, the processes of the subject disclosure are stored in the system memory 2604, the permanent storage device 2602, and / or the ROM 2610. From these various memory units, the one or more processing unit(s) 2612 retrieves instructions to execute and data to process in order to execute the processes of one or more implementations.

[0148] The bus 2608 also connects to the input and output device interfaces 2614 and 2606. The input device interface 2614 enables a user to communicate information and select commands to the electronic system 2600. Input devices that may be used with the input device interface 2614 may include, for example, microphones, alphanumeric keyboards, touchscreens, touchpads, and pointing devices (also called “cursor control devices”). The output device interface 2606 may enable, for example, the display of images generated by electronic system 2600. Output devices that may be used with the output device interface 2606 may include, for example, speakers, printers and display devices, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, a light source, a haptic components, or any other device for outputting information. One or more implementations may include devices that function as both input and output devices, such as a touchscreen. In these implementations, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0149] Finally, as shown in FIG. 26, the bus 2608 also couples the electronic system 2600 to one or more networks and / or to one or more network nodes, such as the server 114 shown in FIG. 1, through the one or more network interface(s) 2616. In this manner, the electronic system 2600 can be a part of a network of computers (such as a LAN, a wide area network (“WAN”), or an Intranet, or a network of networks, such as the Internet. Any or all components of the electronic system 2600 can be used in conjunction with the subject disclosure.

[0150] Implementations within the scope of the present disclosure can be partially or entirely realized using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) encoding one or more instructions. The tangible computer-readable storage medium also can be non-transitory in nature.

[0151] The computer-readable storage medium can be any storage medium that can be read, written, or otherwise accessed by a general purpose or special purpose computing device, including any processing electronics and / or processing circuitry capable of executing instructions. For example, without limitation, the computer-readable medium can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. The computer-readable medium also can include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.

[0152] Further, the computer-readable storage medium can include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more implementations, the tangible computer-readable storage medium can be directly coupled to a computing device, while in other implementations, the tangible computer- readable storage medium can be indirectly coupled to a computing device, e.g., via one or more wired connections, one or more wireless connections, or any combination thereof.

[0153] Instructions can be directly executable or can be used to develop executable instructions. For example, instructions can be realized as executable or non-executable machine code or as instructions in a high-level language that can be compiled to produce executable or non-executable machine code. Further, instructions also can be realized as or can include data. Computer-executable instructions also can be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those of skill in the art, details including, but not limited to, the number, structure, sequence, and organization of instructions can vary significantly without varying the underlying logic, function, processing, and output.

[0154] While the above discussion primarily refers to microprocessor or multi-core processors that execute software, one or more implementations are performed by one or more integrated circuits, such as ASICs or FPGAs. In one or more implementations, such integrated circuits execute instructions that are stored on the circuit itself.

[0155] Those of skill in the art would appreciate that the various illustrative blocks, modules, elements, components, methods, and algorithms described herein may be implemented as electronic hardware, computer software, or combinations of both. To illustrate this interchangeability of hardware and software, various illustrative blocks, modules, elements, components, methods, and algorithms have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. Various components and blocks may be arranged differently (e.g., arranged in a different order, or partitioned in a different way) all without departing from the scope of the subject technology.

[0156] It is understood that any specific order or hierarchy of blocks in the processes disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes may be rearranged, or that all illustrated blocks be performed. Any of the blocks may be performed simultaneously. In one or more implementations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0157] As used in this specification and any claims of this application, the terms “base station”, “receiver”, “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms “display” or “displaying” means displaying on an electronic device.

[0158] As used herein, the phrase “at least one of’ preceding a series of items, with the term “and” or “or” to separate any of the items, modifies the list as a whole, rather than each member of the list (i.e., each item). The phrase “at least one of’ does not require selection of at least one of each item listed; rather, the phrase allows a meaning that includes at least one of any one of the items, and / or at least one of any combination of the items, and / or at least one of each of the items. By way of example, the phrases “at least one of A, B, and C” or “atleast one of A, B, or C” each refer to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0159] The predicate words “configured to”, “operable to”, and “programmed to” do not imply any particular tangible or intangible modification of a subject, but, rather, are intended to be used interchangeably. In one or more implementations, a processor configured to monitor and control an operation or a component may also mean the processor being programmed to monitor and control the operation or the processor being operable to monitor and control the operation. Likewise, a processor configured to execute code can be construed as a processor programmed to execute code or operable to execute code.

[0160] Phrases such as an aspect, the aspect, another aspect, some aspects, one or more aspects, an implementation, the implementation, another implementation, some implementations, one or more implementations, an embodiment, the embodiment, another embodiment, some implementations, one or more implementations, a configuration, the configuration, another configuration, some configurations, one or more configurations, the subject technology, the disclosure, the present disclosure, other variations thereof and alike are for convenience and do not imply that a disclosure relating to such phrase(s) is essential to the subject technology or that such disclosure applies to all configurations of the subject technology. A disclosure relating to such phrase(s) may apply to all configurations, or one or more configurations. A disclosure relating to such phrase(s) may provide one or more examples. A phrase such as an aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to other foregoing phrases.

[0161] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration”. Any embodiment described herein as “exemplary” or as an “example” is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the term “include”, “have”, or the like is used in the description or the claims, such term is intended to be inclusive in a manner similar to the term “comprise” as “comprise” is interpreted when employed as a transitional word in a claim.

[0162] All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. No claim element is to be construed under the provisions of 35 U.S.C. § 112(f) unless theelement is expressly recited using the phrase “means for” or, in the case of a method claim, the element is recited using the phrase “step for”.

[0163] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more”. Unless specifically stated otherwise, the term “some” refers to one or more. Pronouns in the masculine (e.g., his) include the feminine and neuter gender (e.g., her and its) and vice versa. Headings and subheadings, if any, are used for convenience only and do not limit the subject disclosure.

Claims

CLAIMSWhat is claimed is:

1. A system, comprising: a home hub device for a home; one or more cameras disposed at one or more first locations within the home and each having a field-of-view that covers a portion of an interior environment of the home; and one or more fixed-location controllable accessories disposed at one or more second locations within the home, wherein the home hub device is configured to: identify, using the one or more cameras, an enrolled person for the home; track, using the one or more cameras, a location of the enrolled person with respect to a map of the home; and control the one or more fixed-location controllable accessories based on the location of the enrolled person with respect to the map.

2. The system of claim 1, wherein the one or more fixed-location controllable accessories comprise one or more of: a light, a speaker, an electronic display, a thermostat, a security component, or an appliance.

3. The system of claim 2, wherein the home hub device comprises at least one of the one or more cameras.

4. The system of claim 2, wherein the home hub device comprises at least one of the fixed-location controllable accessories.

5. The system of claim 1, wherein the map comprises a two-dimensional map of the home.

6. The system of claim 5, wherein the home hub device is further configured to calibrate the two-dimensional map to the interior environment of the home using the one or more cameras.

7. The system of claim 1, wherein the home hub device is configured to identify the enrolled person by:detecting a torso of the enrolled person in an image captured by the one or more cameras; and identifying the enrolled person based on a previously stored image of the torso of the enrolled person, the previously stored image captured during an enrollment operation for the home.

8. The system of claim 7, wherein the home hub device is configured to track the location of the person by: identifying a plurality of locations of the enrolled person in an image space of, respectively, a plurality of time-separate image frames, the plurality of time-separate image frames obtained from the one or more cameras; transforming the plurality of locations of the enrolled person in the image space to a plurality of respective locations in the map; and tracking the location of the enrolled person using the plurality of respective locations in the map.

9. The system of claim 1, wherein the map of the home identifies a plurality of zones within the home, and wherein the home hub device is configured to control the one or more fixed-location controllable accessories based on the location of the enrolled person with respect to the map by: activating at least one of the one or more fixed-location controllable accessories that is located within a first zone of the plurality of zones when the enrolled person enters the first zone; and deactivating the at least one of the one or more fixed-location controllable accessories that is located within the first zone when the enrolled person leaves the first zone.

10. The system of claim 9, wherein the home hub device is further configured to: determine, based on one or more images from the one or more cameras, that an unenrolled person is in the first zone; refrain from generating any alert with respect to the unenrolled person while the unenrolled person in in the first zone; determine, based on one or more additional images from the one or more cameras, that the unenrolled person is in a second zone of the plurality of zones; and generate, responsive to determining that the unenrolled person is in the second zone, an alert that the unenrolled person is in the second zone.

11. The system of claim 10, wherein the home hub device is configured to perform at least one of: operating one or more of the one or more fixed-location controllable accessories to output the alert, or providing the alert to a device of the enrolled person.

12. The system of claim 1, wherein the one or more cameras comprises one or more fixed-location cameras, the system further comprising a mobile device of the enrolled person, wherein the home hub device is configured to provide the tracked location of the enrolled person to the mobile device of the enrolled person, to facilitate control of the one or more fixed-location controllable accessories with the mobile device based on the tracked location.

13. The system of claim 1, wherein the home hub device comprises one or more machine learning models that have been trained to detect an object in one or more images, and wherein the home hub device is configured to: receive, from the enrolled person, a request to locate the object in the home; detect, using the machine learning model and one or more images from the one or more cameras, the object at a location in the home; and generate an output indicating the location in the home.

14. The system of claim 13, wherein the home hub device is configured to perform at least one of: operating one or more of the one or more fixed-location controllable accessories to provide the output indicating the location, or providing the output indicating the location to a device of the enrolled person, for display on the map by the device.

15. A system, comprising: a home hub device for a home, wherein the home hub device is configured to control a plurality of other devices within the home based at least in part on a video view of the home and based on: a topological model of the home; an operational model of the home; anda behavioral model of the home.

16. The system of claim 15, wherein the topological model comprises a two- dimensional map of the home.

17. The system of claim 15, wherein the operational model of the home comprises a model of a plurality of operational combinations of the plurality of other devices.

18. The system of claim 15, wherein the behavioral model of the home comprises a model of one or more learned activities of one or more enrolled people for the home.

19. The system of claim 18, wherein each of the one or more learned activities is associated with a location within the home.

20. A method, comprising: identifying, by a home hub device for a home and using one or more cameras disposed at one or more first locations within the home and each having a field-of-view that covers a portion of an interior environment of the home, an enrolled person for the home; tracking, by the home hub device using the one or more cameras, a location of the enrolled person with respect to a map of the home; and controlling, by the home hub device and based on the location of the enrolled person with respect to the map, one or more fixed-location controllable accessories that are disposed at one or more second locations within the home.

Images