Queryable virtual models for physical environments

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generate a prediction characterizing an environment. In one aspect, one of the methods include: maintaining, by one or more computers, a virtual model for a physical environment, wherein the physical environment comprises a plurality of physical objects, and wherein the virtual model comprises a virtual object that maps to each of the plurality of physical objects; receiving, by the one or more computers, a query that identifies a particular physical object; determining, by the one or more computers and based on the virtual model, a status of the particular physical object; and providing, by the one or more computers, a response to the query, the response including the status of the particular physical object.

Description

[0001] QUERY ABLE VIRTUAL MODELS FOR PHYSICAL ENVIRONMENTS

[0002] BACKGROUND

[0003] This specification relates to machine learning, and more particularly to using machine learning with smart devices.

[0004] ■‘Smart devices” as used herein refers to devices and / or appliances that are configured for network communication, and hence are remotely controllable using computing devices such as mobile phones, desktop or laptop computers, tablets, digital assistant devices, wearable devices (e.g.. smart watches), etc. Network communications may be carried out using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6L0WPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, etc.) and / or any of a variety7of custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), or any other suitable communication protocol.

[0005] Smart devices include, but are not limited to, smart locks, smart lights, smart thermostats, alarm systems, smart cameras, smart garage door openers, smart electrical outlets, smart faucets, smart sprinkler systems, smart kitchen appliances (e.g., ovens, coffee makers, refrigerators), smart blinds, smart windows, and any other networked appliance that is controllable remotely using a computing device.

[0006] SUMMARY

[0007] This specification describes a service system implemented as computer programs on one or more computers in one or more locations that receives a query that references an object in a physical environment and generates a response to the query by using a virtual model that corresponds to the physical environment. For example, the physical environment can be a smart environment that includes one or more smart devices, and one or more other objects, including non-smart objects.

[0008] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. The techniques described in this specification allow7resident smart devices in a physical environment to share inferences with each other and ith a cloud-based service system, and allow the service system to make better contextual decisions that decide on how to control the operation of the smart devices. By providing the service system with access to a queryable, virtual model corresponding to the physical environment which is continuously updated based on smart device data to reflect the latest state of the environment, the described techniques increase both the usability and the usefulness of various smart devices because the service system gains broader access to contextual information about the environment that the smart devices operate in. For example, instead of making control decisions based solely on a limited amount of locally available data, the service system can make more informed control decisions that control each smart device in a group of multiple smart devices, e.g., control the actions performed by the smart device, based on contextual data shared from other smart devices that have access to different or broader sensor field of view.

[0009] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0010] BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 is an example environment that includes a plurality of smart devices.

[0012] FIG. 2 is a block diagram of an example service system.

[0013] FIG. 3 is a flow diagram of an example process of generating a response to a query.

[0014] FIG. 4 is a flow diagram of an example process of sub-steps of one of the steps of the process of FIG. 3.

[0015] Like reference numbers and designations in the various drawings indicate like elements.

[0016] DETAILED DESCRIPTION

[0017] This specification describes a service system implemented as computer programs on one or more computers in one or more locations that receives a query that references a physical (or real -world) object in a physical (or real-world) environment and generates a response to the query' by using a virtual model that corresponds to the physical environment.

[0018] For example, the physical (or real-world) environment can be a smart environment that includes one or more smart devices, and one or more other physical (or real-world) objects, including non-smart objects. For example, the smart environment can be a user’s home that has various smart devices, an industrial facility that has various smart devices, or the like.

[0019] FIG. 1 is an example environment (referred to below as a “smart environment”) 100 that includes a plurality of smart devices 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 168, and 170. The smart environment 100 includes a network 162, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 162 connects a service system 164. one or more client devices 166, and the plurality of smart devices. The example smart environment 100 can include many different smart devices, as well be described further below.

[0020] A client device 1 6 is an electronic device capable of requesting and receiving data over the network 162. Example client devices 166 include mobile phones, desktop or laptop computers, tablets, gaming devices, mobile communication devices, digital assistant devices, wearable devices (e.g., smart watches), augmented reality devices, virtual reality devices, and other devices that can send and receive data over the network 1 2. A client device 166 typically includes a user application, such as a web browser, to facilitate the sending and receiving of data over the network 162, but native applications (other than browsers) executed by the client device 166 can also facilitate the sending and receiving of data over the network 162.

[0021] The smart environment 100 includes a structure 150 (e.g., ahouse, office building, garage, or mobile home) with various integrated devices. It will be appreciated that devices may also be integrated into a smart environment 100 that does not include an entire structure 150, such as an apartment, condominium, or office space. Further, the smart environment 100 may control and / or be coupled to devices outside of the actual structure 150. Indeed, several devices in the smart environment 100 need not be physically within the structure 150. For example, a device controlling a pool heater 114 or irrigation system 116 may be located outside of the structure 150.

[0022] The depicted structure 150 includes a plurality of rooms 152, separated at least partly from each other via walls 154. The walls 154 may include interior walls or exterior walls. Each room may further include a floor 156 and a ceiling 158. Devices may be mounted on, affixed to, integrated with and / or supported by a wall 154, floor 156 or ceiling 158.

[0023] The smart devices of the smart environment 100 include network-connected devices that integrate with each other in a smart network (e.g., the network 162, or a different network) and / or with a central server or a cloud-computing system, including the service system 164, to provide a variety of useful smart functions. The smart devices are equipped with sensors that provide the devices with sensing capabilities. The sensors can include force sensors; visual sensors, e.g., camera, depth cameras, and lidar; electrical connection sensors; acceleration sensors; audio sensors; gy ros; contact sensors; radar sensors; and proximity sensors, e.g.. infrared proximity sensors, capacitive proximity sensors, or inductive proximity sensors, to name just a few7examples. The smart environment 100 may include one or more multi-sensing and / or network- connected thermostats 102 (referred to below as “smart thermostats 102”), one or more multisensing and / or network-connected hazard detection units 104 (referred to below as “smart hazard detectors 104”), one or more multi -sensing and / or network-connected entry way interface devices 106 and 120 (e.g., “smart doorbells 106” and “smart door locks 120”), and one or more multi-sensing and / or network-connected alarm systems 122 (referred to below as “smart alarm systems 122”).

[0024] In some implementations, the one or more smart thermostats 102 detect ambient climate characteristics (e.g., temperature and / or humidity) and control a HVAC system 103 accordingly. For example, a respective smart thermostat 102 includes an ambient temperature sensor.

[0025] The one or more smart hazard detectors 104 may include thermal radiation sensors directed at respective heat sources (e.g., a stove, oven, other appliances, a fireplace, etc.). For example, a smart hazard detector in a kitchen includes a thermal radiation sensor directed at a stove / oven. A thermal radiation sensor may determine the temperature of the respective heat source (or a portion thereol) at which it is directed and may provide corresponding blackbody radiation data as output.

[0026] The smart doorbell 106 and / or the smart door lock 120 may detect a person’s approach to or departure from a location (e.g., an outer door), control doorbell / door locking functionality (e.g., receive user inputs from a client device 166 1 to actuate bolt of the smart door lock 120), announce a person’s approach or departure via audio or visual means, and / or control settings on a security' system (e.g., to activate or deactivate the security system when occupants go and come).

[0027] The smart alarm system 122 may detect the presence of an individual within close proximity (e.g., using built-in IR sensors), sound an alarm (e.g., through a built-in speaker, or by sending commands to one or more external speakers), and send notifications to entities or users within / outside of the smart environment 100. In some implementations, the smart alarm system 122 also includes one or more input devices or sensors (e.g., keypad, biometric scanner, NFC transceiver, microphone) for verifying the identity of a user, and one or more output devices (e.g., display, speaker). In some implementations, the smart alarm system 122 may also be set to an “armed” mode, such that detection of a trigger condition or event causes the alarm to be sounded unless a disarming action is performed.

[0028] In some implementations, the smart environment 100 includes one or more multisensing and / or network-connected wall switches 108 (referred to below as “smart wall switches 108”), along with one or more multi-sensing and / or network-connected wall plug interfaces 110 (referred to below as "‘smart wall plugsl 10”). The smart wall switches 108 may detect ambient lighting conditions, detect room-occupancy states, and control a power and / or dim state of one or more lights. In some implementations, smart wall switches 108 may also control a power state or speed of a fan, such as a ceiling fan. The smart wall plugs 110 may detect occupancy of a room or enclosure and control supply of power to one or more wall plugs (e.g., such that power is not supplied to the plug if nobody is at home).

[0029] In some implementations, the smart environment 100 includes a plurality of multisensing and / or network-connected appliances 112 (hereinafter referred to as “smart appliances 112”), such as refrigerators, stoves, ovens, televisions, washers, dryers, lights, stereos, intercom systems, garage-door openers, floor fans, ceiling fans, wall air conditioners, pool heaters, irrigation systems, security systems, space heaters, window AC units, motorized duct vents, and so forth. In some implementations, when plugged in, an appliance may announce itself to the smart network, such as by indicating what type of appliance it is, and it may automatically integrate with the controls of the smart environment 100. Such communication by the appliance to the smart environment 100 may be facilitated by either a wired or wireless communication protocol. The smart environment may also include a variety of non-communicating legacy appliances 140, such as old conventional w all air conditioners, washer / dryers, refrigerators, and the like, which may be controlled by smart wall plugs 110. The smart environment 100 may further include a variety of partially communicating legacy appliances 142, such as infrared (“IR”) controlled wall air conditioners or other IR-controlled devices, which may be controlled by IR signals provided by the smart hazard detectors 104 or the smart w all switches 108.

[0030] In some implementations, the smart environment 100 includes one or more network- connected cameras 118 that are configured to provide video monitoring and security in the smart environment 100. The cameras 118 may be used to determine occupancy of the structure 150 and / or particular rooms 152 in the structure 150, and thus may act as occupancy sensors. For example, video captured by the cameras 118 may be processed to identify the presence of an occupant in the structure 150 (e.g., in a particular room 152). Specific individuals may be identified based, for example, on their appearance (e.g., height, face) and / or movement (e.g., their walk / gait). The cameras 118 optionally include one or more sensors (e.g., IR sensors, radar systems, motion detectors), input devices (e g., microphone for capturing audio), and output devices (e.g., speaker for outputting audio). The smart environment 100 may additionally or alternatively include one or more other occupancy sensors (e.g., the smart doorbell 106. smart door locks 120, touch screens, IR sensors, microphones, ambient light sensors, motion detectors, smart nightlights 170, etc.).

[0031] The smart environment 100 may also include communication with devices outside of the physical home but within a vicinity' of the home. For example, the smart environment 100 may include a pool heater monitor 114 that communicates a current pool temperature to other devices within the smart environment 100 and / or receives commands for controlling the pool temperature. Similarly, the smart environment 100 may include an irrigation monitor 116 that communicates information regarding irrigation systems within the smart environment 100 and / or receives control information for controlling such irrigation systems.

[0032] In some implementations, the smart environment 100 includes service robots 168 that are configured to carry out, in an autonomous or semi-autonomous manner, a variety of household tasks. For example, the sendee robots 168 can be respectively configured to perform household tasks including sweep and / or wash floors, deliver objects, wipe tables and / or windows, and arrange chairs.

[0033] A service robot 168 is typically equipped with multiple sensors. The sensors can include perceptual sensors that generate visual sensor data that represent visual characteristics of a surrounding of a robot. For example, in order to achieve better vision capability7, a robot tool can be equipped with multiple cameras, e.g., visible light cameras, infrared cameras, depth cameras, lidars, radars, to name just a few examples. The sensors can also include one or more robot state sensors that generate robot state data that represent physical characteristics of the service robot 168 or a component of the service robot 168. For example, the robot state data can represent force, torque, angles, positions, velocities, and accelerations, of the service robot 168 or respective components of the service robot 168, to name just a few examples.

[0034] By virtue of network connectivity, one or more of the smart devices in the smart environment 100 of FIG. 1 may further allow a user to interact with the device even if the user is not proximate to the device. For example, a user may communicate with a smart device using the client device 166 1 or the client device 166_2. A webpage or application may be configured to receive communications from the user and control the smart device based on the communications and / or to present information about the smart device’s operation to the user. For example, the user may view a current set point temperature for a device (e.g., a stove) and adjust it using the client device. The user may be inside the structure during this communication, or outside the structure. In some implementations, the network interface 160 includes a conventional network device (e.g., a router), and the smart environment 100 of FIG. 1 includes a hub device 180 that is communicatively coupled to the network 162 directly or via the network interface 160. The hub device 180 may be further communicatively coupled to one or more of the smart devices of the smart environment 100. Each of these smart devices optionally communicates with the hub device 180 using one or more radio communication networks available at least in the smart environment 100 (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi and other radio communication netw orks).

[0035] In some implementations, the hub device 180 and smart devices coupled with / to the hub device can be controlled and / or interacted with via an application running on the client device 166 1 or client device 166 2. For example, a user of such controller application can view status of the hub device 180 or coupled smart devices, configure the hub device 180 to interoperate with smart devices newly introduced to the smart netw ork, commission newsmart devices, and adjust or view' settings of connected smart devices, etc.

[0036] In some implementations, a first one of the smart devices communicates with a second one of the smart devices via a wireless router. The smart devices may further communicate with each other via a connection (e.g., network interface 160) to the network 162. Through the network 162, the smart devices can also communicate with the sendee system 164.

[0037] The smart devices can upload data 163 (referred to below as "‘smart device data”) that has been generated, captured, or otherwise processed by the smart devices to the service system 164. In some implementations, the smart device data 163 includes one or more of: sensor data output by one or more sensors of a smart device (e.g., a camera 118, a smart alarm system 122, or a service robot 168) which characterizes a physical environment in a vicinity of the smart device, metadata output by a smart device, settings information for a smart device, usage logs for a smart device, or the like. In some implementations, smart device data 163 is automatically sent from the smart devices to the service system 164 (e.g., when available, upon request, or at predetermined intervals).

[0038] At a high level, the service system 164 is a system implemented as computer programs on one or more computers in one or more locations that receives the smart device data 163 from the smart devices and uses the received data to respond to queries submitted by one or more users. To generate the response to the queries, the service system 164 maintains a virtual model 166 corresponding to the smart environment 100. The virtual model 166 includes a virtual object corresponding to each of multiple physical (or real-world) objects within or outside the structure 150. The service system 164 then uses information that relate to the virtual objects included in the virtual model 166 to respond to queries.

[0039] FIG. 1 depicts the service system 164 as a server system, with some data processing occurring at a server that is physically remote from the structure 150. However, other configurations are possible. For example, the service system 164 can be implemented on a client device, e.g., the client device 166_1 or 166_2, and thus the data processing can occur exclusively on the client device. Furthermore, in some implementations, some of the data processing can be done on the remote sendee system 164 and some of the data processing can occur locally at each of one or more smart devices located within the structure 150.

[0040] A user can generate the query' to submit to the sendee system 164 in any of a variety of different ways. For example, a user can input a query through a client device, e.g.. the client device 166 1 or 166_2. The client device includes one or more input devices that can receive a query' as text-based input (e.g., a query typed using a keyboard), selection-based input (e.g., touchscreen selection, etc.), and audio-based input (e.g., voice input). As another example, a user can input a query through one of the smart devices in the smart environment 100. Some smart devices use input devices including a microphone and voice recognition, or a camera and gesture recognition, to supplement or replace the keyboard.

[0041] Likewise, the response can be generated by the service system 164 in any of a variety' of ways. For example, responses to the queries can be provided to a user in a client device, e.g., the client device 166 1 or 166 2. The client device includes one or more input devices that enable presentation of the responses, e.g., visually on a display for the client device, or audibly through a speaker system of the client device. As another example, responses to the queries can include commands to be sent to one of the smart devices in the smart environment 100 to control the operation of the smart device, e.g., to adjust the heating and cooling settings of a smart thermostat, an on / off status of a smart home appliance, and so on.

[0042] FIG. 2 is a block diagram of an example service system 264. The service system 264 is a system implemented as computer programs on one or more computers in one or more locations that receives the smart device data 263 from smart devices and uses the received data to respond to queries submitted by one or more users. In some situations, the service system 264 can be the same as or similar to the service system 164 of FIG. 1 .

[0043] The service system 264 includes an indexing engine 210. The indexing engine 210 executes pre-defined algorithms to process the smart device data 263 to generate model update data, which can then be used to update the virtual model 266. In general, any suitable algorithms can be used. For example, the pre-defined algorithms can include one or more of: machine learning algorithms, including deep neural networks (e.g., an object detection and / or classification neural network, an object pose estimation neural network, model, an image segmentation neural network), computer visual algorithms (e.g., an object localization algorithm, a depth estimation algorithm, a spatial mapping algorithm), mathematical algorithms, or the like.

[0044] When the pre-defined algorithms include an object classification neural network, for example, the indexing engine 210 can receive an input that includes visual sensor data (e.g., images captured by a camera sensor, or point clouds captured by a depth camera sensor or a lidar sensor), features that have been extracted from the visual sensor data, or both and to process the input to generate an output that identifies objects within a physical (or real-world) environment, e.g., the smart environment 100 of FIG. 1. For example, the output can include data defining one or more bounding boxes in the visual sensor data, and for each of the one or more bounding boxes, a respective confidence score that represents a likelihood that an object belonging to an object category7from a set of one or more object categories is present in the region of the environment shown in the bounding box.

[0045] Examples of object categories include a stationary object, a moving object, a wearable object, a holdable object, a toy, a home appliance, a mechanical device, an electrical device (including a nonportable electrical device, such as a desktop computer, and a portable electrical device, such as a mobile phone or a tablet), a bag, a backpack, a watch, a jewelry, an instrument, a clothing, a belt, a footwear, and other physical (or real -world) objects that might appear in the physical environment.

[0046] When the pre-defined algorithms include an object pose estimation neural network, for example, the indexing engine 210 can estimate the pose of objects in the visual sensor data. Generally, the pose of an object is a combination of the position and orientation of the object in the visual sensor data. For example, the indexing engine 210 can generate as the output a pose vector that includes an estimated location in the visual sensor data of each of a predetermined number of key points of the object.

[0047] When the pre-defined algorithms include an object localization algorithm, e.g., a simultaneous localization and mapping (SLAM) algorithm, for example, the indexing engine 210 can receive an input that includes visual sensor data and, in some implementations, robot state data generated by a sendee robot and to process the input to track a set of objects in the structure and within a view of field of the service robot as the service robot moves within the structure. A SLAM algorithm can be used to construct and update a map of the structure based on the set of objects and track the location and / or a path of movement of the service robot within the map. By receiving visual sensor data corresponding to multiple fields of views, the indexing engine 210 can generate a more holistic view of the structure, which can lead to more objects to be included in the construction and updating of the map. With such an arrangement, the accuracy and robustness of tracking a location of the objects within the structure can be improved.

[0048] When the pre-defined algorithms include an object health prediction algorithm, for example, the indexing engine 210 can receive an input that includes visual sensor data or features that have been extracted from the visual sensor data, and process the input to generate an output that defines a predicted health state of an object described in the visual sensor data. The predicted health state can include one or more of: object usage, maintenance condition, or percentage of operating life remaining of the object. For example, the maintenance condition can indicate whether the object is in normal condition or requires maintenance attention, e.g., requires cleaning, repair, or exchange.

[0049] Other non-computer vision algorithms can also be used. For example, the pre-defined algorithms can include an acoustic signal processing algorithm that receives an input that includes audio sensor data (e.g., audio data captured by a microphone) and to process the input to generate an output that specifies respective values of a set of acoustic properties of an audio described in the audio sensor data. For example, the audio sensor data might capture various sounds when an object is operating (e.g., appliances dinging, humming, or buzzing sound), or is moving (e.g.. key jingling sound). The set of acoustic properties can include one or more of: frequency, amplitude, phase, pitch, duration, loudness, timbre, sonic texture, or spatial location of the audio.

[0050] As another example, the pre-defined algorithms can include an olfactory' signal processing algorithm that receives an input that includes olfactory’ sensor data (e.g., olfactory’ data captured by an olfactory' sensor) and to process the input to generate an output that characterizes a smell, or an odor, described in the olfactory sensor data.

[0051] Given these algorithms in the examples above and other algorithms, the indexing engine 210 is able to detect, based on the smart device data 263, an existence, a shape, a location, and other characteristics of a physical (or real -world) object in the physical environment. The indexing engine 210 can compile the data that characterizes the objects into model update data and use the compiled model update data to update the virtual model 266.

[0052] As mentioned above, examples of a physical object include a stationary object, a moving object, a wearable object, a holdable object, a toy, a home appliance, a mechanical device, an electrical device (including a nonportable electrical device, such as a desktop computer, and a portable electrical device, such as a mobile phone or a tablet), a bag. a backpack, a watch, a jewelry, an instrument, a clothing, a belt, a footwear, or other physical (or real-world) objects that might appear in a physical (or real-world) environment, e.g., the smart environment of FIG. 1.

[0053] The virtual model 266 includes a respective virtual object that corresponds to each of the multiple physical (or real-world) objects in the physical environment. That is, individual virtual objects can map to individual physical objects and can be updated in accordance with the individual physical objects. Each virtual object includes one or more data elements that characterize various aspects of the physical object that the virtual object corresponds to.

[0054] For example, a virtual object can include data elements that represent one or more of: a position and / or location of the physical object corresponding to the virtual object (e.g., an absolute position within the physical environment, or relative position with respect to another stationary object in the physical environment), a visual appearance of the physical object corresponding to the virtual object (e.g., the geometry and / or surface details of the physical object), an acoustic property of the physical object corresponding to the virtual object, or an olfactory property of the physical object corresponding to the virtual object. Optionally, but not necessarily, a virtual object can also include graphical data elements that represent graphical characteristics of the physical object corresponding to the virtual object. The graphical data elements can, for example, include one or more pixels each having one or more respective color or brightness values that, when presented on a display, can visually represent the physical object.

[0055] The virtual model 266 includes or has access to a data store (e.g., implemented on one or more logical or physical memory devices) that associates each virtual object with one or more data elements. In some implementations, the data store is a relational data store that stores the virtual objects and their data elements in a variety of tables. In other implementations, the data store is a non-relational data store, e.g., a key-value store that stores keys and values in association with each other, where the keys represent different virtual objects and. for each key. the associated value represent data elements that each characterize a respective aspect of a physical object corresponding to the virtual object, which is represented by the key.

[0056] As a particular example, since some of the smart devices in the smart environment have data processing and data storage capabilities, the data store can be a distributed hash table, or another distributed key-value data store, that spreads the storage of the keys (virtual objects) and the associated values (data elements) across multiple smart devices. This distributed storage of the keys and values results in enhanced user privacy, as information derived from smart device data 263 that is specific to a smart device can be stored locally on the device, without being transmitted to the cloud server.

[0057] In the example of FIG. 2, the virtual model 266 includes virtual objects A-N that correspond respectively to physical objects A-N in a physical environment. The virtual model 266 includes or has access to a data store. The data store associates each virtual object with one or more data elements that each characterize a respective aspect of the physical object corresponding to the virtual object. As illustrated, virtual object N maps to a hex key (a simple tool used to drive bolts and screws with hexagonal sockets in their heads).

[0058] The data store associates the virtual object N with multiple data elements that represent: (i) a position of the hex key in the physical environment (“on the kitchen counter”), (ii) a visual appearance of the hex key, (iii) an acoustic property of the hex key, (iv) an olfactory property of the hex key, and (v) a maintenance condition of the hex key (“good”). Moreover, the data store associates the virtual object N with metadata indicating a timestamp that a data element of the virtual object N was last modified.

[0059] Thus, as new smart device data 263 becomes available, the indexing engine 210 can process the smart device data 263 to generate model update data which is then used to modify the data stored in the data store. For example, the indexing engine 210 can add new data to the data store, update data in the data store, and delete existing data from the data store.

[0060] In some implementations, once smart device data 263 has been obtained for an object, the indexing engine 210 can search the data store for a similar object. For example, indexing engine 210 could use physical proximity and an embedding vector of visual appearance to assess the similarity of the object relative to the existing virtual objects that are already included in the virtual model 266. If the similarity is greater than a given threshold, the indexing engine 210 determines that another instance of the object has been found. Then, the indexing engine 210 determines whether the object is an identical object (e.g., the same tennis ball) or is another instance of the object (e.g., another tennis ball). One way of making this determination is to analyze latest smart device data obtained at the location where the object was detected. If the object is no longer detected at the location, the indexing engine 210 can determine that it is the same object; otherwise, the indexing engine 210 can treat it as a new' object.

[0061] As a more concrete example, the indexing engine 210 can compile the outputs of the object classification neural network into smart device data. The outputs might indicate the existence of a given physical object (e.g., a given mechanical or electrical device, a given bag, or a given clothing) at a given location in the physical environment.

[0062] In this example, when the given physical object is a new object to the environment, the indexing engine 210 can then use the smart device data to update the data store to insert a new key that represents a virtual object corresponding to the new object, and to associate the new key with a value that represents a data element that specifies the given location at which the new object is located.

[0063] Alternatively, when the given physical object is an existing object to the environment, the indexing engine 210 can then use the smart device data to update the value in the data store that is associated with an existing key representing a virtual object corresponding to the existing object, such that the updated value represents a data element that specifies the new location of the existing object. As such, the indexing engine 210 can use the smart device data to identify the new position of the same object after it has been moved away from its previous position.

[0064] As another example, the indexing engine 210 can compile the outputs of the object health prediction algorithm into smart device data. The outputs might indicate the maintenance or operating condition of a physical object (e.g., a home appliance). The indexing engine 210 can then use the smart device data to update the value in the data store that is associated with an existing key representing a virtual object corresponding to the physical object, such that the updated value represents a data element that specifies the latest maintenance or operating condition of the object.

[0065] The service system 264 also includes a query engine 220. The query engine 220 receives an instruction, a command, a query, or a request (collectively referred to herein as a “query 202”) from a user, analyzes the query 202 to identify one or more search terms 204 to query the virtual model 266, and initiates the query to obtain search results 206. To that end, the query engine 220 can for example include or access a language model neural network, or be configured as a rule-based query7processing engine.

[0066] In some situations, the query 202 may be a text-based input (e.g., a query typed using a keyboard), a selection-based input (e.g., touchscreen selection), or an audio-based input (e.g., voice input), entered by the user through either a smart device (or a client device). In some other situations, the service system receives the query7202 as an automated query7generated by one or more computers, e.g., by the service system itself in accordance with predefined settings. In either situations, the query 202 includes or otherwise specifies one or more search terms 204. The one or more search terms 204 may identify one or more physical objects that are of interest to the user.

[0067] For example, the query 202 may be ‘'Where is my hex key?’’ which includes the search term “hex key.” Upon receiving the query 202, the query engine 220, in turn, uses the search term “hex key” to query the data store. The search results 206 include virtual objects representing physical objects that are relevant to, or match with, the search term. The search results 206 also include the data elements included in the relevant / matching virtual elements.

[0068] It will be appreciated that, to identify the search results based on the search terms, the query’ engine 220 can utilize any of a variety of search algorithms. For example, the search results for the search term “hex key” include the virtual object N that represents a hex key, which matches the search term “hex key”. The search results also include the data elements stored in association with the virtual object N in the data store that represent respective aspects of the hex key.

[0069] The service system 264 further includes a response engine 230. The response engine 230 can for example include or access a language model neural network, or be configured as a rule-based response generation engine. The response engine 230 uses the search results 206 to generate a response 232 to the query 202. Depending on the exact content included in the query’ 202, the response 232 can be in any of a variety’ of formats.

[0070] In some implementations, the response 232 can include a command to cause a smart device to perform an action, a command to change the operating state of the smart device, or both. For example, the response 232 can be a command to cause a smart device to provide the search results 206, information extracted or otherwise derived from the search results 206, or both for presentation to the user, e.g., visually and / or audibly.

[0071] In this example, when the query 202 is a question relating to a status, e.g., a location or condition, of a particular physical object, for example, the response 232 can include a command that causes a first one of the smart devices, e.g., a smart speaker, to generate an audio output that describes the current location or the current condition of the particular physical object. Additionally or alternatively, the response 232 can include a command that causes a second one of the smart devices, e.g.. a smart television, to display content that characterizes the current location or the current condition of the particular physical object.

[0072] In some other implementations, the response 232 can include a command to cause another controllable device, e.g.. a client device (e.g., the client device 166 1 or client device 166 2 in FIG. 1) or a hub device (e.g.. the hub device 180 in FIG. 1). to perform an action. Like the response generated in regard to the smart devices, for example, the response 232 can be a command to cause a client device to provide the search results 206, information extracted or otherwise derived from the search results 206, or both for presentation to the user, e.g., visually and / or audibly.

[0073] FIG. 3 is a flow diagram of an example process 300 of generating a response to a query . For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a service system, e g., the service system 264 of FIG. 2, appropriately programmed, can perform the process 300.

[0074] The service system maintains a virtual model for a physical environment (step 302). The physical environment includes a plurality of physical objects. The plurality of physical objects include a plurality of smart devices. The plurality of physical objects can also include non-smart objects that do not have network communication, sensing, and data processing capabilities.

[0075] The virtual model includes a virtual obj ect that maps to each of the plurality of physical objects. Each virtual object has one or more data elements that each characterize a respective aspect of the physical object corresponding to the virtual object. For example, a virtual object can include data elements that represent one or more of: a position and / or location of the physical object corresponding to the virtual object, a visual appearance of the physical object corresponding to the virtual object, an acoustic property of the physical object corresponding to the virtual object, or an olfactory property of the physical object corresponding to the virtual object. Maintaining such a virtual model is explained in more detail with reference to FIG. 4, which shows sub-steps 402-406 corresponding to step 302.

[0076] FIG. 4 is a flow diagram of sub-steps of one of the steps of the process of FIG. 3.

[0077] The service system obtains smart device data from one or more smart devices (step 402). The smart device data can include any data that been generated, captured, or otherwise processed by the smart devices. In some implementations, the smart device data includes one or more of: sensor data output by one or more sensors of a smart device (which characterizes a physical environment in a vicinity of the smart device), metadata output by a smart device, settings information for a smart device, usage logs for a smart device, or the like.

[0078] The service system processes the sensor data to generate model update data (step 404). In general, any suitable algorithms can be used by the service system to do this. For example, the service system can use one or more of: machine learning algorithms, including deep neural networks, computer visual algorithms, mathematical algorithms, or the like.

[0079] The service system updates the virtual model for the home environment based on the model update data (step 406). In some implementations, the virtual model can include a data store that associates each virtual object with one or more data elements that each characterize a respective aspect of the physical object corresponding to the virtual object. The data store can, for example, be configured as a relational or non-relational data store. In these implementations, updating the virtual model thus involves accessing the data store to modify the data stored in the data store based on the model update data. For example, the service system can add a new virtual object to the data store, update data items associated with a given virtual object in the data store, and delete an existing virtual object from the data store.

[0080] Turning back to FIG. 3, the service system receives a query (step 304). In some situations, the service system receives the query from a client device, e.g., as text-based input, a selection-based input, or an audio-based input), entered by a user of the client device. In some other situations, the service system receives the query as an automated query generated by one or more computers, e.g., by the service system itself in accordance with predefined settings. In either situations, the query includes or otherwise specifies a particular physical object. The query can, for example, be a question relating to the current location, or an operating or maintenance condition, of the particular physical object.

[0081] The service system determines, based on the virtual model, a status of the particular physical object (step 306). In implementations where the virtual model includes a data store, the service system can use one or more search terms derived from the query7to search the data store, and, in response, obtain search results. The search results can include a virtual object representing the particular physical object that match with the search terms. The search results can also include the data elements included in the matching virtual element. From these search results, the system can determine the status of the particular physical object, e.g., the location, the operating or maintenance condition, of the particular physical object.

[0082] For example, when a user of submits a query7"‘Where is my hex key?”, the service system can determine that the search term is “hex key,” identify7a virtual object in the virtual model that maps to the hex key, and obtain various characteristics of the hex key represented by the data elements that are stored in association with the virtual object. The service system can therefore obtain the location of the hex key based on data maintained in the virtual model.

[0083] In response to the query, the sendee system generates, based on the status of the particular physical object, a response (step 308). Depending on the exact content included in the query, the response can be in any of a variety of formats.

[0084] In some implementations, the response can include a command to cause a smart device to perform an action, a command to change the operating state of the smart device, or both. In some other implementations, the response can include a command to cause another controllable device, e.g.. a client device or a hub device, to perform an action.

[0085] In the hex key example above, the response can for example be a command to cause a smart device, e.g., a smart television or a smart speaker, or a client device or a hub device, to visually and / or audibly present content about the location of the hex key to the user. The content can be, for example, “Your hex key is on the kitchen counter.”

[0086] As another example, when the query is an automated query relating to a status, e.g., an operating or maintenance condition, of a home appliance, the response can for example be a command to cause the home application, or another smart device, to visually and / or audibly indicate the status, e.g., the operating or maintenance condition, of the home appliance to the user.

[0087] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0088] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0089] A computer program, which may also be referred to or described as a program, software, a software application, an app. a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0090] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[0091] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0092] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g.. an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0093] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory' can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to. or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0094] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory', media and memory' devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0095] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as w ell; for example, feedback provided to the user can be any form of sensory- feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any' form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by' sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web brow ser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0096] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0097] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a J AX framework.

[0098] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0099] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0100] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0101] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0102] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0103] What is claimed is:

Claims

CLAIMS1. A method comprising: maintaining, by one or more computers, a virtual model for a physical environment, wherein the physical environment comprises a plurality of physical objects, and wherein the virtual model comprises a virtual object that maps to each of the plurality of physical objects; receiving, by the one or more computers, a query' that identifies a particular physical object; determining, by the one or more computers and based on the virtual model, a status of the particular physical object; and providing, by the one or more computers, a response to the query, the response including the status of the particular physical object.

2. The method of claim 1, wherein physical objects comprise a plurality of smart devices.

3. The method of claim 2, wherein maintaining the virtual model for the physical environment comprises: obtaining sensor data characterizing the physical environment that has been generated by one or more sensors of one or more of the plurality of smart devices; processing the sensor data to generate model update data; and updating the virtual model for the home environment based on the model update data.

4. The method of any one of claims 1-3, wherein the status comprises one or both of: a current location of the particular physical object in the physical environment; or an operating or maintenance condition of the particular physical object in the physical environment.

5. The method of any one of claims 1-4, wherein the virtual model comprises, for each virtual object, data elements representing one or more of: a position of a physical object corresponding to the virtual object, a visual appearance of the physical object corresponding to the virtual object. an acoustic property of the physical object corresponding to the virtual object, or an olfactory' property' of the physical object corresponding to the virtual object.

6. The method of claim 5, wherein the virtual model further comprises, for each virtual object:metadata indicating a timestamp that a data element of the virtual object was last modified.

7. The method of any one of claims 5-6, wherein the virtual model comprises data stored in a data store that associates the plurality of virtual objects with the data elements.

8. The method of claim 7 when also dependent on claim 2, wherein the data store is a distributed data store, and wherein each smart device locally stores the virtual objects that map to one or more of the plurality of physical objects.

9. The method of any one of claims 1-8, wherein virtual model comprises a graphical representation of the home environment, and wherein the virtual object that maps to a physical object comprises a visual representation of the physical object.

10. The method of any one of claims 1-9, wherein the query comprises one of: a text or voice query generated by a user of the one or more computers, or an automated query generated by the one or more computers.

11. The method of any one of claims 2-10, wherein the response comprises commands for one or more of the plurality of smart devices to cause one or more actions to be performed by the one or more of the plurality of smart devices.

12. The method of claim 11, wherein the commands for the one or more of the plurality7of smart devices comprise one or both of: a command to cause a smart device to generate an audio output that describes the current location of the particular physical object, or a command to cause a smart device to display content that characterizes the current location of the particular physical object.

13. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of the respective method of any preceding claim.

14. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of any preceding claim.

Images