Intelligent assistant for home automation

Abstract

To provide a system and processing for controlling an electronic device by using a virtual assistant.SOLUTION: In one exemplary processing, a user may speak an input to a user device in a natural language form in order to control one or more electronic devices. The user device may transmit the user speech to a server to convert the user speech to text expression. The server may specify one or more electronic devices and an appropriate command to be executed by the one or more electronic devices on the basis of the text expression. The specified one or more devices and the command to be executed may be returned to the user device, and the user device may transfer the command to one or more electronic devices appropriate for execution. The one or more electronic devices execute the command in accordance with the reception of the command, and may transmit their current states to the user device.SELECTED DRAWING: Figure 1

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority from U.S. Provisional Patent Application No. 62 / 005,893, entitled "INTELLIGENT ASSISTANT FOR HOME AUTOMATION," filed May 30, 2014, and U.S. Nonprovisional Patent Application No. 14 / 503,105, entitled "INTELLIGENT ASSISTANT FOR HOME AUTOMATION," filed September 30, 2014, which are incorporated by reference herein in their entirety for all purposes. [Technical field]

[0002] This application relates generally to natural language processing, and more specifically to the use of virtual assistants with natural language processing to control electronic devices. [Background technology]

[0003] Consumer electronic devices that can be remotely controlled using software applications running on computing devices such as mobile phones, tablet computers, laptop computers, desktop computers, etc. are becoming increasingly popular. For example, a number of manufacturers make light bulbs that can be controlled by software applications running on mobile phones that adjust the brightness and / or color of the light bulb. Other devices such as door locks, thermostats, etc. that have similar controls are also provided.

[0004] While these devices may provide users with a greater degree of control and convenience, as the number of remotely controlled devices in a home and the number of types of remotely controlled devices increases, managing these devices can become very difficult. For example, a typical home may include 40-50 light bulbs located throughout various rooms of the home. Using traditional software applications, each light bulb is given a unique identifier, and a user attempting to control one of these devices must select the appropriate identifier from a list of available devices in a graphical user interface. Remembering the correct identifier for a particular light bulb and finding it from a list of 40-50 identifiers can be a difficult and time-consuming task. In addition to the difficulty of managing and controlling a large number of remotely controlled devices, typically each manufacturer provides a different software application that must be used to control its respective device. Thus, a user must find and open one software application to turn those light bulbs on and off, and then find and open another software application to set the temperature of their thermostat. Summary of the Invention

[0005] A system and process for controlling electronic devices using a virtual assistant is provided. In one exemplary process, a user may utter an input in a natural language format to a user device to control one or more electronic devices. The user device may send the user utterance to a server for conversion into a text representation. The server may identify one or more electronic devices and appropriate commands to be executed by the one or more electronic devices based on the text representation. The identified one or more devices and the commands to be executed may be sent back to the user device, which may forward the command to the appropriate one or more electronic devices for execution. In response to receiving the command, the one or more electronic devices may execute the command and send their current state to the user device. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 illustrates an example environment in which a virtual assistant may be used to control an electronic device, according to various embodiments.

[0007] [Diagram 2] FIG. 1 illustrates an example environment in which a virtual assistant can be used to remotely control electronic devices, according to various embodiments.

[0008] [Diagram 3] FIG. 2 illustrates an exemplary user device in accordance with various embodiments.

[0009] [Figure 4] FIG. 2 illustrates a visual representation of a number of entries used to store information associated with an electronic device, according to various embodiments.

[0010] [Diagram 5] FIG. 1 illustrates an example process for controlling an electronic device using a virtual assistant implemented using a client-server model, according to various embodiments.

[0011] [Figure 6] FIG. 1 illustrates an example process for remotely controlling an electronic device using a virtual assistant implemented using a client-server model, according to various embodiments.

[0012] [Figure 7] FIG. 1 illustrates an example process for controlling an electronic device using a virtual assistant on a standalone user device, according to various embodiments.

[0013] [Figure 8] FIG. 2 illustrates an example process for storing a state of an electronic device as a configuration, according to various embodiments.

[0014] [Figure 9] FIG. 1 illustrates an example process for setting a state of an electronic device using a previously stored configuration, according to various embodiments.

[0015] [Figure 10] FIG. 2 is a functional block diagram of an electronic device configured to control an electronic device according to various embodiments.

[0016] [Figure 11] FIG. 2 is a functional block diagram of an electronic device configured to store a state of the electronic device as a configuration, according to various embodiments.

[0017] [Figure 12] FIG. 2 is a functional block diagram of an electronic device configured to set a state of the electronic device based on a stored configuration, according to various embodiments.

[0018] [Figure 13] FIG. 2 is a functional block diagram of an electronic device configured to control an electronic device according to various embodiments.

[0019] [Figure 14] FIG. 2 is a functional block diagram of an electronic device configured to store a state of the electronic device as a configuration, according to various embodiments.

[0020] [Figure 15] FIG. 2 is a functional block diagram of an electronic device configured to set a state of the electronic device based on a stored configuration, according to various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] In the following description of the embodiments, reference is made to the accompanying drawings in which is shown, by way of illustration, specific embodiments that may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the various embodiments.

[0022] Intelligent automated assistants (or virtual assistants) provide an intuitive interface between a user and an electronic device. These assistants may enable a user to interact with a device or system using natural language in spoken and / or textual form. For example, a user may access the services of an electronic device by providing spoken user input in natural language form to a virtual assistant associated with the electronic device. The virtual assistant may perform natural language processing on the spoken user input to infer the user's intent and to operationalize the user's intent into a task. The task may then be performed by executing one or more functions of the electronic device, and associated output may be returned to the user in natural language form.

[0023] The present application relates to a system and process for controlling electronic devices using a virtual assistant. In one exemplary process, a user may utter an input in a natural language format to a user device to control one or more electronic devices. The user device may transmit the user utterance to a server for conversion into a text representation. The server may identify one or more electronic devices and appropriate commands to be executed by the one or more electronic devices based on the text representation. The identified one or more devices and the commands to be executed may be sent back to the user device, which may forward the command to the appropriate one or more electronic devices for execution. In response to receiving the command, the one or more electronic devices may execute the command and transmit their current state to the user device. System Overview

[0024] FIG. 1 illustrates an example system 100 implementing a virtual assistant to control an electronic device, according to various embodiments. The terms “virtual assistant,” “digital assistant,” “intelligent automated assistant,” or “automated digital assistant” may refer to any information processing system that interprets natural language input in the form of speech and / or text to infer user intent and perform an action based on the inferred user intent. For example, to perform an action based on the inferred user intent, the system may perform one or more of: identifying a task flow including steps and parameters designed to realize the inferred user intent; inputting specific requirements from the inferred user intent into the task flow; executing the task flow by invoking a program, method, service, API, etc.; and generating an output response to the user in an audible (e.g., speech) and / or visual form.

[0025] A virtual assistant may be able to accept user requests, at least in part, in the form of natural language commands, requests, statements, predications, and / or queries. Typically, a user request seeks either an informational answer or the performance of a task by the virtual assistant. A satisfactory response to a user request may include either providing the requested informational answer, performing the requested task, or a combination of the two. For example, a user may ask a virtual assistant a question such as "Where am I right now?" Based on the user's current location, the virtual assistant may respond with "You're in Central Park." The user may also request the performance of a task, for example, "Please remind me to call my mother at 4 p.m. today." In response, the virtual assistant may recognize the request and then create an appropriate reminder item in the user's electronic schedule. During the performance of a requested task, the virtual assistant may interact with the user in a continuous dialog involving multiple exchanges of information over time. There are many other ways to interact with a virtual assistant to request information or the performance of various tasks. In addition to providing verbal responses and taking programmed actions, the virtual assistant may provide other visual or audio forms of responses (e.g., as text, alerts, music, videos, animations, etc.).

[0026] One example of a virtual assistant is shown in Applicant's U.S. patent application Ser. No. 12 / 987,982, for "Intelligent Automated Assistant," filed Jan. 10, 2011, the entire disclosure of which is incorporated herein by reference.

[0027] As shown in FIG. 1, in some embodiments, the virtual assistant may be implemented according to a client-server model. The virtual assistant may include a client-side portion that runs on the user device 102 and a server-side portion that runs on the server system 110. The user device 102 may include any electronic device, such as a mobile phone, tablet computer, portable media player, desktop computer, laptop computer, PDA, television, television set-top box, wearable electronic device, etc., and may communicate with the server system 110 via one or more networks 108, which may include the Internet, an intranet, or any other wired or wireless public or private network. The client-side portion that runs on the user device 102 may provide client-side functionality, such as user-responsive input and output processing and communication with the server system 110. The server system 110 may provide server-side functionality for any number of clients that reside on each user device 102.

[0028] The server system 110 may include one or more virtual assistant servers 114, which may include a client-facing I / O interface 122, one or more processing modules 118, a data and model storage device 120, and an I / O interface 116 to external services. The client-facing I / O interface 122 may facilitate client-facing input and output processing for the virtual assistant server 114. The one or more processing modules 118 may utilize the data and model storage device 120 to determine a user's intent based on natural language input and perform a task based on the inferred user intent. Additionally, the data and model storage device 120 may store a unique identifier, status, type, location, and any other relevant information associated with one or more of the user device 102 and / or electronic devices (e.g., electronic devices 128, 130, and 132) that may be controlled by the server system 110. In some embodiments, the virtual assistant server 114 may communicate with external services 124, such as telephone services, calendar services, information services, messaging services, navigation services, etc., via the network(s) 108 to complete a task or obtain information. An I / O interface 116 to the external service may facilitate such communication.

[0029] Server system 110 may be implemented on one or more standalone data processing devices or a distributed network of computers. In some embodiments, server system 110 may also utilize various virtual devices and / or services of third party service providers (e.g., third party cloud service providers) to provide the underlying computing and / or infrastructure resources of server system 110.

[0030] The user device 102 may further be coupled to electronic devices 128, 130, and 132 via one or more networks 126. The electronic devices 128, 130, and 132 may include any type of remotely controlled electronic device, such as a light bulb (e.g., having a binary on / off state, a numerical dimmable state, a color state, etc.), a garage door (e.g., having a binary open / close state), a door lock (e.g., having a binary locked / unlocked state), a thermostat (e.g., including one or more numerical temperature states, such as high, low, time-based temperature, etc.), an electrical outlet (e.g., having a binary on / off state), a switch (e.g., having a binary on / off state), etc. The network(s) 126 may include a WiFi network or any other wired or wireless public or private local network. Additionally or alternatively, the user device 102 may be coupled to communicate directly with the electronic devices 128, 130, or 132, for example, using Bluetooth, BTLE, line of sight, peer-to-peer, or another radio-based or other wireless communication. Thus, in the illustrated embodiment, the user device 102 may be located in proximity to the electronic devices 128, 130, and 132 so as to be able to communicate directly with them or over the same local network. For example, the user device 102 and the electronic devices 128, 130, and 132 may be located in the same home or building, and the network(s) 126 may include the WiFi network of the home or building. As described in more detail below in connection with FIGS. 5, 8, and 9, the user device 102 may issue commands to control any of the electronic devices 128, 130, and 132 in response to natural language speech input provided by a user to the user device 102.

[0031] Although only three electronic devices 128, 130, and 132 are shown, it should be understood that system 100 may include any number of electronic devices. Furthermore, although the functionality of the virtual assistant is shown in FIG. 1 as including both a client-side portion and a server-side portion, in some embodiments, the functionality of the assistant may be implemented as a standalone application installed on a user device. Furthermore, the distribution of functionality between the client and server portions of the virtual assistant may vary depending on the embodiment. For example, in some embodiments, the client running on user device 102 may be a thin client that provides only user-facing input and output processing functionality and relegates all other functionality of the virtual assistant to a back-end server.

[0032] 2 illustrates another exemplary system 200 for implementing a virtual assistant that remotely controls electronic devices according to various embodiments. Similar to the system 100, the system 200 may include a user device 102, a server system 110, and an external service 124 communicatively coupled by a network(s) 108. However, in contrast to the system 100, the user device 102 may not be coupled to the electronic devices 128, 130, and 132. Instead, the system 200 may include a second user device 134 that is communicatively coupled to the user device 102 and / or the server system 110 via the network(s) 108 and communicatively coupled to the electronic devices 128, 130, and 132 via the network(s) 126. This configuration may represent a situation in which the user and user device 102 are located remotely from the electronic devices 128, 130, and 132 (e.g., the user and user device 102 are located in the user's office and the electronic devices 128, 130, and 132 are located in the user's home).

[0033] Second user device 134 may include any type of electronic device, such as a mobile phone, tablet computer, portable media player, desktop computer, laptop computer, PDA, television, television set-top box, wearable electronic device, etc., and may be configured to receive commands from user device 102 and / or server system 110 and issue commands to electronic devices 128, 130, and 132. As described in more detail below in connection with FIG. 6, second user device 134 may issue commands to control any of electronic devices 128, 130, and 132 in response to natural language speech input provided by a user to user device 102. User Device

[0034] 3 is a block diagram of a user device 102 (or a second user device 134) according to various embodiments. As shown, the user device 102 may include a memory interface 302, one or more processors 304, and a peripherals interface 306. The various components within the user device 102 may be coupled to one another by one or more communication buses or signal lines. The user device 102 may further include various sensors, subsystems, and peripheral devices coupled to the peripherals interface 306. The sensors, subsystems, and peripherals collect information and / or facilitate various functionality of the user device 102.

[0035] For example, the user device 102 may include a motion sensor 310, a light sensor 312, and a proximity sensor 314 coupled to the peripherals interface 306 to facilitate orientation, light, and proximity sensing functions. One or more other sensors 316, such as a positioning system (e.g., a GPS receiver), a temperature sensor, a biometric sensor, a gyroscope, a compass, and an accelerometer, are also connected to the peripherals interface 306 to facilitate related functions.

[0036] In some embodiments, the camera subsystem 320 and optical sensor 322 may be utilized to facilitate camera functions such as taking pictures and recording video clips. Communications functions may be facilitated through one or more wired and / or wireless communication subsystems 324, which may include various communication ports, radio frequency receivers and transmitters, and / or optical (e.g., infrared) receivers and transmitters. The audio subsystem 326 may be coupled to a speaker 328 and a microphone 330 to facilitate voice-enabled functions such as voice recognition functions, voice duplication functions, digital recording functions, and telephone functions.

[0037] In some embodiments, the user device 102 may further include an I / O subsystem 340 coupled to the peripherals interface 306. The I / O subsystem 340 may include a touchscreen controller 342 and / or other input controller(s) 344. The touchscreen controller 342 may be coupled to a touchscreen 346. The touchscreen 346 and touchscreen controller 342 may detect contact and movement or interruption using any of a number of touch sensing technologies, such as, for example, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as proximity sensor arrays. The other input controller(s) 344 may be coupled to other input / control devices 348, such as one or more buttons, rocker switches, thumb wheels, infrared ports, USB ports, and / or pointer devices such as a stylus.

[0038] In some embodiments, the user device 102 may further include a memory interface 302 coupled to a memory 350. The memory 350 may include any electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, a portable computer diskette (magnetic), a random access memory (RAM) (magnetic), a read-only memory (ROM) (magnetic), an erasable programmable read-only memory (EPROM) (magnetic), a portable optical disk such as a CD, a CD-R, a CD-RW, a DVD, a DVD-R, or a DVD-RW, or a flash memory such as a compact flash card, a secure digital card, a USB memory device, a memory stick, or the like. In some embodiments, the non-transitory computer-readable storage medium of memory 350 may be used to store instructions for use by or with an instruction execution system, apparatus, or device (e.g., performing some or all of processes 500, 600, 700, 800, or 900 described below), such as a computer-based system, a system including a processor, or other system that may fetch instructions from an instruction execution system, apparatus, or device and execute those instructions. In other embodiments, the instructions (e.g., performing some or all of processes 500, 600, 700, 800, or 900 described below) may be stored on a non-transitory computer-readable storage medium of server system 110 or distributed between the non-transitory computer-readable storage medium of memory 350 and the non-transitory computer-readable storage medium of server system 110. In the context of this document, a "non-transitory computer-readable storage medium" may be any medium capable of containing or storing a program for use by or in connection with an instruction execution system, apparatus, or device.

[0039] In some embodiments, memory 350 may store an operating system 352, a communications module 354, a graphical user interface module 356, a sensor processing module 358, a telephony module 360, and applications 362. Operating system 352 may include instructions for handling basic system services and instructions for performing hardware-dependent tasks. Communications module 354 may facilitate communication with one or more additional devices, one or more computers, and / or one or more servers. Graphical user interface module 356 may facilitate graphic user interface processing. Sensor processing module 358 may facilitate sensor-related processes and functions. Telephony module 360 ​​may facilitate telephony-related processes and functions. Applications module 362 may facilitate various functions of user applications, such as electronic messaging, web browsing, media processing, navigation, imaging, and / or other processes and functions.

[0040] Memory 350 may also store client-side virtual assistant instructions (e.g., in virtual assistant client module 364) to provide client-side functionality of the virtual assistant, as well as various user data 366 (e.g., user-specific vocabulary data, preference data, and / or other data such as the user's electronic address book, to-do list, shopping list, etc.).

[0041] In various embodiments, the virtual assistant client module 364 may be able to accept voice input (e.g., speech input), text input, touch input, and / or gesture input via various user interfaces (e.g., I / O subsystem 340, audio subsystem 326, etc.) of the user device 102. The virtual assistant client module 364 may also be able to provide output in audio (e.g., speech output), visual, and / or tactile form. For example, the output may be provided as voice, sound, alerts, text messages, menus, graphics, videos, animations, vibrations, and / or combinations of two or more of the above. In operation, the virtual assistant client module 364 may communicate with the virtual assistant server using the communication subsystem 324.

[0042] In some embodiments, the virtual assistant client module 364 may collect additional information from the surrounding environment of the user device 102 using various sensors, subsystems, and peripherals to establish a context associated with the user, the current user interaction, and / or the current user input. In some embodiments, the virtual assistant client module 364 may provide the context information, or a subset thereof, along with the user input to the virtual assistant server to help infer the user's intent. The virtual assistant may also use the context information to determine how to prepare and deliver output to the user.

[0043] In some embodiments, the context information associated with the user input may include sensor information such as lighting, ambient noise, ambient temperature, images or videos of the surrounding environment, distance to another object, etc. The context information may further include information associated with the physical state of the user device 102 (e.g., device orientation, device location, device temperature, power level, speed, acceleration, motion patterns, cellular signal strength, etc.) or the software state of the user device 102 (e.g., running processes, installed programs, past and current network activity, background services, error logs, resource usage, etc.). Any of these types of context information may be provided to the virtual assistant server 114 as context information associated with the user input.

[0044] In some embodiments, the virtual assistant client module 364 may selectively provide information stored on the user device 102 (e.g., user data 366) in response to a request from the virtual assistant server 114. The virtual assistant client module 364 may also elicit additional input from the user via a natural language dialog or other user interface in response to a request by the virtual assistant server 114. The virtual assistant client module 364 may pass additional input to the virtual assistant server 114 to aid the virtual assistant server 114 in inferring intent and / or achieving the user's intent as expressed in the user request.

[0045] The memory 350 may further store electronic device data 370, which may include a unique identifier, status, type, location, and any other relevant information associated with one or more of the electronic devices (e.g., electronic devices 128, 130, and 132) that may be controlled by the user device 102 and / or the server system 110. FIG. 4 shows a visual representation of entries that may be stored in the electronic device data 370 for seven different electronic devices. As shown, each entry includes a unique name, type, and status of the electronic device. The data and model storage 120 of the virtual assistant server 114 may include similar or identical entries for electronic devices that may be maintained separately from those of the electronic device data 370 of the memory 350.

[0046] Returning to FIG. 3 , the memory 350 may further include instructions (e.g., in the daemon module 368) for creating and updating entries for the electronic device in the electronic device data 370, communicating with the electronic device in the system 100, and communicating with the server system 110. For example, to add an electronic device to the system 100, a software application associated with the electronic device may communicate with the processor(s) 304 executing the daemon module 368 to provide the unique name, type, status, location, etc. of the electronic device to the user device 102. The software application may allow the user to enter the unique name in any desired manner. For example, a drop-down box with a common name and / or a free-form text field may be provided within the application to allow the user to name a particular device. The type, status, and / or location of the electronic device may be pre-specified or may be specified by the software application via communication with the electronic device. The processor(s) 304 executing the daemon module 368 may store this information as an entry in the electronic device data 370 or may send this information to the server system 110 for storage in the data and model storage 120. Additionally, the daemon module 368, when executed by the processor(s) 304, may receive commands from the server system 110 over the network(s) to be provided to the electronic devices 128, 130, and 132, and may send those commands to the appropriate electronic devices. The processor(s) 304 executing the daemon module 368 may also receive status updates from the electronic devices 128, 130, and 132, update corresponding entries in the electronic device data 370 to reflect the updated status of the devices, and send the status updates to the server system 110 so that the server system 110 can update corresponding entries in the data and model storage 120 to reflect the updated status of the devices.

[0047] Additionally, the daemon module 368 may include instructions that manage access to the electronic device data 370 by other devices and software applications. For example, the daemon module 368, when executed by the processor(s) 304, may allow the server system 110 to access all of the electronic device data 370, while restricting access by other devices or software applications to only a portion of the electronic device data 370. This may be useful when using the user device 102 to control electronic devices made by different manufacturers. In these situations, the devices or software applications from each manufacturer may communicate with the daemon module 368 using an API, and the daemon module 368 may restrict their access to only the portion of the electronic device data 370 that corresponds to each manufacturer's respective electronic device. For example, company X may manufacture light bulbs that can be controlled by the user device 102, and company Y may manufacture thermostats that can be controlled by the user device 102. The daemon module 368 may facilitate communication between the user device 102 and each of the light bulb and the thermostat such that the user device 102 may issue commands to the electronic devices and receive status information associated with the electronic devices for updating the electronic device data 370. However, the daemon module 368 may limit the access that the light bulb (and associated software applications running on the user device 102) has to information in the electronic device data 370 to only information associated with the light bulb (and potentially any other electronic devices manufactured by company X). Similarly, the daemon module 368 may limit the access that the thermostat (and associated software applications running on the user device 102) has to information in the electronic device data 370 to only information associated with the thermostat (and potentially any other electronic devices manufactured by company Y).However, the daemon module 368 may allow the server system 110 access to all information in the electronic device data 370 .

[0048] In various embodiments, memory 350 may include additional or fewer instructions. Additionally, various functions of user device 102 may be implemented in hardware and / or firmware, including in the form of one or more signal processing circuits and / or application specific integrated circuits. Local control of electronic devices

[0049] 5 illustrates an example process 500 for controlling an electronic device using a virtual assistant. In some embodiments, the process 500 may be performed using a system similar to or identical to the system 100 illustrated in FIG. 1. In these embodiments, the blocks of the process 500 may be performed by both the user device 102 and the server system 110. Specifically, the blocks on the left side of FIG. 5 may be performed by the user device 102, and the blocks on the right side of FIG. 5 may be performed by the server system 110.

[0050] At block 502, audio input including user speech may be received at a user device. In some embodiments, a user device (e.g., user device 102) may receive the audio input including the user speech via a microphone (e.g., microphone 330). The microphone may convert the audio input into an analog or digital representation and provide the audio data to one or more processors (e.g., processor(s) 304).

[0051] In block 504, data corresponding to the audio input received in block 502 may be transmitted to one or more servers for processing. For example, the user device 102 may transmit data corresponding to the audio input via the network(s) 108 to the virtual assistant server 114 of the server system 110.

[0052] In block 506, data corresponding to the audio input transmitted by the user device in block 504 may be received by one or more servers. For example, the virtual assistant server 114 of the server system 110 may receive data corresponding to the audio input transmitted by the user device 102 over the network(s) 108.

[0053] At block 508, speech-to-text conversion may be performed on the data corresponding to the audio input to convert the user utterance into a textual representation of the user utterance. The user utterance may be converted using any known speech-to-text conversion process.

[0054] At block 510, one or more electronic devices may be identified based at least in part on the textual representation generated at block 508. In some embodiments, block 510 may include processing the textual representation of the user input to identify a user intent to issue a command to one or more electronic devices. As described above, server system 110 may include one or more data and model stores 120 that may store unique identifiers, status, type, location, and any other relevant information associated with electronic devices that may be controlled using system 100. Thus, block 510 may include identifying one or more of the electronic devices having associated information stored in data model and store 120.

[0055] The one or more electronic devices may be identified in any number of ways. In some embodiments, the one or more electronic devices may be identified by parsing the textual representation to identify any of a set of nouns that correspond to electronic devices supported by system 100. For example, the set of nouns may include a unique name of the electronic device, possible types of electronic devices and their synonyms (e.g., garage door, thermostat, light, dimmable light, switch, color-changing light, light bulb, lamp, lock, outlet, socket, etc.), possible device state categories (e.g., volume, temperature, brightness, color, etc.), etc. stored in electronic device data 370 and data and model storage 120.

[0056] Illustratively, using the example electronic device entries of FIG. 4, the set of nouns may include unique names of seven electronic devices (e.g., “garage door”, “2nd floor thermostat”, “1st floor thermostat”, “living room lamp 1”, “living room lamp 2”, “front door”, and “toaster outlet”), possible types of electronic devices and their synonyms (e.g., “garage door”, “thermostat”, “light”, “light bulb”, “lamp”, “lock”, “outlet”, “socket”, etc.), and possible device state categories (e.g., temperature). Thus, if the textual representation of the user utterance generated in block 508 includes “lock the front door”, the textual representation may be processed using the processing module 118 and the data and model storage 120 to search for any of the set of nouns. As a result of the search, it may be determined that the textual representation includes the unique name “front door”, and the electronic devices identified in block 510 may include this device. It should be appreciated that more than one electronic device may be identified in block 510 depending on the textual representation of the user utterance. For example, if the textual representation of the user utterance generated in block 508 instead includes "turn off all light bulbs," the textual representation may be processed using the processing module 118 and the data and model storage 120 to search for any of the set of nouns. As a result of the search, it may be determined that the textual representation includes the synonym "light bulb" for the possible device type "lighting," and that the command from the user was to turn off all devices of that type. As a result, the electronic devices identified in block 510 may include both "Living Room Lamp 1" and "Living Room Lamp 2."

[0057] In some examples, it may be difficult to identify the appropriate electronic device using only the set of nouns described above. For example, a textual expression including "turn on the lights" may generate type matches with both "living room lamp 1" and "living room lamp 2." In these examples, block 510 may further include using contextual information received from the user device 102 (e.g., received as part of the data corresponding to the audio input in block 506) to disambiguate between the potentially matching electronic devices. Any type of contextual information may be used, such as sensor information (e.g., lighting, ambient noise, ambient temperature, images or videos of the surrounding environment, distance to another object, etc.), the physical state of the user device 102 (e.g., device orientation, device location, device temperature, power level, speed, acceleration, motion patterns, cellular signal strength, etc.), information associated with the software state of the user device 102 (e.g., running processes, installed programs, past and current network activity, background services, error logs, resource usage, etc.). For example, continuing with the "turn on the lights" example provided above, the location and orientation of the user device 102 when it received the audio input at block 502 may be provided to the server system 110 at blocks 504 and 506. This contextual information may be used to determine the location of the user device 102 and / or the direction in which the user device 102 was pointed when it received the user utterance. The closest light or the light to which the user device 102 was pointing when compared to the known locations of "Living Room Lamp 1" and "Living Room Lamp 2" stored in the data and model storage 120 may be selected as the electronic device at block 510. Other types of contextual information may be used in a similar manner at block 510 to disambiguate between potentially matching electronic devices by identifying contextual information that increases or decreases the likelihood that one or more of the potentially matching electronic devices is referenced by the textual representation of the user utterance.

[0058] In still other embodiments, words associated with states in the textual representation of the user input may be used additionally or alternatively to disambiguate between potentially matching electronic devices or to identify an appropriate electronic device. For example, a textual representation including "set to 68" may not yield any matches when using the set of nouns described above. Thus, in these embodiments, block 510 may further include parsing the textual representation to identify any of a set of words associated with states corresponding to electronic devices supported by system 100. For example, the set of words associated with states may include possible states of an electronic device and their synonyms (e.g., open, closed, shut, on, off, active, inactive, lock, locked, color, etc.), types of state values ​​(e.g., binary, floating point, etc.), queries regarding the state of the device, adjectives associated with particular types of states (e.g., warmer, cooler, brighter, darker, color, etc.), etc. When used to parse "set to 68", the textual representation may be determined to include the floating point value "68". When compared to the entries shown in FIG. 4, it may be determined that only “2nd Floor Thermostat” and “1st Floor Thermostat” accept a floating point value for the state. To disambiguate between the two thermostats, context information such as the location of the user device 102 may be used to select the thermostat closer to the user device 102 as the electronic device identified in block 510. Similarly, if the textual representation instead included “make brighter”, a set of words associated with the state may be used to determine whether the textual representation includes the word “brighter”, which is an adjective that describes the state of a device having a type “lighting”. To disambiguate between the two lights, context information such as the location of the user device 102 may be used to select the light closer to the user device 102 as the electronic device identified in block 510.

[0059] At block 512, a command to be performed by each of the one or more electronic devices identified at block 510 may be identified. The command(s) to be performed may be identified in any number of ways. In some embodiments, the command(s) to be performed may be identified by parsing the textual representation of the user utterance to identify any of a set of words associated with states (e.g., possible states of the electronic device and their synonyms, types of state values, queries regarding the state of the device, adjectives associated with a particular type of state). The identified states or actions may then be used to identify a command to be performed by each of the one or more electronic devices identified at block 510. For example, if the textual representation of the user utterance includes "lock the front door," it may be determined that the textual representation includes the state "locked" from the set of states. Thus, at block 512, a command to transition to the "locked" state may be generated and identified as a command to be performed by the electronic device "front door" identified at block 510. Although in the above examples a single command for a single electronic device is specified, it should be understood that multiple commands may be specified if multiple electronic devices are identified in block 510. For example, the textual expression "Turn on all lights" may result in a command being specified in block 512 to transition each of "Living Room Lamp 1" and "Living Room Lamp 2" to an "on" state.

[0060] In some examples, if the textual representation of the user utterance includes one of the adjectives associated with a particular type of state (e.g., warmer, cooler, brighter, darker, etc.), block 512 may include identifying a command to set the state of the electronic device identified in block 510 to a value relative to its current value. For example, if the textual representation includes "make warmer," the command identified in block 514 may be a command to increase the temperature of the thermostat identified in block 510 by a predetermined amount. Because the actual state of the electronic device may differ from the state stored in data and model storage 120, the command may not be a command to transition the state value to a particular value (e.g., determined using the state of the thermostat stored in data and model storage 120), but rather a command to change the state value by an amount relative to the current value of the electronic device.

[0061] In some embodiments, if the textual representation of the user utterance includes a query regarding the status of a device, block 512 may include identifying instructions to cause the user device 102 to execute a query on the identified electronic device(s). For example, if the textual representation includes "Is the garage door closed?", the command identified in block 514 may be a command to execute a query regarding the status of the electronic device "garage door."

[0062] At block 514, an identifier for each of the one or more electronic devices identified at block 510 and the command(s) to be performed by the one or more electronic devices identified at block 512 may be transmitted to the user device. For example, the server system 110 may transmit a unique identifier associated with each of the electronic devices identified at block 510 and the command(s) to be performed by the one or more electronic devices to the user device 102 over the network(s) 108.

[0063] In some examples, the textual representation of the user utterance generated in block 508 may also be transmitted to the user device in block 514. In these examples, blocks 510 and 514 may also be performed on the user device. The transmitted textual representation may be used by the user device to identify one or more electronic devices and / or to identify the command(s) to be performed by the one or more electronic devices in electronic device data 370. This may be desirable, for example, when electronic device data 370 in the user device is newer than the data on one or more servers. In such cases, the user device may identify electronic devices and / or commands not included in the data on the one or more servers.

[0064] In other examples, the textual representation may be analyzed at one or more servers at blocks 510 and / or 512 to identify significant words or terms that may be suitable for identifying one or more of the electronic devices and / or identifying command(s) to be performed by the one or more electronic devices. In these examples, the analyzed significant words or terms may also be transmitted to the user device at block 514. The transmitted significant words or terms may be used by the user device to identify one or more electronic devices and / or identify command(s) to be performed by the one or more electronic devices in electronic device data 370. This may be desirable, for example, when electronic device data 370 in the user device is newer than data on one or more servers. In such cases, the user device may identify electronic devices and / or commands not included in the data of the one or more servers.

[0065] The respective identifiers and command(s) of the one or more electronic devices sent by the one or more servers may be received by the user device in block 516. For example, the user device 102 may receive the unique identifier(s) and command(s) sent by the server system 110 in block 514 over the network(s) 108.

[0066] In block 518, the user device may transmit the command(s) received in block 516 to the electronic device(s) associated with the identifier(s) received in block 516. For example, if the user device 102 received in block 516 an identifier associated with the electronic device 128 (e.g., “Front Door”) and a command to transition the electronic device to a “LOCKED” state, the user device 102 may transmit the command to transition to the “LOCKED” state to the electronic device 128 over the network(s) 126. If the user device 102 received additional identifiers and commands in block 516, the user device 102 may further transmit those commands to the electronic devices identified in block 518.

[0067] In some embodiments, the actual state of the electronic device may not be the same as the state of the electronic device as stored in the user device 102 (e.g., in memory 350) and / or the server system 110 (e.g., in data and model storage 120). For example, a door lock may be manually opened or closed without using a virtual assistant of the system 100. Thus, in some embodiments, block 518 may be performed regardless of the state of the electronic device as stored in the user device 102 and / or the server system 110. For example, a command may be sent to a door lock to set the state of the door lock to “locked” even if the corresponding entry in the user device 102 and / or the server system 110 indicates that the door is already locked. Furthermore, block 518 may be performed without first querying the electronic device (e.g., between blocks 516 and 518) to determine its actual state in order to reduce the time required to issue a command to the electronic device. For example, a command to set the state of a door key to "locked" may be sent to the door key without first performing a query for its state, thereby reducing the time required to send the command to the door key by an amount equivalent to the time required to send the query to the door key and receive the state from the door key.

[0068] In block 520, after each electronic device executes their respective command, the user device may receive the current state of each electronic device (to which the command was sent in block 518). For example, after the user device 102 executes the command to set the state of the electronic device 128 to “locked”, the user device 102 may receive the updated state of the electronic device 128. In this example, the current state returned to the user device 102 may be the state “locked”. If the command was sent to more than one electronic device in block 518, block 520 may further include receiving the current states from the electronic devices. In some examples, block 520 may further include updating the state of the electronic device in electronic device data 370 based on the received current state. For example, the user device 102 may update the state of the electronic device 128 in electronic device data 370 to “locked”. In some embodiments, block 520 may further include outputting an audio or visual indication of the outcome of the command(s) sent to the electronic device(s) in block 518 based on the sent command(s) and the current state(s) of the received electronic device(s). For example, if the command sent to the electronic device 128 was a command to transition the state of the device to "locked" and the current state of the electronic device 128 received in block 520 was "locked", the outcome indication may be that the electronic device 128 successfully transitioned to the "locked" state. Alternatively, if the current state of the received electronic device (e.g., "unlocked") differs from the desired state (e.g., "locked") indicated in the command, an indication of a failure to transition to the "locked" state may be provided to the user. The current state may further include an error state, such as an unspecified or unavailable state of the device.

[0069] In block 522, the user device may transmit the current state(s) of the electronic device(s) received in block 520 to one or more servers. For example, the user device 102 may transmit the current states of the electronic devices received in block 520 to the server system 110. If the current states of more than one electronic device were received in block 520, block 522 may further include transmitting the current states to the server system 110.

[0070] In block 524, one or more servers may receive the current state(s) of the electronic device(s) sent by the user device in block 522. For example, the server system 110 may receive the current state of the electronic device sent by the user device 102 in block 522. If the user device 102 sent more than one current state, block 524 may further include receiving those current states as well. In some embodiments, block 524 may further include updating the state(s) of the electronic device(s) in the data and model storage 120 based on the received current state(s). For example, the server system 110 may update the state of the electronic device 128 in the data and model storage 120 to “locked”.

[0071] In some examples, process 500 may further include generating a notification associated with a current state of one or more of the electronic devices in response to determining that the predetermined condition is met. For example, in response to one or more of the electronic devices being in a predetermined state (e.g., the "front door" is unlocked) while the location of user device 102 has left a predetermined area (e.g., an area corresponding to the user's home), a notification may be provided to the user via user device 102 indicating that the user has forgotten to lock those doors. Other similar notifications may be generated in response to other predetermined conditions.

[0072] A virtual assistant implemented by a user device may use process 500 to receive natural language commands to set states or perform queries to any number of electronic devices. The natural language commands can query the electronic devices in any desired manner and do not need to include a unique identifier for the electronic device or the type of electronic device. Remote control of electronic devices

[0073] FIG. 6 illustrates an exemplary process 600 for remotely controlling an electronic device using a virtual assistant. In some embodiments, the process 600 may be similar to the process 500, except that the process 600 may be performed using a system similar to or identical to the system 200 illustrated in FIG. 2. For example, the process 600 may be performed by a user device (e.g., the user device 102) located remotely from the electronic device being controlled (e.g., the electronic devices 128, 130, and 132), and a second user device (e.g., the second user device 134) may be used instead to control the electronic device. Thus, each portion of the process 600 may be performed by each of the user device 102, the server system 110, and the second user device 134. Specifically, the blocks on the left side of FIG. 6 may be performed by the user device 102, the blocks in the center of FIG. 6 may be performed by the server system 110, and the blocks on the right side of FIG. 6 may be performed by the second user device 134.

[0074] The blocks of process 600 may be similar or identical to the like-numbered blocks of process 500, except that blocks 516, 518, 520, and 522 of process 600 may instead be performed by a second user device (e.g., second user device 134). Further, so that in block 514, respective identifiers of the one or more electronic devices and the command(s) to be performed by the electronic device(s) may instead be sent to the second user device, and in block 524, the current state(s) of the electronic device(s) may instead be received from the second user device.

[0075] In some embodiments, process 600 may further include transmitting, by one or more servers, the current state(s) of the electronic device(s) received at block 524 to the user device that received the audio input at block 502. Additionally or alternatively, the process may include transmitting an indication of success of the command sent to the electronic device at block 518. For example, a visual or audio output may be generated that notifies the user of successful, partial success, or failure of execution of the command by the electronic device. The determination of success may depend on the command sent to the electronic device and the current state of the device received at block 524. Standalone control of electronic devices

[0076] 7 illustrates an example process 700 for controlling an electronic device using a virtual assistant. In some embodiments, the process 700 may be similar to the process 500, except that the process 700 may be performed using a standalone user device that may perform the functions of both the user device 102 and the server system 110. As a result, all blocks of the process 700 may be performed by a user device (e.g., the user device 102).

[0077] The blocks of process 700 may be similar or identical to the like-numbered blocks of process 500, except that blocks 508, 510, and 512 may instead be performed by a user device (e.g., user device 102). Furthermore, this results in no need to perform blocks corresponding to functions communicating between user device 102 and server system 110 (e.g., blocks 504, 506, 514, 516, 522, and 524). Remembering the configuration of an electronic device

[0078] FIG. 8 illustrates an example process 800 for storing the states of multiple electronic devices as a configuration using a virtual assistant. A configuration may represent a set of stored states of multiple electronic devices that may be referenced in a spoken user input to transition the multiple electronic devices to the states defined in the configuration. For example, a user may create a "sleep" configuration in which the states of all lights are set to off, the state of the thermostat is set to 72°F, the state of all doors is set to locked, and the state of the garage door is set to closed. Thus, when the user goes to bed, the user may provide a command to the user device 102, such as "go to bed," that references the stored configuration, and the system 100 may set the states of the electronic devices based on the stored states in the sleep configuration. In some embodiments, the process 800 may be performed using a system similar to or identical to the system 100 shown in FIG. 1. In these embodiments, the blocks of the process 800 may be performed by both the user device 102 and the server system 110. Specifically, the blocks on the left side of FIG. 8 may be performed by the user device 102, and the blocks on the right side may be performed by the server system 110.

[0079] At block 802, audio input including user utterances may be received at a user device in a manner similar or identical to block 502 of process 500. At block 804, data corresponding to the audio input received at block 802 may be transmitted to one or more servers for processing in a manner similar or identical to block 504 of process 500.

[0080] At block 806, data corresponding to the audio input sent by the user device at block 804 may be received by one or more servers in a manner similar or identical to block 506 of process 500. At block 808, speech-to-text conversion may be performed on the data corresponding to the audio input in a manner similar or identical to block 508 of process 500 to convert the user utterance into a textual representation of the user utterance.

[0081] In block 810, it may be determined that the textual representation of the user utterance represents a user intent to store the state of the electronic device of the system 100 as a configuration. For example, one or more processing modules 118 of the server system 110 may utilize the data and model storage 120 to identify the user intent based on the natural language input. In some embodiments, this may include parsing the textual representation for words that may be related to storing a configuration, such as "save," "store," "name," "keep," "configuration," "scene," and synonyms thereof. For example, if the textual representation includes "store this configuration as sleep," it may be determined in block 810 that the user intends to store the state of the electronic device as a configuration named "sleep." Other textual representations, such as "save this scene as work," may determine in block 810 that the user intends to store the state of the electronic device as a configuration named "work."

[0082] At block 812, in response to determining that the textual representation of the user utterance indicates a user intent to store the state of the electronic device as a configuration in system 100, one or more servers may transmit instructions to the user device to perform a query regarding the state of the electronic device. For example, server system 110 may transmit instructions to user device 102 over network(es) 108 to perform a query regarding the state of electronic devices 128, 130, and 132.

[0083] At block 814, instructions to perform queries regarding the status of the electronic devices sent by one or more servers may be received by the user devices. For example, the user device 102 may receive instructions from the server system 110 over the network(s) 108 to perform queries regarding the status of the electronic devices 128, 130, and 132.

[0084] At block 816, the user device may send a query to each of the electronic devices regarding their current status. For example, user device 102 may send a command over network(s) 126 to each of electronic devices 128, 130, and 132 instructing them to return their current status.

[0085] At block 818, the user device may receive the current status of each of the electronic device(s) (or the current status of all electronic devices capable of transmitting a current status) in response to the query sent at block 516. For example, user device 102 may receive the current status of electronic devices 128, 130, and 132 over network(s) 126 in response to the query sent to each of the devices at block 816. In some embodiments, block 818 may further include updating the status of the electronic devices in electronic device data 370 based on the received current status. The current status may further include an error status, such as an unidentified or unavailable status of the device.

[0086] In block 820, the user device may transmit to one or more servers the current state of the electronic device received in block 818. For example, the user device 102 may transmit the current state of the electronic device received in block 818 to the server system 110.

[0087] At block 822, one or more servers may receive the current states of the electronic devices sent by the user device at block 820. For example, the server system 110 may receive the current states of the electronic devices 128, 130, and 132 from the user device 102 over the network(s) 126.

[0088] At block 824, one or more servers may store the current state of the electronic devices received at block 822 as a configuration. In some examples, the configuration may be assigned a unique identifier, such as “sleep,” “morning,” or “work,” based on the name provided in the textual representation and identified at block 810. For example, in response to receiving a user utterance including “name this configuration sleep” at block 802, server system 110 may store the current state of electronic devices 128, 130, and 132 as a “sleep” configuration in model and data storage 120. Controlling an Electronic Device Using a Stored Configuration - Patent application

[0089] 9 illustrates an example process 900 for using a virtual assistant to set the state of multiple electronic devices using a previously stored configuration (e.g., created using process 800). In some embodiments, process 900 may be performed using a system similar to or identical to system 100 shown in FIG. 1. In these embodiments, the blocks of process 900 may be performed by both user device 102 and server system 110. Specifically, the blocks on the left side of FIG. 9 may be performed by user device 102, and the blocks on the right side may be performed by server system 110.

[0090] At block 902, audio input including user speech may be received at a user device in a manner similar or identical to block 502 of process 500. At block 904, data corresponding to the audio input received at block 902 may be transmitted to one or more servers for processing in a manner similar or identical to block 504 of process 500.

[0091] At block 906, data corresponding to the audio input sent by the user device at block 904 may be received by one or more servers in a manner similar or identical to block 506 of process 500. At block 908, speech-to-text conversion of the audio input to the user utterance may be performed in a manner similar or identical to block 508 of process 500 to convert the user utterance into a textual representation of the user utterance.

[0092] At block 910, it may be determined that the textual representation of the user utterance represents a user intent to set a state of the electronic device of the system 100 based on the stored configuration. For example, one or more processing modules 118 of the server system 110 may utilize the data and model storage 120 to identify the user intent based on the natural language input. In some embodiments, this may include parsing the textual representation for words that may be related to using the stored configuration, such as unique identifiers associated with the stored configuration, such as "set," "configuration," "scene," synonyms thereof, and the like. For example, if the textual representation includes "go to bed," it may be determined at block 910 that the user intends to set a state of the electronic device based on the "sleep" configuration. Similarly, other textual representations, such as "night mode," "set to sleep," and the like, may determine at block 910 that the user intends to set a state of the electronic device based on the "sleep" configuration.

[0093] At block 912, commands may be sent by one or more servers to the user devices to set the state of the electronic devices of the system 100 based on the configuration determined at block 910. Identifiers associated with the commands may be sent to identify the device that will execute each command. For example, the server system 110 may send unique identifiers associated with the electronic devices and commands to be executed by those electronic devices to place the electronic devices in the state specified by the stored configuration.

[0094] At block 914, the command sent by the server(s) to set the state of the electronic device of the system 100 may be received by the user device. For example, the user device 102 may receive the command sent by the server system 110 at block 912 over the network(s) 108.

[0095] At block 916, the user device may transmit the command received at block 914 to an electronic device associated with the command. For example, the user device 102 may transmit a command over the network(s) 126 to the electronic devices 128, 130, and 132 to place the electronic devices in a state specified by the stored configuration.

[0096] In some embodiments, the actual state of the electronic devices may not be the same as the state of the electronic devices as stored in the user device 102 (e.g., in memory 350) and / or the server system 110 (e.g., in data and model storage 120). Thus, in some embodiments, similar to process 500, in order to reduce the time required to issue commands to the electronic devices, block 916 may be performed without first querying the electronic devices (e.g., between blocks 914 and 916) to determine their actual states, regardless of the states of the electronic devices as stored in the user device 102 and / or the server system 110.

[0097] In block 918, the user device may receive an updated state of the electronic device after the electronic device executes the command sent by the user device in block 916. For example, the user device 102 may receive the updated state of the electronic devices after the electronic devices 128, 130, and 132 execute commands that put their state into a state specified by the stored configuration. In some embodiments, block 920 may further include updating the state of the electronic device in the electronic device data 370 based on the received current state. For example, the user device 102 may update the state of the electronic device 128 in the electronic device data 370 to “locked”. In some embodiments, similar to block 520, block 918 may further include outputting an audio or visual indication of the result of the command(s) sent to the electronic device(s) in block 916 based on the sent command(s) and the received current state(s) of the electronic device(s). The updated status may further include error status, such as an unspecified or unavailable status of the device.

[0098] At block 920, the user device may transmit to one or more servers the updated state of the electronic device received at block 918. For example, the user device 102 may transmit the updated state of the electronic device received at block 918 to the server system 110.

[0099] In block 922, one or more servers may receive the updated state of the electronic device sent by the user device in block 920. For example, the server system 110 may receive the updated state of the electronic device sent by the user device 102 in block 920. In some embodiments, block 922 may further include updating the state of the electronic device in the data and model storage 120 based on the received updated state.

[0100] In some embodiments, in addition or alternatively, in response to determining that a predetermined condition is met, a state of the electronic device may be configured using a stored configuration. For example, in response to the location of the user device 102 entering a predetermined area (e.g., an area corresponding to the user's home), a command may be sent to the electronic device to transition to a state specified by a stored configuration (e.g., a "home" configuration). Similarly, in response to the location of the user device 102 leaving a predetermined area (e.g., an area corresponding to the user's home) during a predetermined time frame (e.g., between 8:00 and 9:00 a.m. on a weekday), a command may be sent to the electronic device to transition to a state specified by another stored configuration (e.g., a "work" configuration). Other similar predetermined conditions may be created to cause the system 100 to configure the electronic device based on a stored configuration. Electronic Devices

[0101] According to some embodiments, FIG. 10 illustrates a functional block diagram of an electronic device 1000 configured according to the principles of the various embodiments described. The functional blocks of the device may be implemented by hardware, software, or a combination of hardware and software to execute the principles of the various embodiments described. Those skilled in the art will understand that the functional blocks described in FIG. 10 may be combined or separated into sub-blocks to implement the principles of the various embodiments described. Thus, the description herein optionally supports any possible combination or division or further definition of the functional blocks described herein.

[0102] 10, the electronic device 1000 may include a touchscreen display unit 1002 configured to display a user interface and receive touch input, and an audio receiving unit 1004 configured to receive audio input. In some embodiments, the electronic device 1000 may include a speaker unit 1006 configured to generate audio. The electronic device 1000 may further include a processing unit 1008 coupled to the touchscreen display unit 1002 and the audio receiving unit 1004 (and optionally coupled to the speaker unit 1006). In some embodiments, the processing unit 1008 may include an audio input receiving unit 1010, an audio input sending unit 1012, an identifier and command receiving unit 1014, an identifier and command sending unit 1016, a status receiving unit 1018, a status sending unit 1020, a status update unit 1022, an instruction output unit 1024, a second identifier and command receiving unit 1026, a second identifier and command sending unit 1028, a second status receiving unit 1030, a second status sending unit 1032, and a notification sending unit 1034.

[0103] The processing unit 1008 may be configured to receive an audio input (e.g., using the audio input receiving unit 1010) including a user utterance. The processing unit 1008 may be further configured to transmit data corresponding to the audio input to one or more servers (e.g., using the audio input transmitting unit 1012). The processing unit 1008 may be further configured to receive from the one or more servers (e.g., using the identifier and command receiving unit 1014) an identifier of the first electronic device identified by the one or more servers based on the data corresponding to the audio input, and a first command to be performed by the first electronic device identified by the one or more servers based on the data corresponding to the audio input. The first command may be transmitted to the first electronic device (e.g., using the identifier and command transmitting unit 1016). After transmitting the first command to the first electronic device, a current status of the first electronic device may be received from the first electronic device (e.g., using the status receiving unit 1018). The current status of the first electronic device may be transmitted to the one or more servers (e.g., using the status transmitting unit 1020).

[0104] In some embodiments, the first electronic device includes a light bulb. In other embodiments, the first command includes a command to set an on / off state, dimmable state, or color state of the light. In yet other embodiments, the current state of the first electronic device includes an on / off state, dimmable state, or color state of the light bulb after sending the first command to set the on / off state, dimmable state, or color state of the light bulb.

[0105] In some embodiments, the first electronic device includes a switch. In other embodiments, the first command includes a command to set an on / off state of the switch. In yet other embodiments, the current state of the first electronic device includes an on / off state of the switch after sending the first command to set the on / off state of the switch.

[0106] In some embodiments, the first electronic device includes an electrical outlet. In other embodiments, the first command includes a command to set an active / inactive state of the electrical outlet. In yet other embodiments, the current state of the first electronic device includes an active / inactive state of the electrical outlet after sending the command to set the active / inactive state of the electrical outlet.

[0107] In some embodiments, the first electronic device includes a door key. In other embodiments, the first command includes a command to set a locked / unlocked state of the door key. In yet other embodiments, the current state of the first electronic device includes a locked / unlocked state of the door key after sending the command to set the locked / unlocked state of the door key.

[0108] In some embodiments, the first electronic device includes a garage door. In other embodiments, the first command includes a command to set an open / closed state of the garage door. In yet other embodiments, the current state of the first electronic device includes an open / closed state of the garage door after sending the command to set the open / closed state of the garage door.

[0109] In some embodiments, the first electronic device includes a thermostat. In other embodiments, the first command includes a command to set a value of a temperature setting of the thermostat. In yet other embodiments, the current state of the first electronic device includes a value of a temperature setting of the thermostat after sending the first command to set a value of a temperature setting of the thermostat.

[0110] In some embodiments, the first command includes a query regarding a current state of the first electronic device.

[0111] In some embodiments, the first command may be transmitted (eg, using the identifier and command transmitting unit 1016) to the first electronic device over a local wireless network.

[0112] In some embodiments, the first command may be transmitted directly to the first electronic device by Bluetooth, line of sight, peer-to-peer, or WiFi communication.

[0113] In some examples, the processing unit 1008 may be configured to exclude performing a query to the first electronic device regarding a status of the first electronic device between receiving the first command from the one or more servers and sending the first command to the first electronic device.

[0114] In some embodiments, the electronic device 1000 may further include a database unit 1034 that stores a state of each of the multiple electronic devices, including the first electronic device. In other embodiments, the processing unit 1008 may be further configured to update the state of the first electronic device stored in the database unit 1034 (e.g., using the state update unit 1022) based at least in part on the current state of the first electronic device received from the first electronic device.

[0115] In some embodiments, the processing unit 1008 may be further configured to output (e.g., using the indication output unit 1024) an indication of a result of the first command based on the first command and the current state of the first electronic device received from the first electronic device.

[0116] In some embodiments, electronic device 1000 may include a mobile phone, a desktop computer, a laptop computer, a tablet computer, a portable media player, a television, a television set-top box, or a wearable electronic device.

[0117] In some embodiments, the processing unit 1008 may be further configured to receive from the one or more servers (e.g., using the second identifier and command receiving unit 1026) an identifier of the second electronic device identified by the one or more servers based on the data corresponding to the audio input, and a second command to be performed by the second electronic device identified by the one or more servers based on the data corresponding to the audio input. The processing unit 1008 may be further configured to transmit the second command to the second electronic device (e.g., using the second identifier and command sending unit 1028), and receive a current state of the second electronic device from the second electronic device (e.g., using the second state receiving unit 1030) after transmitting the second command to the second electronic device. The processing unit 1008 may be further configured to transmit the current state of the second electronic device to the one or more servers (e.g., using the second state sending unit 1032).

[0118] In some embodiments, the processing unit 1008 may be further configured to send (e.g., using the notification sending unit 1036) a notification associated with a current state of the first electronic device in response to determining that the predetermined condition is met.

[0119] According to some embodiments, FIG. 11 illustrates a functional block diagram of an electronic device 1100 configured according to the principles of the various embodiments described. The functional blocks of the device may be implemented by hardware, software, or a combination of hardware and software to execute the principles of the various embodiments described. Those skilled in the art will understand that the functional blocks described in FIG. 11 may be combined or separated into sub-blocks to implement the principles of the various embodiments described. Thus, the description herein optionally supports any possible combination or division or further definition of the functional blocks described herein.

[0120] 11, the electronic device 1100 may include a touchscreen display unit 1102 configured to display a user interface and receive touch input, and an audio receiving unit 1104 configured to receive audio input. In some embodiments, the electronic device 1100 may include a speaker unit 1106 configured to generate audio. The electronic device 1100 may further include a processing unit 1108 coupled to the touchscreen display unit 1102 and the audio receiving unit 1104 (and optionally coupled to the speaker unit 1106). In some embodiments, the processing unit 1108 may include an audio input receiving unit 1110, an audio input sending unit 1112, a command receiving unit 1114, a query sending unit 1116, a status receiving unit 1118, and a status sending unit 1120.

[0121] The processing unit 1108 may be configured to receive an audio input (e.g., using the audio input receiving unit 1110) including a user utterance. The processing unit 1108 may be further configured to transmit data corresponding to the audio input to one or more servers (e.g., using the audio input sending unit 1112). The processing unit 1108 may be further configured to receive instructions from the one or more servers (e.g., using the instruction receiving unit 1114) to perform a query regarding a status of each of the plurality of electronic devices identified by the one or more servers based on the data corresponding to the audio input. The status query may be transmitted to each of the plurality of electronic devices (e.g., using the query transmitting unit 1116), and a current status of each of the plurality of electronic devices may be received from the plurality of electronic devices (e.g., using the status receiving unit 1118). The current status of each of the plurality of electronic devices may be transmitted to the one or more servers (e.g., using the status transmitting unit 1120) for storage as a configuration.

[0122] According to some embodiments, FIG. 12 illustrates a functional block diagram of an electronic device 1200 configured according to the principles of the various embodiments described. The functional blocks of the device may be implemented by hardware, software, or a combination of hardware and software to execute the principles of the various embodiments described. It will be understood by those skilled in the art that the functional blocks described in FIG. 12 may be combined or separated into sub-blocks to implement the principles of the various embodiments described. Thus, the description herein optionally supports any possible combination or division or further definition of the functional blocks described herein.

[0123] 12, the electronic device 1200 may include a touchscreen display unit 1202 configured to display a user interface and receive touch input, and an audio receiving unit 1204 configured to receive audio input. In some embodiments, the electronic device 1200 may include a speaker unit 1206 configured to generate audio. The electronic device 1200 may further include a processing unit 1208 coupled to the touchscreen display unit 1202 and the audio receiving unit 1204 (and optionally coupled to the speaker unit 1206). In some embodiments, the processing unit 1208 may include an audio input receiving unit 1210, an audio input sending unit 1212, a command receiving unit 1214, a query sending unit 1216, a status receiving unit 1218, and a status sending unit 1220.

[0124] In some examples, the processing unit 1208 may be further configured to receive audio input including user utterances (e.g., using the audio input receiving unit 1210), transmit data corresponding to the audio input to one or more servers (e.g., using the audio input transmitting unit 1212), and receive a plurality of commands from the one or more servers (e.g., using the command receiving unit 1214) that set a state of each of the plurality of electronic devices identified by the one or more servers based on the stored configuration. The processing unit 1208 may be further configured to transmit the plurality of commands to the plurality of electronic devices (e.g., using the command transmitting unit 1216). The processing unit 1208 may be further configured to receive an updated state of each of the plurality of electronic devices from the plurality of electronic devices (e.g., using the state receiving unit 1218), and transmit the updated state of each of the plurality of electronic devices to the one or more servers (e.g., using the state transmitting unit 1220).

[0125] 13, the electronic device 1300 may include a touchscreen display unit 1302 configured to display a user interface and receive touch input, and an audio receiving unit 1304 configured to receive audio input. In some embodiments, the electronic device 1300 may include a speaker unit 1306 configured to generate audio. The electronic device 1300 may further include a processing unit 1308 coupled to the touchscreen display unit 1302 and the audio receiving unit 1304 (and optionally coupled to the speaker unit 1306). In some embodiments, the processing unit 1308 may include an audio input receiving unit 1310, a speech to text conversion unit 1312, an electronic device identifying unit 1314, a command identifying unit 1316, an identifier and command sending unit 1318, a status receiving unit 1320, a status update unit 1322, a context information receiving unit 1324, an instruction sending unit 1326, a second electronic device identifying unit 1328, a second command identifying unit 1330, a second identifier and command sending unit 1332, and a second status receiving unit 1336.

[0126] The processing unit 1308 may be configured to receive (e.g., using the audio input receiving unit 1310) data corresponding to an audio input including a user utterance. The processing unit 1308 may be further configured to perform speech-to-text conversion on the data corresponding to the audio input (e.g., using the speech-to-text conversion unit 1312) to generate a textual representation of the user utterance. The processing unit 1308 may be further configured to identify (e.g., using the electronic device identifying unit 1314) the first electronic device based on the textual representation of the user utterance. The processing unit 1308 may be further configured to identify (e.g., using the command identifying unit 1316) a first command to be performed by the first electronic device based on the textual representation of the user utterance. The processing unit 1308 may be further configured to transmit (e.g., using the identifier and command transmitting unit 1318) an identifier of the first electronic device and the first command to the user device. The processing unit 1308 may be further configured to receive a current state of the first electronic device (eg, using the state receiving unit 1320).

[0127] In some embodiments, the first electronic device includes a light bulb. In other embodiments, the first command includes a command to set an on / off state, dimmable state, or color state of the light. In yet other embodiments, the current state of the first electronic device includes an on / off state, dimmable state, or color state of the light bulb after sending the first command to set the on / off state, dimmable state, or color state of the light bulb.

[0128] In some embodiments, the first electronic device includes an electrical outlet. In other embodiments, the first command includes a command to set an active / inactive state of the electrical outlet. In yet other embodiments, the current state of the first electronic device includes an active / inactive state of the electrical outlet after sending the command to set the active / inactive state of the electrical outlet.

[0129] In some embodiments, the first electronic device includes a switch. In other embodiments, the first command includes a command to set an on / off state of the switch. In yet other embodiments, the current state of the first electronic device includes an on / off state of the electrical outlet after sending the command to set the on / off state of the switch.

[0130] In some embodiments, the first electronic device includes a door key. In other embodiments, the first command includes a command to set a locked / unlocked state of the door key. In yet other embodiments, the current state of the first electronic device includes a locked / unlocked state of the door key after sending the command to set the locked / unlocked state of the door key.

[0131] In some embodiments, the first electronic device includes a garage door. In other embodiments, the first command includes a command to set an open / closed state of the garage door. In yet other embodiments, the current state of the first electronic device includes an open / closed state of the garage door after sending the command to set the open / closed state of the garage door.

[0132] In some embodiments, the first electronic device includes a thermostat. In other embodiments, the first command includes a command to set a value of a temperature setting of the thermostat. In yet other embodiments, the current state of the first electronic device includes a value of a temperature setting of the thermostat after sending the first command to set a value of a temperature setting of the thermostat.

[0133] In some embodiments, the first command includes a query regarding a current state of the first electronic device.

[0134] In some embodiments, the electronic device 1300 further includes a database unit 1334 that stores the name, type, and status of each of the multiple electronic devices, including the first electronic device. In other embodiments, the processing unit 1308 may be further configured to update the status of the first electronic device stored in the database unit 1334 (e.g., using the status update unit 1322) based on the current status of the first electronic device.

[0135] In some embodiments, processing unit 1308 may be further configured to receive context information from the user device (e.g., using context information receiving unit 1324) in the data corresponding to the audio input. In other embodiments, processing unit 1308 may be further configured to identify the first electronic device (e.g., using electronic device identification unit 1314) based on the context information. In yet other embodiments, processing unit 1308 may be further configured to identify a first command (e.g., using command identification unit 1316) based on the context information. In some embodiments, the context information includes an orientation of the user device when the user device received the audio input, or a location of the user device when the user device received the audio input.

[0136] In some implementations, the textual representation of the user utterance excludes a name of the first electronic device, hi other implementations, the textual representation of the user utterance excludes a type of the first electronic device.

[0137] In some examples, electronic device 1300 includes a mobile phone, a desktop computer, a laptop computer, a tablet computer, a portable media player, a television, a television set-top box, or a wearable electronic device.

[0138] In some embodiments, the processing unit 1308 may be further configured to send an identifier of the first electronic device and a first command to the user device (e.g., using the identifier and command sending unit 1318) and receive a current status of the first electronic device from the user device (e.g., using the status receiving unit 1320).

[0139] In other embodiments, the processing unit 1308 may be further configured to send an identifier of the first electronic device and a first command to the second user device (e.g., using the identifier and command sending unit 1318) and receive a current status of the first electronic device from the second user device (e.g., using the status receiving unit 1320).

[0140] In other embodiments, the processing unit 1308 may be further configured to send (e.g., using the instruction sending unit 1326) to the second user device an indication of a result of the first command based on the first command and a current state of the first electronic device.

[0141] In other embodiments, the processing unit 1308 may be further configured to identify a second electronic device based on the textual representation of the user utterance (e.g., using the second electronic device identification unit 1328), identify a second command to be performed by the second electronic device based on the textual representation of the user utterance (e.g., using the second command identification unit 1330), transmit an identifier of the second electronic device and the second command (e.g., using the second identifier and command sending unit 1332), and receive a current status of the second electronic device (e.g., using the second status receiving unit 1336).

[0142] 14, the electronic device 1400 may include a touchscreen display unit 1402 configured to display a user interface and receive touch input, and an audio receiving unit 1404 configured to receive audio input. In some embodiments, the electronic device 1400 may include a speaker unit 1406 configured to generate audio. The electronic device 1400 may further include a processing unit 1408 coupled to the touchscreen display unit 1402 and the audio receiving unit 1404 (and optionally coupled to the speaker unit 1406). In some embodiments, the processing unit 1408 may include an audio input receiving unit 1410, a speech-to-text conversion unit 1412, a determination unit 1414, a command sending unit 1416, a status receiving unit 1416, a configuration storage unit 1420, a second audio input receiving unit 1422, a second speech-to-text conversion unit 1424, a second determination unit 1426, a command sending unit 1428, and a second status receiving unit 1430.

[0143] The processing unit 1408 may be configured to receive data corresponding to an audio input including a user utterance from a user device (e.g., using the audio input receiving unit 1410). The processing unit 1408 may be further configured to perform speech-to-text conversion on the data corresponding to the audio input (e.g., using the speech-to-text conversion unit 1412) to generate a textual representation of the user utterance. The processing unit 1408 may be further configured to determine (e.g., using the determining unit 1414) that the textual representation of the user utterance represents a user intent to store a state of each of the plurality of electronic devices as a configuration. The processing unit 1408 may be further configured to transmit (e.g., using the instruction transmitting unit 1416) an instruction to perform a query regarding a state of each of the plurality of electronic devices. The processing unit 1408 may be further configured to receive (e.g., using the state receiving unit 1418) a current state of each of the plurality of electronic devices. The processing unit 1408 may be further configured to store (e.g., using the configuration storing unit 1420) the received current state of each of the plurality of electronic devices as a configuration.

[0144] 15, the electronic device 1500 may include a touchscreen display unit 1502 configured to display a user interface and receive touch input, and an audio receiving unit 1504 configured to receive audio input. In some embodiments, the electronic device 1500 may include a speaker unit 1506 configured to generate audio. The electronic device 1500 may further include a processing unit 1508 coupled to the touchscreen display unit 1502 and the audio receiving unit 1504 (and optionally coupled to the speaker unit 1506). In some embodiments, the processing unit 1408 may include an audio input receiving unit 1510, a speech-to-text conversion unit 1512, a determination unit 1514, a command sending unit 1516, a status receiving unit 1516, a configuration storage unit 1520, a second audio input receiving unit 1522, a second speech-to-text conversion unit 1524, a second determination unit 1526, a command sending unit 1528, and a second status receiving unit 1530.

[0145] In some examples, the processing unit 1508 may be further configured to receive (e.g., using the audio input receiving unit 1510) data corresponding to an audio input including a user utterance, perform a speech-to-text conversion on the data corresponding to the audio input (e.g., using the speech-to-text conversion unit 1512) to generate a text representation of the user utterance, determine (e.g., using the determining unit 1514) that the text representation of the user utterance represents a user intention to change a state of each of the multiple electronic devices based on the configuration, send (e.g., using the command sending unit 1516) a plurality of commands that set the states of each of the multiple electronic devices based on the configuration, and receive (e.g., using the status receiving unit 1518) an updated state of each of the multiple electronic devices.

[0146] As mentioned above, one aspect of the present technology is to collect and use data available from various sources to improve the delivery to users of invitee content or any other content that may be of interest to the user. The present disclosure contemplates that in some cases, this collected data may include personal information data that uniquely identifies a specific person or that may be used to contact or locate that person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, home addresses, or any other identifiable information.

[0147] This disclosure recognizes that the use of such personal information data may be used to the benefit of the user in the present technology. For example, personal information data may be used to deliver targeted content that is more appealing to the user. Thus, use of such personal information data allows for intentional control of the content that is delivered. Additionally, other uses of personal information data that benefit the user are contemplated by this disclosure.

[0148] The present disclosure further contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other uses of such personal information data will comply with well-defined privacy policies and / or privacy practices. In particular, such entities should implement and always use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for keeping personal information data strictly confidential. For example, personal information from users should be collected for the entity's lawful and legitimate uses and should not be shared or sold except for those lawful uses. Furthermore, such collection should be carried out only after the user's informed consent is obtained. Furthermore, such entities may take all necessary measures to protect and secure access to such personal information data and ensure that others having access to that personal information data comply with their privacy policies and procedures. Furthermore, such entities may subject themselves to third-party assessments to attest to their compliance with widely accepted privacy policies and practices.

[0149] Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which a user selectively blocks use of or access to personal information data. That is, the present disclosure contemplates that hardware and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case of an ad delivery service, the present technology may be configured to allow a user to select "opt-in" or "opt-out" of participating in the collection of personal information data during registration for the service. In another embodiment, a user may choose not to provide location information to a given content delivery service. In yet another embodiment, a user may choose not to provide precise location information but allow transfer of location zone information.

[0150] Thus, while this disclosure broadly encompasses the use of personal information data to implement one or more of the various disclosed embodiments, this disclosure also contemplates that the various embodiments may also be implemented without requiring access to such personal information data. That is, the various embodiments of the technology are not rendered inoperable by the absence of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences based on non-personal information or minimal personal information, such as content requested by a device associated with the user, other non-personal information provided to a content delivery service, or publicly available information.

[0151] Although the embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art, and such changes and modifications are to be understood as being included within the scope of the various embodiments as defined by the appended claims.

Claims

1. A method for controlling an electronic device using a virtual assistant on a user device, comprising: receiving, by the user device, an audio input comprising user speech; transmitting data corresponding to the audio input to one or more servers; From the one or more servers, an identifier of a first electronic device identified by the one or more servers based on the data corresponding to the audio input; a first command to be executed by the first electronic device identified by the one or more servers based on the data corresponding to the audio input; and and sending the first command to the first electronic device.

2. receiving a current state of the first electronic device from the first electronic device after sending the first command to the first electronic device; The method of claim 1 , further comprising: transmitting the current state of the first electronic device to the one or more servers.

3. The method of claim 2 , wherein the first electronic device comprises a light bulb.

4. The method of claim 3 , wherein the first command comprises a command to set an on / off state, a dimmable state, or a color state of the light bulb.

5. 5. The method of claim 4, wherein the current state of the first electronic device includes the on / off state, dimmable state, or color state of the light bulb after sending the first command to set the on / off state, dimmable state, or color state of the light bulb.

6. The method of claim 2 , wherein the first electronic device comprises an electrical outlet.

7. The method of claim 6 , wherein the first command comprises a command to set an active / inactive state of the electrical outlet.

8. 8. The method of claim 7, wherein the current state of the first electronic device includes the active / inactive state of the electrical outlet after sending the command to set the active / inactive state of the electrical outlet.

9. The method of claim 2 , wherein the first electronic device comprises a switch.

10. The method of claim 9 , wherein the first command comprises a command to set an on / off state of the switch.

11. The method of claim 10 , wherein the current state of the first electronic device includes the on / off state of the electrical outlet after sending the command to set the on / off state of the switch.

12. The method of claim 2 , wherein the first electronic device comprises a door key.

13. The method of claim 12 , wherein the first command includes a command to set a locked / unlocked state of the door.

14. 14. The method of claim 13, wherein the current state of the first electronic device includes the locked / unlocked state of the door key after sending the command to set the locked / unlocked state of the door key.

15. The method of claim 2 , wherein the first electronic device comprises a garage door.

16. The method of claim 15 , wherein the first command includes a command to set an open / closed state of the garage door.

17. 17. The method of claim 16, wherein the current state of the first electronic device includes the open / closed state of the garage door after sending the command to set the open / closed state of the garage door.

18. The method of claim 2 , wherein the first electronic device comprises a thermostat.

19. 20. The method of claim 18, wherein the first command comprises a command to set a numerical value of a temperature setting of the thermostat.

20. 20. The method of claim 19, wherein the current state of the first electronic device includes the value of the temperature setting of the thermostat after sending the first command to set the value of the temperature setting of the thermostat.

21. The method of claim 2 , wherein the first command comprises a query regarding the current state of the first electronic device.

22. The method of claim 2 , further comprising: in response to determining that a predetermined condition is met, outputting a notification associated with the current state of the first electronic device.

23. 3. The method of claim 2, wherein the user device includes a database that stores a state of each of a plurality of electronic devices including the first electronic device, the method further including updating the state of the first electronic device stored in the database based at least in part on the current state of the first electronic device received from the first electronic device.

24. 3. The method of claim 2, further comprising outputting an indication of a result of the first command based on the first command and the current state of the first electronic device received from the first electronic device, the indication of the result being an audio, a visual, or both an audio and visual indication.

25. The method of claim 1 , wherein the user device comprises a mobile phone, a desktop computer, a laptop computer, a tablet computer, a portable media player, a television, a television set-top box, or a wearable electronic device.

26. The method of claim 1 , wherein the first command is transmitted to the first electronic device over a local wireless network.

27. The method of claim 1 , wherein the first command is transmitted directly to the first electronic device by Bluetooth, line of sight, peer-to-peer, or WiFi communication.

28. 2. The method of claim 1 , further comprising excluding querying the first electronic device regarding the status of the first electronic device between receiving the first command from the one or more servers and sending the first command to the first electronic device.

29. From the one or more servers, an identifier of a second electronic device identified by the one or more servers based on the data corresponding to the audio input; and a second command to be executed by the second electronic device identified by the one or more servers based on the data corresponding to the audio input; and and sending the second command to the second electronic device; receiving a current state of the second electronic device from the second electronic device after sending the second command to the second electronic device; The method of claim 1 , further comprising: transmitting the current state of the second electronic device to the one or more servers.

30. A method for controlling an electronic device using a virtual assistant on a user device, comprising: receiving, by the user device, an audio input comprising user speech; transmitting data corresponding to the audio input to one or more servers; receiving instructions from the one or more servers to perform a query regarding a status of each of a plurality of electronic devices identified by the one or more servers based on the data corresponding to the audio input; sending a status query to each of the plurality of electronic devices.

31. receiving a current state of each of the plurality of electronic devices from the plurality of electronic devices; 31. The method of claim 30, further comprising transmitting the current state of each of the plurality of electronic devices to the one or more servers for storage as a configuration.

32. A method for controlling an electronic device using a virtual assistant on a user device, comprising: receiving, by the user device, an audio input comprising user speech; transmitting data corresponding to the audio input to one or more servers; receiving a plurality of commands from the one or more servers to set a state of each of a plurality of electronic devices identified by the one or more servers based on a stored configuration; sending the commands to the electronic devices.

33. receiving an updated status of each of the plurality of electronic devices from the plurality of electronic devices; 33. The method of claim 32, further comprising: transmitting the updated state of each of the plurality of electronic devices to the one or more servers.

34. A method for controlling an electronic device using a virtual assistant on a user device, comprising: receiving, by one or more servers, data from a user device corresponding to audio input including user utterances; performing a speech-to-text conversion on the data corresponding to an audio input to generate a text representation of the user utterance; identifying a first electronic device based on the textual representation of the user utterance; identifying a first command to be executed by the first electronic device based on the textual representation of the user utterance; transmitting an identifier of the first electronic device and the first command.

35. 35. The method of claim 34, further comprising receiving a current state of the first electronic device.

36. 36. The method of claim 35, wherein the first electronic device comprises a light bulb.

37. 37. The method of claim 36, wherein the first command comprises a command to change an on / off state, a dimmable state, or a color state of the light bulb.

38. 38. The method of claim 37, wherein the current state of the first electronic device comprises the on / off state, dimmable state, or color state of the light bulb after the light bulb receives the first command to set the on / off state, dimmable state, or color state of the light bulb.

39. 36. The method of claim 35, wherein the first electronic device comprises an electrical outlet.

40. 40. The method of claim 39, wherein the first command comprises a command to set an active / inactive state of the electrical outlet.

41. 41. The method of claim 40, wherein the current state of the first electronic device comprises the active / inactive state of the electrical outlet after the electrical outlet receives the first command to set the active / inactive state of the electrical outlet.

42. 36. The method of claim 35, wherein the first electronic device comprises a door key.

43. 43. The method of claim 42, wherein the first command includes a command to set a lock / unlock status of the door.

44. 44. The method of claim 43, wherein the current state of the first electronic device includes the locked / unlocked state of the door key after the door key receives the first command to set the locked / unlocked state of the door key.

45. 36. The method of claim 35, wherein the first electronic device comprises a garage door.

46. 46. ​​The method of claim 45, wherein the first command includes a command to set an open / closed state of the garage door.

47. 47. The method of claim 46, wherein the current state of the first electronic device includes the open / closed state of the garage door after the garage door receives the first command to set the open / closed state of the garage door.

48. 36. The method of claim 35, wherein the first electronic device comprises a thermostat.

49. 49. The method of claim 48, wherein the first command comprises a command to set a numerical value of a temperature setting of the thermostat.

50. 50. The method of claim 49, wherein the current state of the first electronic device includes the value of the temperature setting of the thermostat after the thermostat receives the first command to set the value of the temperature setting of the thermostat.

51. The method of claim 35 , wherein the first electronic device comprises a switch.

52. 52. The method of claim 51, wherein the first command comprises a command to set an on / off state of the switch.

53. 53. The method of claim 52, wherein the current state of the first electronic device includes the on / off state of the electrical outlet after sending the command to set the on / off state of the switch.

54. 36. The method of claim 35, wherein the first command comprises a query regarding the current state of the first electronic device.

55. 36. The method of claim 35, wherein the one or more servers include a database that stores the name, type, and status of each of a plurality of electronic devices, including the first electronic device.

56. 56. The method of claim 55, further comprising updating the state of the first electronic device stored in the database based on the current state of the first electronic device.

57. 56. The method of claim 55, wherein the textual representation of the user utterance excludes the name of the first electronic device.

58. 56. The method of claim 55, wherein the textual representation of the user utterance excludes the type of the first electronic device.

59. 35. The method of claim 34, wherein the data corresponding to the audio input includes contextual information associated with the audio input from the user device.

60. 60. The method of claim 59, wherein identifying the first electronic device is further based on the context information.

61. 60. The method of claim 59, wherein identifying the first command is further based on the context information.

62. 60. The method of claim 59, wherein the contextual information includes an orientation of the user device when the user device received the audio input or a location of the user device when the user device received the audio input.

63. 35. The method of claim 34, wherein the user device comprises a mobile phone, a desktop computer, a laptop computer, a tablet computer, a portable media player, a television, a television set-top box, or a wearable electronic device.

64. 35. The method of claim 34, wherein the identifier of the first electronic device and the first command are transmitted to the user device and the current state of the first electronic device is received from the user device.

65. 35. The method of claim 34, wherein the identifier of the first electronic device and the first command are transmitted to a second user device, and the current state of the first electronic device is received from the second user device.

66. 66. The method of claim 65, further comprising sending to the second user device an indication of a result of the first command based on the first command and the current state of the first electronic device.

67. identifying a second electronic device based on the textual representation of the user utterance; determining a second command to be executed by the second electronic device based on the textual representation of the user utterance; and transmitting an identifier of the second electronic device and the second command; 35. The method of claim 34, further comprising: receiving a current state of the second electronic device.

68. A method for controlling an electronic device using a virtual assistant on a user device, comprising: receiving, by one or more servers, data from a user device corresponding to audio input including user utterances; performing a speech-to-text conversion on the data corresponding to the audio input to generate a text representation of the user utterance; determining that the textual representation of the user utterance expresses a user intent to store states of each of a plurality of electronic devices as configurations; and transmitting instructions to the user device to perform a query regarding the status of each of the plurality of electronic devices.

69. receiving a current state of each of the plurality of electronic devices; 70. The method of claim 68, further comprising: storing the received current state of each of the plurality of electronic devices as the configuration.

70. A method for controlling an electronic device using a virtual assistant on a user device, comprising: receiving, by one or more servers, data corresponding to audio input including user utterances; performing a speech-to-text conversion on the data corresponding to the audio input to generate a text representation of the user utterance; determining that the textual representation of the user utterance represents a user intent to modify the state of each of a plurality of electronic devices based on a configuration; and sending a plurality of commands to set the state of each of the plurality of electronic devices based on the configuration.

71. 71. The method of claim 70, further comprising receiving an updated state of each of the plurality of electronic devices.

72. A non-transitory computer readable storage medium comprising instructions for performing the method of any one of claims 1 to 71.

73. 73. A non-transitory computer readable storage medium according to claim 72; one or more processors capable of executing the instructions of the non-transitory computer-readable storage medium; A system comprising:

74. means for receiving audio input comprising user utterances; means for transmitting data corresponding to said audio input to one or more servers; From the one or more servers, an identifier for an electronic device identified by the one or more servers based on the data corresponding to the audio input; commands to be executed by the electronic device identified by the one or more servers based on the data corresponding to the audio input; and and a means for receiving the means for transmitting said command to said electronic device; means for receiving a current state of the electronic device from the electronic device after sending the command to the electronic device; and means for transmitting the current state of the electronic device to the one or more servers.

75. means for receiving an audio input comprising a first user utterance; means for transmitting data corresponding to said audio input to one or more servers; means for receiving instructions from the one or more servers to perform a query regarding a status of each of a plurality of electronic devices identified by the one or more servers based on the data corresponding to the audio input; means for transmitting a status query to each of the plurality of electronic devices; means for receiving a current state of each of the plurality of electronic devices from the plurality of electronic devices; means for transmitting the current state of each of the plurality of electronic devices to the one or more servers for storage as a configuration.

76. means for receiving audio input comprising user utterances; means for transmitting data corresponding to said audio input to one or more servers; means for receiving from the one or more servers a plurality of commands for setting a state of each of a plurality of electronic devices identified by the one or more servers based on a stored configuration; and means for transmitting the plurality of commands to the plurality of electronic devices.

77. means for receiving data from a user device corresponding to audio input including user utterances; means for performing a speech-to-text conversion on the data corresponding to the audio input to generate a text representation of the user utterance; means for identifying a first electronic device based on the textual representation of the user utterance; means for identifying a first command to be executed by the first electronic device based on the textual representation of the user utterance; means for transmitting an identifier of the first electronic device and the first command; and means for receiving a current state of the first electronic device.

78. means for receiving data from a user device corresponding to audio input including user utterances; means for performing a speech-to-text conversion on the data corresponding to the audio input to generate a text representation of the user utterance; means for determining that the textual representation of the user utterance represents a user intent to store states of each of a plurality of electronic devices as a configuration; means for transmitting instructions to the user device to perform a query regarding the status of each of the plurality of electronic devices; means for receiving a current state of each of the plurality of electronic devices; and means for storing the received current state of each of the plurality of electronic devices as the configuration.

79. means for receiving data corresponding to audio input comprising user utterances; means for performing a speech-to-text conversion on the data corresponding to the audio input to generate a text representation of the user utterance; means for determining that the textual representation of the user utterance represents a user intent to modify the state of each of a plurality of electronic devices based on a configuration; and means for sending a plurality of commands to set the state of each of the plurality of electronic devices based on the configuration.

Images