Intelligent assistant for home automation

Abstract

The present application relates to systems and processes for controlling electronic devices using a virtual assistant.SOLUTION: In one example process, a user can speak input in a natural language form to a user device to control one or more electronic devices. The user device can transmit the user speech to a server for conversion into a textual representation. The server may identify the one or more electronic devices and the appropriate commands to be executed by the one or more electronic devices based on the textual representation. The identified one or more devices and the command to be executed may be transmitted back to the user device, which may forward the command to the appropriate one or more electronic devices for execution. In response to receiving the command, the one or more electronic devices may execute the command and transmit their current state to the user device.SELECTED DRAWING: Figure 5

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority from U.S. Provisional Patent Application No. 62 / 005,893, entitled "INTELLIGENT ASSISTANT FOR HOME AUTOMATION," filed May 30, 2014, and U.S. Nonprovisional Patent Application No. 14 / 503,105, entitled "INTELLIGENT ASSISTANT FOR HOME AUTOMATION," filed September 30, 2014, which applications are incorporated by reference herein in their entirety for all purposes. [Technical Field]

[0002] This application relates generally to natural language processing, and more particularly to the use of virtual assistants with natural language processing to control electronic devices. [Background technology]

[0003] Consumer electronic devices that can be remotely controlled using software applications running on computing devices such as mobile phones, tablet computers, laptop computers, desktop computers, etc. are becoming increasingly popular. For example, numerous manufacturers make light bulbs that can be controlled by software applications running on mobile phones that adjust the brightness and / or color of the light bulb. Other devices, such as door locks, thermostats, etc., that have similar controls are also available.

[0004] While these devices may provide users with greater control and convenience, managing these devices can become very difficult as the number of remotely controlled devices in a home and the number of types of remotely controlled devices increase. For example, a typical home may include 40 to 50 light bulbs located throughout various rooms in the home. Using traditional software applications, each light bulb is assigned a unique identifier, and a user attempting to control one of these devices must select the appropriate identifier from a list of available devices in a graphical user interface. Memorizing the correct identifier for a particular light bulb and finding that identifier from a list of 40 to 50 identifiers can be a difficult and time-consuming task. Adding to the difficulty of managing and controlling a large number of remotely controlled devices, each manufacturer typically provides a different software application that must be used to control each of its devices. As a result, users must find and open one software application to turn the light bulbs on and off, and then find and open another software application to set the temperature on their thermostat. Summary of the Invention

[0005] A system and process for controlling electronic devices using a virtual assistant is provided. In one exemplary process, a user may utter input in natural language format to a user device to control one or more electronic devices. The user device may transmit the user utterance to a server for conversion into a text representation. Based on the text representation, the server may identify one or more electronic devices and appropriate commands to be executed by the one or more electronic devices. The identified one or more devices and the commands to be executed may be sent back to the user device, which may then forward the command to the appropriate one or more electronic devices for execution. In response to receiving the command, one or more electronic devices may execute the command and transmit their current state to the user device. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 illustrates an exemplary environment in which a virtual assistant may be used to control electronic devices, according to various embodiments.

[0007] [Figure 2] FIG. 1 illustrates an exemplary environment in which a virtual assistant may be used to remotely control electronic devices, according to various embodiments.

[0008] [Figure 3] FIG. 1 illustrates an exemplary user device in accordance with various embodiments.

[0009] [Figure 4] FIG. 1 illustrates a visual representation of multiple entries used to store information associated with an electronic device, according to various embodiments.

[0010] [Figure 5] FIG. 1 illustrates an example process for controlling an electronic device using a virtual assistant implemented using a client-server model, according to various embodiments.

[0011] [Figure 6] FIG. 1 illustrates an exemplary process for remotely controlling an electronic device using a virtual assistant implemented using a client-server model, according to various embodiments.

[0012] [Figure 7] FIG. 1 illustrates an exemplary process for controlling an electronic device using a virtual assistant on a standalone user device, according to various embodiments.

[0013] [Figure 8] FIG. 1 illustrates an exemplary process for storing a state of an electronic device as a configuration, according to various embodiments.

[0014] [Figure 9] FIG. 1 illustrates an exemplary process for setting the state of an electronic device using a previously stored configuration, according to various embodiments.

[0015] [Figure 10] FIG. 1 is a functional block diagram of an electronic device configured to control an electronic device, according to various embodiments.

[0016] [Figure 11] FIG. 1 is a functional block diagram of an electronic device configured to store a state of the electronic device as a configuration, according to various embodiments.

[0017] [Figure 12] FIG. 1 is a functional block diagram of an electronic device configured to set a state of the electronic device based on a stored configuration, according to various embodiments.

[0018] [Figure 13] FIG. 1 is a functional block diagram of an electronic device configured to control an electronic device, according to various embodiments.

[0019] [Figure 14] FIG. 1 is a functional block diagram of an electronic device configured to store a state of the electronic device as a configuration, according to various embodiments.

[0020] [Figure 15] FIG. 1 is a functional block diagram of an electronic device configured to set a state of the electronic device based on a stored configuration, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0021] In the following description of the embodiments, reference is made to the accompanying drawings, which show, by way of illustration, specific embodiments that may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the various embodiments.

[0022] Intelligent automated assistants (or virtual assistants) provide an intuitive interface between users and electronic devices. These assistants may allow users to interact with devices or systems using natural language in spoken and / or textual form. For example, a user may access the services of an electronic device by providing spoken user input in natural language form to a virtual assistant associated with the electronic device. The virtual assistant may perform natural language processing on the spoken user input to infer the user's intent and actuate the user's intent into a task. The task may then be performed by executing one or more functions of the electronic device, and associated output may be returned to the user in natural language form.

[0023] The present application relates to a system and process for controlling electronic devices using a virtual assistant. In one exemplary process, a user may utter input in natural language format to a user device to control one or more electronic devices. The user device may transmit the user utterance to a server for conversion into a text representation. Based on the text representation, the server may identify one or more electronic devices and appropriate commands to be executed by the one or more electronic devices. The identified one or more devices and the commands to be executed may be sent back to the user device, which may then forward the command to the appropriate one or more electronic devices for execution. In response to receiving the command, the one or more electronic devices may execute the command and transmit their current state to the user device. System Overview

[0024] 1 illustrates an exemplary system 100 implementing a virtual assistant that controls electronic devices, according to various embodiments. The terms “virtual assistant,” “digital assistant,” “intelligent automated assistant,” or “automated digital assistant” may refer to any information processing system that interprets natural language input in the form of speech and / or text to infer user intent and perform an action based on the inferred user intent. For example, to perform an action based on the inferred user intent, the system may perform one or more of: identifying a task flow including steps and parameters designed to achieve the inferred user intent; inputting specific requirements from the inferred user intent into the task flow; executing the task flow by invoking programs, methods, services, APIs, etc.; and generating an output response to the user in an audible (e.g., spoken) and / or visual form.

[0025] A virtual assistant may be able to accept user requests, at least in part, in the form of natural language commands, requests, statements, narratives, and / or queries. Typically, a user request seeks either an informational answer or the performance of a task by the virtual assistant. A satisfactory response to a user request may include providing the requested informational answer, performing the requested task, or a combination of the two. For example, a user may ask a virtual assistant a question such as, "Where am I now?" Based on the user's current location, the virtual assistant may reply, "You're in Central Park." A user may also request the performance of a task, for example, "Please remind me to call my mother at 4 p.m. today." In response, the virtual assistant may recognize the request and then create an appropriate reminder item in the user's electronic schedule. During the performance of a requested task, the virtual assistant may interact with the user in a continuous dialog involving multiple exchanges of information over time. There are many other ways to interact with a virtual assistant to request information or the performance of various tasks. In addition to providing verbal responses and taking programmed actions, the virtual assistant may provide other visual or audio forms of responses (e.g., as text, alerts, music, video, animation, etc.).

[0026] One example of a virtual assistant is shown in Applicant's U.S. patent application Ser. No. 12 / 987,982, filed Jan. 10, 2011, for "Intelligent Automated Assistant," the entire disclosure of which is incorporated herein by reference.

[0027] As shown in FIG. 1, in some embodiments, a virtual assistant may be implemented according to a client-server model. The virtual assistant may include a client-side portion running on a user device 102 and a server-side portion running on a server system 110. The user device 102 may include any electronic device, such as a mobile phone, tablet computer, portable media player, desktop computer, laptop computer, PDA, television, television set-top box, wearable electronic device, etc., and may communicate with the server system 110 via one or more networks 108, which may include the Internet, an intranet, or any other wired or wireless public or private network. The client-side portion running on the user device 102 may provide client-side functionality, such as user-responsive input and output processing and communication with the server system 110. The server system 110 may provide server-side functionality for any number of clients residing on each user device 102.

[0028] The server system 110 may include one or more virtual assistant servers 114, which may include a client-facing I / O interface 122, one or more processing modules 118, a data and model storage device 120, and an I / O interface 116 to external services. The client-facing I / O interface 122 may facilitate client-facing input and output processing for the virtual assistant server 114. The one or more processing modules 118 may utilize the data and model storage device 120 to determine a user's intent based on natural language input and perform a task based on the inferred user intent. Additionally, the data and model storage device 120 may store a unique identifier, status, type, location, and any other related information associated with one or more of the user device 102 and / or electronic devices (e.g., electronic devices 128, 130, and 132) that may be controlled by the server system 110. In some embodiments, the virtual assistant server 114 may communicate with external services 124, such as telephone services, calendar services, information services, messaging services, navigation services, etc., via the network(s) 108 to complete tasks or obtain information. An I / O interface 116 to the external service may facilitate such communication.

[0029] Server system 110 may be implemented on one or more standalone data processing devices or a distributed network of computers. In some embodiments, server system 110 may also utilize various virtual devices and / or the services of third-party service providers (e.g., third-party cloud service providers) to provide the underlying computing and / or infrastructure resources of server system 110.

[0030] User device 102 may be further coupled to electronic devices 128, 130, and 132 via one or more networks 126. Electronic devices 128, 130, and 132 may include any type of remotely controlled electronic device, such as a light bulb (e.g., having a binary on / off state, a numerically dimmable state, a color state, etc.), a garage door (e.g., having a binary open / closed state), a door lock (e.g., having a binary locked / unlocked state), a thermostat (e.g., including one or more numerical temperature states, such as high, low, time-based temperature, etc.), an electrical outlet (e.g., having a binary on / off state), a switch (e.g., having a binary on / off state), etc. Network(s) 126 may include a WiFi network or any other wired or wireless public or private local network. Additionally or alternatively, user device 102 may be coupled to communicate directly with electronic devices 128, 130, or 132, for example, using Bluetooth, BTLE, line-of-sight, peer-to-peer, or another radio-based or other wireless communication. Thus, in the illustrated example, user device 102 may be located near electronic devices 128, 130, and 132 so as to communicate directly with them or over the same local network. For example, user device 102 and electronic devices 128, 130, and 132 may be located within the same home or building, and network(s) 126 may include the home or building's WiFi network. As described in more detail below in connection with FIGS. 5, 8, and 9, user device 102 may issue commands to control any of electronic devices 128, 130, and 132 in response to natural language speech input provided by a user to user device 102.

[0031] Although only three electronic devices 128, 130, and 132 are shown, it should be understood that system 100 may include any number of electronic devices. Furthermore, although the functionality of the virtual assistant is shown in FIG. 1 as including both a client-side portion and a server-side portion, in some embodiments, the functionality of the assistant may be implemented as a standalone application installed on a user device. Furthermore, the distribution of functionality between the client and server portions of the virtual assistant may vary depending on the embodiment. For example, in some embodiments, the client running on user device 102 may be a thin client that provides only user-responsive input and output processing functions and delegates all other functions of the virtual assistant to a backend server.

[0032] 2 shows another exemplary system 200 for implementing a virtual assistant that remotely controls electronic devices according to various embodiments. Similar to system 100, system 200 may include a user device 102, a server system 110, and an external service 124 communicatively coupled by network(s) 108. However, in contrast to system 100, user device 102 may not be coupled to electronic devices 128, 130, and 132. Instead, system 200 may include a second user device 134 coupled to communicate with user device 102 and / or server system 110 via network(s) 108 and to communicate with electronic devices 128, 130, and 132 via network(s) 126. This configuration may represent a situation in which the user and user device 102 are located remotely from electronic devices 128, 130, and 132 (e.g., the user and user device 102 are located in the user's office, and electronic devices 128, 130, and 132 are located in the user's home).

[0033] Second user device 134 may include any type of electronic device, such as a mobile phone, tablet computer, portable media player, desktop computer, laptop computer, PDA, television, television set-top box, wearable electronic device, etc., and may be configured to receive commands from user device 102 and / or server system 110 and issue commands to electronic devices 128, 130, and 132. As described in more detail below in connection with FIG. 6 , second user device 134 may issue commands to control any of electronic devices 128, 130, and 132 in response to natural language speech input provided by a user to user device 102. User Device

[0034] 3 is a block diagram of a user device 102 (or a second user device 134) according to various embodiments. As shown, the user device 102 may include a memory interface 302, one or more processors 304, and a peripherals interface 306. The various components within the user device 102 may be coupled to one another by one or more communication buses or signal lines. The user device 102 may further include various sensors, subsystems, and peripheral devices coupled to the peripherals interface 306. The sensors, subsystems, and peripherals collect information and / or facilitate various functionality of the user device 102.

[0035] For example, the user device 102 may include a motion sensor 310, a light sensor 312, and a proximity sensor 314 coupled to the peripherals interface 306 to facilitate orientation, light, and proximity sensing functions. One or more other sensors 316, such as a positioning system (e.g., a GPS receiver), a temperature sensor, a biometric sensor, a gyroscope, a compass, and an accelerometer, may also be connected to the peripherals interface 306 to facilitate related functions.

[0036] In some embodiments, camera subsystem 320 and optical sensor 322 may be utilized to facilitate camera functions such as taking pictures and recording video clips. Communication functions may be facilitated through one or more wired and / or wireless communication subsystems 324, which may include various communication ports, radio frequency receivers and transmitters, and / or optical (e.g., infrared) receivers and transmitters. Audio subsystem 326 may be coupled to speaker 328 and microphone 330 to facilitate voice-enabled functions such as voice recognition, voice duplication, digital recording, and telephone functions.

[0037] In some embodiments, user device 102 may further include an I / O subsystem 340 coupled to peripherals interface 306. I / O subsystem 340 may include a touchscreen controller 342 and / or other input controller(s) 344. Touchscreen controller 342 may be coupled to a touchscreen 346. Touchscreen 346 and touchscreen controller 342 may detect contact and movement or disruption using any of a number of touch-sensing technologies, such as, for example, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as proximity sensor arrays. Other input controller(s) 344 may be coupled to other input / control devices 348, such as one or more buttons, rocker switches, thumbwheels, infrared ports, USB ports, and / or pointer devices such as styluses.

[0038] In some embodiments, user device 102 may further include a memory interface 302 coupled to memory 350. Memory 350 may include any electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, a portable computer diskette (magnetic), a random access memory (RAM) (magnetic), a read-only memory (ROM) (magnetic), an erasable programmable read-only memory (EPROM) (magnetic), a portable optical disk such as a CD, CD-R, CD-RW, DVD, DVD-R, or DVD-RW, or flash memory such as a compact flash card, a secure digital card, a USB memory device, a memory stick, or the like. In some embodiments, the non-transitory computer-readable storage medium of memory 350 may be used to store instructions for use by or in conjunction with an instruction execution system, apparatus, or device (e.g., to perform some or all of processes 500, 600, 700, 800, or 900 described below), such as a computer-based system, a system including a processor, or other system that may fetch instructions from and execute those instructions from an instruction execution system, apparatus, or device. In other embodiments, the instructions (e.g., to perform processes 500, 600, 700, 800, or 900 described below) may be stored on a non-transitory computer-readable storage medium of server system 110, or distributed between the non-transitory computer-readable storage medium of memory 350 and the non-transitory computer-readable storage medium of server system 110. In the context of this document, a "non-transitory computer-readable storage medium" may be any medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0039] In some embodiments, memory 350 may store an operating system 352, a communications module 354, a graphical user interface module 356, a sensor processing module 358, a telephony module 360, and applications 362. Operating system 352 may include instructions for handling basic system services and for performing hardware-dependent tasks. Communications module 354 may facilitate communication with one or more additional devices, one or more computers, and / or one or more servers. Graphical user interface module 356 may facilitate graphic user interface processing. Sensor processing module 358 may facilitate sensor-related processes and functions. Telephony module 360 ​​may facilitate telephony-related processes and functions. Application module 362 may facilitate various functions of user applications, such as electronic messaging, web browsing, media processing, navigation, imaging, and / or other processes and functions.

[0040] Memory 350 also stores client-side virtual assistant instructions (eg, in virtual assistant client module 364) and various user data 366 (eg, user-specific vocabulary data, preference data, and / or other data such as the user's electronic address book, to-do list, shopping list, etc.) to provide client-side functionality for the virtual assistant.

[0041] In various embodiments, the virtual assistant client module 364 may be able to accept voice input (e.g., speech input), text input, touch input, and / or gesture input via various user interfaces (e.g., I / O subsystem 340, audio subsystem 326, etc.) of the user device 102. The virtual assistant client module 364 may also be able to provide output in audio (e.g., speech output), visual, and / or tactile form. For example, the output may be provided as voice, sound, an alert, a text message, a menu, a graphic, a video, an animation, a vibration, and / or a combination of two or more of the above. During operation, the virtual assistant client module 364 may communicate with the virtual assistant server using the communication subsystem 324.

[0042] In some embodiments, the virtual assistant client module 364 may collect additional information from the surrounding environment of the user device 102 using various sensors, subsystems, and peripherals to establish a context associated with the user, the current user interaction, and / or the current user input. In some embodiments, the virtual assistant client module 364 may provide the context information or a subset thereof to the virtual assistant server along with the user input to help infer the user's intent. The virtual assistant may also use the context information to determine how to prepare and deliver output to the user.

[0043] In some embodiments, the context information associated with the user input may include sensor information such as lighting, ambient noise, ambient temperature, images or videos of the surrounding environment, distance to another object, etc. The context information may further include information associated with the physical state of the user device 102 (e.g., device orientation, device location, device temperature, power level, speed, acceleration, movement patterns, cellular signal strength, etc.) or the software state of the user device 102 (e.g., running processes, installed programs, past and present network activity, background services, error logs, resource usage, etc.). Any of these types of context information may be provided to the virtual assistant server 114 as context information associated with the user input.

[0044] In some embodiments, the virtual assistant client module 364 may selectively provide information stored on the user device 102 (e.g., user data 366) in response to a request from the virtual assistant server 114. The virtual assistant client module 364 may also elicit additional input from the user via a natural language dialog or other user interface in response to a request by the virtual assistant server 114. The virtual assistant client module 364 may pass additional input to the virtual assistant server 114 to assist the virtual assistant server 114 in inferring intent and / or achieving the user's intent expressed in the user request.

[0045] The memory 350 may further store electronic device data 370, which may include a unique identifier, status, type, location, and any other relevant information associated with one or more of the electronic devices (e.g., electronic devices 128, 130, and 132) that may be controlled by the user device 102 and / or server system 110. FIG. 4 shows a visual representation of entries that may be stored in the electronic device data 370 for seven different electronic devices. As shown, each entry includes the unique name, type, and status of the electronic device. The data and model storage device 120 of the virtual assistant server 114 may include similar or identical entries for electronic devices that may be maintained separately from the electronic device data 370 of the memory 350.

[0046] Returning to FIG. 3 , memory 350 may further include instructions (e.g., in daemon module 368) for creating and updating entries for electronic devices in electronic device data 370, communicating with electronic devices in system 100, and communicating with server system 110. For example, to add an electronic device to system 100, a software application associated with the electronic device may communicate with processor(s) 304 executing daemon module 368 to provide the electronic device's unique name, type, status, location, etc. to user device 102. The software application may allow the user to enter the unique name in any desired manner. For example, a drop-down box with a common name and / or a free-form text field may be provided within the application to allow the user to name a particular device. The electronic device's type, status, and / or location may be pre-specified or may be specified by the software application through communication with the electronic device. The processor(s) 304 executing the daemon module 368 may store this information as an entry in electronic device data 370 or may send this information to server system 110 for storage in data and model storage 120. Additionally, the daemon module 368, when executed by the processor(s) 304, may receive commands from server system 110 over network(s) to be provided to electronic devices 128, 130, and 132 and may send those commands to the appropriate electronic devices. The processor(s) 304 executing the daemon module 368 may also receive status updates from electronic devices 128, 130, and 132, update corresponding entries in electronic device data 370 to reflect the updated status of the devices, and send the status updates to server system 110 so that server system 110 can update corresponding entries in data and model storage 120 to reflect the updated status of the devices.

[0047] Additionally, the daemon module 368 may include instructions that manage access to the electronic device data 370 by other devices and software applications. For example, the daemon module 368, when executed by the processor(s) 304, may allow the server system 110 to access all of the electronic device data 370, while restricting access by other devices or software applications to only a portion of the electronic device data 370. This may be useful when using the user device 102 to control electronic devices made by different manufacturers. In these situations, devices or software applications from each manufacturer may communicate with the daemon module 368 using an API, and the daemon module 368 may restrict their access to only the portion of the electronic device data 370 that corresponds to each manufacturer's respective electronic device. For example, company X may manufacture light bulbs that can be controlled by the user device 102, and company Y may manufacture thermostats that can be controlled by the user device 102. The daemon module 368 may facilitate communication between the user device 102 and each of the light bulb and the thermostat so that the user device 102 can issue commands to the electronic devices and receive status information associated with the electronic devices for updating electronic device data 370. However, the daemon module 368 may limit the access that the light bulb (and associated software applications running on the user device 102) has to information in the electronic device data 370 to only information associated with the light bulb (and possibly any other electronic devices manufactured by Company X). Similarly, the daemon module 368 may limit the access that the thermostat (and associated software applications running on the user device 102) has to information in the electronic device data 370 to only information associated with the thermostat (and possibly any other electronic devices manufactured by Company Y).However, the daemon module 368 may allow the server system 110 access to all information in the electronic device data 370 .

[0048] In various embodiments, memory 350 may include additional or fewer instructions. Furthermore, various functions of user device 102 may be implemented in hardware and / or firmware, including in the form of one or more signal processing circuits and / or application specific integrated circuits. Local control of electronic devices

[0049] 5 shows an example process 500 for controlling an electronic device using a virtual assistant. In some embodiments, process 500 may be performed using a system similar to or identical to system 100 shown in FIG. 1. In these embodiments, the blocks of process 500 may be performed by both user device 102 and server system 110. Specifically, the blocks on the left side of FIG. 5 may be performed by user device 102, and the blocks on the right side of FIG. 5 may be performed by server system 110.

[0050] In block 502, audio input including user speech may be received at a user device. In some embodiments, a user device (e.g., user device 102) may receive the audio input including the user speech via a microphone (e.g., microphone 330). The microphone may convert the audio input to an analog or digital representation and provide the audio data to one or more processors (e.g., processor(s) 304).

[0051] In block 504, data corresponding to the audio input received in block 502 may be sent to one or more servers for processing. For example, the user device 102 may send data corresponding to the audio input via the network(s) 108 to the virtual assistant server 114 of the server system 110.

[0052] In block 506, data corresponding to the audio input transmitted by the user device in block 504 may be received by one or more servers. For example, the virtual assistant server 114 of the server system 110 may receive data corresponding to the audio input transmitted by the user device 102 via the network(s) 108.

[0053] At block 508, speech-to-text conversion may be performed on the data corresponding to the audio input to convert the user utterance into a text representation of the user utterance. The user utterance may be converted using any known speech-to-text conversion process.

[0054] At block 510, one or more electronic devices may be identified based at least in part on the textual representation generated at block 508. In some embodiments, block 510 may include processing the textual representation of the user input to identify the user's intent to issue a command to one or more electronic devices. As described above, server system 110 may include one or more data and model stores 120 that may store unique identifiers, status, type, location, and any other relevant information associated with electronic devices that may be controlled using system 100. Accordingly, block 510 may include identifying one or more of the electronic devices that have associated information stored in data model and store 120.

[0055] The one or more electronic devices may be identified in any number of ways. In some embodiments, the one or more electronic devices may be identified by parsing the textual representation to identify any of a set of nouns corresponding to electronic devices supported by system 100. For example, the set of nouns may include a unique name of the electronic device, possible types of electronic devices and their synonyms (e.g., garage door, thermostat, light, dimmable light, switch, color-changeable light, light bulb, lamp, lock, outlet, socket, etc.), possible device state categories (e.g., volume, temperature, brightness, color, etc.), etc. stored in electronic device data 370 and data and model storage 120.

[0056] Using the example electronic device entries of FIG. 4 as an illustration, the set of nouns may include unique names of seven electronic devices (e.g., “garage door,” “second floor thermostat,” “first floor thermostat,” “living room lamp 1,” “living room lamp 2,” “front door,” and “toaster outlet”), possible types of electronic devices and their synonyms (e.g., “garage door,” “thermostat,” “light,” “bulb,” “lamp,” “lock,” “outlet,” “socket,” etc.), and possible device state categories (e.g., temperature). Thus, if the text representation of the user utterance generated in block 508 includes “lock the front door,” the text representation may be processed using processing module 118 and data and model storage 120 to search for any of the set of nouns. As a result of the search, the text representation may be determined to include the unique name “front door,” and the electronic devices identified in block 510 may include this device. It should be understood that more than one electronic device may be identified in block 510 depending on the text representation of the user utterance. For example, if the text representation of the user utterance generated in block 508 instead includes "turn off all light bulbs," the text representation may be processed using processing module 118 and data and model storage 120 to search for any of a set of nouns. As a result of the search, it may be determined that the text representation includes the synonym "light bulb" for the possible device type "lighting," and that the command from the user was to turn off all devices of that type. As a result, the electronic devices identified in block 510 may include both "Living Room Lamp 1" and "Living Room Lamp 2."

[0057] In some examples, it may be difficult to identify the appropriate electronic device using only the set of nouns described above. For example, a text expression including "turn on the lights" may generate type matches with both "living room lamp 1" and "living room lamp 2." In these examples, block 510 may further include using contextual information received from the user device 102 (e.g., received in block 506 as part of the data corresponding to the audio input) to disambiguate between potentially matching electronic devices. Any type of contextual information may be used, such as sensor information (e.g., lighting, ambient noise, ambient temperature, images or videos of the surrounding environment, distance to another object, etc.), the physical state of the user device 102 (e.g., device orientation, device location, device temperature, power level, speed, acceleration, movement patterns, cellular signal strength, etc.), or information associated with the software state of the user device 102 (e.g., running processes, installed programs, past and present network activity, background services, error logs, resource usage, etc.). For example, continuing with the "turn on the lights" example provided above, the location and orientation of the user device 102 when it received the audio input in block 502 may be provided to the server system 110 in blocks 504 and 506. This contextual information may be used to determine the location of the user device 102 and / or the direction the user device 102 was pointed when it received the user utterance. The closest light, or the light the user device 102 was pointing at, when compared to the known locations of "Living Room Lamp 1" and "Living Room Lamp 2" stored in the data and model storage 120 may be selected as the electronic device in block 510. Other types of contextual information may be used in a similar manner in block 510 to disambiguate between potentially matching electronic devices by identifying contextual information that increases or decreases the likelihood that one or more of the potentially matching electronic devices is referenced by the textual representation of the user utterance.

[0058] In still other embodiments, words associated with states in the textual representation of the user input may be used additionally or alternatively to disambiguate between potentially matching electronic devices or to identify an appropriate electronic device. For example, a textual representation including "set to 68" may not yield any matches when using the set of nouns described above. Thus, in these embodiments, block 510 may further include parsing the textual representation to identify any of a set of words associated with states corresponding to electronic devices supported by system 100. For example, the set of words associated with states may include possible states of an electronic device and their synonyms (e.g., open, closed, shut, on, off, active, inactive, lock, locked, color, etc.), value types of the state (e.g., binary, floating-point number, etc.), queries regarding the state of the device, adjectives associated with particular types of states (e.g., warmer, cooler, brighter, darker, color, etc.), etc. When used to parse "set to 68," the textual representation may be determined to include the floating-point value "68." When compared with the entries shown in FIG. 4 , it may be determined that only “2nd Floor Thermostat” and “1st Floor Thermostat” accept floating-point values ​​for the state. To disambiguate between the two thermostats, context information such as the location of the user device 102 may be used to select the thermostat that is closer to the user device 102 as the electronic device identified in block 510. Similarly, if the textual representation instead included “brighter,” a set of words associated with the state may be used to determine whether the textual representation includes the word “brighter,” which is an adjective that describes the state of a device having a type “lighting.” To disambiguate between two lights, context information such as the location of the user device 102 may be used to select the light that is closer to the user device 102 as the electronic device identified in block 510.

[0059] At block 512, a command to be performed by each of the one or more electronic devices identified at block 510 may be identified. The command(s) to be performed may be identified in any number of ways. In some examples, the command(s) to be performed may be identified by analyzing the textual representation of the user utterance to identify any of a set of words associated with states (e.g., possible states of the electronic device and their synonyms, types of state values, queries regarding the state of the device, adjectives associated with particular types of states). The identified states or actions may then be used to identify a command to be performed by each of the one or more electronic devices identified at block 510. For example, if the textual representation of the user utterance includes "lock the front door," it may be determined that the textual representation includes the state "locked" from the set of states. Thus, at block 512, a command to transition to the "locked" state may be generated and identified as a command to be performed by the electronic device "front door" identified at block 510. While the above examples specify a single command for a single electronic device, it should be understood that multiple commands may be specified if multiple electronic devices are specified in block 510. For example, the text expression "Turn on all lights" may specify a command in block 512 to transition each of "Living Room Lamp 1" and "Living Room Lamp 2" to the "On" state.

[0060] In some examples, if the textual representation of the user utterance includes one of the adjectives associated with a particular type of state (e.g., warmer, cooler, brighter, darker, etc.), block 512 may include identifying a command to set the state of the electronic device identified in block 510 to a value relative to its current value. For example, if the textual representation includes "make warmer," the command identified in block 514 may be a command to increase the temperature of the thermostat identified in block 510 by a predetermined amount. Because the actual state of the electronic device may differ from the state stored in data and model storage 120, the command may not transition the state value to a particular value (e.g., determined using the state of the thermostat stored in data and model storage 120), but rather may be a command to change the state value by an amount relative to the current value of the electronic device.

[0061] In some embodiments, if the textual representation of the user utterance includes a query regarding the status of a device, block 512 may include identifying instructions to cause the user device 102 to perform a query on the identified electronic device(s). For example, if the textual representation includes "Is the garage door closed?", the command identified in block 514 may be a command to perform a query regarding the status of the electronic device "garage door."

[0062] At block 514, identifiers for each of the one or more electronic devices identified at block 510 and the command(s) to be performed by the one or more electronic devices identified at block 512 may be transmitted to the user device. For example, the server system 110 may transmit the unique identifiers associated with each of the electronic devices identified at block 510 and the command(s) to be performed by the one or more electronic devices to the user device 102 over the network(s) 108.

[0063] In some examples, the text representation of the user utterance generated in block 508 may also be transmitted to the user device in block 514. In these examples, blocks 510 and 514 may also be performed on the user device. The transmitted text representation may be used by the user device to identify one or more electronic devices and / or to identify command(s) to be performed by one or more electronic devices in electronic device data 370. This may be desirable, for example, if electronic device data 370 in the user device is more recent than the data on one or more servers. In such cases, the user device may identify electronic devices and / or commands not included in the data on one or more servers.

[0064] In other examples, at blocks 510 and / or 512, the text representation may be analyzed at one or more servers to identify significant words or terms that may be suitable for identifying one or more of the electronic devices and / or identifying command(s) to be performed by one or more electronic devices. In these examples, the analyzed significant words or terms may also be transmitted to the user device at block 514. The transmitted significant words or terms may be used by the user device to identify one or more electronic devices and / or identify command(s) to be performed by one or more electronic devices in electronic device data 370. This may be desirable, for example, if electronic device data 370 in the user device is more recent than the data on one or more servers. In such cases, the user device may identify electronic devices and / or commands not included in the data of one or more servers.

[0065] The respective identifiers and command(s) of the one or more electronic devices sent by the one or more servers may be received by the user device in block 516. For example, the user device 102 may receive the unique identifier(s) and command(s) sent by the server system 110 in block 514 over the network(s) 108.

[0066] At block 518, the user device may transmit the command(s) received at block 516 to the electronic device(s) associated with the identifier(s) received at block 516. For example, if the user device 102 received at block 516 an identifier associated with the electronic device 128 (e.g., “Front Door”) and a command to transition the electronic device to a “Locked” state, the user device 102 may transmit the command to transition to the “Locked” state to the electronic device 128 over the network(s) 126. If the user device 102 received additional identifiers and commands at block 516, the user device 102 may further transmit those commands to the electronic devices identified at block 518.

[0067] In some embodiments, the actual state of the electronic device may not be the same as the state of the electronic device as stored in user device 102 (e.g., in memory 350) and / or server system 110 (e.g., in data and model storage 120). For example, a door lock may be manually opened or closed without using a virtual assistant in system 100. Thus, in some embodiments, block 518 may be performed regardless of the state of the electronic device as stored in user device 102 and / or server system 110. For example, a command to set the state of the door lock to “locked” may be sent to the door lock even if the corresponding entry in user device 102 and / or server system 110 indicates that the door is already locked. Furthermore, block 518 may be performed without first querying the electronic device (e.g., between blocks 516 and 518) to determine its actual state, in order to reduce the time required to issue a command to the electronic device. For example, a command to set the state of a door key to "locked" may be sent to the door key without first querying its state, thereby reducing the time required to send the command to the door key by an amount equivalent to the time required to send the query to the door key and receive the state from the door key.

[0068] In block 520, after each electronic device executes its respective command, the user device may receive the current state of each electronic device (to which the command was sent in block 518). For example, after the user device 102 executes a command to set the state of the electronic device 128 to “locked,” the user device 102 may receive the updated state of the electronic device 128. In this example, the current state returned to the user device 102 may be the state “locked.” If the command was sent to more than one electronic device in block 518, block 520 may further include receiving the current states from the electronic devices. In some examples, block 520 may further include updating the state of the electronic device in electronic device data 370 based on the received current state. For example, the user device 102 may update the state of the electronic device 128 in the electronic device data 370 to “locked.” In some embodiments, block 520 may further include outputting an audio or visual indication of the outcome of the command(s) sent to the electronic device(s) in block 518 based on the sent command(s) and the current state(s) of the receiving electronic device(s). For example, if the command sent to electronic device 128 was to transition the device's state to "locked" and the current state of electronic device 128 received in block 520 was "locked," the outcome indication may be that electronic device 128 successfully transitioned to the "locked" state. Alternatively, if the received current state of the electronic device (e.g., "unlocked") differs from the desired state (e.g., "locked") indicated in the command, an indication of a failure to transition to the "locked" state may be provided to the user. The current state may further include an error state, such as an unspecified or unavailable state of the device.

[0069] In block 522, the user device may transmit the current state(s) of the electronic device(s) received in block 520 to one or more servers. For example, the user device 102 may transmit the current states of the electronic devices received in block 520 to the server system 110. If the current states of more than one electronic device are received in block 520, block 522 may further include transmitting the current states to the server system 110.

[0070] At block 524, one or more servers may receive the current state(s) of the electronic device(s) transmitted by the user device at block 522. For example, the server system 110 may receive the current state of the electronic device transmitted by the user device 102 at block 522. If the user device 102 transmitted more than one current state, block 524 may further include receiving those current states as well. In some embodiments, block 524 may further include updating the state(s) of the electronic device(s) in the data and model storage device 120 based on the received current state(s). For example, the server system 110 may update the state of the electronic device 128 in the data and model storage device 120 to “locked.”

[0071] In some examples, process 500 may further include generating a notification associated with a current state of one or more of the electronic devices in response to determining that a predetermined condition is met. For example, in response to one or more of the electronic devices being in a predetermined state (e.g., the "front door" is unlocked) while the location of user device 102 has left a predetermined area (e.g., an area corresponding to the user's home), a notification may be provided to the user via user device 102 indicating that the user has forgotten to lock those doors. Other similar notifications may be generated in response to other predetermined conditions.

[0072] A virtual assistant implemented by a user device may use process 500 to receive natural language commands to set a state or query any number of electronic devices. The natural language commands can query the electronic devices in any desired manner and do not need to include a unique identifier or type of electronic device. Remote control of electronic devices

[0073] FIG. 6 shows an example process 600 for remotely controlling an electronic device using a virtual assistant. In some embodiments, process 600 may be similar to process 500, except that process 600 may be performed using a system similar to or identical to system 200 shown in FIG. 2. For example, process 600 may be performed by a user device (e.g., user device 102) located remotely from the electronic device being controlled (e.g., electronic devices 128, 130, and 132), and a second user device (e.g., second user device 134) may be used instead to control the electronic device. Thus, portions of process 600 may be performed by each of user device 102, server system 110, and second user device 134. Specifically, the left block of FIG. 6 may be performed by user device 102, the center block of FIG. 6 may be performed by server system 110, and the right block of FIG. 6 may be performed by second user device 134.

[0074] The blocks of process 600 may be similar to or identical to the identically numbered blocks of process 500, except that blocks 516, 518, 520, and 522 of process 600 may instead be performed by a second user device (e.g., second user device 134). Further, as a result, in block 514, respective identifiers of one or more electronic devices and command(s) to be performed by the electronic device(s) may instead be sent to the second user device, and in block 524, current state(s) of the electronic device(s) may instead be received from the second user device.

[0075] In some embodiments, process 600 may further include transmitting, by one or more servers, the current state(s) of the electronic device(s) received at block 524 to the user device that received the audio input at block 502. Additionally or alternatively, the process may include transmitting an indication of success of the command sent to the electronic device at block 518. For example, a visual or audio output may be generated that notifies the user of successful, partial success, or failure of execution of the command by the electronic device. The determination of success may depend on the command sent to the electronic device and the current state of the device received at block 524. Standalone control of electronic devices

[0076] 7 shows an example process 700 for controlling an electronic device using a virtual assistant. In some embodiments, process 700 may be similar to process 500, except that process 700 may be performed using a standalone user device that can perform the functions of both user device 102 and server system 110. As a result, all blocks of process 700 may be performed by a user device (e.g., user device 102).

[0077] The blocks of process 700 may be similar to or identical to the identically numbered blocks of process 500, except that blocks 508, 510, and 512 may instead be performed by a user device (e.g., user device 102). Furthermore, as a result, blocks corresponding to functions communicating between user device 102 and server system 110 (e.g., blocks 504, 506, 514, 516, 522, and 524) need not be performed. Storing the configuration of an electronic device

[0078] FIG. 8 shows an example process 800 for storing the states of multiple electronic devices as a configuration using a virtual assistant. A configuration may represent a stored set of states of multiple electronic devices that can be referenced in a spoken user input to transition the multiple electronic devices to the states defined in the configuration. For example, a user may create a "sleep" configuration in which the states of all lights are set to off, the state of the thermostat is set to 72°F, the states of all doors are set to locked, and the state of the garage door is set to closed. Thus, when the user goes to bed, the user may provide a command to the user device 102, such as "go to bed," that references the stored configuration, and the system 100 may set the states of the electronic devices based on the stored states in the sleep configuration. In some embodiments, the process 800 may be performed using a system similar to or identical to the system 100 shown in FIG. 1. In these embodiments, the blocks of the process 800 may be performed by both the user device 102 and the server system 110. Specifically, the blocks on the left side of FIG. 8 may be performed by the user device 102, and the blocks on the right side may be performed by the server system 110.

[0079] At block 802, audio input including user utterances may be received at a user device in a manner similar or identical to block 502 of process 500. At block 804, data corresponding to the audio input received at block 802 may be transmitted to one or more servers for processing in a manner similar or identical to block 504 of process 500.

[0080] At block 806, data corresponding to the audio input sent by the user device at block 804 may be received by one or more servers in a manner similar or identical to block 506 of process 500. At block 808, speech-to-text conversion may be performed on the data corresponding to the audio input in a manner similar or identical to block 508 of process 500 to convert the user utterance into a text representation of the user utterance.

[0081] In block 810, it may be determined that the textual representation of the user utterance represents a user intent to store the state of the electronic device of system 100 as a configuration. For example, one or more processing modules 118 of server system 110 may utilize data and model storage 120 to identify the user intent based on the natural language input. In some embodiments, this may include analyzing the textual representation for words that may be related to storing a configuration, such as "save," "memorize," "name," "keep," "configuration," "scene," and synonyms thereof. For example, if the textual representation includes "store this configuration as sleep," it may be determined in block 810 that the user intends to store the state of the electronic device as a configuration named "sleep." Other textual representations, such as "save this scene as work," may indicate that the user intends to store the state of the electronic device as a configuration named "work."

[0082] In block 812, in response to determining that the textual representation of the user utterance expresses a user intent to store the state of the electronic device as a configuration in system 100, one or more servers may transmit instructions to the user device to perform a query regarding the state of the electronic device. For example, server system 110 may transmit instructions to user device 102 over network(s) 108 to perform a query regarding the state of electronic devices 128, 130, and 132.

[0083] Instructions to perform queries regarding the status of the electronic devices sent by one or more servers may be received by the user devices at block 814. For example, the user device 102 may receive instructions to perform queries regarding the status of the electronic devices 128, 130, and 132 from the server system 110 via the network(s) 108.

[0084] In block 816, the user device may send a query to each of the electronic devices regarding their current status. For example, the user device 102 may send a command over the network(s) 126 to each of the electronic devices 128, 130, and 132 instructing them to return their current status.

[0085] At block 818, the user device may receive the current status of each of the electronic device(s) (or the current status of all electronic devices capable of transmitting a current status) in response to the query sent at block 516. For example, user device 102 may receive the current status of electronic devices 128, 130, and 132 over network(s) 126 in response to the query sent to each of the devices at block 816. In some embodiments, block 818 may further include updating the status of the electronic devices in electronic device data 370 based on the received current status. The current status may further include an error status, such as an unidentified or unavailable status of the device.

[0086] In block 820, the user device may transmit to one or more servers the current state of the electronic device received in block 818. For example, the user device 102 may transmit the current state of the electronic device received in block 818 to the server system 110.

[0087] At block 822, one or more servers may receive the current states of the electronic devices sent by the user devices at block 820. For example, the server system 110 may receive the current states of the electronic devices 128, 130, and 132 from the user device 102 via the network(s) 126.

[0088] At block 824, one or more servers may store the current state of the electronic devices received at block 822 as a configuration. In some examples, the configuration may be assigned a unique identifier, such as “Sleep,” “Morning,” or “Work,” based on the name provided in the textual representation and identified at block 810. For example, in response to receiving a user utterance at block 802 including “Name this configuration Sleep,” server system 110 may store the current state of electronic devices 128, 130, and 132 as a “Sleep” configuration in model and data storage 120. Controlling an electronic device using a stored configuration

[0089] 9 shows an example process 900 for using a virtual assistant to set the state of multiple electronic devices using a previously stored configuration (e.g., created using process 800). In some embodiments, process 900 may be performed using a system similar to or identical to system 100 shown in FIG. 1. In these embodiments, the blocks of process 900 may be performed by both user device 102 and server system 110. Specifically, the blocks on the left side of FIG. 9 may be performed by user device 102, and the blocks on the right side may be performed by server system 110.

[0090] At block 902, audio input including user utterances may be received at a user device in a manner similar or identical to block 502 of process 500. At block 904, data corresponding to the audio input received at block 902 may be transmitted to one or more servers for processing in a manner similar or identical to block 504 of process 500.

[0091] At block 906, data corresponding to the audio input sent by the user device at block 904 may be received by one or more servers in a manner similar or identical to block 506 of process 500. At block 908, speech-to-text conversion of the audio input to the user utterance may be performed in a manner similar or identical to block 508 of process 500 to convert the user utterance into a text representation of the user utterance.

[0092] At block 910, it may be determined that a textual representation of a user utterance represents a user intent to set a state of an electronic device of system 100 based on the stored configuration. For example, one or more processing modules 118 of server system 110 may utilize data and model storage 120 to identify a user intent based on natural language input. In some embodiments, this may include analyzing the textual representation for words that may be related to using the stored configuration, such as unique identifiers associated with the stored configuration, such as “set,” “configuration,” “scene,” synonyms thereof, etc. For example, if the textual representation includes “go to bed,” it may be determined at block 910 that the user intends to set a state of the electronic device based on the “sleep” configuration. Similarly, other textual representations, such as “night mode,” “set to sleep,” etc., may indicate that the user intends to set a state of the electronic device based on the “sleep” configuration.

[0093] At block 912, commands may be sent by one or more servers to the user devices to set the state of the electronic devices of system 100 based on the configuration determined at block 910. Identifiers associated with the commands may be sent to identify the devices that will execute each command. For example, server system 110 may send unique identifiers associated with the electronic devices and commands that will be executed by those electronic devices to place the electronic devices in the state specified by the stored configuration.

[0094] At block 914, a command sent by the server(s) to set the state of an electronic device of the system 100 may be received by the user device. For example, the user device 102 may receive the command sent by the server system 110 at block 912 over the network(s) 108.

[0095] In block 916, the user device may transmit the command received in block 914 to the electronic device associated with the command. For example, the user device 102 may transmit the command over the network(s) 126 to the electronic devices 128, 130, and 132 to place the electronic devices in the state specified by the stored configuration.

[0096] In some embodiments, the actual state of the electronic device may not be the same as the state of the electronic device as stored in user device 102 (e.g., in memory 350) and / or server system 110 (e.g., in data and model storage 120). Thus, in some embodiments, similar to process 500, in order to reduce the time required to issue commands to the electronic device, block 916 may be performed without first querying the electronic device (e.g., between blocks 914 and 916) to determine its actual state, regardless of the state of the electronic device as stored in user device 102 and / or server system 110.

[0097] In block 918, the user device may receive the updated state of the electronic device after the electronic device executes the command sent by the user device in block 916. For example, user device 102 may receive the updated state of electronic devices 128, 130, and 132 after electronic devices execute commands that change their state to the state specified by the stored configuration. In some embodiments, block 920 may further include updating the state of the electronic device in electronic device data 370 based on the received current state. For example, user device 102 may update the state of electronic device 128 to “locked” in electronic device data 370. In some embodiments, similar to block 520, block 918 may further include outputting an audio or visual indication of the result of the command(s) sent to the electronic device(s) in block 916 based on the sent command(s) and the received current state(s) of the electronic device(s). The updated status may also include error status, such as an unspecified or unavailable status of the device.

[0098] In block 920, the user device may transmit to one or more servers the updated state of the electronic device received in block 918. For example, the user device 102 may transmit the updated state of the electronic device received in block 918 to the server system 110.

[0099] At block 922, one or more servers may receive the updated state of the electronic device sent by the user device at block 920. For example, server system 110 may receive the updated state of the electronic device sent by user device 102 at block 920. In some embodiments, block 922 may further include updating the state of the electronic device in data and model storage 120 based on the received updated state.

[0100] In some examples, additionally or alternatively, in response to determining that a predetermined condition is met, the state of the electronic device may be configured using a stored configuration. For example, in response to the location of the user device 102 entering a predetermined area (e.g., an area corresponding to the user's home), a command may be sent to the electronic device to transition to a state specified by a stored configuration (e.g., a "home" configuration). Similarly, in response to the location of the user device 102 leaving a predetermined area (e.g., an area corresponding to the user's home) during a predetermined time frame (e.g., between 8:00 and 9:00 AM on a weekday), a command may be sent to the electronic device to transition to a state specified by another stored configuration (e.g., a "work" configuration). Other similar predetermined conditions may be created to cause the system 100 to configure the electronic device based on a stored configuration. Electronic Devices

[0101] According to some examples, FIG. 10 shows a functional block diagram of an electronic device 1000 configured in accordance with the principles of various described embodiments. The functional blocks of the device may be implemented by hardware, software, or a combination of hardware and software to carry out the principles of various described embodiments. Those skilled in the art will understand that the functional blocks described in FIG. 10 may be combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or division or further definition of the functional blocks described herein.

[0102] 10, electronic device 1000 may include a touchscreen display unit 1002 configured to display a user interface and receive touch input, and an audio receiving unit 1004 configured to receive audio input. In some embodiments, electronic device 1000 may include a speaker unit 1006 configured to generate audio. Electronic device 1000 may further include a processing unit 1008 coupled to touchscreen display unit 1002 and audio receiving unit 1004 (and optionally coupled to speaker unit 1006). In some embodiments, the processing unit 1008 may include an audio input receiving unit 1010, an audio input sending unit 1012, an identifier and command receiving unit 1014, an identifier and command sending unit 1016, a status receiving unit 1018, a status sending unit 1020, a status update unit 1022, an instruction output unit 1024, a second identifier and command receiving unit 1026, a second identifier and command sending unit 1028, a second status receiving unit 1030, a second status sending unit 1032, and a notification sending unit 1034.

[0103] The processing unit 1008 may be configured to receive audio input (e.g., using the audio input receiving unit 1010) including user utterances. The processing unit 1008 may be further configured to transmit data corresponding to the audio input to one or more servers (e.g., using the audio input transmitting unit 1012). The processing unit 1008 may be further configured to receive from the one or more servers (e.g., using the identifier and command receiving unit 1014) an identifier of the first electronic device identified by the one or more servers based on the data corresponding to the audio input, and a first command to be executed by the first electronic device identified by the one or more servers based on the data corresponding to the audio input. The first command may be transmitted to the first electronic device (e.g., using the identifier and command transmitting unit 1016). After transmitting the first command to the first electronic device, a current status of the first electronic device may be received from the first electronic device (e.g., using the status receiving unit 1018). The current status of the first electronic device may be transmitted to one or more servers (e.g., using the status transmitting unit 1020).

[0104] In some embodiments, the first electronic device includes a light bulb. In other embodiments, the first command includes a command to set an on / off state, dimmable state, or color state of the light. In yet other embodiments, the current state of the first electronic device includes the on / off state, dimmable state, or color state of the light bulb after sending the first command to set the on / off state, dimmable state, or color state of the light bulb.

[0105] In some embodiments, the first electronic device includes a switch. In other embodiments, the first command includes a command to set an on / off state of the switch. In yet other embodiments, the current state of the first electronic device includes an on / off state of the switch after sending the first command to set the on / off state of the switch.

[0106] In some embodiments, the first electronic device includes an electrical outlet. In other embodiments, the first command includes a command to set an active / inactive state of the electrical outlet. In yet other embodiments, the current state of the first electronic device includes an active / inactive state of the electrical outlet after sending the command to set the active / inactive state of the electrical outlet.

[0107] In some embodiments, the first electronic device includes a door key. In other embodiments, the first command includes a command to set a locked / unlocked state of the door key. In yet other embodiments, the current state of the first electronic device includes a locked / unlocked state of the door key after sending the command to set the locked / unlocked state of the door key.

[0108] In some embodiments, the first electronic device includes a garage door. In other embodiments, the first command includes a command to set an open / closed state of the garage door. In yet other embodiments, the current state of the first electronic device includes an open / closed state of the garage door after sending the command to set the open / closed state of the garage door.

[0109] In some embodiments, the first electronic device includes a thermostat. In other embodiments, the first command includes a command to set a value of a temperature setting of the thermostat. In yet other embodiments, the current state of the first electronic device includes a value of a temperature setting of the thermostat after sending the first command to set a value of a temperature setting of the thermostat.

[0110] In some embodiments, the first command includes a query regarding the current state of the first electronic device.

[0111] In some embodiments, the first command may be transmitted to the first electronic device (eg, using the identifier and command transmitting unit 1016) over a local wireless network.

[0112] In some embodiments, the first command may be transmitted directly to the first electronic device by Bluetooth, line of sight, peer-to-peer, or WiFi communication.

[0113] In some examples, the processing unit 1008 may be configured to exclude performing a query on the first electronic device regarding a state of the first electronic device between receiving the first command from the one or more servers and sending the first command to the first electronic device.

[0114] In some embodiments, the electronic device 1000 may further include a database unit 1034 that stores a state of each of the multiple electronic devices, including the first electronic device. In other embodiments, the processing unit 1008 may be further configured to update the state of the first electronic device stored in the database unit 1034 (e.g., using the state update unit 1022) based at least in part on the current state of the first electronic device received from the first electronic device.

[0115] In some embodiments, the processing unit 1008 may be further configured to output (e.g., using the indication output unit 1024) an indication of the result of the first command based on the first command and the current state of the first electronic device received from the first electronic device.

[0116] In some embodiments, electronic device 1000 may include a mobile phone, a desktop computer, a laptop computer, a tablet computer, a portable media player, a television, a television set-top box, or a wearable electronic device.

[0117] In some examples, the processing unit 1008 may be further configured to receive from the one or more servers (e.g., using the second identifier and command receiving unit 1026) an identifier of the second electronic device identified by the one or more servers based on the data corresponding to the audio input, and a second command to be performed by the second electronic device identified by the one or more servers based on the data corresponding to the audio input. The processing unit 1008 may be further configured to send the second command to the second electronic device (e.g., using the second identifier and command sending unit 1028), and receive a current state of the second electronic device from the second electronic device (e.g., using the second state receiving unit 1030) after sending the second command to the second electronic device. The processing unit 1008 may be further configured to send the current state of the second electronic device to the one or more servers (e.g., using the second state sending unit 1032).

[0118] In some examples, the processing unit 1008 may be further configured to send (e.g., using the notification sending unit 1036) a notification associated with a current state of the first electronic device in response to determining that the predetermined condition is met.

[0119] According to some examples, FIG. 11 shows a functional block diagram of an electronic device 1100 configured in accordance with the principles of various described embodiments. The functional blocks of the device may be implemented by hardware, software, or a combination of hardware and software to carry out the principles of various described embodiments. Those skilled in the art will understand that the functional blocks described in FIG. 11 may be combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or division or further definition of the functional blocks described herein.

[0120] 11 , electronic device 1100 may include a touchscreen display unit 1102 configured to display a user interface and receive touch input, and an audio receiving unit 1104 configured to receive audio input. In some embodiments, electronic device 1100 may include a speaker unit 1106 configured to generate audio. Electronic device 1100 may further include a processing unit 1108 coupled to touchscreen display unit 1102 and audio receiving unit 1104 (and optionally coupled to speaker unit 1106). In some embodiments, processing unit 1108 may include an audio input receiving unit 1110, an audio input transmitting unit 1112, a command receiving unit 1114, a query transmitting unit 1116, a status receiving unit 1118, and a status transmitting unit 1120.

[0121] The processing unit 1108 may be configured to receive audio input (e.g., using the audio input receiving unit 1110) including user utterances. The processing unit 1108 may be further configured to send data corresponding to the audio input to one or more servers (e.g., using the audio input sending unit 1112). The processing unit 1108 may be further configured to receive instructions from the one or more servers (e.g., using the instruction receiving unit 1114) to perform a query regarding the status of each of the multiple electronic devices identified by the one or more servers based on the data corresponding to the audio input. The status query may be sent to each of the multiple electronic devices (e.g., using the query sending unit 1116), and the current status of each of the multiple electronic devices may be received from the multiple electronic devices (e.g., using the status receiving unit 1118). The current status of each of the multiple electronic devices may be sent to one or more servers (e.g., using the status sending unit 1120) for storage as a configuration.

[0122] According to some examples, FIG. 12 shows a functional block diagram of an electronic device 1200 configured in accordance with the principles of various described embodiments. The functional blocks of the device may be implemented by hardware, software, or a combination of hardware and software to carry out the principles of various described embodiments. Those skilled in the art will understand that the functional blocks described in FIG. 12 may be combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or division or further definition of the functional blocks described herein.

[0123] 12 , electronic device 1200 may include a touchscreen display unit 1202 configured to display a user interface and receive touch input, and an audio receiving unit 1204 configured to receive audio input. In some embodiments, electronic device 1200 may include a speaker unit 1206 configured to generate audio. Electronic device 1200 may further include a processing unit 1208 coupled to touchscreen display unit 1202 and audio receiving unit 1204 (and optionally coupled to speaker unit 1206). In some embodiments, processing unit 1208 may include an audio input receiving unit 1210, an audio input transmitting unit 1212, a command receiving unit 1214, a query transmitting unit 1216, a status receiving unit 1218, and a status transmitting unit 1220.

[0124] In some examples, processing unit 1208 may be further configured to receive audio input including user utterances (e.g., using audio input receiving unit 1210), transmit data corresponding to the audio input to one or more servers (e.g., using audio input transmitting unit 1212), and receive a plurality of commands from one or more servers (e.g., using command receiving unit 1214) that set a state of each of the plurality of electronic devices identified by the one or more servers based on the stored configuration. Processing unit 1208 may be further configured to transmit the plurality of commands to the plurality of electronic devices (e.g., using command transmitting unit 1216). Processing unit 1208 may be further configured to receive updated states of each of the plurality of electronic devices from the plurality of electronic devices (e.g., using status receiving unit 1218) and transmit the updated states of each of the plurality of electronic devices to one or more servers (e.g., using status transmitting unit 1220).

[0125] 13, electronic device 1300 may include a touchscreen display unit 1302 configured to display a user interface and receive touch input, and an audio receiving unit 1304 configured to receive audio input. In some embodiments, electronic device 1300 may include a speaker unit 1306 configured to generate audio. Electronic device 1300 may further include a processing unit 1308 coupled to touchscreen display unit 1302 and audio receiving unit 1304 (and optionally coupled to speaker unit 1306). In some embodiments, the processing unit 1308 may include an audio input receiving unit 1310, a speech-to-text conversion unit 1312, an electronic device identifying unit 1314, a command identifying unit 1316, an identifier and command sending unit 1318, a status receiving unit 1320, a status update unit 1322, a context information receiving unit 1324, an instruction sending unit 1326, a second electronic device identifying unit 1328, a second command identifying unit 1330, a second identifier and command sending unit 1332, and a second status receiving unit 1336.

[0126] The processing unit 1308 may be configured to receive data corresponding to an audio input including a user utterance (e.g., using the audio input receiving unit 1310). The processing unit 1308 may be further configured to perform speech-to-text conversion on the data corresponding to the audio input (e.g., using the speech-to-text conversion unit 1312) to generate a text representation of the user utterance. The processing unit 1308 may be further configured to identify a first electronic device (e.g., using the electronic device identification unit 1314) based on the text representation of the user utterance. The processing unit 1308 may be further configured to identify a first command to be executed by the first electronic device (e.g., using the command identification unit 1316) based on the text representation of the user utterance. The processing unit 1308 may be further configured to transmit an identifier of the first electronic device and the first command to the user device (e.g., using the identifier and command sending unit 1318). The processing unit 1308 may further be configured to receive (eg, using the status receiving unit 1320) a current status of the first electronic device.

[0127] In some embodiments, the first electronic device includes a light bulb. In other embodiments, the first command includes a command to set an on / off state, dimmable state, or color state of the light. In yet other embodiments, the current state of the first electronic device includes the on / off state, dimmable state, or color state of the light bulb after sending the first command to set the on / off state, dimmable state, or color state of the light bulb.

[0128] In some embodiments, the first electronic device includes an electrical outlet. In other embodiments, the first command includes a command to set an active / inactive state of the electrical outlet. In yet other embodiments, the current state of the first electronic device includes an active / inactive state of the electrical outlet after sending the command to set the active / inactive state of the electrical outlet.

[0129] In some embodiments, the first electronic device includes a switch. In other embodiments, the first command includes a command to set an on / off state of the switch. In yet other embodiments, the current state of the first electronic device includes an on / off state of the electrical outlet after sending the command to set the on / off state of the switch.

[0130] In some embodiments, the first electronic device includes a door key. In other embodiments, the first command includes a command to set a locked / unlocked state of the door key. In yet other embodiments, the current state of the first electronic device includes a locked / unlocked state of the door key after sending the command to set the locked / unlocked state of the door key.

[0131] In some embodiments, the first electronic device includes a garage door. In other embodiments, the first command includes a command to set an open / closed state of the garage door. In yet other embodiments, the current state of the first electronic device includes an open / closed state of the garage door after sending the command to set the open / closed state of the garage door.

[0132] In some embodiments, the first electronic device includes a thermostat. In other embodiments, the first command includes a command to set a value of a temperature setting of the thermostat. In yet other embodiments, the current state of the first electronic device includes a value of a temperature setting of the thermostat after sending the first command to set a value of a temperature setting of the thermostat.

[0133] In some embodiments, the first command includes a query regarding the current state of the first electronic device.

[0134] In some embodiments, the electronic device 1300 further includes a database unit 1334 that stores the name, type, and status of each of the multiple electronic devices, including the first electronic device. In other embodiments, the processing unit 1308 may be further configured to update the status of the first electronic device stored in the database unit 1334 (e.g., using the status update unit 1322) based on the current status of the first electronic device.

[0135] In some embodiments, processing unit 1308 may be further configured to receive context information in the data corresponding to the audio input from the user device (e.g., using the context information receiving unit 1324). In other embodiments, processing unit 1308 may be further configured to identify the first electronic device (e.g., using the electronic device identifying unit 1314) based on the context information. In yet other embodiments, processing unit 1308 may be further configured to identify a first command (e.g., using the command identifying unit 1316) based on the context information. In some embodiments, the context information includes an orientation of the user device when the user device received the audio input or a location of the user device when the user device received the audio input.

[0136] In some embodiments, the textual representation of the user utterance excludes the name of the first electronic device. In other embodiments, the textual representation of the user utterance excludes the type of the first electronic device.

[0137] In some examples, electronic device 1300 includes a mobile phone, a desktop computer, a laptop computer, a tablet computer, a portable media player, a television, a television set-top box, or a wearable electronic device.

[0138] In some embodiments, the processing unit 1308 may be further configured to send an identifier of the first electronic device and a first command to the user device (e.g., using the identifier and command sending unit 1318) and receive a current state of the first electronic device from the user device (e.g., using the state receiving unit 1320).

[0139] In other embodiments, the processing unit 1308 may be further configured to send an identifier of the first electronic device and a first command to the second user device (e.g., using the identifier and command sending unit 1318) and receive a current status of the first electronic device from the second user device (e.g., using the status receiving unit 1320).

[0140] In other embodiments, the processing unit 1308 may be further configured to send to the second user device (e.g., using the instruction sending unit 1326) an indication of a result of the first command based on the first command and the current state of the first electronic device.

[0141] In other embodiments, the processing unit 1308 may be further configured to identify the second electronic device based on the text representation of the user utterance (e.g., using the second electronic device identifying unit 1328), identify a second command to be performed by the second electronic device based on the text representation of the user utterance (e.g., using the second command identifying unit 1330), transmit an identifier of the second electronic device and the second command (e.g., using the second identifier and command transmitting unit 1332), and receive a current state of the second electronic device (e.g., using the second state receiving unit 1336).

[0142] 14, electronic device 1400 may include a touchscreen display unit 1402 configured to display a user interface and receive touch input, and an audio receiving unit 1404 configured to receive audio input. In some embodiments, electronic device 1400 may include a speaker unit 1406 configured to generate audio. Electronic device 1400 may further include a processing unit 1408 coupled to touchscreen display unit 1402 and audio receiving unit 1404 (and optionally coupled to speaker unit 1406). In some embodiments, the processing unit 1408 may include an audio input receiving unit 1410, a speech-to-text conversion unit 1412, a determining unit 1414, a command sending unit 1416, a status receiving unit 1416, a configuration storage unit 1420, a second audio input receiving unit 1422, a second speech-to-text conversion unit 1424, a second determining unit 1426, a command sending unit 1428, and a second status receiving unit 1430.

[0143] The processing unit 1408 may be configured to receive data corresponding to an audio input including a user utterance from a user device (e.g., using the audio input receiving unit 1410). The processing unit 1408 may be further configured to perform speech-to-text conversion on the data corresponding to the audio input (e.g., using the speech-to-text conversion unit 1412) to generate a text representation of the user utterance. The processing unit 1408 may be further configured to determine (e.g., using the determining unit 1414) that the text representation of the user utterance represents a user intention to store a state of each of the plurality of electronic devices as a configuration. The processing unit 1408 may be further configured to send (e.g., using the instruction sending unit 1416) an instruction to perform a query regarding the state of each of the plurality of electronic devices. The processing unit 1408 may be further configured to receive a current state of each of the plurality of electronic devices (e.g., using the state receiving unit 1418). The processing unit 1408 may be further configured to store the received current state of each of the plurality of electronic devices as a configuration (e.g., using the configuration storage unit 1420).

[0144] 15, electronic device 1500 may include a touchscreen display unit 1502 configured to display a user interface and receive touch input, and an audio receiving unit 1504 configured to receive audio input. In some embodiments, electronic device 1500 may include a speaker unit 1506 configured to generate audio. Electronic device 1500 may further include a processing unit 1508 coupled to touchscreen display unit 1502 and audio receiving unit 1504 (and optionally coupled to speaker unit 1506). In some embodiments, the processing unit 1408 may include an audio input receiving unit 1510, a speech-to-text conversion unit 1512, a determining unit 1514, a command sending unit 1516, a status receiving unit 1516, a configuration storage unit 1520, a second audio input receiving unit 1522, a second speech-to-text conversion unit 1524, a second determining unit 1526, a command sending unit 1528, and a second status receiving unit 1530.

[0145] In some examples, the processing unit 1508 may be further configured to receive data corresponding to an audio input including a user utterance (e.g., using the audio input receiving unit 1510), perform speech-to-text conversion on the data corresponding to the audio input (e.g., using the speech-to-text conversion unit 1512) to generate a text representation of the user utterance, determine (e.g., using the determining unit 1514) that the text representation of the user utterance represents a user intention to change a state of each of the plurality of electronic devices based on the configuration, send (e.g., using the command sending unit 1516) a plurality of commands to set the state of each of the plurality of electronic devices based on the configuration, and receive (e.g., using the state receiving unit 1518) the updated state of each of the plurality of electronic devices.

[0146] As mentioned above, one aspect of the present technology is the collection and use of data available from various sources to improve the delivery to users of invitee content or any other content deemed to be of interest to the user. The present disclosure contemplates that in some cases, this collected data may include personal information data that uniquely identifies a particular person or that can be used to contact or locate that person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, home addresses, or any other identifiable information.

[0147] This disclosure recognizes that the use of such personal information data may be used to the benefit of the user. For example, personal information data may be used to deliver targeted content that is more likely to interest the user. Thus, use of such personal information data allows for intentional control of the content that is delivered. Additionally, other uses of personal information data that benefit the user are contemplated by this disclosure.

[0148] This disclosure further contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other uses of such personal information data will comply with well-defined privacy policies and / or privacy practices. Specifically, such entities should implement and always use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for maintaining the strict confidentiality of personal information data. For example, personal information from users should be collected for the entity's lawful and legitimate uses and should not be shared or sold except for those lawful uses. Furthermore, such collection should occur only after the user's informed consent is obtained. Furthermore, such entities may take all necessary measures to protect and secure access to such personal information data and to ensure that others with access to that personal information data comply with their privacy policies and procedures. Furthermore, such entities may submit themselves to third-party assessment to attest to their compliance with widely accepted privacy policies and practices.

[0149] Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which a user selectively blocks use of or access to personal information data. That is, the present disclosure contemplates that hardware and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case of an advertising delivery service, the present technology may be configured to allow a user to "opt in" or "opt out" of participating in the collection of personal information data during registration for the service. In another embodiment, a user may choose not to provide location information to a given content delivery service. In yet another embodiment, a user may choose not to provide precise location information but allow the transfer of location zone information.

[0150] Thus, while this disclosure broadly encompasses the use of personal information data to implement one or more various disclosed embodiments, this disclosure also contemplates that the various embodiments may also be implemented without requiring access to such personal information data. That is, various embodiments of the present technology are not rendered inoperable by the absence of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences based on non-personal information or minimal personal information, such as content requested by a device associated with the user, other non-personal information provided to a content delivery service, or publicly available information.

[0151] Although the embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art, and such changes and modifications are to be understood as being included within the scope of the various embodiments as defined by the appended claims.

Claims

1. 1. A method for controlling an electronic device using a virtual assistant on a user device, comprising: receiving, by the user device, an audio input comprising user speech; transmitting data corresponding to the audio input to one or more servers; From the one or more servers, an identifier of a first electronic device identified by the one or more servers based on the data corresponding to the audio input; and a first command to be executed by the first electronic device identified by the one or more servers based on the data corresponding to the audio input; receiving the sending the first command to the first electronic device.

2. receiving a current state of the first electronic device from the first electronic device after sending the first command to the first electronic device; The method of claim 1 , further comprising: transmitting the current state of the first electronic device to the one or more servers.

3. The method of claim 2 , wherein the first electronic device comprises a light bulb.

4. The method of claim 3 , wherein the first command comprises a command to set an on / off state, a dimmable state, or a color state of the light bulb.

5. 5. The method of claim 4, wherein the current state of the first electronic device comprises the on / off state, dimmable state, or color state of the light bulb after sending the first command to set the on / off state, dimmable state, or color state of the light bulb.

6. The method of claim 2 , wherein the first electronic device comprises an electrical outlet.

7. The method of claim 6 , wherein the first command comprises a command to set an active / inactive state of the electrical outlet.

8. 8. The method of claim 7, wherein the current state of the first electronic device includes the active / inactive state of the electrical outlet after sending the command to set the active / inactive state of the electrical outlet.

9. The method of claim 2 , wherein the first electronic device comprises a switch.

10. The method of claim 9 , wherein the first command comprises a command to set an on / off state of the switch.

11. 11. The method of claim 10, wherein the current state of the first electronic device comprises the on / off state of the electrical outlet after sending the command to set the on / off state of the switch.

12. The method of claim 2 , wherein the first electronic device comprises a door key.

13. The method of claim 12 , wherein the first command includes a command to set a lock / unlock status of the door.

14. 14. The method of claim 13, wherein the current state of the first electronic device includes the locked / unlocked state of the door key after sending the command to set the locked / unlocked state of the door key.

15. The method of claim 2 , wherein the first electronic device comprises a garage door.

16. 16. The method of claim 15, wherein the first command comprises a command to set an open / closed state of the garage door.

17. 17. The method of claim 16, wherein the current state of the first electronic device includes the open / closed state of the garage door after sending the command to set the open / closed state of the garage door.

18. The method of claim 2 , wherein the first electronic device comprises a thermostat.

19. 20. The method of claim 18, wherein the first command comprises a command to set a numerical value for a temperature setting of the thermostat.

20. 20. The method of claim 19, wherein the current state of the first electronic device includes the value of the temperature setting of the thermostat after sending the first command to set the value of the temperature setting of the thermostat.

21. The method of claim 2 , wherein the first command comprises a query regarding the current state of the first electronic device.

22. The method of claim 2 , further comprising: outputting a notification associated with the current state of the first electronic device in response to determining that a predetermined condition is met.

23. 3. The method of claim 2, wherein the user device includes a database that stores a state of each of a plurality of electronic devices including the first electronic device, and the method further includes updating the state of the first electronic device stored in the database based at least in part on the current state of the first electronic device received from the first electronic device.

24. 3. The method of claim 2, further comprising outputting an indication of a result of the first command based on the first command and the current state of the first electronic device received from the first electronic device, wherein the indication of the result is an audio, a visual, or both an audio and a visual indication.

25. The method of claim 1 , wherein the user device comprises a mobile phone, a desktop computer, a laptop computer, a tablet computer, a portable media player, a television, a television set-top box, or a wearable electronic device.

26. The method of claim 1 , wherein the first command is transmitted to the first electronic device over a local wireless network.

27. The method of claim 1 , wherein the first command is transmitted directly to the first electronic device by Bluetooth, line-of-sight, peer-to-peer, or WiFi communication.

28. 2. The method of claim 1, further comprising: excluding querying the first electronic device regarding the state of the first electronic device between receiving the first command from the one or more servers and sending the first command to the first electronic device.

29. From the one or more servers, an identifier of a second electronic device identified by the one or more servers based on the data corresponding to the audio input; and a second command to be executed by the second electronic device identified by the one or more servers based on the data corresponding to the audio input; and receiving the sending the second command to the second electronic device; receiving a current state of the second electronic device from the second electronic device after sending the second command to the second electronic device; The method of claim 1 , further comprising: transmitting the current state of the second electronic device to the one or more servers.

30. 1. A method for controlling an electronic device using a virtual assistant on a user device, comprising: receiving, by the user device, an audio input comprising user speech; transmitting data corresponding to the audio input to one or more servers; receiving instructions from the one or more servers to perform a query regarding a state of each of a plurality of electronic devices identified by the one or more servers based on the data corresponding to the audio input; sending a status query to each of the plurality of electronic devices.

31. receiving a current state of each of the plurality of electronic devices from the plurality of electronic devices; 31. The method of claim 30, further comprising transmitting the current state of each of the plurality of electronic devices to the one or more servers for storage as a configuration.

32. 1. A method for controlling an electronic device using a virtual assistant on a user device, comprising: receiving, by the user device, an audio input comprising user speech; transmitting data corresponding to the audio input to one or more servers; receiving a plurality of commands from the one or more servers to set a state of each of a plurality of electronic devices identified by the one or more servers based on the stored configuration; sending the plurality of commands to the plurality of electronic devices.

33. receiving an updated status of each of the plurality of electronic devices from the plurality of electronic devices; 33. The method of claim 32, further comprising: transmitting the updated state of each of the plurality of electronic devices to the one or more servers.

34. 1. A method for controlling an electronic device using a virtual assistant on a user device, comprising: receiving, by one or more servers, data from a user device corresponding to audio input including user utterances; performing a speech-to-text conversion on the data corresponding to the audio input to generate a text representation of the user utterance; identifying a first electronic device based on the textual representation of the user utterance; identifying a first command to be executed by the first electronic device based on the textual representation of the user utterance; transmitting an identifier of the first electronic device and the first command.

35. 35. The method of claim 34, further comprising receiving a current state of the first electronic device.

36. 36. The method of claim 35, wherein the first electronic device comprises a light bulb.

37. 37. The method of claim 36, wherein the first command comprises a command to change an on / off state, a dimmable state, or a color state of the light bulb.

38. 38. The method of claim 37, wherein the current state of the first electronic device comprises the on / off state, dimmable state, or color state of the light bulb after the light bulb receives the first command to set the on / off state, dimmable state, or color state of the light bulb.

39. 36. The method of claim 35, wherein the first electronic device comprises an electrical outlet.

40. 40. The method of claim 39, wherein the first command comprises a command to set an active / inactive state of the electrical outlet.

41. 41. The method of claim 40, wherein the current state of the first electronic device comprises the active / inactive state of the electrical outlet after the electrical outlet receives the first command to set the active / inactive state of the electrical outlet.

42. 36. The method of claim 35, wherein the first electronic device comprises a door key.

43. 43. The method of claim 42, wherein the first command includes a command to set the lock / unlock status of the door.

44. 44. The method of claim 43, wherein the current state of the first electronic device comprises the locked / unlocked state of the door key after the door key receives the first command to set the locked / unlocked state of the door key.

45. 36. The method of claim 35, wherein the first electronic device comprises a garage door.

46. 46. ​​The method of claim 45, wherein the first command includes a command to set an open / closed state of the garage door.

47. 47. The method of claim 46, wherein the current state of the first electronic device comprises the open / closed state of the garage door after the garage door receives the first command to set the open / closed state of the garage door.

48. 36. The method of claim 35, wherein the first electronic device comprises a thermostat.

49. 49. The method of claim 48, wherein the first command comprises a command to set a numerical value for a temperature setting of the thermostat.

50. 50. The method of claim 49, wherein the current state of the first electronic device includes the value of the temperature setting of the thermostat after the thermostat receives the first command to set the value of the temperature setting of the thermostat.

51. 36. The method of claim 35, wherein the first electronic device comprises a switch.

52. 52. The method of claim 51, wherein the first command comprises a command to set an on / off state of the switch.

53. 53. The method of claim 52, wherein the current state of the first electronic device comprises the on / off state of the electrical outlet after sending the command to set the on / off state of the switch.

54. 36. The method of claim 35, wherein the first command comprises a query regarding the current state of the first electronic device.

55. 36. The method of claim 35, wherein the one or more servers include a database that stores the name, type, and status of each of a plurality of electronic devices, including the first electronic device.

56. 56. The method of claim 55, further comprising updating the state of the first electronic device stored in the database based on the current state of the first electronic device.

57. 56. The method of claim 55, wherein the textual representation of the user utterance excludes the name of the first electronic device.

58. 56. The method of claim 55, wherein the textual representation of the user utterance excludes the type of the first electronic device.

59. 35. The method of claim 34, wherein the data corresponding to the audio input includes contextual information associated with the audio input from the user device.

60. 60. The method of claim 59, wherein identifying the first electronic device is further based on the context information.

61. 60. The method of claim 59, wherein identifying the first command is further based on the context information.

62. 60. The method of claim 59, wherein the contextual information includes an orientation of the user device when the user device received the audio input or a location of the user device when the user device received the audio input.

63. 35. The method of claim 34, wherein the user device comprises a mobile phone, a desktop computer, a laptop computer, a tablet computer, a portable media player, a television, a television set-top box, or a wearable electronic device.

64. 35. The method of claim 34, wherein the identifier of the first electronic device and the first command are transmitted to the user device, and the current state of the first electronic device is received from the user device.

65. 35. The method of claim 34, wherein the identifier of the first electronic device and the first command are transmitted to a second user device, and the current state of the first electronic device is received from the second user device.

66. 66. The method of claim 65, further comprising sending to the second user device an indication of a result of the first command based on the first command and the current state of the first electronic device.

67. identifying a second electronic device based on the textual representation of the user utterance; Identifying a second command to be executed by the second electronic device based on the textual representation of the user utterance; and transmitting an identifier of the second electronic device and the second command; 35. The method of claim 34, further comprising: receiving a current state of the second electronic device.

68. 1. A method for controlling an electronic device using a virtual assistant on a user device, comprising: receiving, by one or more servers, data from a user device corresponding to audio input including user utterances; performing a speech-to-text conversion on the data corresponding to the audio input to generate a text representation of the user utterance; determining that the textual representation of the user utterance expresses a user intent to store states of each of a plurality of electronic devices as configurations; and sending instructions to the user device to perform a query regarding the status of each of the plurality of electronic devices.

69. receiving a current state of each of the plurality of electronic devices; 69. The method of claim 68, further comprising: storing the received current state of each of the plurality of electronic devices as the configuration.

70. 1. A method for controlling an electronic device using a virtual assistant on a user device, comprising: receiving, by one or more servers, data corresponding to audio input including user utterances; performing a speech-to-text conversion on the data corresponding to the audio input to generate a text representation of the user utterance; determining that the textual representation of the user utterance expresses a user intent to change the state of each of a plurality of electronic devices based on a configuration; and sending a plurality of commands to set the state of each of the plurality of electronic devices based on the configuration.

71. 71. The method of claim 70, further comprising receiving an updated state of each of the plurality of electronic devices.

72. A non-transitory computer-readable storage medium comprising instructions for performing the method of any one of claims 1 to 71.

73. 73. The non-transitory computer-readable storage medium of claim 72; one or more processors capable of executing the instructions of the non-transitory computer-readable storage medium; A system comprising:

74. means for receiving audio input comprising user utterances; means for transmitting data corresponding to said audio input to one or more servers; From the one or more servers, an identifier of an electronic device identified by the one or more servers based on the data corresponding to the audio input; and commands to be executed by the electronic device identified by the one or more servers based on the data corresponding to the audio input; and and means for receiving the means for transmitting the command to the electronic device; means for receiving a current state of the electronic device from the electronic device after sending the command to the electronic device; means for transmitting the current state of the electronic device to the one or more servers.

75. means for receiving an audio input comprising a first user utterance; means for transmitting data corresponding to said audio input to one or more servers; means for receiving instructions from the one or more servers to perform a query regarding a state of each of a plurality of electronic devices identified by the one or more servers based on the data corresponding to the audio input; means for transmitting a status query to each of the plurality of electronic devices; means for receiving a current state of each of the plurality of electronic devices from the plurality of electronic devices; means for transmitting the current state of each of the plurality of electronic devices to the one or more servers for storage as a configuration.

76. means for receiving audio input comprising user utterances; means for transmitting data corresponding to said audio input to one or more servers; means for receiving from the one or more servers a plurality of commands for setting a state of each of a plurality of electronic devices identified by the one or more servers based on a stored configuration; means for transmitting the plurality of commands to the plurality of electronic devices.

77. means for receiving data from a user device corresponding to audio input including user utterances; means for performing a speech-to-text conversion on the data corresponding to the audio input to generate a text representation of the user utterance; means for identifying a first electronic device based on the textual representation of the user utterance; means for identifying a first command to be executed by the first electronic device based on the textual representation of the user utterance; means for transmitting an identifier of the first electronic device and the first command; means for receiving a current state of the first electronic device.

78. means for receiving data from a user device corresponding to audio input including user utterances; means for performing a speech-to-text conversion on the data corresponding to the audio input to generate a text representation of the user utterance; means for determining that the textual representation of the user utterance expresses a user intent to store states of each of a plurality of electronic devices as a configuration; means for transmitting instructions to the user device to perform a query regarding the status of each of the plurality of electronic devices; means for receiving a current state of each of the plurality of electronic devices; means for storing the received current state of each of the plurality of electronic devices as the configuration.

79. means for receiving data corresponding to audio input comprising user utterances; means for performing a speech-to-text conversion on the data corresponding to the audio input to generate a text representation of the user utterance; means for determining that the textual representation of the user utterance expresses a user intent to change the state of each of a plurality of electronic devices based on a configuration; means for sending a plurality of commands to set the state of each of the plurality of electronic devices based on the configuration.

Images