Media intercommunication over secure device-to-device communication channels

Abstract

The embodiments described herein provide a communication mechanism that enables electronic devices to perform device-to-device communications using a secure encrypted peer-to-peer data connection. The communication mechanism can also be used as a general communication mechanism that enables smart home devices to exchange data, including configuration data. In one embodiment, the general communication mechanism can be used to implement internal communication-like transmission of audio data or video data between electronic devices connected to the communication mechanism.

Description

[0001] Cross-references

[0002] This patent application claims the benefit of U.S. Patent Application No. 16 / 586,538, filed on September 27, 2019, which claims the benefit of U.S. Provisional Patent Application No. 62 / 748,224, filed on October 19, 2018, both of which are hereby incorporated by reference herein. This patent application is related to U.S. Provisional Patent Application No. 62 / 575,373, filed on October 21, 2017, and U.S. Patent Application No. 16 / 147,224, filed on September 28, 2018, both of which are hereby incorporated by reference herein to the extent they do not conflict with the present disclosure. Background Art

[0003] Electronic devices known in the art may include intelligent automated assistant systems that are capable of interfacing with users of the electronic devices. Generally speaking, intelligent automated assistant systems provide digital or virtual assistants that are capable of performing actions on electronic devices or providing requested information to users. These automated assistants can control many operations and functions of electronic devices, such as dialing phone numbers, sending text messages, setting reminders, adding events to calendars, and performing various other operations on behalf of users. The automated assistant system can receive spoken, natural language commands from a user and can speak responses to the user that are generated using a speech synthesis engine.

[0004] The services and operations of an automated assistant may be categorized into various domains that describe the automated assistant's service areas. Enabling full functionality in certain domains may require access to personal or private data associated with, or specific to, a user of an electronic device. It is recognized that use of personally identifiable information should be subject to privacy policies and practices that are recognized to meet or exceed industry or government requirements for maintaining user privacy. Specifically, personally identifiable information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use, and the nature of authorized use should be clearly stated to the user. Summary of the invention

[0005] The embodiments described herein provide a communication mechanism that enables electronic devices to perform device-to-device communications using a secure encrypted peer-to-peer data connection. The communication mechanism can also be used as a general communication mechanism that enables smart home devices to exchange data, including configuration data. In one embodiment, the general communication mechanism can be used to implement internal communication-like transmission of audio data or video data between electronic devices connected to the communication mechanism.

[0006] One embodiment provides an electronic device, which includes a network interface coupled to a bus, a microphone coupled to the bus, a non-transitory machine-readable medium coupled to the bus, and one or more processors coupled to the bus. One or more processors can execute instructions stored on the non-transitory machine-readable medium, wherein the instructions are used to provide an application programming interface (API) so that one or more processors stream internal communication media data from the electronic device to a smart home device connected via a network interface. In order to stream internal communication media data, one or more processors transmit the media data buffer to the smart home device via the network interface through a secure connection established with the smart home device. The secure connection can be established with the smart home device via the API. The media data buffer includes audio data captured via the microphone. The audio data is real-time or substantially real-time audio data that is captured while the buffer is being transmitted in terms of time. In one embodiment, the media data may also include video data captured via a camera on the electronic device.

[0007] One embodiment provides a non-transitory machine-readable medium storing instructions that cause one or more processors of a smart home device to perform operations including: receiving a request to initiate an internal communication media stream from a remote smart home device to the smart home device; verifying a secure connection between the remote smart home device and the smart device, wherein the secure connection is an encrypted peer-to-peer connection; activating a media session for the internal communication media stream, the media session being established with the remote smart home device; receiving recorded media from the remote smart home device; and playing the recorded media via an output device of the smart home device.

[0008] Other features of the present embodiment will be apparent from the accompanying drawings and from the detailed description that follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the present invention are illustrated by way of example and not limitation in the figures of the accompanying drawings in which reference numerals indicate origin numbers, like numerals indicate like elements, and in which:

[0010] Figure 1A to Figure 1B A block diagram of an electronic device system enabling a virtual assistant and a personal data domain according to an embodiment is shown;

[0011] Figure 2 A home network environment including a plurality of smart home devices according to an embodiment is shown;

[0012] Figure 3 is a flow chart of the operations of a discovery process for discovering and pairing a public device with a companion device according to an embodiment;

[0013] Figure 4shows a process of establishing a companion link connection with a paired device according to the embodiments described herein;

[0014] FIG. 5A to FIG. 5B A method of initiating internal communication media streaming from a smart home device is shown;

[0015] Figure 6 is a block diagram of a computing device for use within a smart home device according to an embodiment;

[0016] Figure 7 is a block diagram illustrating an exemplary API architecture that may be used in some embodiments of the present invention;

[0017] FIG. 8A to FIG. 8B is a block diagram of an exemplary API software stack according to an embodiment;

[0018] 9A to 9C A sequence diagram showing an API for enabling internal communication media streaming between smart home devices;

[0019] FIG. 10A to FIG. 10C A sequence diagram showing an API streaming internal communication media from a remote smart home device;

[0020] Fig.11 is a block diagram of a mobile device architecture according to an embodiment;

[0021] Fig.12 is a block diagram of a computing system according to an embodiment;

[0022] Fig.13 is a block diagram of a virtual assistant according to an embodiment;

[0023] Fig.14 A block diagram of a virtual assistant system 1400 is shown according to an embodiment described herein. DETAILED DESCRIPTION

[0024] The network of connected smart home devices may include public devices used by multiple users within the home. Because the device is a public device with multiple users, this may be beneficial to avoid storing personal information specific to any given user of the device. In addition, it may be beneficial to prevent users from performing unauthorized queries on personal data of other users of the public device. In some embodiments, some user data is considered to be inherently personal or private, while other information may be designated as personal information or private information by the user. In order to maintain the security of such data, the communication between the smart home device and the companion device can be performed using a secure device connection that enables encrypted peer-to-peer data and instant messaging channels. Exemplary personal or private user information associated with each user of the public device may include, but is not limited to, contact lists, calendar entries, text messages, emails, call histories, alarms, reminders, and other electronic data that may be considered personal or private user data. In various embodiments, additional personal or private information may include, but is not limited to, location information, family relationships, user preferences, application information, data stored on personal devices, device locations, home device locations, medical information, or health information.

[0025] Communications between public smart home devices and companion devices may be performed over a secure data channel called a companion link. The companion link provides a persistent, low-latency instant messaging system for connected devices in a home network environment. In some embodiments, the companion link supports the linking of fixed public devices within the home, such as speaker devices, with personal mobile devices that are in the home or accessible via the Internet. The companion link enables, for example, a public smart home device to redirect personal requests that the public device would not otherwise be able to handle due to privacy issues for one or more personal devices.

[0026] In addition to enabling personal queries to be redirected to the companion device, in one embodiment, the companion link also provides a universal instant messaging system for devices in the home network environment. The universal instant messaging system enables multiple smart home devices to work together by exchanging messages over the companion link. For example, audio playback between multiple smart speaker devices can be coordinated to enable the devices to perform operations such as, but not limited to, coordinating playback of media items, selectively providing audio notifications to a user via the speaker closest to the user, configuring multiple speakers into a multi-channel audio system, or coordinating audio ducking at the speaker for the duration of a verbal request and response.

[0027] The public device may advertise support for the companion link service via a discovery protocol. A personal user device on the same network as the public device may discover the companion link service advertised by the public device and connect to the public device using the advertised information. The personal device may perform a pairing process with the public device to become a companion device of the user. In one embodiment, the pairing process includes a proximity element, wherein the user device exchanges an identifier, key or secret with the companion device via a short-range wireless communication mechanism. The pairing process may also include the exchange of presence and accessibility information that may facilitate subsequent connections between the public and companion devices on the companion link.

[0028] In another embodiment, the companion link can be used to enable a media internal communication system between smart home devices. The media internal communication system can be implemented within the companion link framework without the need for a large amount of additional program logic at a higher operating system level. The library and daemon that implement the companion link communication can provide an application programming interface (API) that enables smart home devices to form an internal communication system through which audio data and / or video data can be streamed between smart home devices. The data of the internal communication media stream is encrypted and securely transmitted on the companion link. The internal communication stream initiator at one smart home device can enable real-time audio streams and / or video streams to be transmitted to multiple other smart home devices. In one embodiment, the internal communication media stream can be remotely initiated, so that the audio stream or video stream from the remote smart home device can be transmitted to the smart home device of the local or neighboring initiating user. The initiation and termination of the internal communication media stream can be performed using a virtual assistant executed on the smart home device. For example, a voice command can be provided to the virtual assistant to specify the intention of initiating an internal communication media stream to a group of remote smart home devices or from a remote smart home device to a local smart home device.

[0029] In the drawings and the following description, reference numerals indicate the figure in which the referenced element is introduced, such that an element having a reference numeral N00 is first introduced in Figure N. For example, elements having reference numerals between 100 and 199 are first shown in Figure 1, while elements having reference numerals between 200 and 299 are first shown in Figure 2. Figure 2 In the description of a given figure, previously introduced elements may or may not be referenced.

[0030] The processes and operations depicted in the subsequent figures may be performed by processing logic including hardware (e.g., circuit systems, dedicated logic, etc.), software (such as instructions on a non-transitory machine-readable storage medium), or a combination of both hardware and software. Although some processes are described below in terms of sequential operations, it should be understood that some of the operations described may be performed in a different order. In addition, some operations may also be performed in parallel rather than in sequence. In addition, some operations may be indicated as optional and not performed in all embodiments.

[0031] System and Environment

[0032] Figure 1A to Figure 1B A block diagram of an electronic device system enabling a virtual assistant and a personal data domain according to an embodiment is shown. Figure 1A A block diagram of a virtual assistant system 100 is shown according to an embodiment described herein. Figure 1B A block diagram is shown of a companion link system 130 enabling communications between devices within a virtual assistant system 100 according to an embodiment. Figure 1B The companion link system 130 can be used to establish a personal data domain within the virtual assistant system to enable a virtual assistant on a public device to answer queries that require access to personal user data that is not stored on the public device. Figure 1A to Figure 1B The block diagrams are partially logical in that some of the components shown may separate logical components that reside within the same physical server or device.

[0033] Figure 1A A virtual assistant system 100 is shown that provides a processing system that interprets natural language input received in verbal and / or textual form to infer user intent. The virtual assistant system 100 can then perform operations based on the inferred user intent. In various embodiments, the system can perform a set of operations including, but not limited to, identifying a task flow with steps and parameters designed to implement the inferred user intent, inputting specific requirements from the inferred user intent into the task flow; executing the task flow by calling a program, method, service, API, etc.; and generating an output response to the user in an audible (e.g., voice) and / or visual form. As used herein, the term virtual assistant is interchangeable with the term "digital assistant," "intelligent automated assistant," or "automatic digital assistant," and generally refers to any information processing system that interprets natural language input in spoken and / or textual form to perform actions on behalf of a user.

[0034] The virtual assistant may accept user requests in the form of at least partially natural language commands, requests, statements, narrations, and / or inquiries. Typically, a user request may seek an informative answer from the virtual assistant, or seek a virtual assistant to perform a task. A satisfactory response to a user request may be to provide the requested informative answer, perform the requested task, or a combination of the two. For example, a user may ask a virtual assistant questions such as "Where am I now?" Based on the user's current location, the virtual assistant may answer "You are near the west gate of Golden Gate Park." The user may also request to perform a task, such as "Please invite my friends to my girlfriend's birthday party next week." In response, the virtual assistant may confirm the request by saying "OK, right away," and then send a suitable calendar invitation to each of the user's friends listed in the user's electronic address book on behalf of the user. During the execution of the requested task, the virtual assistant may sometimes interact with the user in a continuous conversation involving multiple information exchanges over a long period of time. There are many other methods of interacting with a virtual assistant to request information or perform various tasks. In addition to providing verbal responses and taking programmed actions, the virtual assistant may also provide responses in other visual or audio forms (e.g., as text, alarms, music, video, animation, etc.).

[0035] like Figure 1A As shown, the virtual assistant system 100 may include a server system 108 having one or more server devices. In one embodiment, the server system 108 includes a virtual assistant server 106 that can communicate with multiple virtual assistant clients (VAC) (e.g., VAC101, VAC 102). The virtual assistant client can be executed on multiple electronic devices that can be connected and communicated with the server system 108. The first virtual assistant client (e.g., VAC 101) can be executed on a smart home device 103 (such as a smart speaker device). The second virtual assistant client (e.g., VAC 102) can be executed on a user device 104 such as a smartphone device. The virtual assistant client can communicate with the virtual assistant server 106 via one or more networks 110, which may include a combination of local and wide area networks. Various virtual assistant clients can provide client-side functions, such as user-oriented input and output processing, and can communicate with the virtual assistant server 106. The virtual assistant server 106 can provide server-side functions for multiple virtual assistant clients on any number of user devices connected via one or more networks 110.

[0036] In some embodiments, the virtual assistant server 106 includes an I / O interface 112 to the VA client, one or more processing modules 114, a storage device 116 including data and processing models, and an I / O interface 118 to external services. The client-oriented I / O interface 112 can facilitate client-oriented input and output processing of the virtual assistant server 106. One or more processing modules 114 can perform natural language processing on voice input using the data and processing model 116 to infer user intent. One or more processing modules 114 can then perform task execution based on the inferred user intent. The I / O interface 118 to external services can facilitate communication between the virtual assistant server 106 and the external service 120 through one or more networks 110. In one embodiment, the virtual assistant server 106 can communicate with the external service 120 to complete a task in response to a request received by the VAC 102 on the user device 104, or to obtain information in response to a query received at the VAC 102. The external service 120 may include, but is not limited to, a navigation service, an instant messaging service, an information service, a calendar service, and / or a telephone service, etc. For example, one or more navigation services may be used to enable turn-by-turn navigation on the user device 104 in response to a request received by the VAC 102 on the user device. External services may additionally include location information, weather, financial information, or account information. In various embodiments, the logic for enabling some external services 120 may reside within the server system 108, while some portions of the logic may reside within the user device 104.

[0037] The virtual assistant server 106 may communicate with one or more instant messaging services to send messages in response to voice input received from the virtual assistant client. Information services such as Internet search engines or information databases may be accessed by the virtual assistant server 106 to provide information to the user in response to information queries. In one embodiment, the virtual assistant server 106 may access one or more calendar services to schedule tasks or events, or to satisfy queries for tasks or events. The virtual assistant server 106 may also access one or more telephone services to initiate telephone calls for users.

[0038] In various embodiments, the virtual assistant server 106 can assist the VAC 101, 102 by processing voice input provided by the user to the client. In one embodiment, text-to-speech processing and natural language processing can be performed in part by the VAC 101, 102 and in part on the virtual assistant server 106. The data and processing models 116 of the virtual assistant server 106 can be more complex and high-performance than the corresponding models that are locally accessible to the VAC 101, 102. Therefore, the virtual assistant server 106 may be more suitable for processing some commands, and the results of the processed commands can be sent back to the VAC 101, 102 through one or more networks 110. The operations for building the virtual assistant interaction flow presented to the user can be shared between the virtual assistant server 106 and the VAC 101, 102. In one embodiment, the initial processing of the user input, the presentation of the voice output, and the access to most of the user data can be performed by the VAC 101, 102. In such embodiments, speech recognition and natural language processing can be performed at least in part by the virtual assistant server 106.

[0039] The smart home devices 103 and the user devices 104 can communicate with each other and with the server system 108 via one or more networks 110, local area networks (LANs) and / or wide area networks (WANs), such as the Internet. The one or more networks can be implemented using any known interconnection or network protocols, including various wired or wireless protocols. Exemplary interconnection and / or network protocols include Ethernet, Universal Serial Bus (USB), FIREWIRE, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Long Term Evolution (LTE), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0040] In various embodiments, the server system 108 may be implemented on a standalone data processing device and / or a distributed network of computers. In some examples, the server system 108 may also employ various virtual devices and / or services of a third-party service provider (e.g., a third-party cloud service provider) to provide the potential computing resources and / or infrastructure resources of the server system 108.

[0041] In various embodiments, smart home device 103 may be a smart speaker device, a smart home hub device, a smart home appliance, or another type of smart electronic appliance or device. For example, and in one embodiment, smart home device 103 is a smart speaker device that can be configured as part of a distributed media playback system. A specific example of a smart speaker device is the iPod Touch from Apple Inc. in Cupertino, California. Smart speaker device. In one embodiment, the smart home device 103 can be a smart home hub device configured to manage one or more other devices in a digitally connected smart home system, which includes but is not limited to smart and / or digitally connected devices such as a smart thermostat or a smart lighting system. The smart home device 103 can connect with the user device 104 to exchange data. The connection between the smart home device 103 and the user device 104 can be performed over one or more networks 110. In one embodiment, the smart home device 103 and the user device 104 can dynamically maintain a persistent connection when the user device 104 switches between a LAN and an Internet connection.

[0042] User device 104 may be any suitable electronic device. For example, and in one embodiment, user device 104 may be a portable multifunction device, a multifunction device, or a personal electronic device. A portable multifunction device may be, for example, a mobile phone that also includes other functions, such as a personal data assistant, a music player, and / or an application processor capable of executing applications (e.g., applications, etc.). Specific examples of portable multifunction devices may include the Apple iPod from Apple Inc. (Cupertino, California). Devices, iPod Equipment and Device. Other examples of portable multifunction devices may include, but are not limited to, laptop computers or tablet computers, or wearable electronic devices such as smartwatch devices. In some embodiments, user device 104 may be a non-portable multifunction device, such as, but not limited to, a desktop computer, a game console, a television, or a television set-top box. In one embodiment, user device 104 may be a light fixture in a home, vehicle, motel, etc. In some embodiments, user device 104 may be or work in conjunction with a vehicle door or another entry point to a vehicle, home, or mobile home.

[0043] User device 104 may include a touch-sensitive surface (e.g., a touch screen display and / or a touchpad interface), or one or more physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0044] In one embodiment, the VAC 101 on the smart home device 103 may communicate with the VAC 102 on the user device 104 to facilitate the processing of voice input or to infer the intent of a command received at the smart home device 103. For example, in one embodiment, the smart home device 103 may be a public device used by multiple users within a home. Certain personal or private information of the public device user, such as contact information, calendar information, message data, etc., may be classified as private to a given user. In order to avoid storing such private information on the smart home device 103 as a public device, or enabling the public device to access a server (e.g., virtual assistant server 106) that contains or accesses such information, the VAC 101 on the smart home device 103 may be associated with the user device 104 that may facilitate access to such information. When the VAC 101 on the smart home device 103 executes a command or action that requires access to private information associated with the user, the VAC 101 on the smart home device 103 may communicate with the user device 104 and / or the VAC 102 on the user device 104 to obtain information for performing the requested command or action.

[0045] A single user may be associated with multiple user devices 104. In such a case, a single instance of a user device 104 may be designated as a companion device. The smart home device 103 will then communicate with the companion device to enable processing of certain requests. For example, a user may have multiple instances of a user device 104, such as one or more smart phone devices, one or more tablet devices, or a laptop device. The user may designate one of such user devices 104 as a companion device, and the smart home device 103 will communicate with the companion device to obtain access to private user information.

[0046] In some embodiments, the smart home device 103 and the user device 104 may establish and maintain a companion link 105, which is a persistent or semi-persistent connection that enables the VAC 101 on the smart home device 103 to query the user device 104 and / or the VAC 102 on the user device 104 for private information. In one embodiment, the companion link 105 is an always-connected low-latency instant messaging system that enables communication between devices within the home. Such a connection may be maintained locally on a LAN or may be established over a wide area network such as the Internet. The companion link 105 may enable public devices to redirect personal requests to the user device so that the request can be processed in a confidential manner. The companion link 105 may also enable general device-to-device instant messaging. In one embodiment, the companion link 105 may be used to enable internal communication media streaming between smart home devices.

[0047] Figure 1B Shows Figure 1BThe companion link system 130 enables communication between devices within the virtual assistant system 100. In various embodiments, the companion link system 130 can enable secure and authenticated message exchanges between multiple devices. The multiple devices may include one or more user devices (e.g., user device 104, user device 114), each of which can be a portable multi-function device. The multiple devices may also include one or more fixed devices, such as one or more public smart home devices (e.g., smart home device 103, smart home device 133), which can be used by multiple users. In one embodiment, a set of companion links 105a-105b can enable message-based communication between a user device and one or more public smart home devices. A companion link 105c can also be established between public smart home devices to enable communication between those devices. In one embodiment, a companion link 105d can also be established between user devices to enable peer-to-peer exchange of device data.

[0048] In one embodiment, a set of companion links 105a-105b may be used to cause a public fixed device to redirect personal requests to a personal device associated with a user to allow such requests to be handled in a confidential manner. For example, and in one embodiment, a user may speak a personal query to a public device, such as "When is my next meeting?" The personal query may be received by VAC 101 or VAC 131 on smart home device 103 or smart home device 133. Because this request uses the speaker's calendar, the public device may not have direct access to such personal data. To fulfill such a request, the public device may redirect the query to a designated companion device, such as user device 104, to perform analysis on the query and generate an audio response that may be sent back to the public device for playback to the user.

[0049] In various embodiments, the redirection may be performed in an explicit or implicit manner. For explicit redirection, the VAC 101 on the smart home device 103 or the VAC 131 on the smart home device 133 may receive the request and submit the request at least in part to the virtual assistant server 106 for processing. The virtual assistant server 106 may determine that, in order to satisfy the request, at least some information may be requested from a companion device (e.g., the user device 104). In one embodiment, an explicit redirection may be performed in response to such a determination, in which the virtual assistant server 106 may send a request to the user device 104 via one or more networks 110 to retrieve personal information that will be used to process the request. In one embodiment, a public smart home device may use implicit redirection, in which the virtual assistant client on the smart home device 103 may redirect the virtual assistant request to the VAC 102 on the user device 104. The VAC 102 on the user device 104 may then process the request as if the request was received directly at the user device 104. The user device 104 may then formulate an audio response for playback on the public device.

[0050] In addition to enabling processing queries in a confidential manner, in one embodiment, the companion link 105c can enable universal instant messaging, enabling communication between public devices such as smart home devices 103 and smart home devices 133. Such links enable public devices to operate in conjunction to coordinate operations. For example, smart home devices 103 and smart home devices 133 can be configured as part of a distributed media playback system that can play music or other media. The companion link 105c established between smart home devices 103 and smart home devices 133 can be used to coordinate media playback or coordinate multi-channel audio playback on multiple devices. When a user speaks a query to one of the smart home devices, the device can exchange messages to coordinate the avoidance of audio playback and any related audio responses for the duration of the verbal request. Using the companion link system 130, any number of public smart home devices and user devices can communicate to exchange audio data, media playlists, configuration data, and other information for enabling a connected digital home system.

[0051] In one embodiment, the companion links 105a-105d may be established via a local network connection (e.g., a LAN) via a local wired and / or wireless network connection. In one embodiment, the companion links 105a-105d may also be established via a WAN connection (such as an Internet connection), although policy and configuration determinations may optionally be configured to limit the companion links 105c between the smart home devices 103, 133 to a single network. In one embodiment, in addition to the virtual assistant server 106, the server system 108 may include an online user account server 136 to which the user devices 104, 114 and the smart home devices 103, 133 may be connected via one or more networks 110. The online user account server 136 may include information associated with online user accounts of users associated with various devices. In some embodiments, the online user account server 136 may also include or be associated with server hardware and software logic components to provide online services, including online storage, instant messaging, email, media and streaming services, or navigation services. In one embodiment, the online user account server 136 may also provide or facilitate access to Figure 1B One or more external services in the external services 120. In one embodiment, the online user account server 136 may also provide or facilitate access to online stores, such as online application stores (e.g., application stores) and / or online media stores, such as music, video, or electronic book stores.

[0052] In one embodiment, the companion link 105d between the user devices 104 and 114 can be used to enable peer-to-peer data exchange. In one embodiment, automatic data exchange between devices can be configured to be performed on the companion link 105d. For example, if the user device 104 and the user device 114 are each associated with the same account on the online user account server 136, the credential exchange can be performed via the online account server to enable the automatic establishment of an encrypted communication channel between the devices whenever the devices are within the direct communication range. In one embodiment, data of the cloud service associated with the online user account server 136 can be exchanged between devices via the companion link 105d instead of through one or more networks 110. In one embodiment, the online user account server 136 can maintain a list of devices associated with a single account or a list of devices associated with an associated account family. The device list associated with the account of the account family can be used to facilitate device discovery and establish a companion link connection between devices. The accounts on the online account server can enable common authentication between electronic devices through the exchange of authentication credentials.

[0053] In one embodiment, the online user account server 136 includes or is associated with a registration server to register a unique device identifier associated with each device that has an online account associated with the user. Once the device is registered with the user's online account, an identifier token can be created that enables the device to be located and identified on the network 110. The identifier token may include one or more encryption keys, tokens, certificates, or other data that allows the virtual assistant server 106, the online user account server 136, and other servers within the server system 108 to locate and identify the device on one or more networks 110. In some embodiments, if the user device is outside the home or not connected to the home network of smart home devices, the smart home device 103, 133 can use the identifier token to establish a remote connection with the user device 104. The smart home device 103, 133 can use the identifier token to locate the user device 104 on the mobile wireless network and establish a secure remote companion link connection with the user device 104. Partly via the virtual assistant server 106 and the online user account server 136, it can help locate the user device 104 and establish a connection with the user device. In such embodiments, at least a subset of the companion device functionality of the user device 104 may continue to operate when the user device 104 is away from home, or if the user is at home but not connected to the home network.

[0054] In some embodiments, each virtual assistant client (VAC 101, VAC 102, VAC131) executed on the device may be associated with a virtual assistant identifier. In various embodiments, the virtual assistant identifier may be associated or derived with an identifier token of the host device, an account identifier associated with an online account of a user of the device, and / or another token associated or derived with an account or device identifier. When the virtual assistant clients are communicating with each other or with the virtual assistant server 106, the virtual assistant identifier of the virtual assistant client can uniquely or quasi-uniquely identify the VAC 101, 102, 131 on various devices. The virtual assistant identifier may be associated with the identification or location information of the host device of the virtual assistant, such as the identification token of the host device. After a pairing relationship is established between the smart home devices 103, 133 and the user device 104, the virtual assistant clients (VAC 101, VAC 131) on the smart home devices may store the virtual assistant identifier and use the virtual assistant identifier to locate, identify and communicate with the VAC 102 on the user device 104. The virtual assistant identifier of the VAC 102 may also be sent to the virtual assistant server 106 and stored as a companion virtual assistant identifier. If the virtual assistant server 106 requires access to the private information of the user to perform a processing operation of a request received at the VAC 101, 131 of the smart home device 103, 133, the virtual assistant server 106 may contact the VAC 102 of the user device 104 on behalf of the VAC 101, 131. In various embodiments, the virtual assistant server 106 may then receive from the VAC 102 of the user device 104 the private information from the user device 104, the processing results of the information processing task assigned to the user device 104, or the authority and / or access credentials to access the private data on behalf of the user device 104.

[0055] In some embodiments, the server system 108 includes a relay server 126 that can be used to facilitate remote connections between smart home devices 103, 133 and user devices 104. The relay server 126 can enable a relay service that can relay companion link messages between devices when a local connection or another remote connection (e.g., a remote peer connection) cannot be established. The relay server 126 can enable remote companion link instant messaging based on a relay pairing established between the smart home devices 103, 133 and the user device 104. Keys, certificates, and other identification and verification data associated with various devices can be exchanged during a local pairing process that can be used to establish a relay pairing between the devices. Once a relay pairing is established between the devices, the smart home devices 103, 133 can send a message to the relay server 126, which has a unique relay identifier associated with the user device 104.

[0056] Each device that can be connected to the relay server 126 can have an associated relay identifier, which is a permanent pairing token that can be used to identify and verify the connected device. The relay identifier can be, include, or be derived from a device identifier or identifier token that uniquely identifies the device and can include a certificate and / or signature that enables relay token verification. The relay server 126 can then relay the message to the user device 104. In one embodiment, a persistent and bidirectional connection can be established, enabling the user device 104 to send return messages to the smart home devices 103, 133 via the relay server 126. In one embodiment, the user device 104 can also initiate a connection with the smart home device 103, 133 using the relay identifier associated with the device. In one embodiment, the relay identifier is used for each message exchanged through the relay server 126, enabling the relay server 126 to verify the authenticity of each message relayed through the server and prevent unauthorized devices from transmitting messages via the established relay server connection.

[0057] Figure 2A home network environment 200 including multiple smart home devices according to an embodiment is shown. The home network environment 200 may include a wireless access point 212 to provide access to a wireless network serving the home network environment. The home network environment 200 may be part of a smart home environment. Multiple smart home devices 204, 206, 214, 216, 222A to 222B may be connected to the home network environment 200 via a wireless network or optionally a wired network connection. In various embodiments, the home network environment may include various types of smart home devices. For example, the smart home devices 204, 206, 214 may be smart speaker devices configured to distribute media playback. In addition, the smart home device 216 may be a smart appliance device, such as a smart refrigerator device. Each of the smart home devices 204, 206, 214, 216, 222A-222B may use the network of the home network environment 200 to establish an interconnection companion link to enable the devices to exchange configuration information. For example, the smart home devices 222A-222B may be configured as multi-channel smart speaker devices that may use a companion link to configure multi-channel (e.g., stereo, surround, etc.) audio playback. Additionally, each of the smart home devices 204, 206, 214, 216, 222A-222B may include a virtual assistant client that may interact with a designated companion device via a companion link to facilitate processing of the request when a request to access private user information is required. A user device such as a mobile instance of the user device 104 or a connected vehicle infotainment system may also be configured to connect to the home network environment when in proximity to the wireless access point 212. In one embodiment, the user device 104 may also create a companion link connection with any other user device that may be connected to the home network environment 200 or within direct radio range of the user device 104.

[0058] Devices and accompanying discoveries

[0059] Before establishing a companion link communication channel between the user device 104 and the smart home devices 204, 206, 214, 216, 222A-222B, a companion discovery and pairing process is performed. The companion discovery process enables the smart home device to locate the companion device, the virtual assistant client on the smart home device (e.g., Figure 1A to Figure 1BVAC 101, 131 in the companion device) accesses private user information that can be used to process and / or respond to user requests through the companion device. In some cases, the companion discovery process may also include user authentication that communication will occur between the smart home device and the companion device. In some embodiments, companion discovery can utilize existing service discovery protocols that facilitate locating devices and / or services on wireless or other networks, such as the Simple Service Discovery Protocol (SSDP) developed by the UPnP Forum or the Bonjour network technology developed by Apple (published as IETF RFC 6762 and IETF RFC 6763, referred to herein as "Bonjour"). In the device discovery service, a device can advertise information indicating its existence, address, and optionally additional information about its functionality. Other devices, including other smart home devices or user devices, can browse the announcement and identify the device of interest based on the broadcast information. Using the announcement address, the browsing device can initiate communication with the announcer.

[0060] Depending on the network and the discovery service, the announcement may optionally include real-time broadcast information (e.g., via multicast or beacon signals) and / or provide announcement information to a central repository (e.g., at a network access point) from which other devices can retrieve the information. Announcement browsing may include detecting broadcast announcements and / or retrieving announcement information from a central repository. In some embodiments, a public smart home device that is fixedly attached to a power source (e.g., a power outlet) may continuously perform announcements and discovery of companion link services. Mobile user devices may enable discovery of companion link services based on the location of the user device. For example, and in one embodiment, the geo-fence boundary 203 is configured on the mobile device so that companion link discovery is enabled when the mobile device is located in a geographic location adjacent to a location designated as the home location of the user device 104.

[0061] When a public smart home device is discovered by a user device (e.g., user device 104) acting as a companion device, a network data connection (e.g., TCP protocol, UDP protocol, etc.) may be established between the public smart home device and the companion device. The network data connection may be established using any network layer (e.g., layer 3) protocol. To avoid connection competition between devices that are both announcing and discovering, the device with the lower permanent identifier in lexicographic order initiates the connection between the devices. The permanent identifier of the device is derived from the anonymous identifier announced by the discovery service. In one embodiment, in order to derive the persistent identifier based on the announcement information, data exchanged through a previously performed pairing process is utilized. In such embodiments, a data connection cannot be established using a smart home device until the smart home device is paired with the user device because the permanent identifier used to connect to the public smart home device is unknown. Once the data connection is established, a secure communication session may be established between the public smart home device and the companion device. The public smart home device and the connected companion device may then exchange presence and accessibility information. If the companion device is a mobile device, the companion device may then enter a wake-on-wireless (WoW) state as needed when no data is exchanged on the companion link, while the common device connected to a power source may remain active to reduce first message latency.

[0062] In the event that the connection between devices is lost, the discovery service can be re-enabled and used to search for devices in the local network. If the lost device is rediscovered, the data connection between the devices can be re-established. If the lost device cannot be discovered, the status discovery service information in the lost device record is reconfirmed and cleared. The search device may then attempt to establish communication with the lost device via a secure Internet session. In one embodiment, a portion of the presence and accessibility information exchanged when establishing a data connection includes a device identifier, an identifier token, a relay identifier, or other forms of identification tokens that can be used to connect or enable message exchange with a lost device, such as via a peer or relay Internet connection. If a secure Internet connection can be successfully established using a previously lost device, accompanying link messages can be exchanged over the secure Internet connection.

[0063] In the case where the companion device is connected to the smart home device via an Internet-based connection and a local connection becomes available, the companion link connection may be switched to the local connection. For example, the user device 104 may cross the geo-fence boundary 203 so that a discovery protocol (e.g., SSDP, Bonjour, etc.) can search for devices (e.g., smart home devices 204, 206, 214, 216, 222A-222B). If the user device 104 discovers the availability of a local connection to the smart home device, the user device 104 is connected to the smart home device via an Internet connection, and the user device may convert the Internet connection to a local (e.g., Wi-Fi) connection. In one embodiment, a connection switch may be performed whenever connectivity is lost between connected devices (e.g., the user device 104 leaves the home network environment 200) or if the device determines that a better connection is available (e.g., the user device 104 returns to the home network environment 200). A local connection may be preferred to an Internet connection because the local connection may have lower latency. In addition, from the perspective of the device and infrastructure, maintaining a local connection rather than an Internet connection may take up fewer resources.

[0064] Figure 3 1 is a flowchart of the operation of a discovery process 300 for discovering and pairing electronic devices according to an embodiment described herein. The discovery process 300 shown includes operations performed on an electronic device 302 and an electronic device 304. The electronic device 302 and the electronic device 304 can each be any smart home device described herein (e.g., the smart home devices 103, 133 shown in FIG. 1 , as shown in FIG. 2 ). Figure 2 204, 206, 214, 216, 222A to 222B) as shown, or any personal user device described herein (e.g., user device 104), such as but not limited to a smart phone device, a tablet device, or another user device that can be connected to another device via a companion link as described herein. For the purpose of illustrating the discovery process, electronic device 302 will act as a public device, and electronic device 304 will act as a personal device or companion device.

[0065] In one embodiment, as shown in block 310, the electronic device 302 may set a status bit to indicate that the device is not currently paired, or in other words, is looking for a companion device to pair with. The status bit may be a bit in a status flag indicator listed in the status information advertised by the electronic device 302. At block 312, the electronic device 302 may advertise its presence via a discovery protocol (e.g., SSDP, Bonjour, etc.) because the companion link service is supported. For example, using Bonjour, the electronic device 302 may advertise itself with a name and a service type. The name may be a user-readable name of the companion discovery (e.g., "speaker"); in some instances, the advertised name may be a name specified in the companion discovery information service instance of the device definition record. The service type may be defined for a unified attachment protocol (e.g., service type "_companion-link._tcp"). The announcement may also include additional information. Similar information may be distributed using other service discovery protocols and technologies. For example, using SSDP, companion discovery may include announcing the name and service type URI using a multicast HTTP notification message. The URI may be used by the electronic device 304 to retrieve additional information via a unicast request to the electronic device 302 .

[0066] After the electronic device 302 begins advertising the companion link service via the service discovery protocol, the electronic device 304 may discover public devices while browsing for unpaired devices, as shown at block 314. No specific timing is required between the start of the advertising and the start of the service browsing, but while the electronic device 304 is browsing, the electronic device 304 will not be able to discover the electronic device 302 unless the discovery advertisement is detectable.

[0067] In one embodiment, the electronic device 304 may browse for unpaired devices in response to a trigger, such as a trigger provided by an application executing on the electronic device 304. In one embodiment, the electronic device 304 may browse for unpaired devices when the electronic device 304 is placed in physical proximity to the unpaired device. For example, an out-of-the-box proximity setting for a public device may include data exchange over a short-range wireless communication mechanism (e.g., using Bluetooth and / or Bluetooth low energy, NFC, etc.), which may trigger the electronic device 304 to browse for unpaired devices.

[0068] At block 316, the electronic device 304 may find the device via the discovery service advertisement, for example, by detecting the advertisement performed at block 312. At block 318, the electronic device 304 may determine whether the discovered device is a connection candidate, such as an unpaired electronic device 302, based on the advertisement. In one embodiment, the electronic device 304 may determine that the discovered device is a connection candidate when the discovered device is advertising a companion link service. The electronic device 304 may check the discovery status flag to determine whether the electronic device 302 has been configured with a companion device or paired with a companion device. As another example, the electronic device 304 may check the protocol version of the advertisement to determine whether the companion link protocol version of the electronic device 302 is compatible with the companion link protocol supported by the electronic device 304. If the electronic device 304 determines that the accessory is not a companion device that is being advertised for establishing a companion link with it, the electronic device 304 may return to block 314 and continue browsing. In one embodiment, after a period of time in which no public device is discovered, the browsing operation may time out and browsing of unpaired devices may be interrupted for a period of time.

[0069] At box 322, the electronic device 304 may present information about the electronic device 302 to the user via a user interface such as a display device of the electronic device 304. At box 324, the electronic device 304 may receive input from the user via the user interface about the action performed using the detected device. For example, the user may provide input indicating whether the electronic device 304 should establish a pairing with a public device. The electronic device 304 may present any or all of the information obtained from the notification data provided by the public device, and prompt the user to indicate whether the electronic device 304 should be connected to the electronic device 302. Requesting user confirmation may help avoid false or unwanted pairing between public devices and personal devices. At box 326, the electronic device 304 may interpret the user input received at box 324 and determine whether to pair with the electronic device 302. In addition to the user's approval to initiate the pairing operation, other operations may be performed to complete the pairing operation to minimize the risk of pairing occurring without the approval of the legitimate owner / operator of the electronic device 304. For example, the electronic device 304 and / or the electronic device 302 may request a password known to the user of the electronic device 304 to be entered. In one embodiment, biometric verification (eg, fingerprint, facial recognition, etc.) may be requested by the electronic device 302 and / or the electronic device 304 to complete the pairing operation at block 326 .

[0070] If the user indicates that the electronic device 304 refuses pairing or the user verification operation fails, the electronic device 304 can return to frame 314 to find other accessories or devices. If the electronic device 304 and the electronic device 302 are paired at frame 328 and frame 330, the electronic device 304 and the electronic device 302 can perform the pairing setup process respectively. In some embodiments, the pairing setup process can be used to establish an encryption key to facilitate secure communication between the electronic device 304 and the electronic device 302. In some embodiments, user confirmation can be incorporated into the pairing setup process, and there is no need to perform a separate user confirmation before starting the pairing setup. In one embodiment, the pairing setup process enables a trusted relationship to be established between the electronic device 302 and the electronic device 304. The established trusted relationship can be verified later during the setup of the secure communication session.

[0071] If the pairing setup process is successfully completed, at box 331, electronic device 302 can update device status information to indicate that authorization is now required to communicate with the accessory and / or that the accessory is now paired with at least one personal device, such as by updating a status flag indicator within the notification device data.

[0072] At block 332, the electronic device 304 may obtain and cache a device definition record from the electronic device 302, which the electronic device 302 may provide upon request at block 334. The device definition record may include a set of services supported by the device and / or other relevant characteristics that enable other electronic devices 304 and other connected devices to determine how to control, connect, or otherwise interact with the electronic device 302. In the event that the electronic device 304 caches the device definition record, the information may be used to facilitate detecting a change in state of the electronic device 302. In some embodiments, the electronic device 304 may also cache information from the notification data provided by the electronic device 302.

[0073] At blocks 336 and 338, electronic device 304 and electronic device 302 may begin exchanging data for establishing a secure communication channel. The data exchange may include a key or certificate exchange between devices. The key exchange may be protected by a shared key exchanged between devices, wherein the shared key may be exchanged using an out-of-band communication method. In addition, the data exchange may include an exchange of one or more long-term keys between devices, which may themselves be protected by one or more short-term keys. Once a pairing is established, the pairing may be used to provide end-to-end message encryption so that only paired devices can read messages exchanged between devices. In one embodiment, the secure communication channel is a two-way channel so that either device that does not communicate with the channel initiates a message exchange. During the message exchange, the device that initiates the communication session is called a client device, and the device that accepts the session is called a server device. In one embodiment, the first message exchanged between devices is an information exchange message. The client device may send an initial information message including a feature flag or other device information. The server device may then respond with an information message for supported functions. Once a secure communication channel is established and an information exchange occurs, if there is no pending immediate operation, the communication device may enter an idle state.

[0074] It should be understood that the discovery and pairing process described herein is exemplary, and variations and modifications are possible. The operations described in sequence may be performed in parallel, the order of the steps may be changed, and the steps may be modified, merged, added or omitted. In addition, although SSDP and Bonjour services are used as examples of device discovery services, similar concepts may be applied in the context of other device discovery services. In some embodiments, before determining whether to pair with the electronic device 302 or any other discovered device, the electronic device 304 may request a definition record (or a portion thereof) from a pairing candidate device, which may be requested, for example, via an HTTP request. Depending on the configuration, the pairing candidate device may provide all, some or none of its accessory definition records in response to a request from an unpaired electronic device. Alternatively, the definition record may be accessible before pairing is established and the decision whether pairing may be based on the announcement information provided by the pairing candidate device. The announcement data may include, for example, a local name of the device; a unique attachment identifier; a flag indicating that the attachment is discoverable; a universally unique identifier (UUID) for at least some services; an indicator of the device state; and an indication of whether the device has been paired with any other electronic device.

[0075] In some embodiments, the discovery process 300, or a similar process, may be used to detect a change in the state of a paired device. For example, when a device state changes, the state value advertised by the device may be incremented. When a device (e.g., electronic device 302) announces a state change, other paired devices (e.g., electronic device 304) may announce the state change, for example, by broadcasting an updated Bonjour TXT record, and paired electronic devices that previously cached the device record may detect the change by comparing the broadcast value of the state value with the cached value.

[0076] The secure communication channel established between the devices at box 336 and box 338 can be used to exchange companion link messages. In one embodiment, before the companion link messages are exchanged through the companion link, a secure session is established or re-established between the electronic device 302 and the electronic device 304. In one embodiment, establishing the session includes performing a pairing verification process to verify the pairing between the devices. The pairing verification process relies on the previously established pairing relationship between the electronic device 302 and the electronic device 304. For example, and in one embodiment, establishing a companion link session between devices may require a permanent identifier of the device, which may be derived from an anonymous identifier advertised through a discovery service. However, in order to derive a persistent identifier based on the advertised information, one or more data elements exchanged during the pairing process between the devices can be used. In one embodiment, the pairing verification process may include generating a short-term public key pair for encrypting or signing messages exchanged during the verification process, and verification of a previously exchanged long-term key pair exchanged during the initial pairing of the devices. The pairing verification process may be performed each time a secure session is established between devices using a companion link connection.

[0077] Figure 4 A process 400 is shown for an electronic device to establish a companion link connection with a paired device according to embodiments described herein. Process 400 shows operations in which an electronic device will initiate a connection with another electronic device via a companion link. In one embodiment, during a message exchange via a companion link, regardless of whether the device initiating a communication session with respect to the companion link is a client device and the device accepting a session with respect to the companion link is a server device, any electronic device connectable via a companion link may each initiate or accept a companion link connection.

[0078] At block 402, a determination may be made at a client electronic device that a communication session with another electronic device (e.g., a server electronic device) should be established via a companion link. This determination may be made by any electronic device supporting a companion link as described herein to initiate a data transfer session between devices. A connection may be established at a client device upon device initialization to reestablish a disconnected connection or a connection that has timed out due to non-use.

[0079] At box 404, the client device may attempt to discover other electronic devices via a local network. Discovery may be performed using a discovery service as described herein, including but not limited to Bonjour or SSDP. If other devices are discovered locally at box 406, process 400 continues at box 416, where the client electronic device may attempt to establish a local network connection with the other electronic device. The local network connection may be established via a network protocol such as, but not limited to, the Transmission Control Protocol (TCP). If no other electronic devices are discovered locally, the client electronic device may determine at box 408 whether a remote identifier is available. The remote identifier may be an identifier or identification token that enables an electronic device or other companion link client to locate, connect, and establish a link session with a paired device that does not reside on the same network or is not otherwise discoverable via a discovery service protocol. During a local device discovery and / or pairing process (e.g., such as Figure 3 During the presence and reachability information exchange that occurs during the discovery process 300 in , one or more remote identifiers may be exchanged between companion link devices. Exemplary remote identifiers include virtual assistant identifiers associated with virtual assistant clients on other electronic devices that can be used to communicate with a virtual assistant server (e.g., such as a virtual assistant server). Figure 1B 106); via the use of an online user account server (e.g., Figure 1B and an identifier token established by registering with an online account server 136 in the example; and an identifier token established by registering with a relay server (e.g., Figure 1B The electronic device may use the available remote identifier to query a network address, such as an Internet IP address, which may be used to connect to other electronic devices or to exchange relay messages with other electronic devices via a relay service. At block 418, the electronic device may use the remote identifier to establish a remote network connection with other electronic devices.

[0080] Whether the connection is established via a local network connection at block 416 or via a remote network connection at block 418, the client electronic device can verify pairing with the other electronic device at block 420. A verified pairing is established, and a true pairing relationship exists between the devices annotated by the companion link and utilizing data exchanged during a previous local pairing. In one embodiment, a pairing verification process is performed in which each device demonstrates possession of a long-term private key corresponding to a long-term public key exchanged during pairing setup. The pairing verification process may additionally include establishing a new shared key or session key that can be used to encrypt communications occurring during the pairing verification session.

[0081] At block 422, the other electronic device may determine whether the pairing verification process was successful. If the pairing verification process at block 420 fails to verify the pairing relationship between the electronic devices, the client electronic device will not be able to establish a secure communication session because the security of the communication link may have been compromised. In one embodiment, the client electronic device may notify the user when the secure connection verification with another electronic device fails.

[0082] If the electronic device determines at block 422 that the pairing has been successfully authenticated, the client electronic device may establish an authenticated companion link session with the virtual assistant client on the electronic device via the companion device to service a query to access private data of a user of the companion device at block 424. The authenticated session may enable end-to-end encrypted message exchanges between the devices, enabling messages and data to be exchanged between the devices in a confidential manner. The encrypted message exchanges may be performed to enable, for example, relaying of commands or responses of the virtual assistant with respect to personal or private user data, synchronizing device data between electronic devices, or establishing a secure internal communication media stream with another electronic device.

[0083] As described above, although process 400 is described and illustrated using a client electronic device initiating a connection with another (e.g., server) electronic device, once established, a verified companion link session is a two-way connection that enables a two-way message exchange between the devices. In addition, any mobile device participating in the session may enter an awake radio state when no data is exchanged on the companion link, such that the device may enter a low power state while the radio and associated processor maintain a connection associated with the session.

[0084] Media internal communication on companion link

[0085] FIG. 5A to FIG. 5B A method of initiating internal communication media streaming from a smart home device is shown. Figure 5A A method 500 of streaming media from a local smart home device to a set of remote smart home devices is shown according to an embodiment. Figure 5B A method 510 of initiating streaming from a remote smart home device to a local smart home device according to an embodiment is shown. The smart home device may be, for example, a smart speaker device, a smart home appliance, or another type of smart home device or smart electronic appliance (e.g., a thermostat, a refrigerator, etc.), as described herein. A local smart home device refers to a device that is proximate to an initiating user, while a remote smart home device refers to a device located in a different location, such as a different room in a home or someone else's home that is linked to the initiating user's home.

[0086] like Figure 5AAs shown, method 500 includes a smart home device such as a local smart home device receiving a request to initiate an internal communication media stream to a group of smart home devices (box 501). The group of smart home devices can be remote smart home devices located in a different location from the local smart home device and the initiating user. The request for initiating a media stream can be received in response to touch input, text input, voice input, or another form of input (such as a physical button press). The input can be provided to the local smart home device or another electronic device connected to the smart home device. The voice input can be received and processed via a virtual assistant system as described herein. The request for initiating a media stream can specify the destination of the media stream and the type of media to be streamed. In one embodiment, the source of the media stream can be determined based on the location of the user who is issuing the command. In one embodiment, the source of the media stream can be explicitly stated by the user.

[0087] For example, a user may provide a voice command "Let me talk to the kitchen", and the hardware, firmware, and / or software logic components of the neighboring user on the local smart home device may receive the request. In one embodiment, the voice input may be received and processed by a virtual assistant system as described herein. The request may be interpreted as a request to open a media (e.g., audio) stream from a local smart home device to a smart home device that has been designated as being located in the kitchen at home. Alternatively, the user's electronic device, such as a wearable electronic device or a mobile electronic device, may receive the voice command and relay the command or an associated request based on the command to the local smart home device. In one embodiment, a streaming request may be initiated using an application on a mobile electronic device, a wearable electronic device, or a smart home device. The user may provide a command via an application via voice, touch, or text input, and the command will be interpreted as a request to initiate an audio and / or video stream from a designated local device to a designated destination.

[0088] Once a request indicating or implying a media stream type and a media stream destination is received, method 500 includes the initiator device verifying a secure connection to a specified smart home device group (box 502). Verification of the secure connection may include, for example, verifying a previously established pairing relationship via a pairing verification process described herein. A secure connection is a verified encrypted network connection, such as a companion link connection as described herein. Method 500 then includes the local smart home device activating a media session for an internal communication media stream with one or more smart home devices in a specified smart home device group, for which a secure session can be verified (box 503). Method 500 also includes the local smart home device activating a microphone and / or camera at the local smart home device (box 504). If there are multiple smart home devices local to the initiator, one of the multiple smart home devices may be selected. For example, the nearest smart home device may be selected, or for a video stream, the nearest smart home device with a camera may be selected. Alternatively, if a local source device is specified in the request received at box 501, the microphone and / or camera of the specified local source device may be used to capture media of the media stream.

[0089] The method 500 also includes operations for performing a real-time media streaming loop in which recorded video and / or audio chunks from a local smart home device are transmitted to the group of smart home devices (block 505). The recorded video and / or audio chunks may be buffers of real-time or substantially real-time (e.g., slightly delayed) media data recorded while being transmitted to the group of remote smart home devices. The real-time recording and transmission may continue until the method 500 determines that the streaming media session is complete (block 506). The method 500 may then end the media session with the group of smart home devices (block 507). The method 500 may determine at block 506 that the streaming media session is complete based on a command received from the initiator to end the media session, or via receipt of a touch input, a physical button press, or another indicator to end the media session.

[0090] In addition to streaming media from a local smart home device to one or more remote smart home devices, if remote streaming is enabled for the remote smart home device, the user can initiate media streaming from the remote smart home device to the local smart home device. Remote initiation of streaming can be enabled or disabled in the settings of a given smart home device.

[0091] like Figure 5BAs shown, method 510 includes receiving a request to initiate an internal communication media stream from a smart home device, which may be a remote smart home device (box 511). The request may be relayed from a local smart home device to a remote smart home device. The request may be generated based on a command provided to a local smart home device, a mobile electronic device, or a wearable device associated with the initiating user. The command may be a voice or text command, or may be based on a touch input or a physical button. For example, the initiating user may provide a voice command such as "Let me hear what's going on in the baby's room," which may be interpreted as a request to initiate audio streaming from a smart home device in a location designated as "baby's room." The audio may be streamed to a smart home device adjacent to the initiating user, or in one embodiment, may be streamed to a smart home device designated by the initiating user. In one embodiment, the initiating user may provide a voice command such as "Let me see what's going on in the baby's room," and if a smart home device with a camera function is located in a location designated as "baby's room," a video stream (which may also contain audio) may be enabled. In one embodiment, the voice command may be processed by a virtual assistant system as described herein, which may generate an appropriate request.

[0092] Upon receiving the request indicating the media stream type and the source of the media stream and, in one embodiment, the destination of the media stream, method 510 includes the initiating device verifying a secure connection between the remote smart home device and the local smart home device that will receive the media stream (block 512). Verification of the secure connection may include, for example, verifying a previously established pairing relationship via a pairing verification process described herein. A secure connection is a verified encrypted network connection, such as a companion link connection as described herein. Method 510 also includes verifying that remote initiation of internal communication streaming has been enabled for the remote smart home device (block 513). Remote initiation of streaming may be enabled or disabled in the settings of a given smart home device, and if remote initiation of streaming is disabled, a remote media streaming session will not be established.

[0093] Method 510 also includes the local smart home device activating a media session for an internal communication media stream between the remote smart home device and the local smart home device (box 514). When the media session is activated, the remote smart home device may activate a microphone and / or a camera (box 515). A real-time media streaming loop may then be performed, wherein recorded video and / or audio blocks transmitted from the remote smart home device are received at the local smart home device (box 516). The recorded video and / or audio blocks may be data buffers of real-time or substantially real-time media data that are recorded while the data buffers are being transmitted to the local smart home device. Real-time recording and transmission may continue until the local smart media device determines that the streaming media session is complete (box 517). Method 510 also includes the local smart media device ending the media session with the group of smart home devices (box 518). The local smart media device may determine that the streaming media session is complete based on a command received from the initiator to end the media session at box 517, or via receiving a touch input, a physical button press, or another indicator to end the media session.

[0094] Computing devices for smart home devices

[0095] Figure 6 6 is a block diagram of a computing device 600 for use in a smart home device according to an embodiment. The computing device 600 includes one or more speaker devices 601 to enable media playback. In the case where the computing device 600 is implemented as a smart speaker device, the speaker device 601 may have a higher quality than when the computing device is implemented as a user device or a smart appliance.

[0096] The computing device 600 includes a network interface 602 that supports network communication functions. The network interface 602 can be coupled to one or more wireless radio components 603 to enable wireless communication through one or more wireless networking technologies (such as but not limited to Wi-Fi and Bluetooth). In some specific implementations, the network interface 602 can also support wired network connections. The computing device also includes a processing system 604 having multiple processor devices and a system memory 610, which can be a virtual memory system with an address space, including volatile memory and non-volatile memory.

[0097] In one embodiment, the processing system 604 includes one or more application processors 605 to execute instructions of user and system applications executed on the computing device 600. The processing system may also include a sensor processor to process and monitor a suite of sensor devices 608, which have sensors including, but not limited to, motion sensors, light sensors, proximity sensors, biometric sensors, audio sensors (e.g., microphones), and image sensors (e.g., cameras). The sensor processor 606 may enable low-power monitoring of always-on sensors within the suite of sensor devices 608. When the computing device 600 is not in active use, the sensor processor 606 may allow the application processor 605 to remain in a low-power state while still allowing access to the computing device 600 via voice or gesture input to the virtual assistant 620. In one embodiment, the sensor processor 606 or a similar low-power processor within the processing system may enable low-power processing of media instructions provided by the media player 625. The media player 625 may be a modular media player capable of playing back a variety of different audio and / or video media types, including, but not limited to, MPEG-2, MPEG-4, H.264, and H.265 / HEVC. In one embodiment, other formats may be supported via additional codec plug-ins.The computing device may optionally include a display device (not shown), or may output video to other devices for display using a wireless display protocol.

[0098] Virtual assistant 620 is logic that executes on computing device 600 to provide the intelligent automated assistant system described herein. Virtual assistant 620 may selectively allow access to various software and hardware components within the computing device, including, but not limited to, a network interface 602 for retrieving data via a network, a media playback application for initiating or stopping playback of media files, or a user's calendar data for scheduling calendar events, tasks, reminders, or alarms. When virtual assistant 620 is executed on a smart home device as described herein, the virtual assistant may interact with a companion link module 622 to locate and connect to a companion device or a virtual assistant executing on a companion device upon receiving a request to access the user's personal domain data. Virtual assistant 620 may also interact with a streaming initiator 621 to initiate media streaming to and from computing device 600, such as FIG. 5A to FIG. 5B shown.

[0099] If the computing device 600 is a smart speaker device capable of participating in a distributed playback system, the distributed playback module 612 may perform operations to manage various aspects of media playback, including, but not limited to, a playback queue manager 614 for managing a list of media played via the distributed playback system and a playback routing manager 616 for routing media playback to specific elements of the distributed playback system. In one embodiment, the playback routing manager 616 may connect to different elements of the distributed playback system via connections established using a companion link module 622. The companion link module 622 may facilitate connection establishment and message relaying over a companion link established between a speaker and a device of the distributed playback system to perform operations such as configuring channel outputs for a multi-channel playback system or coordinating volume adjustments across multiple connected speakers. In one embodiment, the companion link module 622 may also perform operations to enable media intercommunication over a companion link connection between the computing device 600 and other smart home devices.

[0100] In one embodiment, event scheduler 630 can exchange data with distributed playback module 612. Data exchange can be performed in response to input received via user interface of computing device 600 or different computing devices participating in distributed playback system. Data exchange can also be performed in response to activity requested via virtual assistant 620. For example and in one embodiment, events scheduled via event scheduler 630 can be associated with media playlists so that when a scheduled event occurs, the playlists can be played via playback queue manager 614. For example, an alarm can be scheduled to wake up a user at a specific time. An alarm can be associated with a playlist so that one or more media elements will be associated with an alarm sound or used as a replacement for an alarm sound to play. In one embodiment, a playlist can be associated with any event scheduled via event scheduler 630, including reminders or timer expiration events.

[0101] In one embodiment, playback queue manager 614 can manage multiple simultaneous playback queues, where a playback queue includes one or more past, current, or future media elements to be played via computing device 600. A playback queue can load a single media element or specify a playlist of multiple media elements. A playback queue can include locally stored media, media to be retrieved for playback via a media server, or media to be streamed from a local or remote media streaming server. Various types of media elements can be played via the distributed playback system by the playback queue manager, including multimedia files, etc., such as, but not limited to, music, music videos, and podcasts, including audio or video podcasts, or audio and / or video clips of current news, weather, or sporting events.

[0102] If one or more podcasts are selected for playback in response to the occurrence of a scheduling event, the podcast selection logic may select a particular episode of the podcast for playback, such as the most recently available podcast episode or the most recently available unplayed podcast. Such selection may be determined by explicit user preferences or based on learned user preference information. The selection may also be performed based on the age of the available unplayed podcasts relative to the current date. In one embodiment, the podcast feed includes metadata indicating whether the podcast feed is associated with a serial podcast or a news-style podcast. Whether to play the oldest unplayed episode or the most recent episode is determined based at least in part on such metadata.

[0103] In one embodiment, for example, when setting a wake-up alarm, a news program or news channel may be selected for playback. The user may select a specific program or channel for playback in response to the wake-up alarm. Alternatively, the user may select a general news category and logic associated with the playback queue manager 614 may select a news program or news channel to play based on the selected user preferences. In one embodiment, news program preferences may be determined based on news topic preferences selected in a news program on a user device.

[0104] In one embodiment, playback routing manager 616 can be used to select playback devices in a distributed playback system for playing playback queues. Depending on the number of playback devices in the distributed playback system, multiple different queues can be active on multiple different playback devices or multiple different playback devices in the distributed playback system can be grouped. Grouped playback devices can share a common playback queue and play the same media at the same time. When providing intelligent playback devices, playback devices can be associated with one or more users and / or one or more user accounts. Intelligent playback devices can also be assigned locations and / or device types. In one embodiment, a residential distributed playback network can be configured, wherein multiple user devices play media via one or more intelligent playback devices in the residence. When intelligent playback devices are added to a residential network, the room or location of each playback device can be specified. Ownership can also be specified for each intelligent playback device, indicating whether the intelligent playback device is associated with a single user or whether the intelligent playback device is a smart home device associated with multiple users.

[0105] As described above, one aspect of the present technology is the collection and use of data obtained from specific and legitimate sources for use with a virtual assistant, as described herein. The present disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include demographic data, location-based data, online identifiers, phone numbers, email addresses, home addresses, data or records related to a user's health or fitness level (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other personal information.

[0106] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used to allow the user to access calendar or reminder data via a virtual assistant. Allowing the virtual assistant to access contact data can enable the virtual assistant to send messages or initiate phone calls. In addition, the present disclosure also anticipates other uses of personal information data that benefit the user. For example, health and fitness data can be used according to user preferences to provide insights into their overall health, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0107] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities will be expected to implement and consistently apply privacy practices that are generally recognized as meeting or exceeding the requirements of the industry or government that maintain user privacy. Such information about the use of personal data should be accessible to users in large quantities and conveniently, and should be updated as the collection and / or use of data changes. The user's personal information should be collected only for legitimate use. In addition, such collection / sharing should only occur after receiving user consent or other legal basis specified in applicable law. In addition, such entities should consider taking any necessary steps to defend and safeguard access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted for specific types of personal information data collected and / or accessed, and applied to applicable laws and standards, including specific considerations that are exclusive to jurisdictions that can be used to impose higher standards. For example, in the United States, collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0108] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, with respect to smart home devices that have access to personal data, the technology of the present invention may be configured to allow users to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for the service. As another example, a user may choose not to enable certain elements of access to personal or private data from a smart home device. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0109] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at a city level rather than at an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods such as differential privacy, where appropriate.

[0110] Therefore, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or part of such personal information data. For example, the enabling of some smart home device functions may be based on aggregated non-personal information data or an absolute minimum amount of personal information, such as content processed only on the user's device or other non-personal information that may be available.

[0111] Exemplary Application Programming Interface Architecture

[0112] Embodiments described herein include one or more application programming interfaces (APIs) in an environment, wherein the calling program code interacts with other program codes called by one or more programming interfaces. Various function calls, messages, or other types of calls may also include various parameters, which may be transmitted via the API between the calling program and the called code. In addition, the API may provide the calling program code with the ability to use data types or categories defined in the API and implemented in the called program code.

[0113] An API allows developers of API-calling components (which may be third-party developers) to utilize specified features provided by an API-implementing component. There may be one API-calling component or there may be more than one such component. An API may be a source code interface provided by a computer system or a program library to support service requests from an application. An operating system (OS) may have multiple APIs to allow applications running on the OS to call one or more of those APIs, and a service (e.g., a program library) may have multiple APIs to allow applications using the service to call one or more of those APIs. An API may be specified in a programming language that can be interpreted or compiled when building an application.

[0114] In some embodiments, the API implementation component may provide more than one API, each API providing a different view or having different aspects that access different aspects of the functionality implemented by the API implementation component. For example, one API of the API implementation component may provide a first set of functions and may be exposed to third-party developers, and another API of the API implementation component may be hidden (not exposed) and provide a subset of the first set of functions, and also provide another set of functions, such as testing or debugging functions that are not in the first set of functions. In other embodiments, the API implementation component itself may call one or more other components via the underlying API, thereby being both an API calling component and an API implementation component.

[0115] The API defines the language and parameters used by the API calling component when accessing and using the specified features of the API implementation component. For example, the API calling component accesses the specified features of the API implementation component through one or more API calls or references exposed by the API (for example, implemented by function or method calls), and uses parameters to pass data and control information via API calls or references. The API implementation component can return values ​​through the API in response to API calls from the API calling component. Although the API defines the syntax and results of API calls (for example, how to cause API calls and what API calls can do), the API may not reveal how the API call completes the function specified by the API call. Various API calls are transmitted via one or more application programming interfaces between the call (API calling component) and the API implementation component. Transmitting API calls may include issuing, initiating, referencing, calling, receiving, returning or responding to function calls or messages; in other words, the transmission can describe the actions of either the API calling component or the API implementation component. Function calls or other references of the API can send or receive one or more parameters through parameter lists or other structures. A parameter may be a constant, a key, a data structure, an object, an object class, a variable, a data type, a pointer, an array, a list, or a pointer to a function or method or another way of referencing data or other items to be passed via an API.

[0116] In addition, data types or classes can be provided by the API and implemented by the API implementation component. Therefore, the API calling component can use the definitions provided in the API to declare variables, use pointers to such types or classes, and use or instantiate constant values ​​of such types or classes.

[0117] Typically, API can be used to access the services or data provided by the API implementation component, or to start the operation or calculation provided by the API implementation component. By way of example, the API implementation component and the API call component can each be any one of an operating system, a library, a device driver, an API, an application, or other modules (it should be understood that the API implementation component and the API call component can be modules of the same or different types). In some cases, the API implementation component can be implemented at least partially in firmware, microcode, or other hardware logic components. In some embodiments, the API can allow client programs to use the services provided by the software development kit (SDK) library. In other embodiments, the application or other client programs can use the API provided by the application framework. In these embodiments, the application or client program can be combined with the call to the function or method provided by the SDK and provided by the API, or use the data type or object defined in the SDK and provided by the API. In these embodiments, the application framework can provide a main event loop for the program, which responds to various events defined by the framework. The API allows the application to specify events and responses to events using the application framework. In some specific implementations, API calls can report the capabilities or status of hardware devices to applications, including capabilities or status related to aspects such as input capabilities and status, output capabilities and status, processing capabilities, power status, storage capacity and status, communication capabilities, etc., and the API can be partially implemented by firmware, microcode, or other low-level logic components that are partially executed on hardware components.

[0118] The API calling component can be a local component (i.e., on the same data processing system as the API implementing component) or a remote component (i.e., on a data processing system different from the API implementing component), which communicates with the API implementing component through the API via a network. It should be understood that the API implementing component can also act as an API calling component (i.e., it can make API calls to APIs exposed by different API implementing components), and the API calling component can also act as an API implementing component by implementing APIs exposed to different API calling components.

[0119] The API may allow multiple API-calling components written in different programming languages ​​to communicate with the API-implementing component (so that the API may include features for converting calls and returns between the API-implementing component and the API-calling component); however, the API may be implemented in a specific programming language. In one embodiment, the API-calling component may call APIs from different providers, such as one set of APIs from an OS provider and another set of APIs from a plug-in provider, and another set of APIs from another provider (e.g., a provider of a software library) or a creator of another set of APIs.

[0120] Figure 7 700 is a block diagram illustrating an API architecture 700 that can be used in some embodiments of the present invention. The API architecture 700 includes an API implementation component 710 (e.g., an operating system, library, device driver, API, application, software, or other module) that implements an API 720. The API 720 specifies one or more functions, methods, classes, objects, protocols, data structures, formats, and / or other features of the API implementation component that can be used by an API calling component 730. The API 720 may specify at least one calling convention that specifies how functions in the API implementation component receive parameters from the API calling component and how functions return results to the API calling component. The API calling component 730 (e.g., an operating system, library, device driver, API, application, software, or other module) makes an API call through the API 720 to access and use the features of the API implementation component 710 specified by the API 720. The API implementation component 710 may return a value to the API calling component 730 through the API 720 in response to an API call.

[0121] It should be understood that the API implementation component 710 may include additional functions, methods, classes, data structures, and / or other features that are not specified by the API 720 and are not available to the API calling component 730. It should be understood that the API calling component 730 may be on the same system as the API implementation component 710, or may be located remotely and access the API implementation component 710 using the API 720 over a network. Figure 7 A single API calling component 730 is shown interacting with API 720 , but it should be understood that other API calling components, possibly written in a different language than API calling component 730 (or the same language), may use API 720 .

[0122] The API implementation component 710, API 720, and API calling component 730 may be stored in a machine-readable medium, which includes any mechanism for storing information in a form readable by a machine (e.g., a computer or other data processing system). For example, a machine-readable medium includes a disk, an optical disk, a random access memory, a read-only memory, a flash memory device, etc.

[0123] FIG. 8A to FIG. 8B is a block diagram of an exemplary API software stack 800, 810 according to an embodiment. Fig. 8A An exemplary API software stack 800 is shown, where an application 802 may use a service API to call service A or service B and an OS API to call an operating system 804. In addition, service A and service B may call the operating system 804 using several OS APIs.

[0124] Figure 8B An exemplary API software stack 810 is shown, including application 1, application 2, service 1, service 2, and operating system 804. As shown, service 2 has two APIs, one of which (service 2 API 1) receives calls from application 1 and returns values, and the other (service 2 API 2) receives calls from application 2 and returns values. Service 1 (e.g., can be a software library) makes calls to OS API 1 and receives returned values, and service 2 (e.g., can be a software library) makes calls to both OS API 1 and OS API 2 and receives returned values. Application 2 makes calls to OS API 2 and receives returned values.

[0125] API for internal communication media streaming

[0126] 9A to 9C A sequence diagram showing an API that enables internal communication media streaming between smart home devices. The sequence and components shown represent Figure 5A A specific implementation of method 500 is shown. Fig.9A An API sequence 900 for initiating streaming from a local smart-home device (or another electronic device) to multiple recipient smart-home devices is shown, according to an embodiment. Fig. 9B An API sequence 930 for a real-time media recording, transfer, and playback loop is shown according to an embodiment. Fig. 9C An API sequence 950 for terminating a real-time media streaming session according to an embodiment is shown. The API sequence shown has an initiator 902, a daemon 904 associated with the initiator 902, and a daemon associated with each recipient device (e.g., daemon 906 for recipient A, daemon 908 for recipient B).

[0127] In one embodiment, the initiator 902 is a software application executing on a processor of a smart home device or another electronic device. In one embodiment, the initiator 902 is a firmware module executing on a microcontroller of a smart home device. The initiator 902 can also be a hardware module on a smart home device connected to a touch input device or a physical device, such as a physical button or switch. In one embodiment, the initiator 902 is Figure 6 600 of the electronic device 600 shown. In some embodiments, daemon 904, daemon 906, and daemon 908 are each software modules associated with an operating system of an electronic device such as a smart home device. In one embodiment, the daemon is a companion link daemon that manages the companion link session on each of the initiator device and the recipient device. For example, each daemon can be Figure 6 Companion link module 622 of computing device 600 is shown or may be a component thereof.

[0128] like Fig.9AAs shown, the initiator 902 may receive a start input 912, which may be based on voice input, text input, touch input, or another type of input, such as a physical button or switch. Voice input may be facilitated via a virtual assistant system as described herein. The start input causes a real-time media session activation message 913 to be sent to or received by a daemon 904 associated with the initiator 902. In the case where the initiator 902 is a hardware or firmware module, the initiator may detect the start input 912 and trigger an electrical signal that causes an association interruption with the real-time media session activation message 913. In the case where the initiator 902 is a software module, an inter-process communication API provided by an operating system or framework may be used to send the real-time media session activation message 913 to the daemon 904. The daemon 904 may trigger an AV media controller setup operation 914 in response to receiving a real-time media session activation message 913. The AV media controller setup operation 914 may include various sub-operations to configure a media (e.g., audio / video) controller on an electronic device associated with the daemon 904, thereby preparing to record media, such as audio and / or video. Daemon 904 may also send start real-time media request messages 915A to 915B via a secure network connection (e.g., a companion link) established with multiple recipients. Daemon 906 on recipient A may receive start real-time media request message 915A, and daemon 908 on recipient B may receive start real-time media request message 915B. In response to receiving the start real-time media request message, daemon 906 and daemon 908 may perform corresponding AV media engine start operations 916 and 918, each of which configures the media engine of the corresponding device to accept incoming audio and / or video streams. Daemon 906 and daemon 908 may then each send corresponding start real-time media response messages 925A to 925B to daemon 904. Daemon 904 may then send real-time media session activation message 923 to initiator 902 and trigger AV media controller to start recording operation 924. A recording and transmission cycle may then be started, in which the initiator streams the media containing audio and / or video data to the recipient.

[0129] like Fig. 9BAs shown, a media recording, transmission and playback cycle may occur, wherein the daemon 904 on the initiator device performs operation 934 to buffer the captured media (e.g., audio and / or video) data. Daemon 904 then sends a media message 935A to daemon 906 on receiver A and a media message 935B to daemon 908 on receiver B. Receiver A performs decoding, scheduling and playing media operations 936, while receiver B performs similar decoding, scheduling and playing media operations 938. Daemon 906 and daemon 908 can decode, schedule and play the received media by sending messages to other software, hardware and firmware modules on the electronic device to facilitate the decoding, scheduling and playing of the media. The manner in which the received media message is decoded, scheduled and played may vary depending on the receiver, because receiver A and receiver B may be different types of smart home devices. For example, receiver A may be a smart speaker device, while receiver B may be a smart appliance device. Additionally, one or both of the recipients may include a display device, while other types of smart home devices may not have a display or include a low-resolution display.

[0130] Although some embodiments may only enable unidirectional media streams, at least one embodiment enables bidirectional media streams. For bidirectional media streams, one or both of the recipient A and recipient B may use the microphone and / or camera on the recipient device to record real-time audio and / or video. Daemon 906 may perform operation 946 to buffer captured media data, and daemon 908 may perform additional operation 948 to buffer captured media data. The recipient's daemon may perform operation 945A to 945B to send media messages to daemon 904, thereby performing operation 944 to decode, schedule and play the media received from the recipient, for example, by using the internal API provided by the device associated with initiator 902. In one embodiment, the recipient's daemon may also perform operation 947A to 947B to exchange media messages with each other, which may be decoded, scheduled and played immediately, or during the play media operation 936 and play media operation 938 of the next cycle, decoded, scheduled and played.

[0131] like Fig. 9CAs shown, the real-time media session may terminate when the initiator receives a stop input 952, which may be any type of input described herein. In response to receiving the stop input, the initiator 902 may send a real-time media session invalidation message 953 to the daemon 904, which causes the daemon 904 to perform operations that invalidate and terminate the media session. Such operations include the daemon 904 performing an AV controller stop recording operation 954, which causes the media controller on the initiator's smart home device to stop the recording operation at the microphone and / or camera. The daemon 904 may then send a stop real-time media request message 955A and a stop real-time media request message 955B to the daemon 906 on the receiver A and the daemon 908 on the receiver B, respectively. The daemon 906 and the daemon 908 may perform an AV media controller stop operation 956. The daemon 908 may perform a similar AV media controller stop operation 958, which may include sub-operations similar to or different from the AV media controller stop operation 956 performed by the daemon 906. Daemon process 906 and daemon process 908 may then send stop real-time media response messages 965A to 965B to daemon process 904 when confirming stop real-time media request messages 955A to 955B. Daemon process 904 may then send real-time media session invalidation message 963 to initiator 902 to notify the initiator that the real-time media session has been invalidated and terminated, which will end the media streaming between the initiator and the recipient.

[0132] FIG. 10A to FIG. 10C A sequence diagram showing an API for streaming media from a remote smart home device. The sequence and components shown represent Figure 5B A specific implementation of method 510 is shown. Fig. 10A An API sequence 1000 is shown for initiating streaming from a source smart home device to a local smart home device (or another electronic device), according to an embodiment. Fig. 10B An API sequence 1040 for a real-time media recording, transfer, and playback loop is shown, according to an embodiment. Fig. 10C An API sequence 1060 for terminating a real-time media streaming session with a remote source is shown according to an embodiment. The API sequence shown has an initiator 1002, a daemon 1004 associated with the initiator 1002, and a daemon 1006 associated with the remote source device. In one embodiment, the sequence and components are similar to 9A to 9C , except that if remote initiation of streaming is enabled for a remote smart home device, then FIG. 10A to FIG. 10C The API enables an initiator to initiate streaming from the device. For example, the initiator 1002 may be Figure 6The streaming initiator 621 of the computing device 600 may be or may be included therein. The daemon process 1004 and the daemon process 1006 may be Figure 6 The companion link module 622 of the computing device 600 may be included therein.

[0133] like Fig. 10A As shown, the initiator 1002 may receive a start input 1012. In response to receiving the start input 1012, the initiator 1002 may send a real-time media session remote activation message 1013 to a daemon 1004 associated with the initiator 1002. The daemon 1004 may perform a remote verification operation 1014 through a companion link connection with a daemon 1006, which is associated with a remote smart home device that will be the source of the remotely initiated media stream. The daemon 1006 on the remote smart home device may also perform a remote verification operation 1016 to verify to the daemon 1004 that remote initiation of media streams has been enabled for the remote smart home device that will be the source of the remote media stream. The remote verification operation 1016 and the remote verification operation 1014 may also verify that the initiator's smart home device (or other electronic device) is authorized to initiate a remote media stream. The remote verification operation may also verify that the initiating smart home device is associated with a user account that is authorized to initiate a remote media stream.

[0134] If remote initiation of media streams has been enabled for the remote smart home device that is to be the source of the media stream, the daemon 1006 may perform an AV media controller setup operation 1026 and send a start real-time media request message 1015 to the daemon 1004. The daemon 1004 may then perform an AV media engine start operation 1024 and send a start real-time media response message 1025 to the daemon 1006. The daemon 1006 may reply to the start real-time media response message 1025 by sending a real-time media session activation message 1022 to the initiator 1002 to indicate that the real-time media session has begun. The daemon 1006 may then perform an AV media controller start recording operation 1036 to begin recording audio and / or video for transmission to the smart home device associated with the initiator 1002.

[0135] like Fig. 10BAs shown, a media recording, transmission and playback loop may occur, wherein the daemon 1006 on the remote source device performs operation 1046 to buffer the captured media (e.g., audio and / or video) data. The daemon 1006 then sends a media message 1045 to the daemon 1004 on the initiator device. The daemon 1004 may then use an internal API provided by the system on which the daemon 1004 and / or the initiator 1002 are executed to perform decoding, scheduling and playing media operations 1044. The manner in which the received media message is decoded, scheduled and played may vary depending on the specific device associated with the initiator 1002 and / or the daemon 1004, because such a device may be, for example, a smart speaker device, a smart appliance device, or another type of smart home device. In one embodiment, a bidirectional stream may alternatively be enabled, wherein the daemon 1004 associated with the initiator device may perform a buffer captured media data 1054 operation and send a media message 1055 to the daemon 1006 of the source device. The daemon 1006 may then perform decode, schedule, and play media operations 1056 .

[0136] like Fig. 10C As shown, the real-time media session may be terminated when the initiator receives a stop input 1062, which may be any type of input described herein. In response to receiving the stop input, the initiator 1002 may send a real-time media session invalidation message 1063 to the daemon 1004. The daemon 1004 may then send a real-time media session remote invalidation message 1063 to the daemon 1006 on the remote smart home device that is the source of the remote media stream. The daemon 1006 may then perform an AV controller stop recording operation 1066, which will stop the recording operation of the microphone and / or camera on the remote source. The daemon 1006 may then send a stop real-time media request message 1075 to the daemon 1004. The stop real-time media request message 1075 may cause the daemon 1004 to perform an AV media controller stop operation 1064, which causes the AV media controller on the initiator to stop the process responsible for decoding, scheduling, and playing the incoming media stream. Daemon 1004 then responds to daemon 1006 via the companion link with a stop real-time media response message 1085, which will confirm to daemon 1006 that the initiator has performed an action in response to stop real-time media request message 1075. Daemon 1006 may then send a real-time media session remote invalidate message 1095 to daemon 1004. Daemon 1004 may then notify initiator 1002 via real-time media session invalidate message 1073, which will notify the initiator that the remote real-time media session has been invalidated and terminated.

[0137] In addition to the above-mentioned real-time media stream, in one embodiment, a "send as a virtual assistant" function is provided, in which a message can be sent to a remote device to be spoken with the voice of a virtual assistant, which is executed on the remote device or otherwise associated with the remote device. The user can issue a voice command to the virtual assistant on the local smart home device to speak the message at the remote smart home device. The local smart home device can perform a speech-to-text operation on the input message and transmit the text of the input message to the remote smart home device via the accompanying link. The remote smart home device can then perform a text-to-speech operation on the received message text. For example, a user can issue a command to the local smart home device, such as "Let me say a word to the children's room." The virtual assistant on the local smart home device can reply "Okay. What do you want to say?" The user can reply "Come to the kitchen, dinner will be ready soon." The audio data of the message can be translated into text data and transmitted to the smart home device in the location designated as the "children's room." The smart home device in the children's room can then speak the message "<User> said, 'Come to the kitchen, dinner will be ready soon'". <User> can be the name of the sending user or another designated title, such as but not limited to "Mom", "Dad", "Grandma", "Grandpa", etc. The selected title can be predefined by the user. The virtual assistant can select the title to use based on detecting the relationship between the sending user account and the user account associated with the target smart home device. If the source smart home device and the target smart home device use the same user account, or the relationship between the sender and recipient of the message cannot be determined in other ways, the virtual assistant can omit the "<User> says" portion of the message, or can say the source of the message, such as, for example, "Come to the kitchen, dinner will be ready soon." Or "A message came from the kitchen saying 'Come to the kitchen, dinner will be ready soon'."

[0138] Fig.11 1 is a flow chart illustrating a method 1100 for sending a virtual assistant message according to an embodiment. The method 1100 may be implemented on a smart home device as described herein using a virtual assistant as described herein, and as described below. Fig.14. Method 1100 includes a virtual assistant on a smart home device receiving input defining a request that causes a remote smart home device to speak a message (box 1101). The remote smart home device can speak the message via the virtual assistant executing on the remote smart home device. The input defining the request can specify the remote home device through which the message is to be spoken and the message is to be spoken. In one embodiment, the input is voice input, but other types of input, such as text input, can be received. For example, a text message can be sent to the virtual assistant on the smart home device, where the text message defines a request that causes the remote smart home device to speak the message. However, if the input is received as voice input (box 1102), the virtual assistant can perform a speech-to-text operation on the voice input (box 1103). In one embodiment, the speech-to-text operation can include the following: Figure 1A to Figure 1B The method 1100 also includes the smart home device transmitting a companion link message including the message text to the remote smart home device (block 1104). The companion link message transmission can be performed using various techniques for establishing and verifying the security of the companion link as described herein. The remote smart home device, for example, via the virtual assistant, can speak the message text using the text-to-speech function of the virtual assistant (block 1105).

[0139] Additional Exemplary Electronic Devices

[0140] Fig.12 1 is a block diagram of a device architecture 1200 for a mobile or embedded device according to an embodiment. The device architecture 1200 may be included in a mobile device as described herein, such as a smart phone device, a desktop computing device, and / or a wearable electronic device. Aspects of the device architecture 1200 may also be used alone or in combination with Figure 6 Aspects of the computing device 600 are included in combination in a smart home device as described herein. The device architecture 1200 includes a memory interface 1202, a processing system 1204 including one or more data processors, an image processor and / or a graphics processing unit, and a peripheral device interface 1206. The various components can be coupled via one or more communication buses or signal lines. The various components can be separate logical components or devices or can be integrated in one or more integrated circuits, such as a system-on-chip integrated circuit.

[0141] The memory interface 1202 can be coupled to the memory 1250, which may include high-speed random access memory such as static random access memory (SRAM) or dynamic random access memory (DRAM) and / or non-volatile memory such as but not limited to flash memory (e.g., NAND flash, NOR flash, etc.).

[0142] Sensors, devices, and subsystems may be coupled to the peripheral device interface 1206 to facilitate multiple functions. For example, a motion sensor 1210, a light sensor 1212, and a proximity sensor 1214 may be coupled to the peripheral device interface 1206 to facilitate mobile device functions. There may also be one or more biometric sensors 1215, such as a fingerprint scanner for fingerprint recognition or an image sensor for facial recognition. Other sensors 1216 may also be connected to the peripheral device interface 1206, such as a positioning system (e.g., a GPS receiver), a temperature sensor, or other sensing devices to facilitate related functions. A camera subsystem 1220 and an optical sensor 1222 (e.g., a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) optical sensor) may be utilized to facilitate camera functions, such as taking pictures and video clips.

[0143] Communication functions may be facilitated by one or more wireless communication subsystems 1224, which may include radio frequency receivers and transmitters and / or optical (e.g., infrared) receivers and transmitters. The specific design and implementation of the wireless communication subsystem 1224 may depend on the communication network over which the mobile device is intended to operate. For example, a mobile device including the illustrated device architecture 1200 may include a wireless communication subsystem 1224 designed to operate over a GSM network, a CDMA network, an LTE network, a Wi-Fi network, a Bluetooth network, or any other wireless network. Specifically, the wireless communication subsystem 1224 may provide a communication mechanism in which a media playback application may retrieve resources from a remote media server or retrieve scheduled events from a remote calendar or event server.

[0144] The audio subsystem 1226 may be coupled to a speaker 1228 and a microphone 1230 to facilitate voice-enabled functions such as voice recognition, voice replication, digital recording, and telephone functions. Where the illustrated device architecture is included in a smart home device as described herein, particularly in a smart home device that is customized for media playback, the audio subsystem 1226 may be a high-quality audio system that includes support for virtual surround sound and / or multi-channel audio. In some embodiments, multi-channel audio may be enabled using multiple smart home devices that are each configured to play a specific audio channel. Audio channel configuration may be automatically performed by the smart home device via communications performed over a companion link.

[0145] I / O subsystem 1240 may include touch screen controller 1242 and / or other input controller 1245. For computing devices that include a display device, touch screen controller 1242 may be coupled to a touch-sensitive display system 1246 (e.g., a touch screen). Touch-sensitive display system 1246 and touch screen controller 1242 may detect contact and motion or pressure using, for example, any of a variety of touch and pressure sensing technologies, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch-sensitive display system 1246. Display output of touch-sensitive display system 1246 may be generated by display controller 1243. In one embodiment, display controller 1243 may provide frame data to touch-sensitive display system 1246 at a variable frame rate.

[0146] In one embodiment, a sensor controller 1244 is included to monitor, control, and / or process data received from one or more motion sensors 1210, light sensors 1212, proximity sensors 1214, or other sensors 1216. The sensor controller 1244 may include logic to interpret the sensor data to determine the occurrence of one of a plurality of motion events or activities by analyzing the sensor data from the sensors.

[0147] In one embodiment, the I / O subsystem 1240 includes other input controllers 1245 that may be coupled to other input / control devices 1248 such as one or more buttons, rocker switches, thumb wheels, infrared ports, USB ports, and / or pointer devices such as a stylus or controls / devices such as up / down buttons for volume controls for the speaker 1228 and / or microphone 1230.

[0148] In one embodiment, memory 1250 coupled to memory interface 1202 can store instructions for an operating system 1252, including portable operating system interface (POSIX) compliant and non-compliant operating systems or embedded operating systems. Operating system 1252 can include instructions for handling basic system services and for performing hardware-related tasks. In some specific implementations, operating system 1252 can be a kernel.

[0149] The memory 1250 may also store communication instructions 1254 to facilitate communication with one or more additional devices, one or more computers and / or one or more servers, such as obtaining web resources from a remote web server. The memory 1250 may also include user interface instructions 1256, including graphical user interface instructions to facilitate graphical user interface processing.

[0150] In addition, the memory 1250 may store sensor processing instructions 1258 to facilitate sensor-related processing and functions; phone instructions 1260 to facilitate processes and functions related to phone calls; instant messaging instructions 1262 to facilitate processes and functions related to electronic message processing; web browsing instructions 1264 to facilitate processes and functions related to web browsing; media processing instructions 1266 to facilitate processes and functions related to media processing; location service instructions including GPS and / or navigation instructions 1268 and Wi-Fi-based location instructions to facilitate location-based functionality; camera instructions 1270 to facilitate processes and functions related to cameras; and / or other software instructions 1272 to facilitate other processes and functions, such as security processes and functions and system-related processes and functions. The memory 1250 may also store other software instructions, such as web video instructions to facilitate processes and functions related to web videos; and / or web shopping instructions to facilitate processes and functions related to web shopping. In some specific implementations, the media processing instructions 1266 are divided into audio processing instructions and video processing instructions to facilitate processes and functions related to audio processing and processes and functions related to video processing, respectively. A mobile device identifier, such as an International Mobile Equipment Identity (IMEI) 1274 or similar hardware identifier may also be stored in memory 1250 .

[0151] Each of the instructions and applications identified above may correspond to an instruction set for performing one or more of the functions described above. These instructions need not be implemented as separate software programs, processes, or modules. Memory 1250 may include additional instructions or fewer instructions. In addition, various functions may be performed in hardware and / or software, including in one or more signal processing and / or application specific integrated circuits.

[0152] Fig.13 is a block diagram of one embodiment of a computing system 1300 . Fig.13 The computer system shown is intended to represent a range of computing systems (wired or wireless), including, for example, one or more implementations of a desktop computer system, a laptop computer system, a tablet computer system, a cellular phone, a personal digital assistant (PDA) (including a cellular-enabled PDA), a set-top box, an entertainment system or other consumer electronic device, a smart appliance device, or a smart media playback device. Alternative computing systems may include more, fewer, and / or different components. Fig.13 The computing system may be used to provide computing devices and / or server devices to which the computing devices may be connected.

[0153] The computer system 1300 includes a bus 1335 or other communication device for communicating information, and a processor 1310 coupled to the bus 1335 that can process information. Although the computing system 1300 is illustrated as having a single processor, the computing system 1300 may include multiple processors and / or coprocessors. The computing system 1300 may also include a memory 1320, such as a random access memory (RAM) or other dynamic storage device coupled to the bus 1335. The memory 1320 may store information and instructions that may be executed by the processor 1310. The memory 1320 may also be used to store temporary variables or other intermediate information during the execution of instructions by the processor 1310.

[0154] The computing system 1300 may also include a read-only memory (ROM) 1330 and / or other data storage device 1340 coupled to the bus 1335 that may store information and instructions for the processor 1310. The data storage device 1340 may be or include a variety of storage devices, such as a flash memory device, a magnetic disk, or an optical disk and may be coupled to the computing system 1300 via the bus 1335 or via a remote peripheral interface.

[0155] The computing system 1300 may also be coupled to a display device 1350 via bus 1335 to display information to a user. The computing system 1300 may also include an alphanumeric input device 1360, which includes alphanumeric and other keys, which may be coupled to the bus 1335 to send information and command options to the processor 1310. Another user input device includes a cursor control 1370 device, such as a touch pad, mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 1310 and controlling cursor movement on the display device 1350. The computing system 1300 may also receive user input from a communicatively coupled remote device via one or more network interfaces 1380.

[0156] The computing system 1300 may also include one or more network interfaces 1380 to provide access to a network such as a local area network. The network interface 1380 may include, for example, a wireless network interface having an antenna 1385, which may represent one or more antennas. The computing system 1300 may include multiple wireless network interfaces, such as Wi-Fi and A combination of near field communication (NFC) and / or a cellular phone interface. Network interface 1380 may also include, for example, a wired network interface to communicate with a remote device via a network cable 1387, which may be, for example, an Ethernet cable, a coaxial cable, a fiber optic cable, a serial cable, or a parallel cable.

[0157] In one embodiment, the network interface 1380 may provide access to a local area network, for example by complying with the IEEE 802.11 standard, and / or the wireless network interface may provide access to a personal area network, for example by complying with the Bluetooth standard. Other wireless network interfaces and / or protocols may also be supported. In addition to or in lieu of communicating via the wireless LAN standard, the network interface 1380 may provide wireless communications using, for example, a time division multiple access (TDMA) protocol, a global system for mobile communications (GSM) protocol, a code division multiple access (CDMA) protocol, a long term evolution (LTE) protocol, and / or any other type of wireless communication protocol.

[0158] The computing system 1300 may also include one or more energy sources 1305 and one or more energy measurement systems 1345. The energy source 1305 may include an AC / DC adapter coupled to an external power source, one or more batteries, one or more charge storage devices, a USB charger, or other energy sources. The energy measurement system includes at least one voltage or current measurement device that can measure the energy consumed by the computing system 1300 over a predetermined time period. In addition, one or more energy measurement systems may be included to measure, for example, the energy consumed by a display device, a cooling subsystem, a Wi-Fi subsystem, or other commonly used or high energy consumption subsystems.

[0159] Virtual assistant system for smart home devices

[0160] Fig.14 A block diagram of a virtual assistant system 1400 according to an embodiment described herein is shown. The virtual assistant system 1400 shown is an example of an embodiment and is not limited to all embodiments described herein. The virtual assistants utilized by the various embodiments described herein may include additional, fewer and / or different components or features than those shown. The virtual assistant system 1400 includes a virtual assistant 1402, which can accept user input 1404, such as oral or typed language, process the input, and generate output 1408 to the user and / or perform 1410 actions on behalf of the user. The virtual assistant 1402 can use context information to supplement natural language or gesture input from the user. Context information can be used to clarify the user's intentions and reduce the number of candidate interpretations of the user input. Context information can also reduce the need for users to provide too much clarifying input. Context can include any available information available to the assistant to supplement explicit user input to constrain information processing problems and / or personalized processing results. Context can be used to constrain solutions in various stages of processing, including, for example, speech recognition, natural language processing, task flow processing, and dialog box generation.

[0161] Virtual assistant 1402 may utilize any of a variety of different context sources of knowledge and data (e.g., a lexicon, a domain model, and / or a task model). From the perspective of the presently described embodiments, such context sources may be internal to virtual assistant 1402, or may be collected from one or more remote databases. In addition to user input 1404 and context sources, virtual assistant 1402 may also utilize information from several sources of context, including, for example, device sensor data 1456, application preferences and usage history 1472, conversation history and assistant memory 1452, personal database 1458, personal acoustic context data 1480, current application context 1460, and event context 1406.

[0162] In one embodiment, the physical device running the virtual assistant 1402 (such as a user device, smart media playback device, or smart appliance as described herein) has one or more sensor devices. Such sensors can provide a source of contextual information in the form of device sensor data 1456. Examples of sensor information include, but are not limited to, the user's current location; the local time of the user's current location; the location, orientation, and motion of the device with which the user is interacting; the current light level, temperature, and other environmental measurements; the properties of the microphone and camera in use; the signatures of the network currently in use and the connected networks, including Ethernet, Wi-Fi, and Bluetooth. The signature includes the MAC address of the network access point, the assigned IP address, the device identifier, such as the Bluetooth name, frequency channel, and other properties of the wireless network. The sensor can be of any type, including, for example, an accelerometer, a compass, a GPS unit, an altitude detector, a light sensor, a thermometer, a barometer, a clock, a network interface, a battery test circuit, and the like.

[0163] Current application context 1460 refers to an application state or similar software state related to the user's current activity. For example, a user may be using a text instant messaging application to chat with a particular person. Virtual assistant 1402 need not be specific to the user interface or portion of the user interface of a text instant messaging application. Instead, virtual assistant 1402 may receive context from any number of applications, each of which promotes its context to notify virtual assistant 1402. If the user is currently using an application when virtual assistant 1402 is called, the state of the application may provide useful context information. For example, if virtual assistant 1402 is called from an email application, the context information may include sender information, recipient information, send date and / or time, subject, data, etc. extracted from the email content, mailbox or folder name.

[0164] In one embodiment, information describing user application preferences and usage history 1472 includes preferences and settings for various applications and usage history associated with those applications. Application preferences and usage history 1472 serves as context for interpreting and / or implementing user intent or other functionality of virtual assistant 1402. Examples of such application preferences and usage history 1472 include, but are not limited to: a collection of shortcuts, favorites, bookmarks, friend lists, or any other user data about people, companies, addresses, phone numbers, places, websites, email messages, or any other references; recent calls on the device; recent text messaging conversations, including the parties to the conversations; recent requests for maps or directions; recent web searches and URLs; stocks listed in the Stocks application; recently played songs or videos or other media; the names of alarms set on the Alarms application; the names of applications or other digital objects on the device; and the user's preferred language or the language being used at the user's location.

[0165] Another source of contextual data is a user's personal database 1458 on a device such as a phone, such as, for example, an address book containing names and phone numbers. In one embodiment, the user's personal information obtained from personal database 1458 is used as context for interpreting and / or implementing the user's intent or other functionality of virtual assistant 1402. For example, data in the user's contacts database can be used to reduce ambiguity in interpreting user commands when the user refers to someone only by name. Examples of contextual information that may be obtained from personal database 1458 include, but are not limited to: a user's contact database (address book) about people, companies, organizations, places, websites, and other entities to which the user may refer, including information about names, phone numbers, physical addresses, network addresses, account identifiers, important dates; the user's own name, preferred pronunciation, address, phone number, etc.; the user's named relationships, such as mother, father, sister, boss, etc.; the user's calendar data, including calendar events, names of specific days, or any other named entries to which the user may refer; the user's reminder or task list, including a list of things to accomplish, remember, or obtain to which the user may refer; song names, genres, playlists, and other data associated with the user's music library to which the user may refer; people, places, categories, tags, labels, or other symbolic names of photos or videos or other media in the user's media library; titles, authors, genres, or other symbolic names in books or other documents in the user's personal library.

[0166] Another source of contextual data is the user's conversation history with virtual assistant 1402. Such history may include, for example, references to domains, people, places, and the like. For example, a user may ask, "What time is it in New York?" Virtual assistant 1402 may respond by providing the current time in New York City. The user may then ask, "What's the weather like?" Virtual assistant 1402 may use the previous conversation history to infer that the location used for the weather query is the last location mentioned in the conversation history.

[0167] Examples of contextual information from the conversation history and virtual assistant memory include, but are not limited to, people mentioned in the conversation; places and locations mentioned in the conversation; the current time frame in focus; the current application domain in focus, such as email or calendar; the current task in focus, such as reading email or creating a calendar entry; the current domain object in focus, such as the email message just read or the calendar entry just created; the current state of the conversation or transaction flow, such as whether a question is asked and possible answers are expected; the history of user requests; the history of results of user requests, such as a set of restaurants returned; the history of phrases used by the assistant in the conversation; and facts told by the user to the assistant.

[0168] In one embodiment, personal acoustic context data 1480 can be used to select from possible statistical language models that can be used to understand the user's speech, or to otherwise tune speech recognition to optimize the recognized acoustic context. When interpreting speech input, the virtual assistant 1302 can tune the speech-to-text service to take into account the acoustic environment of the input speech. For example, the noise profile of a quiet office is different from the noise profile of a car or a public place. If the speech recognition system is capable of recognizing and storing acoustic configuration data, this data can also be provided as context information. When combined with other context information such as the properties of the microphone in use, the current location and the current conversation state, the acoustic context can help recognize and interpret the input.

[0169] In the above description, the present invention has been described for specific embodiments of the present invention. However, it is obvious that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present invention. Accordingly, the description and the accompanying drawings should be regarded as having illustrative rather than restrictive meanings. The specific details in the provided description and examples may be used anywhere in one or more embodiments. The various features of different embodiments or examples may be combined differently with some of the included features and other features excluded to accommodate a variety of different applications. Examples may include subject matter, such as methods, components for performing the behavior of the method, at least one machine-readable medium including instructions, the instructions causing the machine to perform the behavior of the method when executed by the machine, or the actions of the device or system according to the embodiments and examples described herein. In addition, the various components described herein may be components for performing the operations or functions described according to the embodiments.

[0170] Embodiments described herein provide a communication mechanism that enables a public electronic device, such as a smart speaker device or another smart home device, to relay or redirect a virtual assistant request involving personal user data to a companion device, which is a personal user device associated with the user requesting access to the private data. The communication mechanism may also function as a general purpose communication mechanism that enables smart home devices to exchange data, including configuration data.

[0171] Communications between the public device and the companion device may be performed over a secure data channel called a companion link. The companion link provides a persistent, low latency instant messaging system for connected devices in a home network environment. In some embodiments, the companion link supports the linking of fixed public devices within the home, such as speaker devices, to personal mobile devices in the home or accessible via the Internet. The companion link enables the public device to redirect personal requests that the public device cannot handle due to privacy issues for one or more personal devices. Redirecting personal requests to a personal device associated with a user enables a virtual assistant on the public device to receive and request in a confidential manner.

[0172] In addition to enabling personal queries to be redirected to the companion device, in one embodiment, the companion link also provides a universal instant messaging system for devices in the home network environment. The universal instant messaging system enables multiple home devices to work together by exchanging messages over the companion link. For example, audio playback between multiple smart speaker devices can be coordinated to enable the devices to perform operations such as, but not limited to, coordinating playback of media items, selectively providing audio notifications to a user via the speaker closest to the user, configuring multiple speakers into a multi-channel audio system, or coordinating audio ducking at the speaker for the duration of a verbal request and response.

[0173] The public device may advertise support for the companion link service via a discovery protocol. A personal user device on the same network as the public device may discover the companion link service advertised by the public device and connect to the public device using the advertised information. The personal device may perform a pairing process with the public device to become a companion device of the user. In one embodiment, the pairing process includes a proximity element, wherein the user device exchanges an identifier, key or secret with the companion device via a short-range wireless communication mechanism. The pairing process may also include the exchange of presence and accessibility information that may facilitate subsequent connections between the public and companion devices on the companion link.

[0174] The embodiments described herein provide a communication mechanism that enables a public electronic device such as a smart speaker device or another smart home device to relay or redirect a virtual assistant request involving personal user data to a personal user device for processing. The communication mechanism can also be used as a general communication mechanism that enables smart home devices to exchange data, including configuration data. However, it is obvious that various modifications and changes can be made to it without departing from the broader essence and scope of the present invention. Accordingly, the specification and the drawings should be regarded as having illustrative rather than restrictive meanings. The specific details in the description and examples provided can be used anywhere in one or more embodiments. Various features of different embodiments or examples can be combined differently with some of the included features and other excluded features to accommodate a variety of different applications. Examples may include subjects, such as methods, components for performing the behavior of the method, at least one machine-readable medium including instructions, which, when executed by a machine, causes the machine to perform the behavior of the method, or the actions of an apparatus or system according to the embodiments and examples described herein. In addition, the various components described herein may be components for performing operations or functions described according to the embodiments.

[0175] One embodiment provides a data processing system for a public electronic device, the data processing system comprising a memory device storing instructions and one or more processors executing the instructions stored on the memory device. The instructions cause the one or more processors to provide a virtual assistant to receive a command at a public electronic device, wherein the virtual assistant, via the one or more processors, is configured to receive a command at a public electronic device and determine whether the command is to access personal data of a user associated with the public electronic device. In response to determining that the command is to access the personal data of the user, the virtual assistant may send a request to the user's personal electronic device to process at least a portion of the command.

[0176] In the embodiments described herein, the user's personal data includes a contact list, text messages, emails, call history, alarms, reminders, communication history, settings, preferences or location history, and the public electronic device includes a smart speaker device. Personal data can be stored on the user's personal electronic device and the virtual assistant can request the personal electronic device to access the personal data on behalf of the public electronic device. The virtual assistant can then receive the output of the request sent to the user's personal electronic device and complete the processing of the command based on the output. The command can be a voice command spoken by the user or a text command transmitted to the public device. In one embodiment, sending a request to the user's personal electronic device includes redirecting the command to the user's personal electronic device, wherein the personal electronic device processes the command on behalf of the public electronic device. The virtual assistant can receive an audio response generated by the personal electronic device and play the audio response as a response to the command. If the command is a command to send a message to the user's contact, the audio response can be a notification received to reply to the message.

[0177] In one embodiment, before enabling the public electronic device to send a request to the personal electronic device, the public electronic device may establish a pairing and / or a trusted relationship with the personal electronic device. The virtual assistant may send the request to the user's personal electronic device through a verified data connection with the personal electronic device. A verified data connection may be established based on a trusted relationship with the personal electronic device, wherein the verified data is verified by data exchanged during the establishment of the trusted relationship. In one embodiment, the verified data connection is established via a remote data connection established via a wide area network such as the Internet. The user's personal data is data specific to the user and may include data that is inherently personal or private, or data designated as personal or private. Personal data may also include data that can be used to specifically identify the user.

[0178] One embodiment provides a non-transitory machine-readable medium storing instructions that cause one or more processors to perform operations including determining that a communication session is to be established between a public electronic device and a companion device, wherein the companion device is an electronic device that has access to personal data associated with a user. The operations additionally include establishing a data connection with the companion device, verifying a trusted relationship between the public electronic device and the companion device, establishing an encrypted communication session between the public electronic device and the companion device after verifying the trusted relationship, and relaying commands received at the public electronic device to the companion device via the encrypted communication session.

[0179] One embodiment provides an electronic device comprising a network interface connected to one or more networks and a memory device storing instructions and personal data associated with a user of the electronic device. The electronic device comprises one or more processors to execute instructions stored on the memory device, wherein the instructions cause the one or more processors to provide a virtual assistant that receives commands. The virtual assistant, via one or more processors, may receive commands redirected from a public electronic device via the network interface. The command may include or specify a request for accessing personal data associated with a user of the electronic device. The virtual assistant may then process at least a portion of the command on behalf of the public electronic device and transmit the output of the processing performed on behalf of the public electronic device to the public electronic device via the network interface.

[0180] Although a companion link connection between a common electronic device such as a smart speaker device and a companion device such as a smart phone device is described, a companion link connection can also be established between multiple companion devices, such as multiple smart phone devices, tablet computing devices, or between a smart phone device and a tablet computing device.

[0181] One embodiment provides a data processing system for an electronic device, the data processing system comprising a memory device storing instructions and one or more processors executing the instructions stored on the memory device. The instructions, when executed, cause the one or more processors to enable an encrypted data channel between electronic devices. In order to enable an encrypted data channel, the one or more processors are configured to determine that a communication session is to be established between a first electronic device and a second electronic device, wherein the first electronic device and the second electronic device are each associated with a cloud service account. The one or more processors may be further configured to establish a peer-to-peer data connection between the first electronic device and the second electronic device, verify the trusted relationship between the first electronic device and the second electronic device, and establish an encrypted communication session between the first electronic device and the second electronic device after verifying the trusted relationship. The encrypted communication session may then be used to exchange data between the first electronic device and the second electronic device through the encrypted communication session.

[0182] In another embodiment, one or more processors of the data processing system may establish a peer-to-peer data connection between the first electronic device and the second electronic device via a short-range wireless connection, and / or establish an encrypted communication session via a network layer protocol via a wireless network connection. To verify the trusted relationship between the first electronic device and the second electronic device may include verifying a previously established trusted relationship, which may be established via one or more of an exchange of credentials between the first electronic device and the second electronic device via a short-range wireless connection and / or an exchange of credentials of a cloud service account associated with the first electronic device and the second electronic device, the credentials enabling mutual authentication between the first electronic device and the second electronic device.

[0183] In a further embodiment, one or more processors may be configured to determine that a communication session will be established between a first electronic device and a second electronic device in response to discovering a second electronic device at a first electronic device via a device discovery protocol, and to establish a peer-to-peer data connection between the first electronic device and the second electronic device after discovering the second electronic device. A communication session may also be established based on a list of devices associated with a cloud service account, with which the first device and the second device are associated. In one embodiment, a communication session may be established based on a list of devices associated with a cloud service account family, wherein a first electronic device is associated with a first account, a second electronic device is associated with a second account, and the cloud service account family includes the first account and the second account. In one embodiment, before the first electronic device is enabled to send a request for data exchange to the second electronic device, the first electronic device establishes a trusted relationship with the second electronic device.

[0184] One embodiment provides a non-transitory machine-readable medium storing instructions that cause one or more processors to perform operations, including determining that a communication session is to be established between a first electronic device and a second electronic device, wherein the first electronic device and the second electronic device are each associated with a cloud service account. The instructions may further cause the one or more processors to perform additional operations, including establishing a peer-to-peer data connection between the first electronic device and the second electronic device, verifying a trusted relationship between the first electronic device and the second electronic device, establishing an encrypted communication session between the first electronic device and the second electronic device after verifying the trusted relationship, and exchanging data between the first electronic device and the second electronic device through the encrypted communication session.

[0185] One embodiment provides a method performed on a computing device or data processing system described herein. The method includes determining to establish a communication session between a first electronic device and a second electronic device, wherein the first electronic device and the second electronic device are each associated with a cloud service account, and establishing a peer-to-peer data connection between the first electronic device and the second electronic device through a radio device, and verifying a trusted relationship between the first electronic device and the second electronic device, wherein verifying the trusted relationship between the first electronic device and the second electronic device includes verifying a previously established trusted relationship. The method further includes establishing an encrypted communication session between the first electronic device and the second electronic device via a network layer protocol through a wireless network connection, the encrypted communication session being established after verifying the trusted relationship, and exchanging data between the first electronic device and the second electronic device through the encrypted communication session. For example, data can be exchanged to synchronize device data between the first electronic device and the second electronic device, wherein the device data is associated with a cloud service account.

[0186] In another embodiment, the method further includes determining that a communication session is established between a first electronic device and a second electronic device based on a device list associated with a cloud service account family, wherein the first electronic device is associated with a first account, the second electronic device is associated with a second account, and the cloud service account family includes the first account and the second account. The first electronic device can discover the second electronic device via a device discovery protocol and establish a peer-to-peer data connection with the second electronic device after discovering the second electronic device.

[0187] In additional embodiments, the general communication mechanism described herein can be used to implement intercommunication-like transmission of audio data or video data between electronic devices connected to the communication mechanism. The media intercommunication system can be implemented within the companion link framework without requiring a large amount of additional program logic at a higher operating system level. The library and daemon that implement the companion link communication can provide an application programming interface (API) that enables smart home devices to form an intercommunication system through which audio data and / or video data can be streamed between smart home devices.

[0188] One embodiment provides an electronic device, which includes a network interface coupled to a bus, a microphone coupled to the bus, a non-transitory machine-readable medium coupled to the bus, and one or more processors coupled to the bus. One or more processors can execute instructions stored on the non-transitory machine-readable medium, wherein the instructions are used to provide an application programming interface (API) so that one or more processors stream internal communication media data from the electronic device to a smart home device connected via a network interface. In order to stream internal communication media data, one or more processors transmit the media data buffer to the smart home device via the network interface through a secure connection established with the smart home device. The secure connection can be established with the smart home device via the API. The media data buffer includes audio data captured via the microphone. The audio data is real-time or substantially real-time audio data that is captured while the buffer is being transmitted in terms of time. In one embodiment, the media data may also include video data captured via a camera on the electronic device.

[0189] One embodiment provides a non-transitory machine-readable medium storing instructions that cause one or more processors of a smart home device to perform operations including: receiving a request to initiate an internal communication media stream from a remote smart home device to the smart home device; verifying a secure connection between the remote smart home device and the smart device, wherein the secure connection is an encrypted peer-to-peer connection; activating a media session for the internal communication media stream, the media session being established with the remote smart home device; receiving recorded media from the remote smart home device; and playing the recorded media via an output device of the smart home device.

[0190] Other embodiments may provide systems or methods that perform the techniques of the above-described embodiments.

[0191] Other features of the present embodiment will become apparent from the accompanying drawings and from the above detailed description. Therefore, the true scope of the invention will become apparent to the skilled practitioner upon studying the drawings, the specification and the appended claims.

Claims

1. An electronic device, comprising: a network interface coupled to the bus; a microphone coupled to the bus; a non-transitory machine-readable medium coupled to the bus; as well as One or more processors coupled to the bus, the one or more processors executing instructions stored on the non-transitory machine-readable medium, wherein the instructions are to: providing an application programming interface (API) so that the one or more processors stream the internal communication media data from the electronic device to a smart home device connected to the electronic device via the network interface, receiving a request via the API to initiate a media stream from the electronic device to the smart home device, the electronic device connecting to the smart home device via the network interface and establishing a secure connection with the smart home device via the API; Verifying the secure connection established with the smart home device; as well as activating a media session with the smart home device, the media session establishing context data to facilitate transmission of a buffer of media data to the smart home device; wherein streaming the internal communication media data comprises transmitting the media data buffer to the smart home device via the network interface through the secure connection established with the smart home device, The media data buffer includes audio data captured via the microphone, and the audio data is captured while the media data buffer is being transmitted.

2. The electronic device of claim 1, wherein the secure connection established with the smart home device is an encrypted peer-to-peer connection established via the API.

3. An electronic device according to claim 1 or 2, wherein the smart home device is a smart speaker device or another connected media playback device.

4. The electronic device according to claim 1 or 2, wherein the smart home device is a smart home appliance device. 5 . The electronic device of claim 1 , wherein verifying the secure connection established with the smart home device comprises verifying a trusted relationship between the electronic device and the smart home device. 6 . The electronic device of claim 5 , wherein verifying the trusted relationship between the electronic device and the smart home device comprises verifying a previously established trusted relationship between the electronic device and the smart home device. 7 . The electronic device according to claim 1 , further comprising a camera device, wherein the media data buffer further comprises video data captured via the camera device, the video data being captured while the media data buffer is being transmitted.

8. The electronic device of claim 1, wherein the smart home device is a first smart home device, and the one or more processors further transmit the media data buffer via the network interface through a secure connection established with a second smart home device.

9. The electronic device of claim 1, wherein the smart home device is a first smart home device, and the API causes the one or more processors to transmit the media data buffer via the network interface through a secure connection established with a second smart home device.

10. The electronic device of claim 1, wherein the API enables the one or more processors to receive a media data buffer from the smart home device, and the one or more processors also decode the media data buffer and play the decoded media data via one or more output devices of the electronic device.

11. The electronic device according to any one of claims 7 to 10, wherein the smart home device is a smart speaker device or a smart electrical appliance device.

12. A method performed by one or more processors of a smart home device, the method comprising: receiving a request to initiate an internal communication media stream from a remote smart home device to the smart home device, the smart home device being connected to the remote smart home device via a network interface; verifying a secure connection between the remote smart home device and the smart home device, wherein the secure connection is an encrypted peer-to-peer connection; activating a media session for the internal communication media stream, the media session being established with the remote smart home device, the media session establishing context data to facilitate receipt of recorded media from the remote smart home device, the recorded media comprising a media data buffer; receiving recorded media from the remote smart home device; as well as The recorded media is played via an output device of the smart home device.

13. The method of claim 12, further comprising verifying that remote initiation of streaming is enabled for the remote smart home device prior to activating the media session for the internal communication media stream, wherein remote initiation of streaming is enabled via a configuration setting of the remote smart home device.

14. The method of claim 12, wherein the recorded media from the smart home device includes one or more of audio data and video data.

15. The method of claim 14, wherein the request to initiate an internal communication media stream is based on a voice command received at the smart home device, the voice command being processed by a virtual assistant executing on the smart home device.

16. The method of claim 15, wherein the voice command is used to specify the remote smart home device, the remote smart home device being specified via a location associated with the smart home device.

17. The method of claim 16, wherein the location associated with the smart home device is a designated room location within a smart home environment.

18. The method of claim 17, further comprising determining whether the internal communication media stream contains audio data or video data based on the voice command.

19. The method of claim 12, wherein the smart home device and the remote smart home device are smart speaker devices or smart electrical appliances.

20. A non-transitory machine-readable medium storing instructions that cause one or more processors to perform the method of any one of claims 12 to 19.

21. A system comprising means for performing the method according to any one of claims 12 to 19.

Images