Secure device-to-device communication channel

Abstract

The present invention relates to secure device-to-device communication channels. One embodiment provides a data processing system on a public electronic device, the data processing system comprising a memory device storing instructions; one or more processors executing the instructions stored on the memory device, the instructions causing the one or more processors to provide a virtual assistant to receive a command at the public electronic device, wherein the virtual assistant, via the one or more processors, receives the command at the public electronic device; determine whether the command is to access personal data of a user associated with the public electronic device; and in response to determining that the command is to access the personal data of the user, send a request to a personal electronic device of the user to process at least a portion of the command.

Description

[0001] This application is a divisional application of the invention patent application with application number 201811226436.7, application date October 19, 2018, and invention title "Secure Device-to-Device Communication Channel". Technical Field

[0002] This disclosure relates to secure device-to-device communication channels. Background Technology

[0003] Electronic devices known in the art may include intelligent automated assistant systems capable of interacting with a user of the electronic device. Generally, intelligent automated assistant systems provide digital or virtual assistants capable of performing actions on the electronic device or providing requested information to the user. These automated assistants can control many operations and functions of the electronic device, such as making phone calls, sending text messages, setting reminders, adding events to a calendar, and performing various other operations on behalf of the user. Automated assistant systems can receive verbal, natural language commands from the user and can respond to the user with responses generated using a speech synthesis engine.

[0004] Automated assistant services and operations can be categorized into various domains describing the service areas of automated assistants. Enabling full functionality in some domains may require access to personal or private data associated with or specific to the user of the electronic device. Such data may be stored on a personal user device or at a remote location accessible to the user device. However, some specific implementations of automated assistants may be included on public devices that may be used by more than one user. For privacy reasons, it may be advantageous to avoid storing personal or private user information on public devices. Therefore, automated assistant services that require access to users' personal information may not be available on public devices. Summary of the Invention

[0005] The embodiments described herein provide a communication mechanism that enables public electronic devices, such as smart speaker devices or other smart home devices, to relay or redirect virtual assistant requests involving personal user data to the personal user device for processing. The communication mechanism can also be used as a general communication mechanism that enables smart home devices to exchange data, including configuration data.

[0006] One embodiment provides a data processing system on an electronic device, the data processing system including a memory device storing instructions and one or more processors executing the instructions stored on the memory device. When executed, the instructions cause the one or more processors to enable an encrypted data channel between the electronic devices. To enable the encrypted data channel, the one or more processors are configured to determine to establish a communication session between a first electronic device and a second electronic device, wherein the first electronic device and the second electronic device are each associated with a cloud service account. The one or more processors may be further configured to establish a peer-to-peer data connection between the first electronic device and the second electronic device, verify a trust relationship between the first electronic device and the second electronic device, and establish an encrypted communication session between the first electronic device and the second electronic device after verifying the trust relationship. The encrypted communication session can then be used to exchange data between the first electronic device and the second electronic device through the encrypted communication session.

[0007] One implementation provides a non-transitory machine-readable medium storing instructions to cause one or more processors to perform operations, including determining that a communication session should be established between a first electronic device and a second electronic device, wherein each of the first and second electronic devices is associated with a cloud service account. The instructions may also cause the one or more processors to perform additional operations, including establishing a peer-to-peer data connection between the first and second electronic devices, verifying a trust relationship between the first and second electronic devices, establishing an encrypted communication session between the first and second electronic devices after verifying the trust relationship, and exchanging data between the first and second electronic devices through the encrypted communication session.

[0008] One embodiment provides a method performed on a computing device or data processing system as described herein. The method includes determining to establish a communication session between a first electronic device and a second electronic device, wherein the first electronic device and the second electronic device are each associated with a cloud service account, and establishing a peer-to-peer data connection between the first electronic device and the second electronic device via a wireless device; and verifying a trust relationship between the first electronic device and the second electronic device, wherein verifying the trust relationship includes verifying a previously established trust relationship. The method further includes establishing an encrypted communication session between the first electronic device and the second electronic device via a wireless network connection through a network layer protocol, the encrypted communication session being established after verifying the trust relationship, and exchanging data between the first electronic device and the second electronic device through the encrypted communication session. For example, data may be exchanged to synchronize device data between the first electronic device and the second electronic device, wherein the device data is associated with a cloud service account.

[0009] Other features of this embodiment will become apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0010] Embodiments of the present invention are illustrated by way of example and are not limited to the figures in the accompanying drawings, in which reference numerals indicate origin numbers, similar reference numerals indicate similar elements, and wherein:

[0011] Figure 1A-Figure 1B A block diagram illustrating the enabling of virtual assistants and personal data domains in an electronic device system according to an implementation scheme is shown;

[0012] Figure 2 A home network environment including multiple smart home devices according to an implementation scheme is shown;

[0013] Figure 3 It is a flowchart of the operation of the discovery process for discovering and pairing common equipment with accompanying equipment, according to the implementation plan;

[0014] Figures 4A-4C This is a flowchart illustrating the process of enabling a virtual assistant client on a public device to process queries in a privacy-preserving manner;

[0015] Figure 5 The process of establishing a companion link connection with a paired device according to the implementation scheme described herein is illustrated;

[0016] Figure 6 A process for allowing access to personal data on an accompanying device according to an embodiment described herein is illustrated;

[0017] Figure 7 It is a block diagram of a computing device used in a smart home device according to the implementation plan;

[0018] Figure 8 This is a block diagram illustrating an exemplary API architecture that can be used in some embodiments of the present invention;

[0019] Figures 9A-9B This is a block diagram of an exemplary API software stack according to the implementation plan;

[0020] Figure 10 A block diagram of the mobile device architecture according to the implementation plan;

[0021] Figure 11 Here is a block diagram of the computing system according to the implementation plan; and

[0022] Figure 12 A block diagram of the virtual assistant according to the implementation plan. Detailed Implementation

[0023] A network of connected smart home devices may include a common device used by multiple users within the home. Because the device is a common device with multiple users, this may help avoid storing personal information specific to any given user of the device. Furthermore, it may be advantageous to prevent users from making unauthorized inquiries about the personal data of other users of the common device. Exemplary personal or private user information associated with each user of the common device may include, but is not limited to, contact lists, calendar entries, text messages, emails, call history, alarms, reminders, and other electronic data that may be considered personal or private user data. In various embodiments, additional personal or private information may include, but is not limited to, location information, family relationships, user preferences, application information, data stored on personal devices, device location, home device location, medical information, or health information. In some embodiments, some user data is considered inherently personal or private, while other information may be designated as personal or private information by the user.

[0024] The implementation described herein provides functionality that enables a virtual assistant on a public device to access a personal domain via a paired user device. The paired user device can serve as a companion device to the public device. Pairing between the public device and the companion device allows the devices to establish trusted and secure communication over a secure data channel. When the virtual assistant on the public device receives a query requiring access to personal user information not stored on the public device or accessible on the public device, the public device can connect to the companion device and request the companion device to perform an operation related to the personal user data.

[0025] Communication between public devices and accompanying devices can be performed on a secure data channel known as an accompanying link. The accompanying link provides a persistent, low-latency, instant messaging system for connected devices in a home network environment. In some implementations, the accompanying link supports links between fixed public devices within the home, such as speaker equipment, and personal mobile devices within the home or accessible via the internet. The accompanying link enables public devices to redirect personal requests that they cannot handle due to privacy concerns from one or more personal devices. Redirecting personal requests to personal devices associated with a user allows virtual assistants on public devices to receive and request requests confidentially.

[0026] In addition to redirecting individual queries to accompanying devices, in one implementation, the accompanying link also provides a universal instant messaging system for devices in a home network environment. The universal instant messaging system enables multiple home devices to work collaboratively by exchanging messages on the accompanying link. For example, audio playback between multiple smart speaker devices can be coordinated, enabling the devices to perform actions such as, but not limited to, coordinating the playback of media items, selectively providing audio notifications to the user via the speaker closest to the user, configuring multiple speakers into a multi-channel audio system, or coordinating audio avoidance at the speaker during the duration of verbal requests and responses.

[0027] Public devices can announce support for companion link services via discovery protocols. Personal user devices (PUEs) on the same network as the public device can discover the companion link services announced by the public device and connect to the public device using the announced information. The PUE can perform a pairing process with the public device to become the user's companion device. In one embodiment, the pairing process includes a proximity element, where the user device exchanges identifiers, keys, or secrets with the companion device via a short-range wireless communication mechanism. The pairing process may also include the exchange of information regarding the presence and accessibility of subsequent connections between the public and companion devices via the companion link.

[0028] In the accompanying drawings and the following description, reference numerals indicate the figures in which referenced elements are introduced, such that elements with reference numeral N00 are first introduced in Figure N. For example, elements with reference numerals between 100 and 199 are shown in Figure N. Figure 1A and 1B The first part is shown, while the element with reference numerals between 200 and 299 is first shown in Figure 2 The diagram shows, etc. Previously introduced elements may or may not be referenced in the description of a given diagram.

[0029] The processes and operations depicted in the following figures can be performed by processing logic comprising hardware (e.g., circuit systems, special-purpose logic, etc.), software (e.g., instructions on a non-transitory machine-readable storage medium), or a combination of both. While some processes are described below in sequence, it should be understood that some operations described may be performed in a different order. Furthermore, some operations may be performed in parallel rather than sequentially. Additionally, some operations may be designated as optional and not performed in all embodiments.

[0030] Systems and Environment

[0031] Figure 1A-Figure 1B A block diagram is shown illustrating the enabling of virtual assistants and personal data domains in an electronic device system according to an implementation scheme. Figure 1A A block diagram of a virtual assistant system 100 according to the embodiment described herein is shown. Figure 1BA block diagram is shown illustrating how the accompanying link system 130 enables communication between devices within the virtual assistant system 100 according to an implementation scheme. Figure 1B The accompanying link system 130 can be used to establish a personal data domain within the virtual assistant system, enabling the virtual assistant on the public device to answer queries that require access to personal user data not stored on the public device. Figure 1A-Figure 1B The block diagram is logical because some of the components shown can be separate logical components residing within the same physical server or device.

[0032] Figure 1A A virtual assistant system 100 is illustrated, which provides a processing system that interprets natural language input received in spoken and / or text form to infer user intent. The virtual assistant system 100 can then perform operations based on the inferred user intent. In various embodiments, the system can perform a set of operations including, but not limited to, identifying a task flow with steps and parameters designed to implement the inferred user intent; inputting specific requirements from the inferred user intent into the task flow; executing the task flow by invoking programs, methods, services, APIs, etc.; and generating output responses to the user in an audible (e.g., voice) and / or visual form. As used herein, the term virtual assistant is used interchangeably with the terms "digital assistant," "intelligent automated assistant," or "automatic digital assistant," and generally refers to any information processing system that interprets natural language input in spoken and / or text form to perform actions on behalf of the user.

[0033] Virtual assistants can accept user requests, at least in part, in the form of natural language commands, requests, statements, narration, and / or inquiries. Typically, user requests seek an informative answer from the virtual assistant or request the virtual assistant to perform a task. A satisfactory response to a user request can be providing the requested informative answer, performing the requested task, or a combination of both. For example, a user might ask a virtual assistant a question such as, "Where am I now?" Based on the user's current location, the virtual assistant could answer, "You are near the west gate of Golden Gate Park." A user might also request to perform a task, such as, "Please invite my friends to my girlfriend's birthday party next week." In response, the virtual assistant could confirm the request by saying "Okay, coming right away" and then send the appropriate calendar invitations to each of the user's friends listed in their electronic address book. During the performance of a requested task, a virtual assistant can sometimes interact with the user in a sustained conversation involving multiple exchanges of information over extended periods. Many other methods exist for interacting with a virtual assistant to request information or perform various tasks. In addition to providing verbal responses and taking programmed actions, virtual assistants can also provide responses in other forms of video or audio (e.g., as text, alarms, music, videos, animations, etc.).

[0034] like Figure 1A As shown, the virtual assistant system 100 may include a server system 108 having one or more server devices. In one embodiment, the server system 108 includes a virtual assistant server 106 capable of communicating with multiple virtual assistant clients (VACs) (e.g., VAC 101, VAC 102). The virtual assistant clients may execute on multiple electronic devices that can connect to and communicate with the server system 108. A first virtual assistant client (e.g., VAC 101) may execute on a smart home device 103 (such as a smart speaker device). A second virtual assistant client (e.g., VAC 102) may execute on a user device 104, such as a smartphone device. The virtual assistant clients may communicate with the virtual assistant server 106 via one or more networks 110, which may include a combination of local and wide area networks. Various virtual assistant clients may provide client-side functionality, such as user-oriented input and output processing, and may communicate with the virtual assistant server 106. The virtual assistant server 106 may provide server-side functionality for multiple virtual assistant clients on any number of user devices connected via one or more networks 110.

[0035] In some implementations, the virtual assistant server 106 includes an I / O interface 112 to a VA client, one or more processing modules 114, a storage device 116 including data and processing models, and an I / O interface 118 to an external service. The client-facing I / O interface 112 facilitates client-facing input and output processing of the virtual assistant server 106. One or more processing modules 114 may utilize the data and processing models 116 to perform natural language processing on voice input to infer user intent. The one or more processing modules 114 may then perform task execution based on the inferred user intent. The I / O interface 118 to the external service facilitates communication between the virtual assistant server 106 and the external service 120 via one or more networks 110. In one implementation, the virtual assistant server 106 may communicate with the external service 120 to complete a task in response to a request received on the VAC 102 on the user equipment 104, or to obtain information in response to a query received on the VAC 102. The external service 120 may include, but is not limited to, navigation services, instant messaging services, information services, calendar services, and / or telephone services. For example, one or more navigation services may be used to enable turn-by-turn navigation on user equipment 104 in response to a request received on VAC 102 on the user equipment. External services may additionally include location information, weather, financial information, or account information. In various implementations, the logic for enabling some external services 120 may reside within server system 108, while some portions of the logic may reside within user equipment 104.

[0036] The virtual assistant server 106 can communicate with one or more instant messaging services to send messages in response to voice input received from a virtual assistant client. Information services such as internet search engines or information databases can be accessed by the virtual assistant server 106 to provide information to the user in response to information queries. In one embodiment, the virtual assistant server 106 can access one or more calendar services to schedule tasks or events, or to satisfy queries for tasks or events. The virtual assistant server 106 can also access one or more telephone services to initiate telephone calls for the user.

[0037] In various implementations, virtual assistant server 106 can assist VAC 101, 102 by processing voice input provided by the user to the client. In one implementation, text-to-speech processing and natural language processing may be performed partly by VAC 101, 102 and partly on virtual assistant server 106. The data and processing model 116 of virtual assistant server 106 may be more complex and perform better than the corresponding model of locally accessible VAC 101, 102. Therefore, virtual assistant server 106 may be better suited to process some commands and may send the results of processed commands back to VAC 101, 102 via one or more networks 110. Operations for constructing the virtual assistant interaction stream presented to the user may be shared between virtual assistant server 106 and VAC 101, 102. In one implementation, initial processing of user input, presentation of voice output, and access to most user data may be performed by VAC 101, 102. In such implementations, speech recognition and natural language processing may be performed at least partially by virtual assistant server 106.

[0038] Smart home device 103 and user device 104 can communicate with each other and with server system 108 via one or more networks 110, local area networks (LANs) and / or wide area networks (WANs), such as the Internet. The one or more networks can be implemented using any known interconnect or network protocols, including various wired or wireless protocols. Exemplary interconnect and / or network protocols include Ethernet, Universal Serial Bus (USB), FireWire, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Long Term Evolution (LTE), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.

[0039] In various implementations, server system 108 may be implemented on a standalone data processing device and / or a distributed network of computers. In some examples, server system 108 may also utilize various virtual devices and / or services from third-party service providers (e.g., third-party cloud service providers) to provide potential computing and / or infrastructure resources for server system 108.

[0040] In various implementations, the smart home device 103 may be a smart speaker device, a smart home hub device, or another type of smart electronic device. For example, and in one implementation, the smart home device 103 is a smart speaker device that can be configured as part of a distributed media playback system. A specific example of a smart speaker device is Apple Inc. from Cupertino, California. A smart speaker device. In one embodiment, smart home device 103 may be a smart home hub device configured to manage one or more other devices in a digitally connected smart home system, including but not limited to smart and / or digitally connected devices such as smart thermostats or smart lighting systems. Smart home device 103 may connect to user device 104 to exchange data. The connection between smart home device 103 and user device 104 may be performed on one or more networks 110. In one embodiment, smart home device 103 and user device 104 may dynamically maintain a persistent connection as user device 104 switches between LAN and Internet connections.

[0041] User equipment 104 can be any suitable electronic device. For example, and in one embodiment, user equipment 104 can be a portable multifunction device, a multifunction device, or a personal electronic device. A portable multifunction device can be, for example, a mobile phone that also includes other functions such as a personal data assistant, a music player, and / or an application processor capable of executing applications (e.g., applications, etc.). Specific examples of portable multifunction devices may include those from Apple Inc. (Cupertino, California). Devices, iPod Equipment and Devices. Other examples of portable multi-functional devices may include, but are not limited to, laptop computers or tablets, or wearable electronic devices such as smartwatches. In some embodiments, user equipment 104 may be a non-portable multi-functional device, such as, but not limited to, a desktop computer, game console, television, or set-top box. In one embodiment, user equipment 104 may be a light fixture in a home, vehicle, motel, etc. In some embodiments, user equipment 104 may work in conjunction with or at a vehicle door or another entrance to a vehicle, home, or mobile home.

[0042] User equipment 104 may include a touch-sensitive surface (e.g., a touchscreen display and / or touchpad interface), or one or more physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0043] In one implementation, VAC 101 on smart home device 103 may communicate with VAC 102 on user device 104 to facilitate the processing of voice input or to infer the intent of commands received at smart home device 103. For example, in one implementation, smart home device 103 may be a common device used by multiple users within a home. Certain personal or private information of the common device users, such as contact information, calendar information, message data, etc., may be categorized as private to a given user. To avoid storing such private information on smart home device 103 as a common device, or to enable the common device to access a server (e.g., virtual assistant server 106) containing or accessing such information, VAC 101 on smart home device 103 may be associated with user device 104 that facilitates access to such information. When VAC 101 on smart home device 103 executes a command or action that requires access to private information associated with a user, VAC 101 on smart home device 103 may communicate with user device 104 and / or VAC 102 on user device 104 to obtain information for executing the requested command or action.

[0044] A single user can be associated with multiple user devices 104. In this case, a single instance of user device 104 can be designated as a companion device. The smart home device 103 will then communicate with the companion device to enable the processing of certain requests. For example, a user can have multiple instances of user device 104, such as one or more smartphone devices, one or more tablet devices, or laptop devices. The user can designate one of such user devices 104 as a companion device, and the smart home device 103 will communicate with the companion device to obtain access to private user information.

[0045] In some implementations, smart home device 103 and user device 104 may establish and maintain an accompanying link 105, which is a persistent or semi-persistent connection, enabling VAC 101 on smart home device 103 to query private information from user device 104 and / or VAC 102 on user device 104. In one implementation, accompanying link 105 is an always-connected low-latency instant messaging system that enables communication between devices within the home. Such a connection may be maintained locally on a LAN or established over a wide area network such as the Internet. Accompanying link 105 may enable public devices to redirect personal requests to the user device, allowing requests to be processed confidentially.

[0046] Figure 1B It shows Figure 1B The accompanying link system 130 enables communication between devices within the virtual assistant system 100. In various embodiments, the accompanying link system 130 can enable secure and authenticated message exchange between multiple devices. The multiple devices may include one or more user devices (e.g., user device 104, user device 115), each of which may be a portable multi-functional device. The multiple devices may also include one or more fixed devices, such as one or more public smart home devices (e.g., smart home device 103, smart home device 133), which can be used by multiple users. In one embodiment, a set of accompanying links 105a-105b enables message-based communication between user devices and one or more public smart home devices. Accompanying links 105c may also be established between public smart home devices to enable communication between those devices. In one embodiment, accompanying links 105d may also be established between user devices to enable peer-to-peer exchange of device data.

[0047] In one implementation, a set of accompanying links 105a-105b may be used to enable a public fixed device to redirect personal requests to a personal device associated with the user, allowing such requests to be processed confidentially. For example, and in one implementation, a user may speak a personal query to the public device, such as “When is my next meeting?” This personal query may be received via VAC 101 or VAC 131 on smart home device 103 or smart home device 133. Since this request uses the speaker’s calendar, the public device may not have direct access to such personal data. To satisfy such a request, the public device may redirect the query to a designated accompanying device, such as user device 104, to perform analysis on the query and generate an audio response that can be sent back to the public device for playback to the user.

[0048] In various implementations, redirection can be performed explicitly or implicitly. For explicit redirection, VAC 101 on smart home device 103 or VAC 131 on smart home device 133 can receive the request and at least partially submit it to virtual assistant server 106 for processing. Virtual assistant server 106 can determine that, in order to satisfy the request, at least some information is available from an accompanying device (e.g., user device 104). In one implementation, explicit redirection can be performed in response to such a determination, in which virtual assistant server 106 can send a request over one or more networks 110 to user device 104 to retrieve personal information to be used to process the request. In one implementation, implicit redirection can be used on public smart home devices, where a virtual assistant client on smart home device 103 can redirect the virtual assistant request to VAC 102 on user device 104. VAC 102 on user device 104 can then process the request as if it were received directly at user device 104. User device 104 can then configure an audio response for playback on a public device.

[0049] In addition to enabling confidential query processing, in one implementation, the accompanying link 105c enables universal instant messaging, allowing communication between common devices such as smart home device 103 and smart home device 133. Such links enable common devices to cooperate and coordinate operations. For example, smart home device 103 and smart home device 133 may be configured as part of a distributed media playback system capable of playing music or other media. The accompanying link 105c established between smart home device 103 and smart home device 133 can be used to coordinate media playback or coordinate multi-channel audio playback across multiple devices. When a user queries one of the smart home devices, that device can exchange messages to coordinate the avoidance of audio playback and any associated audio responses for the duration of the verbal request. Using the accompanying link system 130, any number of common smart home devices and user devices can communicate to exchange audio data, media playlists, configuration data, and other information for enabling connected digital home systems.

[0050] In one embodiment, accompanying links 105a-105d can be established via a local area network (e.g., LAN) through a local wired and / or wireless network connection. In one embodiment, accompanying links 105a-105d can also be established via a WAN connection (such as an Internet connection), although policies and configurations may optionally be configured to restrict accompanying links 105c between smart home devices 103, 133 to a single network. In one embodiment, in addition to the virtual assistant server 106, the server system 108 may include an online account server 136, and user devices 104, 115 and smart home devices 103, 133 may be connected to the online account server 136 via one or more networks 110. The online account server 136 may include information associated with online accounts of users associated with various devices. In some embodiments, the online account server 136 may also include or be associated with server hardware and software logic to provide online services, including online storage, instant messaging, email, media and streaming services, or navigation services. In one embodiment, the online account server 136 may also provide or facilitate access to… Figure 1B One or more of the external services 120. In one embodiment, the online account server 136 may also provide or facilitate access to online stores, such as online app stores (e.g., application stores) and / or online media stores, such as music, video, or ebook stores.

[0051] In one implementation, the accompanying link 105d between user devices 104 and 115 can be used to enable peer-to-peer data exchange. In one implementation, automatic data exchange between devices can be configured to be performed on the accompanying link 105d. For example, if user devices 104 and 115 are each associated with the same account on an online account server 136, credential exchange can be performed through the online account server to enable the automatic establishment of an encrypted communication channel between devices whenever the devices are within direct communication range. In one implementation, data from cloud services associated with the online account server 136 can be exchanged between devices via the accompanying link 105d instead of via one or more networks 110. In one implementation, the online account server 136 can maintain a list of devices associated with individual accounts or a list of devices associated with a family of associated accounts. The list of devices associated with accounts in a family of accounts can be used to facilitate device discovery and the establishment of accompanying link connections between devices. Accounts on the online account server can enable mutual authentication between electronic devices through the exchange of authentication credentials.

[0052] In one implementation, the online account server 136 includes or is associated with a registration server to register a unique device identifier associated with each device, each device having an online account associated with a user. Once a device registers with the user's online account, an identifier token is created that enables the device to be located and identified on network 110. The identifier token may include one or more encryption keys, tokens, certificates, or other data that allows the virtual assistant server 106, online account server 136, and other servers within server system 108 to locate and identify the device on one or more networks 110. In some implementations, if the user device is outside the home or not connected to a home network of smart home devices, smart home devices 103, 133 may use the identifier token to establish a remote connection with user device 104. Smart home devices 103, 133 may use the identifier token to locate user device 104 on a mobile wireless network and establish a secure remote companion link connection with user device 104. The virtual assistant server 106 and online account server 136 may assist in locating user device 104 and establishing a connection with the user device. In such implementations, at least a subset of the accompanying device functions of user equipment 104 may continue to operate when user equipment 104 is away from home, or if the user is at home but not connected to the home network.

[0053] In some implementations, each virtual assistant client (VAC 101, VAC 102, VAC 131) executing on the device may be associated with a virtual assistant identifier. In various implementations, the virtual assistant identifier may be associated with or derived from an identifier token of the host device, an account identifier associated with the user's online account on the device, and / or another token associated with or derived from an account or device identifier. When the virtual assistant clients are communicating with each other or with the virtual assistant server 106, the virtual assistant identifier of the virtual assistant client can uniquely or quasi-uniquely identify VAC 101, 102, 131 on various devices. The virtual assistant identifier may be associated with the identification or location information of the virtual assistant's host device, such as the host device's identifier token. After a pairing relationship is established between smart home devices 103, 133 and user device 104, the virtual assistant clients (VAC 101, VAC 131) on the smart home devices may store the virtual assistant identifier and use it to locate, identify, and communicate with VAC 102 on user device 104. The virtual assistant identifier of VAC 102 may also be sent to the virtual assistant server 106 and stored as an accompanying virtual assistant identifier. If the virtual assistant server 106 requests access to private information related to a user performing a processing operation received at VAC 101, 131 of smart home devices 103, 133, the virtual assistant server 106 may contact VAC 102 of user device 104 on behalf of VAC 101, 131. In various embodiments, the virtual assistant server 106 may then receive from VAC 102 of user device 104 private information from user device 104, the processing results of information processing tasks assigned to user device 104, or permissions and / or access credentials for accessing private data on behalf of user device 104.

[0054] In some implementations, server system 108 includes a relay server 126 that can be used to facilitate remote connections between smart home devices 103, 133 and user device 104. Relay server 126 can enable a relay service that can relay accompanying link messages between devices when a local connection or another remote connection (e.g., a remote peer-to-peer connection) cannot be established. Relay server 126 can enable instant messaging of remote accompanying links based on a relay pairing established between smart home devices 103, 133 and user device 104. Keys, certificates, and other identification and authentication data associated with the various devices can be exchanged during a local pairing process that can be used to establish a relay pairing between devices. Once a relay pairing is established between the devices, smart home devices 103, 133 can send messages to relay server 126, which has a unique relay identifier associated with user device 104.

[0055] Each device that can connect to relay server 126 may have an associated relay identifier, which is a permanent pairing token that can be used to identify and verify the connected device. This relay identifier may be, include, or be derived from a device identifier or identifier token that uniquely identifies the device and may include a certificate and / or signature that enables relay token verification. Relay server 126 can then relay messages to user device 104. In one embodiment, a persistent and bidirectional connection may be established, enabling user device 104 to send return messages to smart home devices 103, 133 via relay server 126. In one embodiment, user device 104 may also initiate a connection with smart home devices 103, 133 using the relay identifier associated with the device. In one embodiment, the relay identifier is used for each message exchanged through relay server 126, enabling relay server 126 to verify the authenticity of each message relayed through the server and prevent unauthorized devices from transmitting messages via the established relay server connection.

[0056] Figure 2 A home network environment 200 comprising multiple smart home devices is illustrated according to an embodiment. The home network environment 200 may include a wireless access point 212 to provide access to a wireless network serving the home network environment. Multiple smart home devices 204, 206, 214, 216, 222A-222B may be connected to the home network environment 200 via a wireless network or alternatively a wired network connection. In various embodiments, the home network environment may include various types of smart home devices. For example, smart home devices 204, 206, 214 may be smart speaker devices configured to distribute media playback. Additionally, smart home device 216 may be a smart appliance, such as a smart refrigerator. Each of the smart home devices 204, 206, 214, 216, 222A-222B may use the network of the home network environment 200 to establish an interconnecting link, enabling the devices to exchange configuration information. For example, smart home devices 222A-222B can be configured as multi-channel smart speaker devices that can use accompanying links to configure multi-channel (e.g., stereo, surround, etc.) audio playback. Additionally, each of smart home devices 204, 206, 214, 216, and 222A-222B may include a virtual assistant client that can interact with a designated accompanying device via an accompanying link to facilitate processing requests for access to private user information. Mobile instances of user devices such as user device 104 or connected vehicle infotainment systems can also be configured to connect to the home network environment when near wireless access point 212. In one embodiment, user device 104 may also establish accompanying link connections with any other user device that can connect to the home network environment 200 or is within direct radio range of user device 104.

[0057] Devices and accompanying discoveries

[0058] Before establishing a companion link communication channel between user equipment 104 and smart home devices 204, 206, 214, 216, 222A-222B, a companion discovery and pairing process is performed. The companion discovery process enables the smart home devices to locate companion devices, such as virtual assistant clients on the smart home devices (e.g., ...). Figure 1A-Figure 1B VAC 101, 131) allows access to private user information that can be used to process and / or respond to user requests via an accompanying device. In some cases, the accompanying discovery process may also include user authentication that will occur between the smart home device and the accompanying device. In some implementations, accompanying discovery may utilize existing service discovery protocols that facilitate the location of devices and / or services over wireless or other networks, such as the Simple Service Discovery Protocol (SSDP) developed by the UPnP Forum or the Bonjour networking technology developed by Apple Inc. (published in IETF RFC 6762 and IETF RFC 6763, referred to herein as “Bonjour”). In a device discovery service, a device may advertise information indicating its presence, its address, and alternatively, additional information about its capabilities. Other devices, including other smart home devices or user devices, may browse the advertisement based on the broadcast information and identify the device of interest. Using the advertised address, the browsing device may initiate communication with the advertiser.

[0059] Depending on the network and discovery service, advertisements may optionally include real-time broadcast information (e.g., via multicast or beacon signals) and / or notification information to a central repository (e.g., at a network access point) from which other devices can retrieve information. Ad browsing may include detecting broadcast advertisements and / or retrieving advertisement information from the central repository. In some embodiments, public smart home devices permanently attached to a power source (e.g., a power outlet) may continuously perform notification and discovery of the accompanying link service. Mobile user equipment may enable the discovery of the accompanying link service based on the user equipment's location. For example, and in one embodiment, a geofence boundary 203 is configured on the mobile device such that accompanying link discovery is enabled when the mobile device is located in a geographic location adjacent to the location designated as the home location of user equipment 104.

[0060] When a public smart home device is discovered by a user device (e.g., user device 104) acting as a companion device, a network data connection (e.g., TCP, UDP, etc.) can be established between the public smart home device and the companion device. Any network layer (e.g., Layer 3) protocol can be used to establish the network data connection. To avoid a connection race between devices that are both announcing and discovering, the device with the lowest lexicographically ordered permanent identifier initiates the connection between the devices. The device's permanent identifier is derived from an anonymous identifier announced through the discovery service. In one embodiment, data exchanged through a previously performed pairing process is utilized to derive the persistent identifier based on the announcement information. In such embodiments, a data connection cannot be established using the smart home device until it is paired with the user device because the permanent identifier used to connect to the public smart home device is unknown. Once the data connection is established, a secure communication session can be established between the public smart home device and the companion device. The public smart home device and the connected companion device can then exchange presence and accessibility information. If the accompanying device is a mobile device, it can then enter a Wake-Up Wireless (WoW) state as needed when no data is exchanged on the accompanying link, while the public device connected to a power source can remain active to reduce first message delay.

[0061] In the event of a lost connection between devices, the discovery service can be reactivated and used to search for devices on the local network. If the lost device is rediscovered, the data connection between the devices can be re-established. If the lost device cannot be discovered, the status discovery service information in the lost device's record is reconfirmed and cleared. The searching device can then attempt to establish communication with the lost device via a secure internet session. In one implementation, a portion of the presence and accessibility information exchanged when establishing a data connection includes a device identifier, identifier token, relay identifier, or other form of identification token that can be used to access or enable message exchange with the lost device, such as via a peer-to-peer or relay internet connection. If a secure internet connection can be successfully established using the previously lost device, accompanying link messages can be exchanged over the secure internet connection.

[0062] When an accompanying device connects to a smart home device via an internet-based connection and a local connection becomes available, the accompanying link connection can be switched to a local connection. For example, user equipment 104 may cross geofence boundaries 203, enabling discovery protocols (e.g., SSDP, Bonjour, etc.) to search for devices (e.g., smart home devices 204, 206, 214, 216, 222A-222B). If user equipment 104 discovers the availability of a local connection for the smart home device, and user equipment 104 connects to the smart home device via an internet connection, the user equipment can switch from the internet connection to a local (e.g., Wi-Fi) connection. In one implementation, the connection switch can be performed whenever connectivity between connected devices is lost (e.g., user equipment 104 leaves the home network environment 200) or if a device determines that a better connection is available (e.g., user equipment 104 returns to the home network environment 200). A local connection may be preferred over an internet connection because it may have lower latency. Furthermore, it may require fewer device and infrastructure resources to maintain a local connection compared to an internet connection.

[0063] Figure 3 This is a flowchart illustrating the operation of a discovery process 300 for discovering and pairing a public device with a companion device according to the embodiments described herein; the discovery process 300 shown includes operations performed on a public device 302 and a personal device 304. The public device 302 may be any smart home device described herein (e.g., such as...). Figure 1A and 1B The smart home devices 103 and 133 shown are, for example... Figure 2 The smart home devices shown are 204, 206, 214, 216, 222A-222B. Personal device 304 can be any personal user device described herein (e.g., user device 104), such as, but not limited to, a smartphone, tablet, or other user device storing personal data accessible by the public device 302 via an accompanying link. While the discovery process 300 is shown and described as being performed between the public device 302 and the personal device 304, variations of the shown process can also be performed between multiple instances of the public device 302 or between multiple instances of the personal device 304.

[0064] In one implementation, as shown in box 310, public device 302 may set status bits to indicate that the device is currently unpaired, or in other words, it is searching for a companion device to pair with. The status bits may be bits in the status flag indicator listed in the status information announced by public device 302. At box 312, public device 302 may announce its presence via a discovery protocol (e.g., SSDP, Bonjour, etc.) because it supports companion link services. For example, using Bonjour, public device 302 may announce itself with a name and service type. The name may be a user-readable name for companion discovery (e.g., “speaker”); in some instances, the announced name may be a name specified in the companion discovery information service instance in the device definition record. The service type may be defined for a uniform attachment protocol (e.g., service type “_companion-link._tcp”). The announcement may also include additional information. Similar information may be distributed using other service discovery protocols and technologies. For example, using SSDP, companion discovery may include announcing the name and service type URI using a multicast HTTP notification message. The URI can be used by personal device 304 to retrieve additional information via a unicast request to public device 302.

[0065] After public device 302 begins advertising the accompanying link service via the service discovery protocol, personal device 304 can discover public device 302 when browsing unpaired devices, as shown in box 314. No specific timing is required between the start of the advertisement and the start of the service browsing; however, when personal device 304 is browsing, it will not be able to discover public device 302 unless the public device 302's discovery advertisement is detectable.

[0066] In one embodiment, personal device 304 may browse unpaired devices in response to triggers, such as triggers provided by an application executed on personal device 304. In one embodiment, personal device 304 may browse unpaired devices when placed in physical proximity to them. For example, out-of-the-box setup for public devices may include data exchange on short-range wireless communication mechanisms (e.g., using Bluetooth and / or Bluetooth Low Energy, NFC, etc.) that may trigger personal device 304 to browse unpaired devices.

[0067] In box 316, personal device 304 can find devices via discovery service advertisements, for example, by detecting the announcement performed in box 312. At box 318, personal device 304 can determine, based on the announcement, that the discovered device is a connection candidate, such as unpaired public device 302. In one embodiment, personal device 304 can determine that the discovered device is a connection candidate when it is advertising the companion link service. Personal device 304 can check the discovery status flag to determine if public device 302 has been configured or paired with a companion device. Alternatively, personal device 304 can check the protocol version of the announcement to determine if the companion link protocol version of public device 302 is compatible with the companion link protocol supported by personal device 304. If personal device 304 determines that the companion device is not the one being advertised for establishing a companion link, personal device 304 can return to box 314 and continue browsing. In one embodiment, after a period of no public device discovery, the browsing operation may time out and browsing of unpaired devices may be interrupted for a period of time.

[0068] At box 322, personal device 304 may present information about public device 302 to the user via a user interface, such as a display device for personal device 304. At box 324, personal device 304 may receive input from the user via the user interface regarding actions to be performed with the detected device. For example, the user may provide input instructing whether personal device 304 should pair with public device 302. Personal device 304 may present any or all information obtained from notification data provided by public device 302 and prompt the user to instruct whether personal device 304 should connect to public device 302. Requesting user confirmation can help avoid false or unwanted pairing between public device 304 and personal device 302. At box 326, personal device 304 may interpret the user input received at box 324 and determine whether to pair with public device 302. In addition to user approval to initiate the pairing operation, other operations may be performed to complete the pairing operation to minimize the risk of pairing occurring without the approval of the legitimate owner / operator of personal device 304, as completing the pairing process allows public device 302 to accept queries regarding the personal characteristics of the user of personal device 304. For example, personal device 304 and / or public device 302 may request a password known to the user of personal device 304. In one embodiment, public device 302 and / or personal device 304 may request biometric verification (e.g., fingerprint, facial recognition, etc.) to complete the pairing operation at box 326.

[0069] If the user instructs personal device 304 to refuse pairing or the user authentication operation fails, personal device 304 may return to box 314 to locate other accessories or devices. If personal device 304 and public device 302 pair at boxes 328 and 330, personal device 304 and public device 302 may each perform a pairing setup procedure. In some embodiments, this pairing setup procedure may be used to establish an encryption key to facilitate secure communication between personal device 304 and public device 302. In some embodiments, user authentication may be integrated into the pairing setup procedure, and separate user authentication is not required before initiating pairing setup. In one embodiment, the pairing setup procedure enables the establishment of a trusted relationship between public device 302 and personal device 304. The established trusted relationship may be verified later during the setup of the secure communication session.

[0070] If the pairing setup process is successfully completed, at box 331, the public device 302 may update the device status information to indicate that authorization to communicate with the accessory is now required and / or the accessory is now paired with at least one personal device, for example by updating the status flag indicator within the notification device data.

[0071] At block 332, personal device 304 may obtain and cache a device definition record from public device 302, which may be provided upon request at block 334. The device definition record may include a set of services supported by the device and / or other relevant characteristics that enable other personal devices 304 and other connected devices to determine how to control, connect to, or otherwise interact with public device 302. If personal device 304 caches the device definition record, the information can be used to facilitate the detection of state changes of public device 302. In some embodiments, personal device 304 may also cache information from notification data provided by public device 302, which can also be used to detect state changes of the public device.

[0072] At boxes 336 and 338, personal device 304 and public device 302 may begin exchanging data for establishing a secure communication channel. Data exchange may include the exchange of keys or certificates between the devices. Key exchange may be protected by a shared key exchanged between the devices, which may be exchanged using out-of-band communication methods. Additionally, data exchange may include the exchange of one or more long-term keys between the devices, which may themselves be protected by one or more short-term keys. Once a pairing is established, the pairing can be used to provide end-to-end message encryption, allowing only the paired device to read the messages exchanged between the devices. In one embodiment, the secure communication channel is a bidirectional channel, such that neither device initiates the message exchange. During message exchange, the communication session initiated by whichever device initiates the session is called the client device, while the device receiving the session is called the server device. In one embodiment, the first message exchanged between the devices is an information exchange message. The client device may send an initial information message including characteristic identifiers or other device information. The server device may then respond with an information message for supported functions. Once a secure communication channel is established and information exchange has occurred, the communicating devices may enter an idle state if there are no pending immediate actions.

[0073] In one implementation, public device 302 may perform additional operations at block 340 to verify that the connected personal device 304 is a companion device. In such implementations, the discovery process 300 may be performed with multiple devices, which are legitimate personal devices of the user but may not be personal devices already designated as companion devices for use with public device 302. In one implementation, only one of a plurality of possible personal devices will be designated as a companion device of public device 302. The user's personal device 304 designated as a companion device can be determined via a variety of mechanisms. In some implementations, the user may enable a setting on personal device 304 that indicates that the personal device can be configured as a companion device. In one implementation, this setting may be a specific companion device setting that indicates that the user has designated that particular personal device as a companion device for connecting smart home devices. In one implementation, the companion device selection may be determined indirectly based on other settings on personal device 304. For example, if the personal device is configured as a source of the user's location information, public device 302 may determine that personal device 304 is designated as a companion device.

[0074] It should be understood that the discovery and pairing process described herein is exemplary, and variations and modifications are possible. The operations described sequentially can be performed in parallel, the order of steps can be changed, and steps can be modified, combined, added, or omitted. Furthermore, while SSDP and Bonjour services are used as examples of device discovery services, similar concepts can be applied in the context of other device discovery services. In some implementations, before determining whether to pair with public device 302 or any other discovered device, personal device 304 may request a definition record (or a portion thereof) from a pairing candidate device, such as via an HTTP request. Depending on the configuration, a pairing candidate device may provide all, some, or none of its attachment definition records in response to a request from an unpaired personal device. Alternatively, the definition record may be accessible before pairing is established, and the decision to pair may be based on announcement information provided by the pairing candidate device. Announcement data may include, for example, the device's local name; a unique attachment identifier; a flag indicating that the attachment is discoverable; a universally unique identifier (UUID) for at least some services; an indicator of the device's status; and an indication of whether the device has already been paired with any other personal device.

[0075] In some implementations, discovery process 300, or a similar process, can be used to detect state changes in paired devices. For example, when a device state changes, the state value announced by the device can increment. When a device (e.g., public device 302) announces a state change, other paired devices (e.g., personal device 304) can announce the state change, for example, by broadcasting an updated BonjourTXT record, and paired personal devices that previously cached the device record can detect the change by comparing the broadcast value of the state number with the cached value.

[0076] The secure communication channel established between the devices at boxes 336 and 338 can be used to exchange companion link messages. In one embodiment, a secure session is established or re-established between public device 302 and personal device 304 before exchanging companion link messages on the companion link. In one embodiment, establishing a session includes performing a pairing verification process to verify the pairing between the devices. The pairing verification process relies on a previously established pairing relationship between public device 302 and personal device 304. For example, and in one embodiment, establishing a companion link session between devices may require a persistent identifier for the devices, which may be derived from an anonymous identifier announced through a discovery service. However, to derive a persistent identifier based on the announced information, one or more data elements exchanged during the pairing process between the devices can be used. In one embodiment, the pairing verification process may include generating short-lived public key pairs for encrypting or signing messages exchanged during the verification process, and verifying previously exchanged long-lived key pairs exchanged during the initial pairing of the devices. The pairing verification process can be performed each time a secure session is established between the devices using a pairing link connection.

[0077] Figures 4A-4C This is a flowchart of the process 400, 430, and 460 for enabling a virtual assistant client on a public device to process queries in a privacy-preserving manner. Figures 4A-4C The operation performed on common equipment 402 and accompanying equipment 404 is illustrated. In various embodiments, common equipment 402 may be as follows: Figure 3 A variation of the public equipment 302 shown. The accompanying equipment 404 can be as follows: Figure 3 The personal device 304 shown is undergoing a pairing process with the public device 402.

[0078] Figure 4A A process 400 according to some embodiments described herein is illustrated, wherein a virtual assistant client on a public device 402 may be connected to an accompanying device 404 to facilitate the processing of virtual assistant requests for access to data categorized as personal user data. Figure 4B A process 450 according to some embodiments described herein is illustrated, wherein a virtual assistant client on a public device 402 may implicitly redirect virtual assistant queries to an accompanying device 404. Figure 4C A process 460 for sending a message to a contact via a verbal request to a public device, according to some embodiments described herein, is illustrated. The use of a particular process may vary depending on policy, configuration, and network status. In one embodiment, the selection of a particular process may be determined based on the type of operation to be performed. For example, a request to read a received message may be based on... Figure 4A The process 400 is executed, while the request to send a message (e.g., Figure 4C Process 460) can be based on Figure 4B The process is executed in step 430.

[0079] like Figure 4A As shown, public device 402 can receive requests at a virtual assistant client running on the public device, as shown in box 410. At box 412, public device 402 can at least partially process the request to determine whether the request accesses a user's private data. The request can be a text-based request or a verbal request interpreted through natural language processing. In various embodiments, natural language processing can be partially on public device 402, entirely on the public device, or on a virtual assistant server (e.g., such as...). Figure 1A-Figure 1B The request is executed on the virtual assistant server 106. After the request has been at least partially processed, the public device 402 (or the virtual assistant server (if processing is offloaded)) may determine whether the request requires access to private data, as shown in box 414.

[0080] Determining whether a request uses personal user data at box 414 can be performed after the request has been processed and the intent of the request has been determined. For example, a request for the current weather forecast for the location of public facility 402 may not require access to data classified as personal user data. Such requests can be further processed, at least in part, by public facility 402, which can complete request processing and formulate a response to the request, as shown in box 418.

[0081] If public device 402 determines at box 414 that further processing of the request will require access to personal user data, public device 402 may submit a task associated with the request to the accompanying device for processing, as shown in box 415. Accompanying device 404 may receive the processing request at box 422. The task submitted by public device 402 at box 415 and received by the accompanying device at box 422 can be transmitted via the accompanying link established between public device 402 and accompanying device 404. For example, and in one embodiment, a virtual assistant client executing on public device 402 and accompanying device 404 may use public accompanying link messages to exchange data via an application programming interface (API).

[0082] In one implementation, at box 414, it is determined whether the request uses personal user data that can be communicated by a virtual assistant server (e.g., such as...) Figure 1A and 1B The virtual assistant server 106 shown is executed. When performing processing operations on a public device, the public device 402 can provide the virtual assistant identifier of the accompanying device 404. If the virtual assistant server will need to access private data, the virtual assistant server can use the virtual assistant identifier of the accompanying device to contact the accompanying device 404 on behalf of the public device 402.

[0083] In box 424, companion device 404 can access personal user data in response to a request from public device 402. Exemplary requests that can be facilitated by companion device 404 include requests to access the user's calendar, such as adding an event or querying the occurrence of the next calendar event. Companion device 404 can also access data within the user's contacts, such as looking up contact information for a contact in the user's contacts. Companion device 404 can access personal user data stored locally on the device. In one embodiment, companion device 404 can communicate with one or more servers (e.g., such as...). Figure 1B The online account server 136 communicates with the accompanying device to access private data that can be remotely accessed by the accompanying device based on online account credentials stored on or associated with the accompanying device. In one embodiment, this accesses personal user data.

[0084] At box 426, the accompanying device 404 can process the requested personal data portion. For example, the accompanying device 404 can perform a lookup for a set of scheduled tasks or events on a user's calendar, or add a task or event to the user's calendar. The accompanying device 404 can also send a text or email message to the user in response to a request received at the public device 402. For example, the accompanying device 404 can access the user's calendar data in response to a request to read or write a set of scheduled tasks or events for the user. The accompanying device 404 can also access data in the user's contacts, such as looking up phone numbers, email addresses, or instant messaging identifiers of contacts, and sending text or email messages to contacts.

[0085] The personal data portion of the request may also include interaction with one or more applications on the accompanying device 404, or with one or more underlying frameworks that provide backend services to those applications. In one implementation, the accompanying device 404 may provide licenses and / or credentials to the virtual assistant server (e.g., such as...). Figure 1A-Figure 1B The virtual assistant server 106 in the middle is enabled to use personal user data to perform processing tasks on behalf of the accompanying device 404.

[0086] At block 428, the accompanying device 404 may provide the output of a requested processing task to the common device 402 via an accompanying link established between the devices. The output provided to the common device 402 may vary across implementations and / or depending on the type of request received at the common device 402. In one implementation, the output of the requested processing task may be raw information, such as the time and date of the next calendar event, or data from another type of specific request. In one implementation, the output may be a subset of voice data output by the common device 402, such as voice output that can be integrated into the final output provided by the common device 402.

[0087] At block 416, public device 402 can receive the output of the requested processing task from accompanying device 404 via the accompanying link established between the devices. In one embodiment, personal data received from accompanying device 404 is not persistently stored or cached on public device 402 and can be discarded after use. Public device 402 can use the personal data to complete the processing of the virtual assistant request and formulate a response to the request at block 418.

[0088] In one implementation, to fulfill this request, the processing at box 418 may include data exchange between public device 402 and the virtual assistant server. Additional References Figure 1A In one implementation scheme, Figure 4A The public device 402 is a smart home device 103 with a virtual assistant client (e.g., VAC 101). VAC 101 can communicate with the virtual assistant server 106 via an I / O interface with the VA client 112. The processing module 114 can use data and processing model 116 to determine whether the received request is for a local weather forecast for the area of ​​the smart home device. The virtual assistant server 106 can then utilize an I / O interface to an external service 118 to access an external service 120 via one or more networks 110. For example, the external service 120 (e.g., an information service) can be accessed to retrieve the requested weather forecast. Information that can be used to formulate a response to the request can be output via the I / O interface with the VA client 112, and VAC 101 on the smart home device 103 can formulate a response to the request. Alternatively, in some cases, the virtual assistant server 106 can formulate a response to the received request and provide the response to the output via the I / O interface with the VA client 112. The VAC 101 on the smart home device 103 can incorporate the output into the final response.

[0089] Return to Figure 4A In one implementation, public device 402 may formulate a response to a request received at the virtual assistant client based on public information retrieved from a virtual assistant server associated with public device 402 and cached on public device 402, or based on personal information received from accompanying device 404. When the output is voice output, the voice output may be generated entirely on public device 402 using a local text-to-speech engine. In one implementation, a subset of the voice output may be generated by accompanying device 404 and incorporated into the response formulated by public device 402.

[0090] At box 420, public device 402 may output a response to a request. The response may include a verbal component output via a speaker on public device 402. In some embodiments, the output may include a visual component, such as animation, color changes, or other visual indications that may be indicated by public device 402. In some specific embodiments, textual information or other visual data may be displayed on a display associated with public device 402, although some specific embodiments of public device 402 may lack a display.

[0091] like Figure 4B As shown, in some embodiments, public device 402 and accompanying device 404 can perform process 430, in which the public device redirects an incoming virtual assistant client request to the accompanying device for processing. In one embodiment, public device 402 may be configured to explicitly have access to personal user data while implicitly redirecting all processing of one or more virtual assistant client requests to accompanying device 404. At block 440, public device 402 may receive the request at the virtual assistant client of the device that needs access to personal user data. At block 442, public device 402 may then redirect the request to accompanying device 404 via an accompanying link established with the accompanying device. In one embodiment, all or substantially all processing of the virtual assistant request may be performed on accompanying device 404 in a manner transparent to the user of the public device. Processing of virtual assistant requests on the accompanying device may also be partially performed on a virtual assistant server as described herein (e.g., such as...). Figure 1A-Figure 1B The virtual assistant server (106) performs these functions. For example, speech recognition and natural language processing can be performed at least partially by the virtual assistant server.

[0092] At box 452, companion device 404 may receive a request from public device at a virtual assistant client executed on companion device. Companion device 404 may then process the request on behalf of public device at box 454 and formulate a response to the output of the virtual assistant client on public device at box 456. In one embodiment, the processing and response generated on companion device 404 may appear as if the request was received directly on the virtual assistant client rather than companion device. At box 458, companion device may then provide a response to public device 402 for output. In one embodiment, the response provided to public device 402 may include specific voice output transmitted by public device 402. Alternatively, the provided response may be other data that can be rendered as voice output by public device via a voice engine on public device.

[0093] At box 459, if the request includes modification of such data, the companion device 404 may perform one or more virtual assistant actions on the personal user data. For example, the user's calendar, alarm clock, or reminder data may be updated on the companion device based on the request, an application may be opened on the companion device, configuration settings may be adjusted on the companion device, or any other virtual assistant action may be performed as if the request were received directly by the virtual assistant on companion device 404. Although box 459 is shown as being executed after box 478, the requested virtual assistant action may be executed relative to any order in which a response to the common device is provided for the output at box 458. For example, a request to open an application may be executed after an audio response confirmation request is provided to the common device for playback. Alternatively, a request for a schedule and event or alarm clock may be executed before or simultaneously with providing a confirmation response to the output of the common device.

[0094] At box 444, common device 402 can receive a response to the redirection request. Common device 402 can then output a response to the request at box 446. The output of the response can be compared with... Figure 4A The output performed at box 420 is performed in a similar manner and may include the output of voice data generated by accompanying device 404 or the output of voice data locally generated on common device 402 based on the output received from accompanying device 404. In cases where a request requires multiple rounds of interaction to complete, a multi-round request redirection can be performed between common device 402 and accompanying device 404 using the accompanying link established between the devices.

[0095] like Figure 4C As shown, in some embodiments, the public device 402 and the accompanying device 404 can perform a process 460 whereby the accompanying device redirects a virtual assistant client request to send a message (e.g., a text message, email message, or another direct message) to the accompanying device. The public device 402 can receive a verbal request at the virtual assistant client on the public device to send a message to a contact. The message can be a text message, an email message, or other types of direct messages that can be sent between user devices. In various embodiments, the text message can be a Short Message Service (SMS) message and / or other types of text-based messages, such as those provided by Apple Inc. in Cupertino, California. The system sends messages. Some implementations allow text messages to be sent via other types of non-SMS text messaging services. In one implementation, the message may be a recorded voice message. The contact to whom the message is sent may be a contact listed in a contact list, where the contact list may be considered part of the user's personal domain. In various implementations, the public device 402 may at least partially handle verbal requests and may also employ similar methods. Figure 4AThe operation at box 412 will offload at least a portion of the processing to the virtual assistant server (e.g., as...). Figure 1A and 1B (Virtual assistant server 106 in the diagram). In one embodiment, natural language processing operations on the virtual assistant server can determine whether personal user data will be used to process a verbal request received at box 470. For example, the virtual assistant server can determine that processing the verbal request will access data or services falling within the user's personal domain, such as the user's contact information and associated instant messaging services. Access to the user's instant messaging services may include access to the instant messaging service account associated with the user. To facilitate the processing of the verbal request received at box 470, public device 402 may send the instant messaging request to accompanying device 404 at box 472. At box 480, accompanying device 404 may receive the instant messaging request at the device's virtual assistant client. In one embodiment, the instant messaging request may be received indirectly from public device 402 at box 482 via the virtual assistant server that processes a portion of the verbal request.

[0096] The companion device 404 receives instant messaging requests directly or indirectly from the public device 402, and can process these requests via a virtual assistant client on the companion device 404. Processing the instant messaging request may include sending a message on behalf of the public device to an identified contact, as shown in box 482. Sending a message on the companion device 404 may include sending a text message via an SMS service, sending a message via an email client, or sending a message via another instant messaging service or application on the companion device 404. In one embodiment, the type of message to be sent may be specified within a verbal request received by the public device 402 at box 470. In one embodiment, message sending may be facilitated based on a user's online account, for example, with... Figure 1B The online account associated with the online account server 136 in the system. For example, an instant messaging service that sends messages to contacts could be an instant messaging service associated with a user's online account.

[0097] Accompanying device 404 may receive a response to a message sent at block 484. In one embodiment, as shown at block 484, if a response to a message is received within a time period, accompanying device 404 may send a response notification to public device 402 at block 486. The time period for which a notification will be sent for a received message may be specified within a configuration or policy associated with the virtual assistant system or may be configured based on user settings of preferences associated with the user, public device 402, or accompanying device 404.

[0098] In one implementation, companion device 404 will send a notification at block 486 to public device 402 that received the instant message request. In one implementation, where multiple instances of companion device 404 are connected to public device 402, the specific public device to which it is notified at block 486 can be determined using one of several techniques. As described above, one technique is to send the notification to the public device that received the message request. Another technique is to send the notification to the public device that the user recently interacted with. For example, a user may request a first public device in a first location to send a message to a contact. The user may then move to a second location and interact with a second public device. If a response to the message is received within the time period and the user recently interacted with the second public device, the notification may be sent to the second public device. Another technique that can be used is to send the notification to the public device closest to the user. In one implementation, one or more ranging techniques may be used to periodically determine the distance between companion device 404 and public device 402. When a response to a recently sent message is received, companion device 404 may send a notification to public device 402 that is closest to the public device, or associated wearable electronic device, according to one or more ranging techniques.

[0099] In block 476, the public device 402, which received the response notification in block 474, can output a notification of the received response in block 476. In one embodiment, the output of the notification may be provided via a verbal notification played through one or more speaker devices of the public device 402. In one embodiment, other notification technologies, including visual notification technologies, may be used. In one embodiment, the verbal notification may include text-to-speech conversion of one or more components of the received message, such as a spoken contact name. In one embodiment, the verbal notification may include the verbal content of the received message. In one embodiment, where the received message includes recording media, the recording media (e.g., audio, video) associated with the received message may be output. In some embodiments, playback of the verbal content of the received text message or the recorded media is performed only in response to a request provided by a user. In this embodiment, a verbal request to play the content of the response may be sent to an accompanying device, which can be based on... Figure 4A Process 400 or Figure 4B The process 430 handles the request.

[0100] Figure 5 A process 500 for establishing a companion link connection with a paired device according to the embodiment described herein is illustrated. Process 500 illustrates the operation in which the common device initiates a connection with the companion device via the companion link (e.g., Figures 4A to 4CThe common device 402 and the accompanying device 404 are mentioned. However, similar operations can be performed, where the accompanying device initiates a connection with the common device. In one embodiment, during message exchange via the accompanying link, regardless of whether the device initiating the communication session with respect to the accompanying link is a client device and the device accepting the session with respect to the accompanying link is a server device, the common device 402 and the accompanying device 404 can each initiate or accept the accompanying link connection. Furthermore, in one embodiment, this can be done within a home network environment (e.g., such as...). Figure 2 In a home network environment (200), accompanying link connections are established between multiple public devices. Additionally, accompanying link connections can be established between multiple accompanying devices, such as... Figure 1B The user equipment 104 and user equipment 114 shown are located between each other.

[0101] At box 502, it can be determined that a communication session should be established between the public device and the accompanying device via a companion link. This determination can be made by either the public device or the accompanying device. For illustrative purposes, process 500 shows the public device as a client and the accompanying device as a server, but the implementation is not limited to this. At box 504, the public device attempts to discover the accompanying device via its local network. Discovery can be performed using discovery services as described herein, including but not limited to Bonjour or SSDP. If the device is discovered locally at box 506, process 500 continues at box 516, where the public device can attempt to establish a local network connection with the accompanying device. The local network connection can be established via a network protocol such as, but not limited to, Transmission Control Protocol (TCP). If the device is not discovered locally, the public device can determine at box 508 whether a remote identifier is available. The remote identifier can be an identifier or identification token that enables the public device or other companion link clients to locate, connect to, and establish a link session with the paired device, which is not residing on the same network or otherwise cannot be discovered via a discovery service protocol. The local device discovery and / or pairing process (e.g., as described in...) Figure 3 During the presentation and accessibility information exchange that occurs during the discovery process (300) in the accompanying link device, one or more remote identifiers may be exchanged between the accompanying link devices. Exemplary remote identifiers include a virtual assistant identifier associated with a virtual assistant client on the accompanying device, which can be used via a virtual assistant server (e.g., such as...). Figure 1B Communicates with the virtual assistant server 106 in the system; communicates with the online account server (e.g., Figure 1B The identifier token established by registering with the online account server 136 in the network; and the identifier token established by registering with the relay server (e.g., such as Figure 1BThe relay identifier associated with the relay server (126). The public device can use the available remote identifier to query network addresses, such as Internet IP addresses, which can be used to connect to the accompanying device or to exchange relayed messages with the accompanying device via the relay service. In box 518, the public device can use the remote identifier to establish a remote network connection with the accompanying device.

[0102] In box 516, a connection is established via a local network connection, or in box 518, a connection is established via a remote network connection. The public device can then verify pairing with the accompanying device in box 520. Establishing a verified pairing ensures a genuine pairing relationship exists between the devices linked by the accompanying link and utilizes data exchanged during previous local pairing. In one embodiment, a pairing verification process is performed where each device demonstrates possession of a long-term private key corresponding to the long-term public key exchanged during the pairing setup. The pairing verification process may additionally include establishing a new shared key or session key, which can be used to encrypt communications occurring during the pairing verification session.

[0103] At box 522, the companion device can determine whether the pairing verification process was successful. If the pairing verification process at box 520 fails to verify the pairing relationship between the public device and the companion device, the companion device will be directed to box 510, where, at least for the user associated with the companion device, the virtual assistant's access to personal information will be unavailable.

[0104] If, as determined at box 508, the remote identifier is unavailable for the public device used to access the accompanying device, then the virtual assistant's access to personal information will also be unavailable. In such cases, if a query is given that requires access to the user's personal information without a paired and / or valid accompanying device, the virtual assistant on the public device will be unable to process the query.

[0105] If the public device determines that pairing has been successfully verified at box 522, it can establish a verified companion link session with the virtual assistant client on the public device via the companion device to serve queries for the private data of the user accessing the companion device at box 524. The verified session can enable end-to-end encrypted message exchange between the devices, allowing messages and data to be exchanged confidentially between them. Encrypted message exchange can be performed to enable, for example, relaying commands or responses for the virtual assistant against personal or private user data. Encrypted message exchange can also be performed to synchronize device data between electronic devices, such as device data associated with cloud service accounts, application data, or configuration data.

[0106] As described above, while process 500 is described as having a common device as the initiating connection device, accompanying devices may also initiate local or remote sessions with the common device. In one embodiment, once the verified accompanying link session is established, it is a bidirectional connection, enabling bidirectional message exchange between the devices. Furthermore, when no data is being exchanged on the accompanying link, any mobile device participating in the session can enter a wake-up wireless state, allowing the device to enter a low-power state while the radio and associated processor maintain the connection associated with the session.

[0107] In some implementations, the companion device described herein may be configured by default to restrict access to one or more elements of personal data. In one implementation, restriction may occur on certain elements of the personal data until the use of the virtual assistant is enabled on the companion device. In one implementation, restriction may occur on certain elements of the personal data while the device is locked. In one implementation, restriction may be applied until at least one successful unlock authorization has occurred since the device has been powered on. For example, in one implementation, personal data on the user device may be encrypted, and access to the key used to access such personal data may be restricted when the device is locked. Therefore, depending on the configuration of the companion device, if the request is received while the companion device is locked, the companion device may be unable to access the personal data used to process the request from the public device. In one implementation, it is possible to use... Figure 6 The process shown enables access to personal data on the accompanying device.

[0108] Figure 6 A process 600 for allowing access to personal data on an accompanying device according to an embodiment described herein is illustrated. Process 600 can be executed by logic within the accompanying device as described herein, wherein the accompanying device can be any of the specified personal user devices described herein. The accompanying device can receive a request from a public device to process a request for access to personal user data, as shown in box 602. The accompanying device can determine whether personal data access is enabled on the personal device. In various embodiments, access to personal data on the accompanying device can be restricted in multiple ways. In one embodiment, the device can be configured to restrict programmed access to personal user data when the device is locked. If personal data access is determined to be restricted (e.g., not enabled) at box 603, the accompanying device can prompt the user to access the personal data, as shown in box 604. The prompt for access to personal data can be a general prompt for access or a prompt specific to the type of personal data to be accessed. In various embodiments, the prompt at box 604 can be presented in various forms. In one embodiment, the prompt is a pop-up notification or another type of notification presented on the user interface of the accompanying device. In one embodiment, the prompt is a verbal prompt that can be spoken by the accompanying device.

[0109] If the user grants access to personal data at box 605, the accompanying device can access the personal data and process requests from the public device at box 608. The manner in which the user grants access to personal data can depend on the characteristics of the access point. For example, and in one implementation, the user may grant access via a selected user interface element or by providing verbal confirmation of the request to the accompanying device. If the user does not grant access to personal data, the accompanying device can reject the request from the public device at box 606.

[0110] Figure 7 This is a block diagram of a computing device 700 for a distributed media playback system according to an embodiment. In one embodiment, the computing device 700 includes hardware and software adaptable to user devices or smart home devices as described herein, such as public smart speaker devices or smart appliances with media playback capabilities. The computing device 700 includes one or more speaker devices 701 to enable media playback. When the computing device 700 is implemented as a smart speaker device, the speaker device 701 may have higher quality compared to when the computing device is implemented as a user device or smart appliance.

[0111] The computing device 700 includes a network interface 702 that supports network communication functions. The network interface 702 can be coupled to one or more wireless devices 703 to enable wireless communication over one or more wireless networking technologies, such as, but not limited to, Wi-Fi and Bluetooth. In some implementations, the network interface 702 may also support wired network connections. The computing device also includes a processing system 704 with multiple processor devices and a system memory 710, which may be a virtual memory system with an address space, including volatile and non-volatile memory.

[0112] In one embodiment, the processing system 704 includes one or more application processors 705 to execute instructions from user and system applications running on a computing device. The processing system may also include a sensor processor to process and monitor a suite of sensor devices 708, which have sensors including, but not limited to, motion sensors, light sensors, proximity sensors, biometric sensors, audio sensors (e.g., microphones), and image sensors (e.g., cameras). The sensor processor 706 may enable low-power monitoring of sensors within the suite of sensor devices 708 that are always on. When the computing device 700 is not in active use, the sensor processor 706 may allow the application processor 705 to remain in a low-power state while still allowing the computing device 700 to be accessed via voice or gesture input to the virtual assistant 720. In one embodiment, the sensor processor 706 or a similar low-power processor within the processing system may enable low-power processing of media instructions provided by a media player 725. The media player 725 may be a modular media player capable of playing back a variety of different audio and / or video media types, including but not limited to MPEG-2, MPEG-4, H.264, and H.265 / HEVC. In one implementation, other formats can be supported via additional codec plugins.

[0113] Virtual Assistant 720 is logic executed on computing device 700 to provide the intelligent automated assistant system described herein. Virtual Assistant 720 may selectively allow access to various software and hardware components within the computing device, including but not limited to network interface 702 for retrieving data over a network, media playback applications for initiating or stopping playback of media files, or user calendar data for scheduling calendar events, tasks, reminders, or alarms. When Virtual Assistant 720 executes on a common device as described herein, the virtual assistant may interact with accompanying link module 722 to locate and connect to the accompanying device or a virtual assistant executing on the accompanying device upon receiving a request to access the user's personal domain data.

[0114] If the computing device 700 is a smart speaker device capable of participating in a distributed playback system, the distributed playback module 712 can perform operations to manage various aspects of media playback, including but not limited to a playback queue manager 714 to manage the list of media played through the distributed playback system and a playback routing manager 716 to route media playback to specific elements of the distributed playback system. In one embodiment, the playback routing manager 716 can be connected to different elements of the distributed playback system via connections established using the accompanying link module 722. The accompanying link module 722 can facilitate the establishment of accompanying links between the speaker and devices in the distributed playback system, connection establishment, and message relay to perform operations such as configuring channel outputs for a multi-channel playback system or coordinating volume adjustment across multiple connected speakers.

[0115] In one embodiment, the event scheduler 730 can exchange data with the distributed playback module 712. Data exchange can be performed in response to input received via a user interface of computing device 700 or different computing devices participating in the distributed playback system. Data exchange can also be performed in response to activities requested via virtual assistant 720. For example, and in one embodiment, an event scheduled by the event scheduler 730 can be associated with a media playlist, such that when a scheduled event occurs, the playlist can be played via playback queue manager 714. For example, an alarm clock can be scheduled to wake a user at a specific time. The alarm clock can be associated with a playlist, such that one or more media elements will be played associated with or as a replacement for the alarm clock sound. In one embodiment, the playlist can be associated with any event scheduled by the event scheduler 730, including reminders or timer expiration events.

[0116] In one implementation, a playback queue manager 714 can manage multiple simultaneous playback queues, wherein each playback queue includes one or more past, current, or future media elements to be played via computing device 700. Playback queues can load a single media element or specify a playlist of multiple media elements. Playback queues may include locally stored media, media retrieved for playback from a media server, or media streamed from a local or remote media streaming server. Various types of media elements can be played via a distributed playback system through the playback queue manager, including multimedia files such as, but not limited to, music, music videos, and podcasts, including audio or video podcasts, or audio and / or video clips of current news, weather, or sports events.

[0117] If one or more podcasts are selected for playback in response to a scheduling event, the podcast selection logic can choose a specific episode of the podcast for playback, such as the most recently available podcast episode or the most recently available unplayed podcast. Such selection can be determined by explicit user preferences or by learned user preference information. Selection can also be based on the age of available unplayed podcasts relative to the current date. In one implementation, the podcast feed contains metadata indicating whether the podcast feed is associated with a serial podcast or a news-style podcast. Whether to play the earliest unplayed episode or the most recent episode is determined at least in part based on such metadata.

[0118] In one implementation, for example, when setting a wake-up alarm, a news program or news channel can be selected for playback. The user can select a specific program or channel for playback in response to the wake-up alarm. Alternatively, the user can select a general news category, and the logic associated with the playback queue manager 714 can select the news program or news channel to play based on the selected user preference. In one implementation, news program preferences can be determined based on news topic preferences selected in news programs on the user's device.

[0119] In one implementation, when an existing playback queue is active during a scheduling event that causes playback in a different playback queue, the playback queue manager 714 can manage the queue based on user-selected preferences. In one instance, the playback queue manager 714 can be configured to replace an existing playback queue with a new one, delete queue items from an existing playback queue, and replace them with items from the new one. In such cases, items selected for playback in response to an alarm or other scheduling event are replaced and reassigned to previous queue items. Alternatively, the playback queue manager 714 can be configured to perform a momentary playback queue, which is active only before the scheduling event is canceled. After the scheduling event is canceled, playback items from previously existing playback queues are restored to the active queue, and playback of previously existing queues can be resumed. The scheduling event can be canceled via voice or text commands from a virtual assistant, through a user interface on computing device 700, or through a user interface of a user device connected to computing device 700.

[0120] In one implementation, a playback routing manager 716 can be used to select playback devices within a distributed playback system for playback queues. Depending on the number of playback devices within the distributed playback system, multiple different queues may be active on multiple different playback devices, or multiple different playback devices within the distributed playback system may be grouped. Grouped playback devices may share a common playback queue and play the same media simultaneously. When smart playback devices are provided, they may be associated with one or more users and / or one or more user accounts. Smart playback devices may also be assigned a location and / or device type. In one implementation, a residential distributed playback network can be configured where multiple user devices play media via one or more smart playback devices within a residence. When a smart playback device is added to a residential network, a room or location for each playback device can be specified. Ownership may also be specified for each smart playback device, indicating whether the smart playback device is associated with a single user or whether the smart playback device is a common device associated with multiple users.

[0121] This disclosure recognizes that personal information data including biometric data in the present invention can be used to benefit users. For example, biometric authentication data can be used to conveniently access device features without using a password. In other examples, user biometric data is collected to provide users with feedback on their health or fitness levels. Furthermore, this disclosure envisions other uses for personal information data including biometric data that benefit users.

[0122] This disclosure also envisions that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of those legitimate uses. Furthermore, such collection should only be conducted with the user's informed consent. Additionally, such entities should take any necessary steps to safeguard and protect access to such personal information data and ensure that others with access to such personal information data comply with their privacy policies and procedures. Furthermore, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices.

[0123] Regardless of the foregoing, this disclosure also contemplates implementation schemes for users to selectively block the use or access to personal information data. That is, this disclosure contemplates providing hardware and / or software components to prevent or block access to such personal information data. For example, with respect to advertising delivery services, the technology of this invention can be configured to allow users to choose to "join" or "opt out" of the collection of personal information data during service registration. As another example, users can choose not to provide location information for a targeted content delivery service. Yet another example is that users can choose not to provide precise location information but allow the transmission of location area information.

[0124] The implementation described herein includes one or more application programming interfaces (APIs) in an environment, where calling program code interacts with other program code invoked through one or more programming interfaces. Various function calls, messages, or other types of calls may also include various parameters, which can be transferred via the API between the calling program and the called program code. Furthermore, the API may provide the calling program code with the ability to use data types or categories defined in the API and implemented in the called program code.

[0125] An API allows developers of API-calling components (which can be third-party developers) to utilize specified features provided by the API-implemented components. There can be one API-calling component or more such components. An API can be a source code interface provided by a computer system or library to support service requests from applications. An operating system (OS) can have multiple APIs to allow applications running on the OS to call one or more of those APIs, and a service (such as a library) can have multiple APIs to allow applications using the service to call one or more of those APIs. APIs can be specified according to the programming language that can be interpreted or compiled when the application is built.

[0126] In some implementations, the API implementation component may provide more than one API, each API providing a different view or having different aspects that access different aspects of the functionality implemented by the API implementation component. For example, one API of the API implementation component may provide a first set of functions and be exposed to third-party developers, while another API of the API implementation component may be hidden (not exposed) and provide a subset of the first set of functions, and also provide another set of functions, such as test or debug functions not in the first set. In other implementations, the API implementation component itself may call one or more other components via a lower-level API, thus acting as both an API calling component and an API implementation component.

[0127] An API defines the language and parameters used by an API-invoking component when accessing and using specified features of an API-implemented component. For example, an API-invoking component accesses specified features of an API-implemented component through one or more API calls or references exposed by the API (e.g., implemented by function or method calls), and uses parameters to pass data and control information via these API calls or references. An API-implemented component may return a value via the API in response to an API call from an API-invoking component. While an API defines the syntax and results of API calls (e.g., how an API call is initiated and what it can do), it may not reveal how the API call completes the function specified by the API call. Various API calls are transmitted via one or more application programming interfaces between the invoking component (API-invoking component) and the API-implemented component. Transmitting API calls may include issuing, initiating, referencing, calling, receiving, returning, or responding to function calls or messages; in other words, transmission can describe the actions of either the API-invoking component or the API-implemented component. API function calls or other references may send or receive one or more parameters via parameter lists or other structures. Parameters can be constants, keys, data structures, objects, object classes, variables, data types, pointers, arrays, lists, or pointers to functions or methods, or references to data or other items to be passed via the API.

[0128] Furthermore, data types or classes can be provided by the API and implemented by API implementation components. Therefore, API calling components can utilize the definitions provided in the API to declare variables, use pointers to such types or classes, and use or instantiate constant values ​​of such types or classes.

[0129] Typically, APIs can be used to access services or data provided by API implementation components, or to initiate operations or computations provided by API implementation components. By way of example, API implementation components and API calling components can each be any of an operating system, library, device driver, API, application, or other module (it should be understood that API implementation components and API calling components can be the same or different types of modules). In some cases, API implementation components may be implemented at least partially in firmware, microcode, or other hardware logic components. In some implementations, an API may allow client programs to use services provided by a Software Development Kit (SDK) library. In other implementations, applications or other client programs may use APIs provided by an application framework. In these implementations, applications or client programs may incorporate calls into functions or methods provided by both the SDK and the API, or use data types or objects defined in the SDK and provided by the API. In these implementations, the application framework may provide a main event loop for the program, which responds to various events defined by the framework. APIs allow applications to utilize the application framework to specify events and responses to events. In some implementations, API calls can report the capabilities or status of hardware devices to applications, including capabilities or status related to input capabilities and status, output capabilities and status, processing capabilities, power status, storage capacity and status, communication capabilities, etc., and the API may be implemented in part by firmware, microcode, or other low-level logic components that execute in part on the hardware components.

[0130] API invocation components can be local components (i.e., on the same data processing system as the API implementation component) or remote components (i.e., on a different data processing system than the API implementation component), communicating with the API implementation component via a network through the API. It should be understood that an API implementation component can also act as an API invocation component (i.e., it can make API calls to APIs exposed by different API implementation components), and an API invocation component can also act as an API implementation component by implementing APIs exposed to different API invocation components.

[0131] An API can allow multiple API calling components written in different programming languages ​​to communicate with an API implementation component (therefore, the API may include features for translating calls and returns between the API implementation component and the API calling component); however, the API may be implemented in a specific programming language. In one implementation, the API calling component may call APIs from different providers, such as one set of APIs from an OS provider and another set of APIs from a plugin provider, as well as yet another set of APIs from another provider (e.g., a software library provider) or the creator of another set of APIs.

[0132] Figure 8 This is a block diagram illustrating an exemplary API architecture that can be used in some embodiments of the present invention. For example... Figure 8 As shown, API architecture 800 includes an API implementation component 810 (e.g., an operating system, library, device driver, API, application, software, or other module) that implements API 820. API 820 specifies one or more functions, methods, classes, objects, protocols, data structures, formats, and / or other characteristics of the API implementation component that can be used by API calling component 830. API 820 can specify at least one calling convention that specifies how functions in the API implementation component receive parameters from the API calling component and how functions return results to the API calling component. API calling component 830 (e.g., an operating system, library, device driver, API, application, software, or other module) makes API calls through API 820 to access and use the characteristics of API implementation component 810 specified by API 820. API implementation component 810 may return values ​​to API calling component 830 through API 820 in response to API calls.

[0133] It should be understood that API implementation component 810 may include additional functions, methods, classes, data structures, and / or other features not specified through API 820 and not available to API calling component 830. It should be understood that API calling component 830 may be on the same system as API implementation component 810, or may be remotely located and accessed via a network using API 820. Although Figure 8 The example shows a single API call component 830 interacting with API 820, but it should be understood that other API call components, written in different languages ​​(or the same language) and different from API call component 830, can use API 820.

[0134] API implementation component 810, API 820, and API call component 830 may be stored in machine-readable media, including any mechanism for storing information in a machine-readable form (e.g., a computer or other data processing system). For example, machine-readable media include disks, optical disks, random access memory, read-only memory, flash memory devices, etc.

[0135] Figures 9A-9B This is a block diagram of an exemplary API software stack 900, 910 according to the implementation scheme. Figure 9A An exemplary API software stack 900 is illustrated, in which application 902 can use service APIs to invoke service A or service B and use OS APIs to invoke operating system 904. Furthermore, services A and B can use several OS APIs to invoke operating system 904.

[0136] Figure 9B An exemplary API software stack 910 is illustrated, including application 1, application 2, service 1, service 2, and operating system 904. As shown, service 2 has two APIs, one (service 2 API 1) receiving calls from application 1 and returning values, and the other (service 2 API 2) receiving calls from application 2 and returning values. Service 1 (e.g., could be a software library) calls OS API 1 and receives the returned values, and service 2 (e.g., could be a software library) calls both OS API 1 and OS API 2 and receives the returned values. Application 2 calls OS API 2 and receives the returned values.

[0137] Additional exemplary computing devices

[0138] Figure 10 This is a block diagram of a device architecture 1000 for a mobile or embedded device according to an implementation scheme. Device architecture 1000 includes a memory interface 1002, a processing system 1004 including one or more data processors, an image processor and / or graphics processing unit, and a peripheral device interface 1006. Various components can be coupled via one or more communication buses or signal lines. These components can be individual logic components or devices or can be integrated into one or more integrated circuits, such as system-on-a-chip (SoC) integrated circuits.

[0139] The memory interface 1002 may be coupled to the memory 1050, which may include high-speed random access memory such as static random access memory (SRAM) or dynamic random access memory (DRAM) and / or non-volatile memory such as, but not limited to, flash memory (e.g., NAND flash, NOR flash, etc.).

[0140] Sensors, devices, and subsystems can be coupled to peripheral interface 1006 to facilitate multiple functions. For example, motion sensor 1010, light sensor 1012, and proximity sensor 1014 can be coupled to peripheral interface 1006 to facilitate mobile device functions. One or more biometric sensors 1015 may also be present, such as fingerprint scanners for fingerprint recognition or image sensors for facial recognition. Other sensors 1016 may also be connected to peripheral interface 1006, such as positioning systems (e.g., GPS receivers), temperature sensors, or other sensing devices to facilitate related functions. Camera subsystem 1020 and optical sensor 1022 (e.g., charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) optical sensors) can be used to facilitate camera functions such as taking photos and video clips.

[0141] Communication functionality can be facilitated by one or more wireless communication subsystems 1024, which may include radio frequency receivers and transmitters and / or optical (e.g., infrared) receivers and transmitters. The specific design and implementation of the wireless communication subsystem 1024 may depend on the communication network through which the mobile device intends to operate. For example, a mobile device including the illustrated device architecture 1000 may include a wireless communication subsystem 1024 designed to operate over a GSM network, CDMA network, LTE network, Wi-Fi network, Bluetooth network, or any other wireless network. Specifically, the wireless communication subsystem 1024 may provide a communication mechanism in which a media playback application can retrieve resources from a remote media server or retrieve scheduled events from a remote calendar or event server.

[0142] The audio subsystem 1026 can be coupled to the speaker 1028 and microphone 1030 to facilitate voice-enabled functions such as speech recognition, speech copying, digital recording, and telephone functionality. In the smart media device described herein, the audio subsystem 1026 can be a high-quality audio system that includes support for virtual surround sound.

[0143] I / O subsystem 1040 may include touchscreen controller 1042 and / or other input controller 1045. For computing devices including display devices, touchscreen controller 1042 may be coupled to touch-sensitive display system 1046 (e.g., touchscreen). Touch-sensitive display system 1046 and touchscreen controller 1042 may, for example, use any of a variety of touch and pressure sensing technologies to detect contact and movement or pressure, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch-sensitive display system 1046. Display output of touch-sensitive display system 1046 may be generated by display controller 1043. In one embodiment, display controller 1043 may provide frame data to touch-sensitive display system 1046 at a variable frame rate.

[0144] In one embodiment, a sensor controller 1044 is included to monitor, control, and / or process data received from one or more motion sensors 1010, light sensors 1012, proximity sensors 1014, or other sensors 1016. The sensor controller 1044 may include logic to interpret the sensor data to determine the occurrence of one of a plurality of motion events or activities by analyzing the sensor data from the sensors.

[0145] In one implementation, the I / O subsystem 1040 includes other input controllers 1045 that can be coupled to other input / control devices 1048, such as one or more buttons, rocker switches, thumbwheels, infrared ports, USB ports, and / or pointing devices such as styluses, or up / down buttons for volume controls such as speakers 1028 and / or microphones 1030.

[0146] In one implementation, memory 1050 coupled to memory interface 1002 may store instructions for operating system 1052, including POSIX-compliant and incompatible operating systems or embedded operating systems. Operating system 1052 may include instructions for handling basic system services and for performing hardware-related tasks. In some specific implementations, operating system 1052 may be a kernel.

[0147] The memory 1050 may also store communication instructions 1054 to facilitate communication with one or more additional devices, one or more computers, and / or one or more servers, such as retrieving web resources from a remote web server. The memory 1050 may also include user interface instructions 1056, including graphical user interface instructions to facilitate graphical user interface processing.

[0148] Furthermore, memory 1050 may store sensor processing instructions 1058 to facilitate sensor-related processing and functions; telephone instructions 1060 to facilitate telephone-related processes and functions; instant messaging instructions 1062 to facilitate electronic message processing and functions; web browser instructions 1064 to facilitate web browsing-related processes and functions; media processing instructions 1066 to facilitate media processing-related processes and functions; location service instructions including GPS and / or navigation instructions 1068 and Wi-Fi-based location instructions to facilitate location-based functionality; camera instructions 1070 to facilitate camera-related processes and functions; and / or other software instructions 1072 to facilitate other processes and functions, such as security processes and functions, and system-related processes and functions. Memory 1050 may also store other software instructions, such as web video instructions to facilitate web video-related processes and functions; and / or web shopping instructions to facilitate web shopping-related processes and functions. In some specific implementations, media processing instructions 1066 are divided into audio processing instructions and video processing instructions, used to facilitate audio processing-related processes and functions and video processing-related processes and functions, respectively. Mobile device identifiers, such as International Mobile Equipment Identity (IMEI) 1074 or similar hardware identifiers, may also be stored in memory 1050.

[0149] Each of the instructions and applications identified above corresponds to an instruction set used to perform one or more of the functions described above. These instructions do not need to be implemented as a separate software program, process, or module. Memory 1050 may include additional instructions or fewer. Furthermore, various functions can be performed in hardware and / or software, including in one or more signal processing and / or application-specific integrated circuits.

[0150] Figure 11 A block diagram of one implementation of the computing system 1100. Figure 11 The computer systems illustrated are intended to represent one or more implementations of a range of computing systems (wired or wireless), including, for example, desktop computer systems, laptop computer systems, tablet computer systems, cellular phones, personal digital assistants (PDAs) including cellular-enabled PDAs, set-top boxes, entertainment systems or other consumer electronic devices, smart appliances, or smart media playback devices. Alternative computing systems may include more, fewer, and / or different components. Figure 11 The computing system can be used to provide computing devices and / or server devices that may be connected to the computing devices.

[0151] Computer system 1100 includes a bus 1135 or other communication device for transmitting information, and a processor 1110 coupled to the bus 1135 for processing information. Although computing system 1100 is illustrated as having a single processor, computing system 1100 may include multiple processors and / or coprocessors. Computing system 1100 may also include memory 1120, such as random access memory (RAM) or other dynamic storage devices coupled to the bus 1135. Memory 1120 may store information and instructions executable by processor 1110. During the execution of instructions by processor 1110, memory 1120 may also be used to store temporary variables or other intermediate information.

[0152] The computing system 1100 may also include a read-only memory (ROM) 1130 and / or other data storage device 1140 coupled to the bus 1135 for storing information and instructions for the processor 1110. The data storage device 1140 may be or include various storage devices, such as flash memory devices, disks, or optical disks, and may be coupled to the computing system 1100 via the bus 1135 or via a remote peripheral interface.

[0153] The computing system 1100 can also be coupled to a display device 1150 via a bus 1135 to display information to a user. The computing system 1100 may also include a numeric-alphanumeric input device 1160, which includes numeric keys and other keys, and can be coupled to the bus 1135 to send information and command options to the processor 1110. Another user input device includes a cursor control device 1170, such as a touchpad, mouse, trackball, or cursor arrow keys, for transmitting directional information and command selections to the processor 1110 and controlling cursor movement on the display device 1150. The computing system 1100 can also receive user input from a communicatively coupled remote device via one or more network interfaces 1180.

[0154] The computing system 1100 may also include one or more network interfaces 1180 to provide access to a network such as a local area network (LAN). The network interface 1180 may include, for example, a wireless network interface with an antenna 1185, which may represent one or more antennas. The computing system 1100 may include multiple wireless network interfaces, such as Wi-Fi and... A combination of Near Field Communication (NFC) and / or a cellular telephone interface. The network interface 1180 may also include, for example, a wired network interface for communicating with remote devices via a network cable 1187, which may be, for example, an Ethernet cable, a coaxial cable, a fiber optic cable, a serial cable, or a parallel cable.

[0155] In one implementation, network interface 1180 may provide access to a local area network (LAN) for example, by conforming to IEEE 802.11b and / or IEEE 802.11g standards, and / or wireless network interface may provide access to a personal area network (PAN) for example, by conforming to Bluetooth standards. Other wireless network interfaces and / or protocols may also be supported. In addition to or instead of communicating via wireless LAN standards, network interface 1180 may use, for example, Time Division Multiple Access (TDMA), Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), and / or any other type of wireless communication protocol to provide wireless communication.

[0156] The computing system 1100 may also include one or more energy sources 1105 and one or more energy measurement systems 1145. The energy source 1105 may include an AC / DC adapter coupled to an external power source, one or more batteries, one or more charge storage devices, a USB charger, or other energy sources. The energy measurement system includes at least one voltage or current measuring device capable of measuring the energy consumed by the computing system 1100 over a predetermined time period. Furthermore, one or more energy measurement systems may be included to measure, for example, the energy consumed by a display device, cooling subsystem, Wi-Fi subsystem, or other commonly used or high-energy-consuming subsystems.

[0157] Figure 12 A block diagram of a virtual assistant system 1200 according to an embodiment described herein is shown. The illustrated virtual assistant system 1200 is an exemplary embodiment and is not limited to all embodiments described herein. Virtual assistants utilized in various embodiments described herein may include additional, fewer, and / or different components or features than those shown. The virtual assistant system 1200 includes a virtual assistant 1202 that can accept user input 1204, such as spoken or typed language, process the input, and generate output 1208 for the user and / or perform actions 1210 on behalf of the user. The virtual assistant 1202 may use contextual information to supplement natural language or gesture input from the user. Contextual information can be used to clarify the user's intent and reduce the number of candidate interpretations of the user input. Contextual information can also reduce the need for the user to provide excessive clarifying input. Context may include any available information available to the assistant to supplement explicit user input to constrain information processing issues and / or personalize processing outcomes. Context can be used to constrain schemes at various stages of processing, including, for example, speech recognition, natural language processing, task flow processing, and dialog box generation.

[0158] Virtual assistant 1202 may utilize any of a variety of different knowledge and data context sources (e.g., dictionaries, domain models, and / or task models). From the perspective of the currently described implementation, such context sources may be internal to virtual assistant 1202 or may be collected from one or more remote databases. In addition to user input 1204 and context sources, virtual assistant 1202 may also utilize information from several context sources, including, for example, device sensor data 1256, application preferences and usage history 1272, conversation history and assistant memory 1252, personal database 1258, personal acoustic context data 1280, current application context 1260, and event context 1206.

[0159] In one implementation, the physical device running the virtual assistant 1202, such as a user device, playback device, or intelligent media playback device as described herein, has one or more sensor devices. Such sensors can provide a source of contextual information in the form of device sensor data 1256. Examples of sensor information include, but are not limited to, the user's current location; the local time of the user's current location; the location, orientation, and motion of the device the user is interacting with; current light levels, temperature, and other environmental measurements; attributes of the microphone and camera in use; and signatures of the network currently in use and connected networks, including Ethernet, Wi-Fi, and Bluetooth. The signature includes the MAC address of the network access point, the assigned IP address, the device identifier, and other attributes such as the Bluetooth name, frequency channel, and wireless network. Sensors can be of any type, including, for example, accelerometers, compasses, GPS units, altimeters, light sensors, thermometers, barometers, clocks, network interfaces, battery test circuitry, etc.

[0160] The current application context 1260 refers to the application state or similar software state related to the user's current activity. For example, a user may be using a text messaging application to chat with a specific person. The virtual assistant 1202 is not necessarily specific to the user interface or part of the user interface of the text messaging application. Instead, the virtual assistant 1202 may receive context from any number of applications, each of which facilitates its context to notify the virtual assistant 1202. If the user is currently using an application when the virtual assistant 1202 is invoked, the state of that application can provide useful contextual information. For example, if the virtual assistant 1202 is invoked from an email application, the contextual information may include sender information, recipient information, sending date and / or time, subject, data extracted from email content, mailbox or folder name, etc.

[0161] In one implementation, the information describing a user's application preferences and usage history 1272 includes preferences and settings for various applications, as well as the usage history associated with those applications. Application preferences and usage history 1272 are used to interpret the context of user intent or other functions of the virtual assistant 1202. Examples of such application preferences and usage history 1272 include, but are not limited to, shortcuts, favorites, bookmarks, friend lists, or any other collection of user data about people, companies, addresses, phone numbers, places, websites, email messages, or any other references; recent calls on the device; recent text message conversations, including the parties to the conversation; recent requests for maps or directions; recent web searches and URLs; stocks listed in stock applications; recently played songs or videos or other media; alarm names set on alarm clock applications; names of applications or other digital objects on the device; and the user's preferred language or currently used language at their location.

[0162] Another source of contextual data is the user's personal database 1258 on a device such as a telephone, including, for example, an address book containing names and phone numbers. In one implementation, the user's personal information obtained from the personal database 1258 is used as context for interpreting and / or operating the virtual assistant 1202's intent or other functions. For example, data from the user's contact database can be used to reduce ambiguity when interpreting user commands when the user refers to someone only by their name. Examples of contextual information can be obtained from the personal database 1258, including, but not limited to, the user's contact database (address book)—including name information, phone number, physical address, network address, account identifier, important dates—about people, companies, organizations, places, websites, and other entities the user may refer to; the user's own name, preferred pronunciation, address, phone number, etc.; the user's naming relationships, such as mother, father, sister, boss, etc.; the user's calendar data, including calendar events, names of specific days, or any other named entries the user may refer to; the user's reminders or task lists, including lists of things to complete, remember, or retrieve that the user may refer to; song titles, genres, playlists, and other data associated with the user's music library that the user may refer to; people, places, categories, tags, labels, or other symbolic names of photos or videos or other media in the user's media library; titles, authors, genres, or other symbolic names of books or other documents in the user's personal library.

[0163] Another source of contextual data is the user's conversation history with the virtual assistant 1202. This history can include, for example, references to domains, people, places, and so on. For example, a user might ask, "What time is it in New York?" The virtual assistant 1202 could respond by providing the current time in New York City. The user could then ask, "How's the weather?" The virtual assistant 1202 could use the previous conversation history to infer that the location used for the weather query was the last location mentioned in the conversation history.

[0164] Examples of contextual information from the conversation history and virtual assistant memory include, but are not limited to, people mentioned in the conversation; places and locations mentioned in the conversation; the current time frame of focus; the current application domain of focus, such as email or calendar; the current task of focus, such as reading an email or creating a calendar entry; the current domain object of focus, such as an email message just read or a calendar entry just created; the current state of the conversation or transaction flow, such as whether a question was asked and the expected possible answer; the history of user requests; the history of results of user requests, such as the returned collection of restaurants; the history of phrases used by the assistant in the conversation; and facts that the user tells the assistant.

[0165] In one implementation, personal acoustic context data 1280 can be used to select from possible statistical language models that can be used to understand the user's speech, or otherwise tune speech recognition to optimize the acoustic context for recognition. When interpreting voice input, the virtual assistant 1202 can adjust the voice-to-text service to take into account the acoustic environment of the input speech. For example, the noise profile of a quiet office differs from that of a car or public place. If the speech recognition system is able to recognize and store acoustic configuration data, this data can also be provided as contextual information. When combined with other contextual information such as the properties of the microphone in use, current location, and current conversation state, acoustic context can help in recognizing and interpreting input.

[0166] In the foregoing specification, the invention has been described with respect to specific embodiments thereof. However, it will be apparent that various modifications and alterations may be made thereto without departing from the broader spirit and scope of the invention. Accordingly, the specification and drawings should be considered illustrative rather than restrictive. Specific details provided in the description and examples can be used anywhere in one or more embodiments. Various features of different embodiments or examples may be combined differently with some of the included features and others excluded features to suit a variety of different applications. Examples may include subjects such as methods, means for performing the actions of the method, at least one machine-readable medium including instructions that, when executed by a machine, cause the machine to perform the actions of the method, or actions of a means or system according to the embodiments and examples described herein. Furthermore, the various components described herein may be means for performing the operations or functions described according to the embodiments.

[0167] The embodiments described herein provide a communication mechanism that enables public electronic devices, such as smart speaker devices or other smart home devices, to relay or redirect virtual assistant requests involving personal user data to an accompanying device, which is a personal user device associated with the user requesting access to private data. The communication mechanism can also be used as a general communication mechanism that enables smart home devices to exchange data, including configuration data.

[0168] Communication between public devices and accompanying devices can be performed on a secure data channel known as an accompanying link. The accompanying link provides a persistent, low-latency, instant messaging system for connected devices in a home network environment. In some implementations, the accompanying link supports links between fixed public devices within the home, such as speaker equipment, and personal mobile devices within the home or accessible via the internet. The accompanying link enables public devices to redirect personal requests that they cannot handle due to privacy concerns from one or more personal devices. Redirecting personal requests to personal devices associated with a user allows virtual assistants on public devices to receive and request requests confidentially.

[0169] In addition to redirecting individual queries to accompanying devices, in one implementation, the accompanying link also provides a universal instant messaging system for devices in a home network environment. The universal instant messaging system enables multiple home devices to work collaboratively by exchanging messages on the accompanying link. For example, audio playback between multiple smart speaker devices can be coordinated, enabling the devices to perform actions such as, but not limited to, coordinating the playback of media items, selectively providing audio notifications to the user via the speaker closest to the user, configuring multiple speakers into a multi-channel audio system, or coordinating audio avoidance at the speaker during the duration of verbal requests and responses.

[0170] Public devices can announce support for companion link services through discovery protocols. Personal user devices (PMIs) on the same network as the public device can discover the companion link services announced by the public device and connect to the public device using the announced information. The PMI can perform a pairing process with the public device to become the user's companion device. In one embodiment, the pairing process includes a proximity element, where the user device exchanges identifiers, keys, or secrets with the companion device via a short-range wireless communication mechanism. The pairing process may also include the exchange of information facilitating the subsequent connection and accessibility of the companion link between the public and companion devices.

[0171] The embodiments described herein provide a communication mechanism that enables public electronic devices, such as smart speaker devices or other smart home devices, to relay or redirect virtual assistant requests involving personal user data to the personal user device for processing. The communication mechanism can also be used as a general communication mechanism that enables smart home devices to exchange data, including configuration data. However, it will be apparent that various modifications and variations can be made thereto without departing from the broader spirit and scope of the invention. Accordingly, the specification and drawings should be considered illustrative rather than restrictive. Specific details in the provided descriptions and examples can be used anywhere in one or more embodiments. Various features of different embodiments or examples can be combined differently with some of the included features and others excluded features to suit a variety of different applications. Examples may include subjects such as methods, means for performing the actions of the method, at least one machine-readable medium including instructions that, when executed by a machine, cause the machine to perform the actions of the method, or actions of means or systems according to the embodiments and examples described herein. Furthermore, the various components described herein may be means for performing the operations or functions described according to the embodiments.

[0172] One embodiment provides a data processing system on a public electronic device, the data processing system including a memory device storing instructions and one or more processors executing the instructions stored on the memory device. The instructions cause the one or more processors to provide a virtual assistant to receive commands from the public electronic device, wherein the virtual assistant, via the one or more processors, is configured to receive commands from the public electronic device and determine whether the command is to access personal data of a user associated with the public electronic device. In response to determining that the command is to access the user's personal data, the virtual assistant may send a request to the user's personal electronic device to process at least a portion of the command.

[0173] In the embodiments described herein, the user's personal data includes a contact list, text messages, emails, call history, alarms, reminders, communication history, settings, preferences, or location history, and the public electronic device includes a smart speaker device. Personal data may be stored on the user's personal electronic device, and the virtual assistant may request the personal electronic device to access the personal data on behalf of the public electronic device. The virtual assistant may then receive the output of the request sent to the user's personal electronic device and process the command based on that output. The command may be a voice command spoken by the user or a text command transmitted to the public device. In one embodiment, sending a request to the user's personal electronic device includes redirecting the command to the user's personal electronic device, whereby the personal electronic device processes the command on behalf of the public electronic device. The virtual assistant may receive an audio response generated by the personal electronic device and play the audio response as a response to the command. If the command is to send a message to the user's contacts, the audio response may be a notification of a received reply to the message.

[0174] In one implementation, the public electronic device (PED) may establish a pairing and / or trusted relationship with the PED before enabling the PED to send a request to the PED. The virtual assistant can send the request to the user's PED via a verified data connection with the PED. The verified data connection can be established based on the trusted relationship with the PED, wherein the verified data is verified through data exchanged during the establishment of the trusted relationship. In one implementation, the verified data connection is established via a remote data connection established via a wide area network such as the Internet. The user's personal data is user-specific data and may include data that is inherently personal or private, or data designated as personal or private.

[0175] One embodiment provides a non-transitory machine-readable medium storing instructions to cause one or more processors to perform operations, including determining that a communication session should be established between a public electronic device and an accompanying device, wherein the accompanying device is an electronic device capable of accessing personal data associated with a user. The operations further include establishing a data connection with the accompanying device, verifying a trusted relationship between the public electronic device and the accompanying device, establishing an encrypted communication session between the public electronic device and the accompanying device after verifying the trusted relationship, and relaying commands received at the public electronic device to the accompanying device via the encrypted communication session.

[0176] One embodiment provides an electronic device including a network interface connected to one or more networks and a memory device storing instructions and personal data associated with a user of the electronic device. The electronic device includes one or more processors to execute instructions stored on the memory device, wherein the instructions cause the one or more processors to provide a virtual assistant that receives commands. The virtual assistant can receive commands redirected from a public electronic device via the network interface through the one or more processors. The commands may include or specify a request for accessing personal data associated with a user of the electronic device. The virtual assistant can then process at least a portion of the commands on behalf of the public electronic device and transmit the output of the processing performed on behalf of the public electronic device back to the public electronic device via the network interface.

[0177] While the accompanying link connection between public electronic devices such as smart speaker devices and accompanying devices such as smartphones is described, the accompanying link connection can also be established between multiple accompanying devices, such as multiple smartphones, tablet computing devices, or between a smartphone and a tablet computing device.

[0178] One embodiment provides a data processing system on an electronic device, the data processing system including a memory device storing instructions and one or more processors executing the instructions stored on the memory device. When executed, the instructions cause the one or more processors to enable an encrypted data channel between the electronic devices. To enable the encrypted data channel, the one or more processors are configured to determine to establish a communication session between a first electronic device and a second electronic device, wherein the first electronic device and the second electronic device are each associated with a cloud service account. The one or more processors may be further configured to establish a peer-to-peer data connection between the first electronic device and the second electronic device, verify a trust relationship between the first electronic device and the second electronic device, and establish an encrypted communication session between the first electronic device and the second electronic device after verifying the trust relationship. The encrypted communication session can then be used to exchange data between the first electronic device and the second electronic device through the encrypted communication session.

[0179] In another embodiment, one or more processors of the data processing system may establish a peer-to-peer data connection between the first electronic device and the second electronic device via a short-range wireless connection, and / or establish an encrypted communication session via a network layer protocol through a wireless network connection. Verifying the trusted relationship between the first electronic device and the second electronic device may include verifying a previously established trusted relationship, which may be established via one or more of the exchange of credentials between the first and second electronic devices via a short-range wireless connection and / or via the exchange of credentials of cloud service accounts associated with the first and second electronic devices, the credentials enabling mutual authentication between the first and second electronic devices.

[0180] In a further embodiment, one or more processors may be configured to determine that a communication session will be established between the first and second electronic devices in response to the discovery of a second electronic device at the first electronic device via a device discovery protocol, and that a peer-to-peer data connection will be established between the first and second electronic devices after the discovery of the second electronic device. The communication session may also be established based on a list of devices associated with a cloud service account, with the first and second devices associated with that cloud service account. In one embodiment, the communication session may be established based on a list of devices associated with a cloud service account family, wherein the first electronic device is associated with a first account, the second electronic device is associated with a second account, and the cloud service account family includes both the first and second accounts. In one embodiment, a trusted relationship is established between the first and second electronic devices before the first electronic device is enabled to send a request for data exchange to the second electronic device.

[0181] One implementation provides a non-transitory machine-readable medium storing instructions to cause one or more processors to perform operations including determining that a communication session should be established between a first electronic device and a second electronic device, wherein each of the first and second electronic devices is associated with a cloud service account. The instructions may further cause the one or more processors to perform additional operations including establishing a peer-to-peer data connection between the first and second electronic devices, verifying a trusted relationship between the first and second electronic devices, establishing an encrypted communication session between the first and second electronic devices after verifying the trusted relationship, and exchanging data between the first and second electronic devices through the encrypted communication session.

[0182] One embodiment provides a method performed on a computing device or data processing system as described herein. The method includes determining to establish a communication session between a first electronic device and a second electronic device, wherein the first electronic device and the second electronic device are each associated with a cloud service account, and establishing a peer-to-peer data connection between the first electronic device and the second electronic device via a wireless device; and verifying a trust relationship between the first electronic device and the second electronic device, wherein verifying the trust relationship includes verifying a previously established trust relationship. The method further includes establishing an encrypted communication session between the first electronic device and the second electronic device via a wireless network connection through a network layer protocol, the encrypted communication session being established after verifying the trust relationship, and exchanging data between the first electronic device and the second electronic device through the encrypted communication session. For example, data may be exchanged to synchronize device data between the first electronic device and the second electronic device, wherein the device data is associated with a cloud service account.

[0183] In another implementation, the method further includes determining whether a communication session is established between a first electronic device and a second electronic device based on a list of devices associated with a cloud service account family, wherein the first electronic device is associated with a first account, the second electronic device is associated with a second account, and the cloud service account family includes both the first and second accounts. The first electronic device can discover the second electronic device via a device discovery protocol and establish a peer-to-peer data connection with the second electronic device after discovery.

[0184] Other features of this embodiment will become apparent from the accompanying drawings and the detailed description described above. Therefore, the true scope of the invention will be apparent to those skilled in the art upon studying the drawings, specification, and appended claims.

Claims

1. A data processing system for a public electronic device, the data processing system comprising: A memory device that stores instructions; One or more processors execute the instructions stored on the memory device, the instructions causing the one or more processors to provide a virtual assistant associated with the public electronic device: Receive commands at the public electronic device; The command is processed at least in part by the one or more processors to determine whether the command accesses personal data of a user associated with the public electronic device, wherein the personal data includes data specific to the user and stored on the user's personal electronic device; In response to determining that the command requires access to the user's personal data, a request is sent to the user's personal electronic device to process at least a portion of the command on behalf of the public electronic device; In response to the request, receive from the personal electronic device the result of processing at least a portion of the command at the personal electronic device; as well as In response to determining that the command does not require the user's personal data, the command is processed by the one or more processors without being sent to the personal electronic device.

2. The data processing system according to claim 1, wherein the user's personal data includes a contact list, text messages, emails, call history, alarms, reminders, communication history, settings, preferences, or location history, and the public electronic device includes a smart speaker device.

3. The data processing system according to claim 1 or 2, wherein the virtual assistant requests the personal electronic device to access personal data on behalf of the public electronic device.

4. The data processing system according to claim 1 or 2, wherein the virtual assistant receives the output of a request sent to the user's personal electronic device and performs command processing based on the output, wherein the command is a voice command or a text command.

5. The data processing system of claim 1 or 2, wherein sending a request to the user's personal electronic device includes redirecting a command to the user's personal electronic device to process on behalf of the public electronic device.

6. The data processing system according to claim 5, wherein the virtual assistant receives an audio response generated by the personal electronic device and plays the audio response as a response to the command.

7. The data processing system of claim 6, wherein the command is a command to send a message to the user's contact and the audio response is a notification of a reply to the message.

8. The data processing system according to claim 7, wherein the public electronic device establishes a trust relationship with the personal electronic device before the public electronic device is able to send the request to the personal electronic device, and the virtual assistant sends the request to the user's personal electronic device through a verified data connection with the personal electronic device, the verified data connection being established based on the trust relationship with the personal electronic device.

9. The data processing system of claim 8, wherein the verified data connection is verified via data exchanged during the establishment of the trust relationship.

10. The data processing system according to claim 8, wherein the verified data connection is established on a remote data connection established via a wide area network.

11. The data processing system according to claim 1 or 2, wherein the personal data includes inherent personal or private data.

12. The data processing system of claim 1, wherein the personal data includes data designated as personal or private.

13. A non-transitory machine-readable medium storing instructions that cause one or more processors to perform operations including: Determine whether to establish a communication session between a public electronic device and an accompanying device, wherein the accompanying device is an electronic device capable of accessing personal data associated with a user among a plurality of users of the public electronic device, wherein the personal data includes data specific to the user and stored on the user's personal electronic device, wherein the determination is made as to whether the user's personal data is required for processing of a command based on a virtual assistant implemented on the public electronic device, the command being received at the public electronic device; Establish a data connection between the public electronic device and the accompanying device; Verify the trust relationship between the public electronic device and the accompanying device; After verifying the trust relationship, an encrypted communication session is established between the public electronic device and the accompanying device; When the command requires the user's personal data, at least a portion of the command is relayed to the accompanying device via the encrypted communication session; Receive, from the accompanying device, the result of processing at least a portion of the command at the accompanying device; as well as When the command does not require the user's personal data, the command is processed by accessing an information service via a wide area network.

14. The non-transitory machine-readable medium of claim 13, further comprising: Attempt to discover the accompanying device on the local network; as well as Once the accompanying device is discovered on the local network, a data connection is established between the public electronic device and the accompanying device via the local network.

15. The non-transitory machine-readable medium of claim 13, further comprising: Attempt to discover the accompanying device on the local network; When the accompanying device is found to have failed on the local network, it is determined whether the remote identifier of the accompanying device is available at the public electronic device. as well as In response to determining that a remote identifier for the accompanying device is available at the public electronic device, a data connection with the accompanying device is established via a wide area network using the remote identifier.

16. The non-transitory machine-readable medium of claim 13, the operation further comprising receiving a response to the command at the common electronic device, the response to the command being generated by the accompanying device, wherein the command is a voice command or a text command.

17. An electronic device comprising: A network interface that connects to one or more networks; A memory device that stores instructions and personal data associated with a user of the electronic device, wherein the personal data includes user-specific data; and One or more processors execute instructions stored on the memory device, the instructions causing the one or more processors to provide a virtual assistant for receiving commands, the virtual assistant being accessed via the one or more processors to: The network interface receives commands that are at least partially redirected from a virtual assistant implemented on a public electronic device, the at least partially redirected commands accessing the personal data of the user, who is one of a plurality of users associated with the public electronic device, wherein the personal data includes data specific to the user and stored on the user's electronic device, and wherein commands that do not require the user's personal data are not redirected by the public electronic device. The public electronic device processes at least a portion of the command, which requires access to the user's personal data; as well as The output of the processing performed on behalf of the public electronic device is sent to the public electronic device via the network interface.

18. The electronic device of claim 17, wherein accessing personal data on behalf of the public electronic device includes accessing personal data after successful unlocking authorization on the public electronic device.

19. The electronic device of claim 17 or 18, wherein at least a portion of processing the command on behalf of the public electronic device includes accessing personal data on behalf of the public electronic device.

20. The electronic device of claim 17 or 18, wherein at least a portion of processing the command on behalf of the public electronic device includes accessing remotely stored personal data on behalf of the public electronic device via the network interface.

21. The electronic device of claim 17 or 18, further comprising a unique identifier that enables the receiving of the command redirected from the public electronic device via a wide area network.

22. The electronic device of claim 17 or 18, further comprising a unique identifier for a virtual assistant server to identify the virtual assistant, the virtual assistant being connected to the virtual assistant server via the network interface to process at least a portion of the command redirected from the public electronic device, wherein the command is a voice command or a text command.

23. The electronic device of claim 17 or 18, wherein accessing personal data on behalf of the public electronic device includes requesting authorization to access the personal data on behalf of the public electronic device, wherein requesting authorization to access the personal data on behalf of the public electronic device includes displaying an authorization request via a display or playing an audio request for authorization via a speaker device.

24. The electronic device of claim 17 or 18, wherein the personal data includes data designated as personal or private, or inherently personal or private data.

Images