Connection sequencing including automated assistants and peripheral devices

Abstract

Implementations described herein relate to ordering connections between a peripheral device and two or more applications, which may include an automated assistant application. Connections can be ordered such that an application designated for a peripheral device takes priority over an automated assistant application when the peripheral device becomes available for connection with the application. The designated application can be, for example, a streaming application designated for a media streaming peripheral device or a health application for a health management peripheral device. The designated application can connect with the peripheral device via the Bluetooth Classic protocol and then provide an indication to other applications confirming completion of the Bluetooth Classic connection. In response, the other applications can initiate connections with the peripheral device via the Bluetooth Classic protocol or the Bluetooth Low Energy protocol.

Description

[Background technology]

[0001] Humans may conduct human-computer interactions using interactive software applications referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, humans (who may be referred to as "users" when interacting with an automated assistant) may provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances) and / or by providing textual (e.g., typed) natural language input, which in some cases may be converted to text and then processed.

[0002] In some examples, a user may interact with an automated assistant via a peripheral device, such as a smartwatch, a vehicle interface, an exercise device (e.g., an exercise bicycle), and / or any other suitable device. However, when other applications are available to connect to the peripheral device, establishing a connection between the peripheral device and another computing device can be problematic for the automated assistant. For example, an exercise application provided by the exercise bicycle manufacturer may attempt to connect to the exercise bicycle via Bluetooth® when the computing device (e.g., a mobile phone) is sufficiently close to the exercise bicycle. However, the automated assistant on the computing device may simultaneously attempt to connect to the exercise bicycle, resulting in both connection attempts being delayed or otherwise failing. This can be particularly problematic when some applications connect via different protocols, such as Bluetooth Classic and Bluetooth Low Energy (BLE). For example, when an interruption in a Bluetooth Classic connection occurs, application processes that rely on the BLE connection may also be interrupted because the application is generally notified about the connection interruption. As a result, certain application processes may be repeated and / or certain data may not be realized, thereby inefficiently utilizing computational resources, such as processing bandwidth and memory. Summary of the Invention

[0003] Embodiments described herein relate to an automated assistant that operates to connect to peripheral devices according to an ordering process. When the ordering process involves connecting to other computing devices, the ordering process can prioritize connecting to native applications on the other computing devices over the automated assistant. In this manner, when a peripheral device provides communication to the computing device to initialize pairing, an application on the computing device can connect to the peripheral device according to a particular priority. In some embodiments, when multiple devices are available to connect to the computing device, a particular device can also be prioritized according to a particular ordering process. Subsequently, when a particular device disconnects from the computing device, a disconnection sequence can also be executed to ensure that a particular application service does not reconnect until a particular connection sequence is initiated.

[0004] For example, a peripheral device may include a heart rate monitor attached to a user's wrist and connected to the user's computing device via Bluetooth Low Energy (BLE). The computing device may be a smartphone that includes multiple different applications, such as a health application, which may be a native or designated application of the heart rate monitor, and an automated assistant application. In some implementations, the heart rate monitor can accept input from a user (e.g., tap input from the user) on an interface of the heart rate monitor to invoke the automated assistant and can render output from the automated assistant on another interface of the heart rate monitor (e.g., a display interface). Such operation of the automated assistant can be achieved by allowing an instance of the automated assistant application to communicate with another instance of the automated assistant application on the smartphone via BLE. Similarly, an instance of a health application can run on the heart rate monitor, and another instance of the health application can run on the smartphone. The smartphone can communicate with the heart rate monitor via BLE to facilitate certain features of the heart rate monitor.

[0005] When the heart rate monitor is powered on or otherwise becomes available to connect with a smartphone, the heart rate monitor can transmit a beacon or other data indicating that the heart rate monitor is available for connection with another device (e.g., a smartphone). When the smartphone detects the beacon from the heart rate monitor, the smartphone can provide Bluetooth availability data to an application on the smartphone. In some implementations, the automated assistant application on the smartphone can be reconfigured to limit attempts to connect to an available Bluetooth device (e.g., a heart rate monitor) until the Bluetooth device's native application (e.g., a health application) indicates that the native application is connected to the Bluetooth device. In other words, the automated assistant avoids or delays connecting to the heart rate monitor until the health application or the heart rate monitor indicates to the automated assistant that the health application has successfully connected to the heart rate monitor. In this way, by requiring one or more applications to connect according to a specific sequence, errors that occur when establishing connections for multiple applications can be mitigated.

[0006] In some implementations, connection and / or disconnection ordering may occur for peripheral devices operating multiple Bluetooth protocols, such as Bluetooth Classic and BLE. A peripheral device, such as, without limitation, a smartwatch or athletic device, may connect to a particular application and / or device via Bluetooth Classic and / or BLE, depending on the services that may be implemented by the peripheral device. For example, a smartwatch may connect to a computing device's native smartwatch application via the Bluetooth Classic protocol and to a health application and automated assistant via the BLE protocol. In some implementations, when the smartwatch provides a beacon for connecting to a computing device (e.g., a smartphone), the health application and automated assistant may receive an indication of the smartwatch's availability from the computing device's operating system. In response, and based on the indication and / or one or more characteristics of the smartwatch, the health application may avoid attempting to connect to the smartwatch until the native smartwatch application connects to the computing device via Bluetooth Classic. In this manner, when the native smartwatch application has a specific encrypted connection to establish between the smartwatch and the computing device, the encrypted connection may be established before other connection requests are processed. Thereafter, when the native smartwatch application indicates that a Bluetooth Classic connection has been established, the native smartwatch application can communicate with the computing device's health application and / or automated assistant, in response to which the health application can provide a connection request to the smartwatch, and the automated assistant can subsequently provide a separate connection request to the smartwatch.

[0007] In some implementations, each device can contribute state machine data managed by a state machine available to interconnected devices. The state machine can be a module and / or other executable portion of the state machine data that characterizes the connection status and / or connection priority of each device and / or application available to connect to other devices. For example, a computing device such as a smartphone can include a state machine with default settings for ordering connections from particular applications to particular peripheral devices. For example, the state machine can prioritize any native applications of the peripheral device over automated assistant applications by default. The information managed by the state machine can then be shared with any peripheral devices attempting to securely connect with the computing device, thereby ensuring that any peripheral devices follow the same priority and sequence.

[0008] In some implementations, the connection sequence established by a particular state machine can be determined by the operating system and / or other applications available on the computing device, depending on the context of the computing device. For example, the sequence for connecting an application to a particular peripheral device may differ depending on the other peripheral devices to which the application can connect. Alternatively or additionally, a sequence for connecting and / or disconnecting an application to a particular peripheral device with a particular interface profile (e.g., a hands-free profile) can be implemented. As an example, an automated assistant can connect to earbuds according to a first profile (e.g., a hands-free profile or a media streaming profile) but connect to a smartwatch according to a second profile (e.g., a profile that allows touch input and hands-free input). When the earbuds are no longer being used by a user (e.g., when the user returns the earbuds to their case), the earbuds and / or earbud case can provide a disconnection beacon for the hands-free profile to any applications and / or devices that were using the hands-free profile with the earbuds. In response, each application and / or device can follow a protocol to disconnect any application and / or device from the earbuds without affecting any other profiles being utilized. For example, an earbud hands-free profile may utilize the Bluetooth Classic protocol, while other profiles, such as a diagnostic profile, may utilize BLE to communicate battery status updates from the earbuds and / or their cases to any associated devices and / or applications. In this manner, an application utilizing the earbud hands-free profile may follow a specific sequence when disconnecting from the earbud hands-free profile while maintaining a connection with the diagnostic profile.For example, when a profile utilizing Bluetooth Classic is no longer available to stream music from a music application and communicate with an automated assistant, the music application and automated assistant can use a profile utilizing BLE to maintain a relatively uninterrupted connection to provide earbud charging information via the application interface.

[0009] The above description is provided as a summary of some embodiments of the present disclosure. Details of these and other embodiments are described in more detail below.

[0010] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., central processing unit (CPU)(s), graphics processing unit (GPU)(s), and / or tensor processing unit (TPU)(s)) to perform one or more methods, such as those described above and / or elsewhere herein. Still other embodiments may include one or more computer systems including one or more processors operable to execute stored instructions, for example, to perform one or more of the methods described above and / or elsewhere herein.

[0011] It should be understood that all combinations of the foregoing concepts, and additional concepts described in more detail herein, are contemplated as part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. [Brief explanation of the drawings]

[0012] [Figure 1A] 1 illustrates a view of a connection sequence utilized by an automated assistant connecting to a peripheral device and one or more other applications. [Figure 1B]1 illustrates a view of a connection sequence utilized by an automated assistant connecting to a peripheral device and one or more other applications. [Figure 1C] 1 illustrates a view of a connection sequence utilized by an automated assistant connecting to a peripheral device and one or more other applications. [Figure 2] SUMMARY OF THE INVENTION A system for operating an automated assistant that connects with peripheral devices according to a sequence that can prioritize applications assigned to the peripheral devices is presented. [Figure 3] A method is shown for ordering connections of applications, including automated assistants, to one or more peripheral devices to prioritize designated applications to connect first with one or more peripheral devices. [Figure 4] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE INVENTION

[0013] 1A, 1B, and 1C illustrate views 100, 120, and 140 of a connection sequence utilized by an automated assistant 108 connecting to a peripheral device 112 and one or more other applications. The automated assistant 108 may be an application accessible via the computing device 104, and the automated assistant 108 may include one or more other applications loaded thereon. The one or more applications may include, for example, a designated application 106 that may be provided by an entity that also provides the peripheral device 112. For example, the peripheral device 112 may be an exercise bicycle with an integrated computing device, and the designated application 106 may be a health management application developed by the same entity that developed the exercise bicycle. In such a case, the designated application 106 may process peripheral device data generated by the peripheral device 112 and may allow other applications on the computing device 104 to access the peripheral device data and / or any other data resulting from the processing of the peripheral device data.

[0014] In some implementations, when the user 102 repositions the computing device 104 so that it is within range of the peripheral device 112 in order to connect with the peripheral device 112, the peripheral device 112 can communicate connection data 114 to the peripheral device 112. The connection data 114 can be a beacon communicated by the peripheral device 112, indicating to nearby devices that the peripheral device 112 is available to connect with the particular device via a particular communication protocol (e.g., a first communication protocol, a second communication protocol, etc.). When the computing device 104 receives the beacon, an operating system or other application of the computing device 104 can determine whether the designated application 106 is present on the peripheral device 112. If the designated application 106 is available on the computing device 104, the designated application 106 can be prioritized over other applications for connecting to the peripheral device 112.

[0015] 1B , connection data 124 may be generated to instruct designated application 106 to connect to peripheral device 112 before any other applications connect to peripheral device 112. In some implementations, connection data 124 executes on peripheral device 112 and / or on computing device 104 to enable designated application 106 to connect to peripheral device 112. In some implementations, connection data 122 may also be communicated to one or more other applications on computing device 104, such as automated assistant 108. This connection data 122 may include connection sequence data that instructs the automated assistant to request a connection with peripheral device 112 only after designated application 106 has completed connecting with peripheral device 112. In some implementations, connection data 122 and connection data 124 may be provided by peripheral device 112 and / or designated application 106. Alternatively or additionally, instances of connection data may be generated on computing device 104 and may adapt over time according to applications available on computing device 104.

[0016] As each application receives the connection data, the prioritized application can initiate a connection with the peripheral device 112. For example, as shown in FIG. 1C , the designated application 106 can be the prioritized application that first provides a connection request 144 to the peripheral device 112. In some implementations, the connection request 144 can be provided via the Bluetooth Classic protocol to establish a communication channel (e.g., a first communication channel) between the designated application 106 and the peripheral device 112. Once the peripheral device 112 completes the connection with the designated application 106, the peripheral device 112, the designated application 106, and / or the computing device 104 can communicate a connection confirmation 146 to one or more other applications that may connect with the peripheral device 112. In some implementations, the connection confirmation 146 can include connection sequence data that indicates the sequence in which each application should connect with the peripheral device 112.

[0017] In some embodiments, the designated application 106 and / or the computing device 104 can communicate a connection confirmation 146 to the automated assistant 108, causing the automated assistant 108 to initiate a connection with the peripheral device 112. In some embodiments, the connection confirmation 146 can be communicated over a channel separate from the channel over which the designated application 106 connected with the peripheral device 112 (e.g., a second communication channel, such as a separate Bluetooth channel). In some embodiments, the connection confirmation 146 can be communicated over a communication modality separate from the communication modality over which the designated application 106 connected with the peripheral device 112 (e.g., Bluetooth). In response to receiving the connection request 144 or otherwise determining that the preferred application has connected with the peripheral device 112, the automated assistant 108 can initiate a connection with the peripheral device 112. For example, the automated assistant 108 can provide the connection request 142 to the peripheral device 112 over a communication channel separate from the communication channel over which the connection request 144 was provided to the peripheral device 112. By sequencing connections in this manner, applications can connect with peripheral devices 112 in a manner that reduces connection drops and shortens connection timeouts.

[0018] FIG. 2 illustrates a system 200 for operating an automated assistant 204 that connects with peripheral devices according to a sequence that can prioritize applications assigned to the peripheral devices. The automated assistant 204 can operate as part of an assistant application provided on one or more computing devices, such as the computing device 202 and / or a server device. A user can interact with the automated assistant 204 through assistant interface(s) 220. The assistant interface(s) 220 can be a microphone, a camera, a touchscreen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user can initialize the automated assistant 204 by providing verbal, textual, and / or graphical input to the assistant interface 220, causing the automated assistant 204 to initiate one or more actions (e.g., provide data, control a peripheral device, access an agent, generate input and / or output, etc.). Alternatively, the automated assistant 204 can be initialized based on processing of context data 236 using one or more trained machine learning models. The context data 236 can characterize one or more features of an environment accessible to the automated assistant 204 and / or one or more features of a user who is predicted to intend to interact with the automated assistant 204. The computing device 202 can include a display device, which can be a display panel including a touch interface for receiving touch input and / or gestures to allow a user to control the application 234 of the computing device 202 via the touch interface. In some implementations, the computing device 202 may lack a display device, thereby providing an audible user interface output rather than a graphical user interface output.Additionally, computing device 202 may provide a user interface, such as a microphone, to receive spoken natural language input from a user. In some implementations, computing device 202 may include a touch interface and may not have a camera, but may optionally include one or more other sensors.

[0019] Computing device 202 and / or other third-party client devices can communicate with the server device over a network, such as the Internet. Additionally, computing device 202 and any other computing devices can communicate with each other over a local area network (LAN), such as a Wi-Fi network. Computing device 202 can offload computing tasks to a server device to conserve computing resources at computing device 202. For example, the server device can host automated assistant 204 and / or computing device 202 can send input received at one or more assistant interfaces 220 to the server device. However, in some implementations, automated assistant 204 can be hosted at computing device 202, and various processes that can be associated with automated assistant operation can occur at computing device 202.

[0020] In various embodiments, all or fewer of the aspects of automated assistant 204 may be implemented on computing device 202. In some of these embodiments, aspects of automated assistant 204 are implemented via computing device 202 and may interface with a server device that may implement other aspects of automated assistant 204. The server device may optionally provide services to multiple users and their associated assistant applications via multiple threads. In embodiments in which all or fewer of the aspects of automated assistant 204 are implemented via computing device 202, automated assistant 204 may be an application separate from the operating system of computing device 202 (e.g., installed "on top" of the operating system), or alternatively, may be implemented directly by the operating system of computing device 202 (e.g., considered an operating system application but integral to the operating system).

[0021] In some implementations, automated assistant 204 may include input processing engine 206, which may utilize multiple different modules to process input and / or output of computing device 202 and / or a server device. For example, input processing engine 206 may include speech processing engine 208, which may process audio data received at assistant interface 220 to identify text embodied in the audio data. The audio data may be transmitted from computing device 202 to a server device, for example, to conserve computational resources at computing device 202. Additionally or alternatively, the audio data may be processed exclusively at computing device 202.

[0022] The process for converting audio data to text may include speech recognition algorithms that can utilize neural networks and / or statistical models to identify groups of audio data that correspond to words or phrases. The text converted from the audio data may be analyzed by data analysis engine 210 and made available to automated assistant 204 as text data that can be used to generate and / or identify command phrase(s), intent(s), action(s), slot value(s), and / or any other content specified by the user. In some implementations, output data provided by data analysis engine 210 may be provided to parameter engine 212 to determine whether the user has provided input that corresponds to a particular intent, action, and / or routine that can be performed by automated assistant 204 and / or an application or agent accessible via automated assistant 204. For example, assistant data 238 may be stored on server device and / or computing device 202 and may include data defining one or more actions that automated assistant 204 can perform and the parameters required to perform the action. The parameter engine 212 can generate one or more parameters for the intent, action, and / or slot value and provide the one or more parameters to the output generation engine 214. The output generation engine 214 can use the one or more parameters to communicate with the assistant interface 220 to provide output to the user and / or to communicate with one or more applications 234 to provide output to the one or more applications 234.

[0023] In some implementations, the automated assistant 204 may be an application that can be installed “on top of” the operating system of the computing device 202 and / or that itself may form part of (or the entire) the operating system of the computing device 202. The automated assistant application may include and / or have access to on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, on-device speech recognition may be performed using an on-device speech recognition module that processes audio data (detected by the microphone(s)) using an end-to-end voice recognition machine learning model stored locally on the computing device 202. The on-device speech recognition generates recognized text for speech utterances (if any) present in the audio data. Also, for example, on-device natural language understanding (NLU) may be performed using an on-device NLU module that processes the recognized text generated using on-device speech recognition, and optionally contextual data, to generate NLU data.

[0024] The NLU data may include intent(s) corresponding to the spoken utterance and, optionally, parameter(s) of the intent(s) (e.g., slot values). On-device fulfillment can be performed using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU) and, optionally, other local data, to determine action(s) to take to resolve the intent(s) (and, optionally, parameter(s) for the intent(s)) of the speech utterance. This can include determining a local and / or remote response (e.g., answer) to the speech utterance, interaction(s) with locally installed application(s) based on the speech utterance, command(s) to send to Internet of Things (IoT) device(s) (directly or via corresponding remote system(s)) based on the speech utterance, and / or other resolution action(s) to take based on the speech utterance. The on-device fulfillment can then initiate local and / or remote performance / execution of the determined action(s) to resolve the speech utterance.

[0025] In various embodiments, remote speech processing, remote NLU, and / or remote fulfillment can be at least selectively available. For example, recognized text can be at least selectively sent to a remote automated assistant component(s) for remote NLU and / or remote fulfillment. For example, recognized text can be optionally sent for remote performance in parallel with on-device performance or in response to a failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution can be prioritized because they can at least reduce latency when resolving a speech utterance (because client-server round trip(s) are not required to resolve the speech utterance). Furthermore, on-device functionality can be the only functionality available in situations where network connectivity is absent or limited.

[0026] In some implementations, computing device 202 may include one or more applications 234 that may be provided by a third-party entity different from the entity that provided computing device 202 and / or automated assistant 204. The application state engine of automated assistant 204 and / or computing device 202 may access application data 230 to determine one or more actions that may be taken by one or more applications 234, as well as the state of each application of one or more applications 234 and / or the state of each device associated with computing device 202. The device state engine of automated assistant 204 and / or computing device 202 may access device data 232 to determine one or more actions that may be taken by computing device 202 and / or one or more devices associated with computing device 202. Additionally, the application data 230 and / or any other data (e.g., device data 232) can be accessed by the automated assistant 204 to generate context data 236 that can characterize the context in which a particular application 234 and / or device is running, and / or the context in which a particular user is accessing the computing device 202, the application 234, and / or any other device or module.

[0027] While one or more applications 234 are executing on the computing device 202, the device data 232 can characterize the current operational state of each application 234 executing on the computing device 202. Additionally, the application data 230 can characterize one or more features of the executing applications 234, such as the content of one or more graphical user interfaces rendered at the direction of the one or more applications 234. Alternatively or additionally, the application data 230 can characterize action schemas, which can be updated by each application and / or by the automated assistant 204 based on the current operational state of each application. Alternatively or additionally, the one or more action schemas for one or more applications 234 can remain static but can be accessed by the application state engine to determine suitable actions to initialize via the automated assistant 204.

[0028] Computing device 202 may further include assistant invocation engine 222, which can process application data 230, device data 232, context data 236, and / or any other data accessible to computing device 202 using one or more trained machine learning models. Assistant invocation engine 222 can process this data to determine whether to wait for the user to explicitly speak an invocation phrase to invoke automated assistant 204, or whether to consider the data as indicative of the user's intent to invoke the automated assistant, instead of requiring the user to explicitly speak an invocation phrase. For example, one or more trained machine learning models can be trained using instances of training data based on scenarios in which a user is in an environment in which multiple devices and / or applications exhibit various operating states. Instances of training data can be generated to capture training data. The training data characterizes contexts in which a user invokes an automated assistant and other contexts in which the user does not invoke an automated assistant. Once the one or more trained machine learning models have been trained according to these training data instances, assistant invocation engine 222 can cause automated assistant 204 to detect or limit the detection of an invocation phrase spoken by a user based on features of the context and / or environment. Additionally or alternatively, assistant invocation engine 222 can cause automated assistant 204 to detect or limit the detection of one or more assistant commands from a user based on features of the context and / or environment. In some implementations, assistant invocation engine 222 can disable or limit based on computing device 202 detecting an assistant that suppresses output from other computing devices.In this manner, when computing device 202 detects an assistant that suppresses output, automated assistant 204 is not invoked based on context data 236. This would otherwise cause automated assistant 204 to be invoked if an assistant that suppresses output had not been detected.

[0029] In some implementations, system 200 may include a device detection engine 216 that can determine whether a device, such as a peripheral device, is present within range of computing device 202 so that computing device 202 can establish a wireless connection with the device. For example, device detection engine 216 may be a module that is part of automated assistant 204 or a module separate from automated assistant 204 and can determine whether computing device 202 receives a beacon signal from a peripheral device. In some implementations, device detection engine 216 can determine whether a beacon signal is received from a peripheral device that can control or otherwise interface with the automated assistant. When device detection engine 216 determines that a peripheral device is available for connection with computing device 202, device detection engine 216 can interact with sequence generation engine 218 to generate connection sequence data.

[0030] The connection sequence data generated by the sequence generation engine 218 may indicate the priority that a particular application has over other applications when requesting a connection with a peripheral device detected by the device detection engine 216. In some embodiments, the sequence generation engine 218 may generate the connection sequence data based on determining whether the application 234 includes a designated application on the peripheral device. If it is determined that the application 234 includes a designated application on the peripheral device, the connection sequence data may be generated to prioritize connecting the designated application with the peripheral device over connecting the automated assistant 204 with the peripheral device. In some embodiments, the connection sequence data may be generated with further instructions for each application that successfully connects to the peripheral device and, upon completion of the connection, provide an indication of completion to other applications with next priority. In some embodiments, the connection sequence data may vary over time for a peripheral device according to the characteristics that a particular application exhibits when connecting to the peripheral device. For example, an application that exhibits a threshold number of timeouts when attempting to connect with the peripheral device may be given a lower priority the next time connection sequence data is generated for the peripheral device. Thereafter, if the number of timeouts or other characteristics of the lower priority application improves, the application may be assigned a relatively higher priority the next time connection sequence data is generated for the peripheral device.

[0031] In some implementations, system 200 may include a profile connection engine 226 that can control whether a particular profile of the automated assistant and / or other applications 234 connects or disconnects to a peripheral device. For example, when device detection engine 216 receives an indication that a peripheral device is available, profile connection engine 226 can determine whether automated assistant 204 is already connected to the peripheral device via a particular profile (e.g., a hands-free profile) and / or whether any other profiles are available for use with the peripheral device. When profile connection engine 226 determines that automated assistant 204 is already connected to the peripheral device and / or that no other profiles are available for connecting with the peripheral device, profile connection engine 226 can cause automated assistant 204 to avoid attempting to reconnect with the peripheral device. In other words, connection sequence data may or may not be generated in response to detection of the peripheral device, but the automated assistant and / or any other associated applications can avoid attempting to reconnect with the peripheral device in accordance with the connection sequence data.

[0032] In some implementations, the system 224 may include a state machine engine 224 capable of generating state machine data that can be synchronized with multiple different devices and / or applications. The state machine engine 224 may adapt the state machine data over time as the application 234 and / or automated assistant 204 connects or disconnects from different devices and / or utilizes different profiles with different devices. In this manner, the operating system of a computing device seeking to connect with a peripheral device and / or the operating system of the peripheral device may maintain a synchronized record of the connection status of the devices and applications and / or the respective profiles utilized for the corresponding connections.

[0033] FIG. 3 illustrates a method 300 for ordering connections of applications, including an automated assistant, to one or more peripheral devices, prioritizing designated applications for connecting first with one or more peripheral devices. Method 300 may be performed by one or more computing devices, applications, and / or any other device or module that may be associated with an automated assistant. Method 300 may include operation 302 of determining whether a separate device is available for connecting to a computing device that includes two or more applications. The computing device may be, without limitation, a mobile phone or other smart device, but the separate device may include other computing devices that can communicate via one or more communication protocols (e.g., Bluetooth Classic communication protocol, Bluetooth Low Energy communication protocol, etc.). The separate device may determine that it is available for connecting with the computing device in response to receiving a beacon signal or other data from the separate device and / or based on the computing device otherwise detecting the separate device. If it is determined that the separate device is available, method 300 may proceed from operation 302 to operation 304. Otherwise, the computing device or an application on the computing device may continue to determine whether the separate device is available.

[0034] Act 304 may include having two or more applications provide a request to connect to the separate device. In some implementations, the two or more applications may include a designated application provided by a third-party entity that also provides (e.g., sells or manufactures) the separate device and an automated assistant application that can respond to inputs at the separate device and the computing device. The request may be provided by the two or more applications in response to the two or more applications receiving data from the operating system of the computing device, from the separate device via other wireless communication modalities (e.g., Wi-Fi), and / or from other applications or devices. For example, the data received from the separate device may indicate profiles that can be operated via the separate device, and therefore certain applications may provide requests according to whether they can be controlled according to such profiles.

[0035] From operation 304, method 300 may proceed to operation 306, where it determines whether the request was received from a designated application for the separate device. For example, when the separate device is an athletic device for generating athletic data during an exercise routine, the designated application may be an athletic application that processes the athletic data. Thus, the athletic application may provide a request to connect with the separate device, and the request may be considered to have originated from the application designated for the separate device. Based on the request being received from the designated application, method 300 may proceed from operation 306 to operation 308. Otherwise, method 300 may proceed from operation 306 to operation 310.

[0036] Operation 308 may include prioritizing the designated application in connection sequence data, which may be utilized to determine a connection sequence for connecting two or more applications to the separate devices. In some implementations, the connection sequence data may be generated to indicate a priority or order in which applications should connect to the separate devices available for connection. In this manner, each application may not connect to the separate device until a higher priority application has completed the connection process with the separate device. When the designated application is indicated as the preferred application for connecting to the separate device, method 300 may proceed from operation 308 to operation 310.

[0037] Act 310 may include generating an order in which two or more applications should attempt to connect to the separate device. In some embodiments, the order may be characterized by connection sequence data, which may be generated by the separate device and / or the operating system of the computing device and / or any other applications associated with the separate device and / or the computing device. The order may prioritize designated applications over automated assistants accessible through the computing device and controllable through the separate device. In this manner, the automated assistant may at least temporarily avoid attempting to connect to the separate device until the designated application completes the connection process with the separate device. In some embodiments, the sequence data may also include instructions to instruct each application attempting to connect to the separate device to communicate with one or more other applications in the sequence to connect with the separate device. In this manner, each application may be notified that a connection is or is not occurring and may initialize its respective connection process upon receiving notification that the separate device is ready to connect.

[0038] Method 300 may proceed from operation 310 to operation 312, where it causes two or more applications to connect with the separate devices according to the generated connection sequence data. In some implementations, the two or more applications may initialize connections with the separate devices by first allowing a designated application to establish an encrypted communication channel with the separate device. Thereafter, other applications, such as an automated assistant, may separately establish encrypted communication channels with the separate devices. Sequencing the creation of encrypted channels may conserve computational resources, given that establishing such encrypted channels can consume memory and processing bandwidth. In some implementations, the two or more applications may initialize connections with the separate devices by first allowing a designated application to connect with the separate device via the Bluetooth Classic protocol (or other protocol) and allowing other applications to connect via the Bluetooth Classic protocol or a different protocol.

[0039] 4 is a block diagram 400 of an exemplary computer system 410. The computer system 410 typically includes at least one processor 414 that communicates with a number of peripheral devices via a bus subsystem 412. These peripheral devices may include a storage subsystem 424 (e.g., including memory 425 and a file storage subsystem 426), a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices enable user interaction with the computer system 410. The network interface subsystem 416 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.

[0040] The user interface input devices 422 may include keyboards, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, scanners, touchscreens integrated into displays, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information onto the computer system 410 or communications network.

[0041] The user interface output devices 420 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 410 to a user or to another machine or computer system.

[0042] Storage subsystem 424 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 424 may include logic for performing selected aspects of method 300 and / or logic for implementing one or more of system 200, computing device 104, automated assistant, peripheral device 112, and / or any other applications, devices, apparatuses, and / or modules described herein.

[0043] These software modules generally execute on processor 414 alone or in combination with other processors. The memory 425 used by storage subsystem 424 may include several memories, including a main random access memory (RAM) 430 for storing instructions and data during program execution and a read-only memory (ROM) 432 in which fixed instructions are stored. The file storage subsystem 426 may provide persistent storage for program files and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that perform the functions of a particular embodiment may be stored by file storage subsystem 426 in storage subsystem 424, or may be stored on another machine accessible by processor(s) 414.

[0044] Bus subsystem 412 provides a mechanism for allowing the various components and subsystems of computer system 410 to communicate with each other as intended. Although bus subsystem 412 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0045] Computing device 410 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system 410 shown in Figure 4 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computer system 410 can have more or fewer components than the computer system shown in Figure 4.

[0046] In situations where the systems described herein collect or may use personal information about users (or, as is sometimes referred to herein as "participants"), users may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how content that may be more relevant to the user is received from content servers. Also, certain data may be processed in one or more ways to remove personally identifiable information before it is stored or used. For example, a user's identity may be processed so that information that can identify the user individually cannot be determined, or if geographic location information (such as to the city, zip code, or state level) is obtained, the user's geographic location may be generalized so that the user's specific geographic location cannot be determined. Thus, users may control how information about them is collected and / or used.

[0047] While several embodiments have been described and illustrated herein, various other means and / or structures may be utilized to perform the functions and / or obtain the results and / or one or more advantages described herein, and each such variation and / or modification is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be illustrative, and the actual parameters, dimensions, materials, and / or configurations will depend on the specific application(s) for which the teachings are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. Accordingly, it should be understood that the foregoing embodiments are presented by way of example only, and that, within the scope of the appended claims and their equivalents, embodiments may be practiced other than as specifically described and claimed. Embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, unless such features, systems, articles, materials, kits, and / or methods are mutually inconsistent.

[0048] In some implementations, a method executed by one or more processors is described as including operations such as determining that a peripheral device is available for connection with a computing device, the computing device including an automated assistant application that provides access to an automated assistant, and the peripheral device including an interface for a user to interact with the automated assistant when the automated assistant application is connected to the peripheral device. The method may further include causing two or more applications on the computing device to provide a request to connect to the peripheral device based on the peripheral device being available for connection with the computing device, the two or more applications including the automated assistant application. The method may further include providing connection sequence data to the automated assistant application based on the request from the two or more applications, the connection sequence data instructing the automated assistant application to connect with the peripheral device after a separate application of the two or more applications has connected with the peripheral device. The method may further include causing the automated assistant to connect with the peripheral device after a separate application of the two or more applications has connected with the peripheral device in response to providing the connection sequence data to the automated assistant application.

[0049] In some embodiments, the separate application connects with the peripheral device over a channel that is encrypted differently from other channels over which the automated assistant connects with the peripheral device. In some embodiments, the automated assistant application is provided by a separate entity that is different from a separate entity that provides the separate application and the peripheral device. In some embodiments, the separate application connects with the peripheral device over a Bluetooth Classic protocol, and the automated assistant application connects with the peripheral device over a Bluetooth Low Energy protocol. In some embodiments, after the separate application of the two or more applications connects with the peripheral device, causing the automated assistant application to connect with the peripheral device includes causing the automated assistant application to connect with the peripheral device in response to the automated assistant application receiving a connection confirmation from the separate application of the two or more applications.

[0050] In some embodiments, after a separate application of the two or more applications connects with the peripheral device, causing the automated assistant application to connect with the peripheral device includes causing the automated assistant application to connect with the peripheral device in response to the automated assistant application receiving a connection confirmation from the peripheral device. In some embodiments, the connection confirmation indicates that the peripheral device has completed connection with the separate application, and the connection confirmation is communicated to the automated assistant application through a different communication modality than the peripheral device connecting with the separate application.

[0051] In another embodiment, a method executed by one or more processors is described as including operations such as determining, in a computing device, that a peripheral device is available for connection with the computing device, wherein, when the peripheral device is in communication with the computing device, the peripheral device selectively communicates with a designated application of the computing device according to a first communication protocol and with a separate application of the computing device according to a second communication protocol. The method further includes providing connection sequence data to the separate application of the computing device based on the peripheral device being available for connection with the computing device, the sequence data instructing the separate application to connect with the peripheral device after the designated application of the computing device connects with the peripheral device according to the first communication protocol. The method may further include causing, by the computing device, the separate application to connect with the peripheral device via the second communication protocol after the designated application connects with the peripheral device via the first communication protocol.

[0052] In some embodiments, the first communication protocol includes a Bluetooth Classic protocol and the second communication protocol includes a Bluetooth Low Energy protocol. In some embodiments, the separate application includes an automated assistant application that connects with the peripheral device via the Bluetooth Classic protocol and the Bluetooth Low Energy protocol. In some embodiments, after the designated application connects with the peripheral device via the first communication protocol, causing the separate application to connect with the peripheral device via the second communication protocol includes providing an indication to the separate application that the designated application has completed connection with the peripheral device via the first communication protocol.

[0053] In yet another embodiment, a method executed by one or more processors is described as including operations such as determining, by a peripheral device, that the computing device is available to connect with the peripheral device via a wireless communication modality, wherein the computing device provides access to an automated assistant application responsive to a voice utterance from a user. The method may further include providing connection sequence data to the computing device based on the computing device's availability to connect with the peripheral device, the connection sequence data indicating a sequence for at least a designated application of the computing device and the automated assistant application to connect with the peripheral device. The method may further include receiving, from the computing device, a request for the designated application to connect with the peripheral device via a first communication channel established via the wireless communication modality. The method may further include, in response to receiving the request, causing the designated application to connect with the peripheral device via the first communication channel. The method may further include causing the designated application or the peripheral device to provide an indication to the automated assistant application that the designated application will connect to the peripheral device based on the peripheral device connecting with the designated application. The method may further include, in response to the automated assistant application receiving an indication that the designated application is connected to the peripheral device based on the connection sequence data, causing the automated assistant application to connect with the peripheral device over a second communication channel.

[0054] In some embodiments, the peripheral device includes an interface for receiving voice utterances for the automated assistant application when the automated assistant application connects to the peripheral device via a second communication channel. In some embodiments, when the peripheral device connects with the designated application, the peripheral device controls the designated application according to a first profile, and when the peripheral device connects with the automated assistant application, the peripheral device controls the automated assistant application according to a second profile different from the first profile. In some embodiments, the first profile includes a streaming profile and the second profile includes a hands-free profile. In some embodiments, having the designated application or the peripheral device provide an indication to the automated assistant application includes causing the indication to be provided through a wireless communication modality separate from the wireless communication modality.

[0055] In some embodiments, the wireless communication modality includes Bluetooth, and the separate wireless communication modality includes Wi-Fi. In some embodiments, the first wireless communication channel operates according to the Bluetooth Classic protocol, and the second wireless communication channel operates according to the Bluetooth Low Energy protocol. In some embodiments, the method may further include generating state machine data based on a designated application connecting with the peripheral device and the automated assistant application, the state machine data characterizing a connection status between the designated application and the peripheral device and a separate connection status between the automated assistant application and the peripheral device. In some embodiments, the method further includes providing the state machine data to the computing device by the peripheral device via the wireless communication modality or the separate wireless communication modality.

Claims

1. 1. A method executed by one or more processors, the method comprising: determining that a peripheral device is available for connection with the computing device; The computing device includes an automated assistant application that provides access to an automated assistant, and the peripheral device includes an interface for a user to interact with the automated assistant when the automated assistant application is connected to the peripheral device, and the method further comprises: causing two or more applications on the computing device to provide a request to connect to the peripheral device based on the peripheral device being available for connection with the computing device; the two or more applications include the automated assistant application, and the method further comprises: providing connection sequence data to the automated assistant application based on the requests from the two or more applications; The connection sequence data instructs the automated assistant application to connect to the peripheral device after a separate application of the two or more applications connects to the peripheral device, and the method further comprises: In response to providing the connection sequence data to the automated assistant application, causing the automated assistant to connect with the peripheral device after the separate applications of the two or more applications connect with the peripheral device.

2. The method of claim 1 , wherein the separate application connects to the peripheral device over a channel that is encrypted differently from other channels over which the automated assistant connects to the peripheral device.

3. The method of claim 1 or 2, wherein the automated assistant application is provided by a separate entity that is distinct from a separate entity that provides the separate application and the peripheral device.

4. 10. The method according to any of the preceding claims, wherein the separate application connects to the peripheral device via a Bluetooth Classic protocol and the automated assistant application connects to the peripheral device via a Bluetooth Low Energy protocol.

5. causing the automated assistant application to connect to the peripheral device after the separate applications of the two or more applications connect to the peripheral device, causing the automated assistant application to connect with the peripheral device in response to the automated assistant application receiving a connection confirmation from the separate one of the two or more applications; 10. A method according to any preceding claim, comprising:

6. causing the automated assistant application to connect to the peripheral device after the separate applications of the two or more applications connect to the peripheral device, causing the automated assistant application to connect with the peripheral device in response to the automated assistant application receiving a connection confirmation from the peripheral device; The method according to any one of claims 1 to 4, comprising:

7. 7. The method of claim 6, wherein the connection confirmation indicates that the peripheral device has completed connection with the separate application, and the connection confirmation is communicated to the automated assistant application through a different communication modality by which the peripheral device is connected with the separate application.

8. 1. A method executed by one or more processors, the method comprising: determining, in a computing device, that a peripheral device is available for connection with the computing device; When the peripheral device is in communication with the computing device, the peripheral device selectively communicates with a designated application of the computing device according to a first communication protocol and selectively communicates with a separate application of the computing device according to a second communication protocol, the method further comprising: providing connection sequence data to the separate application on the computing device based on the peripheral device being available for connection with the computing device; The sequence data instructs the separate application to connect to the peripheral device after the designated application on the computing device connects to the peripheral device according to the first communication protocol, and the method further comprises: and after the designated application connects to the peripheral device via the first communication protocol, causing the computing device to connect the separate application to the peripheral device via the second communication protocol.

9. The method of claim 8 , wherein the first communication protocol comprises a Bluetooth Classic protocol and the second communication protocol comprises a Bluetooth Low Energy protocol.

10. The method of claim 9 , wherein the separate application includes an automated assistant application that connects with the peripheral device via the Bluetooth Classic protocol and the Bluetooth Low Energy protocol.

11. causing the separate application to connect to the peripheral device via the second communication protocol after the designated application connects to the peripheral device via the first communication protocol; A method according to any one of claims 8 to 10, comprising providing an indication to the separate application that the designated application has completed a connection with the peripheral device via the first communications protocol.

12. 1. A method executed by one or more processors, the method comprising: determining, by the peripheral device, that the computing device is available to connect with said peripheral device via a wireless communication modality; The computing device provides access to an automated assistant application responsive to voice utterances from a user, and the method further comprises: providing connection sequence data to the computing device based on the availability of the computing device to connect with the peripheral device; The connection sequence data indicates a sequence for at least a designated application and an automated assistant application of the computing device to connect with the peripheral device, and the method further comprises: receiving, from the computing device, a request for the designated application to connect with the peripheral device over a first communication channel established over the wireless communication modality; In response to receiving the request, causing the designated application to connect with the peripheral device over the first communication channel; Based on the peripheral device connecting to the designated application, causing the designated application or the peripheral device to provide an indication to the automated assistant application that the designated application is connected to the peripheral device; In response to the automated assistant application receiving the indication that the designated application will connect to the peripheral device based on the connection sequence data, causing the automated assistant application to connect to the peripheral device via a second communication channel; A method comprising:

13. The method of claim 12 , wherein the peripheral device includes an interface for receiving the voice utterance for the automated assistant application when the automated assistant application is connected to the peripheral device via the second communication channel.

14. 14. The method of claim 12 or 13, wherein when the peripheral device connects to the designated application, the peripheral device controls the designated application according to a first profile, and when the peripheral device connects to the automated assistant application, the peripheral device controls the automated assistant application according to a second profile different from the first profile.

15. The method of claim 14 , wherein the first profile comprises a streaming profile and the second profile comprises a hands-free profile.

16. causing the designated application or the peripheral device to provide the indication to the automated assistant application, causing the indication to be provided through a wireless communication modality separate from the wireless communication modality; The method according to any one of claims 12 to 15, comprising:

17. The method of claim 16, wherein the wireless communication modality includes Bluetooth and the separate wireless communication modality includes Wi-Fi.

18. 18. The method of claim 17, wherein the first wireless communication channel operates according to a Bluetooth Classic protocol and the second wireless communication channel operates according to a Bluetooth Low Energy protocol.

19. generating state machine data based on the designated application connecting with the peripheral device and the automated assistant application; The method according to any one of claims 12 to 18, wherein the state machine data characterizes a connection status between the designated application and the peripheral device and a separate connection status between the automated assistant application and the peripheral device.

20. 20. The method of claim 19, further comprising the peripheral device providing the state machine data to the computing device via the wireless communication modality or a separate wireless communication modality.

21. 1. A computing device comprising: one or more processors; a memory storing instructions that, when executed by said one or more processors, cause said one or more processors to perform the operations of any one of claims 1 to 20; A computing device comprising:

22. A computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of any one of claims 1 to 20.

Images