Techniques for communication between a hub device and multiple endpoints

Abstract

Techniques for coordinating interactions between a user device and a plurality of accessory devices are disclosed. In one example, a user device receives information identifying one or more accessory devices in communication with the user device. The user device can implement an accessory interaction instance for each of the identified accessories. A first accessory interaction instance can be associated with a first accessory of the identified accessories, and receive a first audio input corresponding to a user request from the first accessory. The first accessory interaction instance can process a portion of the received audio input and receive a first response from a server computer. The user device can then transmit the first response to the first accessory device.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. nonprovisional application 17 / 718,977, filed April 12, 2022, entitled “TECHNIQUES FOR COMMUNICATION BETWEEN HUB DEVICE AND MULTIPLE ENDPOINTS,” which claims priority under 35 U.S.SC §119(e) to U.S. provisional application 63 / 175,473, filed April 15, 2021, also entitled “TECHNIQUES FOR COMMUNICATION BETWEEN HUB DEVICE AND MULTIPLE ENDPOINTS.” The contents of these applications are incorporated herein by reference.

[0003] This application claims the benefit of U.S. nonprovisional application 17 / 718,984, filed April 12, 2022, entitled “TECHNIQUES FOR LOAD BALANCING WITH A HUB DEVICE AND MULTIPLE ENDPOINTS,” which claims priority under 35 U.S.SC §119(e) to U.S. provisional application 63 / 175,478, filed April 15, 2021, entitled “TECHNIQUES FOR LOAD BALANCING WITH A HUB DEVICE AND MULTIPLE ENDPOINTS.”

[0004] This application claims the benefit of U.S. non-provisional application 17 / 719,086, filed April 12, 2022, entitled “TECHNIQUES FOR ESTABLISHING COMMUNICATIONS WITH THIRD-PARTY ACCESSORIES,” which claims priority under 35 U.S. SC §119(e) to U.S. provisional application 63 / 175,480, filed April 15, 2021, entitled “TECHNIQUES FORESTABLISHING COMMUNICATIONS WITH THIRD-PARTY ACCESSORIES.” Background Technology

[0005] Technologies exist for communication between multiple user devices in residential environments. For example, a user can interact with a device that provides a digital assistant program. This device can communicate with other devices via the digital assistant to execute requests from the user; these other devices include those controlling smart accessory devices such as light switches, speakers, and thermostats. However, controlling the functionality of smart devices remains challenging. Users may not be able to directly access their user devices using a digital assistant to provide the desired interaction. Accessory devices can have many different features and capabilities and are manufactured by different companies. In a residential environment, users may want to interact with accessories via voice commands, in the same way they interact with user devices using a device assistant. Summary of the Invention

[0006] Embodiments of this disclosure may provide methods, systems, and computer-readable media for managing the interaction between an accessory device that receives audio user requests and a user device that processes those requests. In some examples, the user device may be associated with one or more accessories, and a separate instance of a device assistant application may be implemented to manage user requests at the accessory.

[0007] According to one implementation, a method can be performed by a computer system within a residential environment. The computer system can be a user device, such as a smartphone, tablet, smart TV (TV) media streaming device, smart hub speaker, etc. The user device can receive information identifying one or more accessories present within the residential environment. The user device can use this information to form an association with the identified accessories. In forming the association, the user device can implement instances of one or more processes or other applications corresponding to the associated accessories. These instances can be device assistant applications or other processes used to analyze human speech or other audio signals.

[0008] In some examples, a user equipment may receive audio input from one or more associated accessory devices. The audio input may be audio data streamed from the accessory to the user equipment. At least a portion of the audio input may correspond to an audio trigger or wake word. A first accessory interaction instance corresponding to the accessory transmitting the audio input may receive and process the audio. In some embodiments, processing may include transmitting some or all of the received audio input to a server computer or cloud service for robust language analysis. The server computer may parse and analyze the audio to determine whether the audio corresponds to an identified user within the residence, whether the audio is a user request or command, in what language any spoken audio is presented, what the appropriate response should be, and whether the user or the transmitting accessory device is authorized to issue the identified request or receive the determined response. The first interaction instance may then receive the response and transmit it to the accessory device.

[0009] In other implementations, the accessory interaction instance may also execute another process or operation as part of the received response. This could include setting a timer, instructing another device to take an action (such as turning off a light), or invoking a music streaming service to transmit audio to the accessory device. The accessory instance may also delegate the execution of the response to another device, including another accessory that may be more suitable for the response.

[0010] In some implementations, the user equipment may receive a second audio input from a second accessory device. Because each associated accessory has its own interaction instance at the user equipment, the second audio input may be processed simultaneously with the first audio input. At least a portion of the second audio input may correspond to a trigger or wake word. The second accessory interaction instance may process this portion of the second audio input to determine if a wake word is present. The instance may also determine whether the second accessory is authorized to interact with it. If a wake word is present and the second accessory is authorized, the second interaction instance may process the remainder of the second audio input in a manner similar to the processing of the first audio.

[0011] To enable interaction between accessory devices and user devices, in some implementations, each accessory device may include a software development kit (“SDK”) within its own memory. This software may be provided by an entity associated with the user device (e.g., a manufacturer), enabling communication with a specific user device regardless of the accessory device’s manufacturer. Accessory interaction instances may be configured to communicate with the SDK, which may include transferring accessory settings from the accessory to the user device for management. The SDK may also provide additional features, including wake word detection for audio input received at the accessory device.

[0012] Embodiments of this disclosure may provide methods, systems, and computer-readable media for providing load balancing management between accessory devices and hub devices. In some examples, a user device may act as a leader device to assign accessory devices to hub devices and provide information corresponding to that assignment.

[0013] According to one implementation, a method can be performed by a computer system within a residential environment. The computer system can be user equipment, such as a smartphone, tablet, smart TV (TV) media streaming device, smart hub speaker, etc. The user equipment can receive an allocation request from an accessory device within the residential environment. The user equipment can then select a hub device within the residential environment to connect to the accessory device. This selection can be based on determining which hub device is the best hub device to connect to the accessory.

[0014] In some implementations, the user equipment (UE) may receive information from an accessory device identifying accessory characteristics corresponding to the features or functions of the accessory. The UE may also receive information from one or more hub devices within the residential environment. Hub information may correspond to attributes of each hub, including the characteristics and capabilities of each hub. Hub information may also include the number of accessory connection slots available at the hub device. The UE can then score the hub device by comparing the accessory characteristics with the hub attributes to obtain a score corresponding to the suitability of the hub for connecting to the accessory device. This score can then be multiplied by the number of available accessory slots at each hub to obtain a final connectivity score. The hub with the highest connectivity score can be assigned to the accessory.

[0015] In another implementation, due to changes at the hub device (including increased processing load at the hub device), the hub device may update its available connection slots. The updated slots may cause currently assigned accessories to be discarded from the hub device. The discarded accessories can then request new assignments from the leader device.

[0016] In another implementation, a user equipment acting as the leader device may receive information indicating that it is no longer suitable to act as the leader device. This information may be determined by the user equipment based on its own current attributes, including that the user equipment is experiencing a higher processing load and is no longer able to effectively manage the hub device. The user equipment may transmit a request to the server device to select another user equipment to act as the leader device. The server device may then select the second user equipment as the leader device. The server device may then instruct the second user equipment to assume management control of the hub device and accessory devices. The first user equipment may then transmit current hub information and accessory allocation information to the second user equipment.

[0017] In some implementations, a user equipment acting as the leader device can obtain information about the current processing capacity of the first hub device. This information can indicate that the first hub device is unable to respond to an accessory assigned to it within a threshold time period. The user equipment can then obtain hub information from other hub devices and compare this hub information with the accessory characteristics currently associated with the first hub device to obtain a connection score. The user equipment can then instruct the first hub device to discard the accessory device and instruct the other hub device with the highest score to connect to the accessory device.

[0018] Embodiments of this disclosure provide methods, systems, and computer-readable media for providing communication between an accessory device and a cellular-enabled device. In some examples, a controller device may receive a call request from the accessory device and select an appropriate cellular-enabled device to make the call. The cellular-enabled device may then establish an audio connection with the accessory device to relay call audio to and from the accessory device.

[0019] According to one implementation, a method can be performed by a computer system within a residential environment. The computer system can be user equipment, such as a smartphone, tablet, smart TV (TV) media streaming device, smart hub speaker, etc. The user equipment can receive a call request from an accessory device within the residential environment. The user equipment can then select a cellular-enabled device within the residential environment to initiate the call and connect to the accessory device. This selection can be based on determining which cellular-enabled devices are associated with the user making the call request.

[0020] In some implementations, one or more accessory devices may be associated with a user device. The user device may implement instances of one or more processes or other applications corresponding to the associated accessory. This instance may be a device assistant application or other process for analyzing human speech or other audio signals. The set of processes in each instance may correspond to a software ecosystem on the user device.

[0021] In another implementation, when an accessory device participates in a call using a cellular-enabled device, the user equipment may enter a call listening state. While in call listening state, the instance corresponding to the accessory device may have voice processing capabilities limited to detecting only end words. In other implementations, when the first accessory device is participating in a call, interaction instances corresponding to other accessories associated with the user equipment may be unrestricted. When a call is not in progress, other instances may receive and process user requests from other accessories, according to normal operation of the accessories and the user equipment. Attached Figure Description

[0022] Figure 1 This is a simplified block diagram of an exemplary method according to some implementation schemes.

[0023] Figure 2 This is a schematic diagram of a residential environment including user equipment and accessory equipment, based on some implementation schemes.

[0024] Figure 3 This is another simplified block diagram illustrating at least some methods for coordinating communication between user equipment and accessory equipment according to some implementation schemes.

[0025] Figure 4This is a block diagram illustrating at least some technologies used for communication between an accessory device and a user device.

[0026] Figure 5 This is a flowchart illustrating an exemplary process by which an accessory device and a user device, according to an implementation scheme, detect a user request and take action based on the user request.

[0027] Figure 6 This is a simplified block diagram illustrating an exemplary architecture of a system for detecting user requests and taking actions based on user requests, according to some implementation schemes.

[0028] Figure 7 This is another simplified block diagram illustrating an example of an accessory device that receives and processes multiple communications from a user equipment according to some implementation schemes.

[0029] Figure 8 This is a flowchart illustrating the process by which an accessory device, according to some implementation schemes, determines which of a plurality of accessory devices will respond to a user request.

[0030] Figure 9 This is a flowchart illustrating the process by which a user device coordinates its interaction with multiple accessory devices according to some implementation schemes.

[0031] Figure 10 This is a simplified block diagram of an exemplary method according to some implementation schemes.

[0032] Figure 11 This is a schematic diagram of a residential environment including hub equipment and accessory equipment, based on some implementation schemes.

[0033] Figure 12 This is another simplified block diagram illustrating at least some methods of linking management accessory devices and hub devices according to some implementation schemes.

[0034] Figure 13 This is a simplified block diagram illustrating an exemplary architecture of a system for detecting user requests and taking actions based on user requests, according to some implementation schemes.

[0035] Figure 14 This is a flowchart illustrating an exemplary process for assigning an accessory device to a selected hub device according to an implementation scheme.

[0036] Figure 15 This is another flowchart illustrating an exemplary process for reassigning an accessory device from one hub device to another, according to an implementation scheme.

[0037] Figure 16This is another flowchart illustrating an exemplary process for reassigning an accessory device from one hub device to another, according to an implementation scheme.

[0038] Figure 17 This is another flowchart illustrating an exemplary process for transferring hub management from one user device to another, according to an implementation scheme.

[0039] Figure 18 This is a flowchart illustrating the process by which a user device assigns an accessory device to a selected hub device according to some implementation schemes.

[0040] Figure 19 This is a simplified block diagram of an exemplary process according to some implementation schemes.

[0041] Figure 20 This is a schematic diagram of a residential environment including user equipment and accessory equipment, based on some implementation schemes.

[0042] Figure 21 This is another simplified block diagram illustrating at least some methods for establishing communication between an accessory device and a cellular-enabled device according to some implementation schemes.

[0043] Figure 22 This is a simplified block diagram illustrating at least some technologies used for communication between an accessory device and a cellular-enabled device.

[0044] Figure 23 This is another simplified block diagram illustrating an exemplary architecture of a system for establishing communication between an accessory device and a cellular-enabled device according to an implementation scheme.

[0045] Figure 24 This is another flowchart illustrating an exemplary process, according to an embodiment, for requesting a telephone call at an accessory device and making that call at a cellular-enabled device.

[0046] Figure 25 This is another flowchart illustrating an exemplary process, according to an embodiment, for requesting termination of a telephone call at an accessory device and ending the call at a cellular-enabled device.

[0047] Figure 26 This is a flowchart illustrating the process by which a user device establishes a connection between an accessory device and a cellular-enabled device according to some implementation schemes. Detailed Implementation

[0048] In the following description, various examples will be described. For illustrative purposes, many specific configurations and details are presented to provide a thorough understanding of the examples. However, it will also be apparent to those skilled in the art that some examples can be implemented without these specific details. Furthermore, well-known features may be omitted or simplified to avoid confusion with the examples described herein.

[0049] The embodiments of this disclosure provide techniques for coordinating interactions between a user equipment and multiple accessory devices. As a first example, consider a residential environment corresponding to a residence. A person in the residence may want to know the current time. This person can use a verbal request (e.g., “What time is it?”) to query an accessory device within the residential environment (e.g., a nearby smart speaker). The accessory device can determine that the request is directed to that device and then transmit the received audio information to the user equipment (e.g., a hub speaker). The user equipment can process the audio information to determine the nature of the request and prepare a corresponding response (e.g., “It is 10:30 PM”). Alternatively, or in part in combination with the above, the user equipment can transmit some or all of the verbal request to a server computer (e.g., a service provider), where the service provider can determine the nature of the request and / or prepare a corresponding response. The user equipment can then transmit the response back to the accessory device for playback to the user. In another example, the user may have a request that does not require a response (e.g., “Turn off the lights”). The user equipment can process the audio request to identify one or more other devices corresponding to the request and transmit instructions to another device to execute the request (e.g., instructing the device to control the lights to turn off). Similar to the alternatives mentioned above, the user equipment can transmit requests that do not require a response to a service provider, which can then transmit instructions to another device or return instructions for another device to the user equipment. In the latter case, the user equipment will then send the server-generated instructions to the other device.

[0050] As an example of the above examples, a residential environment may include numerous “smart” devices, such as electronic devices with features that allow them to interact and operate autonomously to some extent. Smart devices can have a variety of functions, including cameras, speakers, thermostats, headphones and headsets, telephones, or media players. Smart devices may also have various network communication capabilities, including WiFi, Ethernet, Bluetooth, Zigbee, cellular, etc. These devices may be manufactured by different companies. In some cases, smart devices can be categorized as user devices and accessory devices. User devices can be residential-resident devices (e.g., smart speakers, smart digital media players configured to control a television (TV), mobile phones, etc.). While not always the case, in some examples, it is expected that residential devices reside within the residence and are not frequently moved (e.g., moved within or outside the residence). User devices may have capabilities equal to or greater than those of accessory devices. For example, a user device could be a mobile phone that may include wireless (e.g., WiFi) and cellular communication capabilities, multimedia capabilities, and a device assistant. In the same example, an accessory device could be a smart speaker that may include audio media and wireless communication capabilities but lacks a device assistant. A device assistant can be a virtual assistant program configured to interact with a user. In these examples, depending on its capabilities, a smart speaker can be a user device or an accessory device. In some examples, if the accessory is manufactured by a different entity than the one that manufactures the user device, the accessory may not initially be configured to have the ability to communicate with the user device. In some cases, the user device manufacturer may provide an accessory development kit (“ADK”) for installation on the accessory, which enables such communication after the accessory has been manufactured, sold, supplied, or used.

[0051] In some implementations, a user equipment (UE) may obtain information about accessory devices present in a residential environment. This information may be obtained by a UE that communicates directly with accessory devices sharing the same network within the residential environment. In other implementations, information about accessory devices may be sent to the UE by a second UE, a UE configured as a leader device, or a remote server device (e.g., a service provider). For example, a user in a residence may add a new accessory device to the residential environment. As part of this process, the user may interact with a second UE (e.g., a mobile phone) to configure the new accessory device and send information about the new accessory device to a first UE. As another example, a leader device in the residential environment may have information about multiple accessory devices in the residential environment and report information about some or all of these accessory devices to the UE. The UE can then use this information to form associations with the corresponding accessory devices. The accessory information may be stored by the UE.

[0052] A user equipment (UE) can associate with multiple accessory devices by creating an accessory interaction instance for each accessory device. An interaction instance can be a software module or process configured to perform tasks on the UE. In some embodiments, the interaction instance may each implement and / or communicate with a device assistant. For example, the UE may receive information about accessory smart speakers and smart thermostats located in a residential environment. The UE may create two interaction instances corresponding to the device assistant, one for each of the smart speaker and smart thermostat. In some embodiments, the interaction instance may be a copy of the device assistant, while in other embodiments, the instance may be a collection of modules including the device assistant and other processes for performing tasks on the UE. Depending on the associated accessory and its capabilities, the interaction instance may include different modules or processes. It should be understood that any suitable combination of processes running on the UE may be included in the interaction instance corresponding to the accessory device.

[0053] Continuing with the first example above, a user can make a request to the accessory. For example, a user could speak into the microphone of a nearby smart speaker (or thermostat, light bulb, etc.) and say, “Computer, what time is it?” In this example, the request (“What time is it?”) corresponds to a portion of the user’s audio input to the smart speaker. The start phrase (“Computer”) corresponds to a second portion of the user’s audio input and can be a trigger or wake word. In some implementations, the smart speaker can perform speech recognition processing on the wake word. Based on this processing, the smart speaker can determine whether the user’s voice is intended to be a request or command that the speaker should respond to. The wake word processing at the accessory device can be performed at a first level sufficient to recognize that the user’s audio input may contain a command or request. If such recognition is made, the smart speaker can then transmit the user’s audio to a user device running an accessory interaction instance corresponding to the smart speaker. In some implementations, the accessory device can temporarily store a copy of the audio input for transmission to the user device after processing the wake word portion. In other implementations, while processing the wake word portion of the audio input, the accessory device can establish a streaming audio connection with the user device to relay the portion of the user’s audio input after the wake word. In these implementations, the accessory device may transmit a copy of the wake word portion of the stored audio input to the user device for additional processing.

[0054] When receiving audio input from a smart speaker, the user device may perform additional processing on both the wake word portion of the audio input and the portion corresponding to the request or command. For example, the user device may perform natural language processing (“NLP”) on the wake word. The wake word processing at the user device may be performed at a second level, sufficient to confirm the presence of the wake word with a higher probability than wake word processing performed at the attachment (e.g., at the first level). Based on this wake word processing, the user device may then process the portion of the audio corresponding to the request. If the user device determines that the wake word portion is not actually the accurate wake word, it may ignore the remainder of the audio or terminate the audio stream from the attachment. In some embodiments, the voice processing module on the user device may be part of the attachment interaction instance. The interaction instance may also transmit all or part of the audio input to another device for analysis (e.g., to a service provider device). The service provider device may be a remote server computer or cloud device that performs voice processing and parses the request to provide an appropriate response. In some cases, the user device performs wake word processing while remotely processing the remainder of the audio. Parsing the request involves determining the content and context of the user's dictated audio and providing a response so that the user device can take action. In the current example, the response will be a time indication, which can be determined and prepared by the user device using an appropriate process or by a remote server device or a combination of both.

[0055] Once a response has been determined, the user equipment can execute it. This may include preparing an audio response for transmission back to the accessory device for playback to the user. Response preparation and execution can occur within the interaction instance corresponding to the accessory. Responses requiring specific actions may be delegated from the interaction instance to another process on the user equipment or to another device with which the user equipment can communicate, depending on the circumstances. Any response audio may be generated by a text-to-speech process within the interaction instance and transmitted to the accessory. The accessory can then play the response. Some implementations may provide various pre-generated audio responses corresponding to frequently encountered requests that do not require a unique response. These responses may be stored at the user equipment or the accessory device. For example, if the accessory device fails to connect to the user equipment and cannot transmit user audio, the accessory device may provide an audio response stored at the accessory indicating that the request cannot be processed.

[0056] Expanding on the example just described, consider another scenario where a second user in the residence also wants to make a request to a second accessory smart thermostat. This request could suitably be, "Computer, increase the temperature by 3℉." As previously described, the thermostat can process a portion of the user's audio input to determine the presence of a wake word, and if detected, transmit the wake word and request to the user device associated with the smart thermostat. Once received by the user device, instances of a voice processing module and an accessory interaction module, distinct from the interaction module associated with the previously executive smart speaker, can process the audio. Thus, the user device can simultaneously process and execute requests from multiple accessories. The accessory interaction instance corresponding to the smart thermostat could include a thermostat management module configured to manage the surrounding environment of the residence (e.g., heating or air conditioning). Upon processing a user request, the thermostat management module could execute the request and instruct the smart thermostat to increase its temperature setting by 3℉. In other embodiments, the thermostat management module could exist as a single instance on the user device. The accessory interaction instance corresponding to the smart thermostat can then delegate the execution of its requests to the single management module, as is desirable in systems containing multiple smart thermostat accessories but requiring unified management of the surrounding environment on the user equipment. In several embodiments, the architecture of the accessory interaction instance and other software modules on the user equipment can be configured in any suitable manner to improve the efficiency of request processing and execution of the interaction instance on the user equipment. This can include various combinations of modules and processes related to the characteristics and capabilities of the accessory device and the user equipment.

[0057] Figure 1 This is a simplified block diagram 101 of an exemplary implementation. Process 100 is an exemplary high-level process flow for a system including user equipment 110, which may be associated with various accessory devices 111 to receive user requests from the accessories. Figure 101 illustrates the state of the system corresponding to the blocks of process 100. Process 100 may be performed within a residential environment containing multiple user devices 110 and accessories. As described herein, user equipment 110 may be a hub speaker, while accessory devices 111 may be a smart thermostat 112, a camera 114, or a smart speaker 116. Although described as a particular device, it will be apparent that accessory devices 111 may be several types of smart devices in various combinations and numbers. Similarly, although a hub speaker is depicted as user equipment 110 performing process 100, other suitable devices may perform one or more operations in process 100. For example, a smartphone, media device (e.g., a smart TV) or tablet (connected to a cellular network, connected to a local area network via WiFi from a residential network, or connected to a wide area network (“WAN”)) may perform one or more of the operations of process 100.

[0058] In more detail, referring to process 100, at block 102, user equipment 110 may create one or more accessory interaction instances corresponding to one or more associated accessory devices 111. Each accessory interaction instance represents one or more software modules or processes running on user equipment 110 to enable accessory devices 111 to interact with user equipment 110. For example... Figure 1 As shown, accessory interaction instance 122 can correspond to smart thermostat 112, accessory interaction instance 124 can correspond to camera 114, and accessory interaction instance 126 can correspond to smart speaker 116.

[0059] At box 104, an accessory device, shown as smart speaker 116, receives audio input 120. In some embodiments, audio input 120 may include an audio portion corresponding to a user request or command (e.g., “What time is it?”) and a second portion corresponding to a wake word (e.g., “computer”). The wake word need not be a single word and may be a word or phrase that signals to the system that the user has or will speak a request, command, or other audible interaction to the system. Audio input may also be other sounds not made by the user, including the sound of breaking glass or a baby crying. In these cases, as described herein, the wake word may be a trigger sound corresponding to a portion of audio input 120. In other embodiments, the portion of audio input 120 corresponding to the wake word and the user request may be received by accessory 116 at intervals. This interval may be sufficient for the user to speak the wake word and receive an acknowledgment response from accessory 116 before speaking the user request. Upon receiving input containing the wake word, accessory 116 may process this portion of audio input 120 at a first level to determine the presence of the wake word. The first level processing may determine the timing and resource-efficient manner in which the wake word may be present. For example, the accessory can perform voice pattern matching using stored voice patterns corresponding to the user's spoken wake word. The stored patterns can be associated with users in the residential environment containing the system, or they can be general patterns applicable to a large number of possible users. In this way, the accessory device 116 does not bear the burden of a complex voice detection process, nor does it respond to every external audio input received by the user or other sources in its vicinity.

[0060] Moving down to box 106, upon detecting a wake word, the accessory device 116 can transmit the received audio input 120 to the user device 110, where it will be processed. As shown, the smart speaker 116 has a corresponding accessory interaction instance 126 on the user device 110, enabling the accessory interaction instance 126 to manage the processing of the audio input 120 received from the smart speaker 116. See below for reference. Figure 4In more detail, the accessory interaction example 126 may include modules configured to process audio input 120. For example, accessory interaction example 126 may include a voice detection module that analyzes the portion of audio input 120 corresponding to a wake word. This analysis may be performed at a second level where the presence of a wake word can be confirmed with a higher probability than wake word detection at smart speaker 116. Furthermore, in some embodiments, the voice detection module may determine the user's language and perform wake word detection based on the determined language. If the voice detection module of accessory interaction example 126 does not detect a wake word, user device 110 may ignore the audio input.

[0061] The Attachment Interaction Instance 126 may also include a module configured to communicate with the Remote Service 130. The Remote Service 130 may be provided by a remote server associated with the residential environment of the User Equipment 110 via a WAN or other network, and in some embodiments may be a cloud server. The remote service may include NLP or other speech analysis services. If the Attachment Interaction Instance 126 does detect a wake word, it may process that portion of the audio input 120 corresponding to the user request by transmitting it to the Remote Service 130. The Remote Service 130 may analyze the request to determine the type of request, an appropriate response, and one or more devices that will execute the response. In some embodiments, the Remote Service 130 may also determine the identity of the user making the request. This identity may be determined based on user profile information accessed by the Remote Service 130. As part of the Attachment Interaction Instance 126's processing of the audio input 120, the user profile information may be stored at the User Equipment and transmitted to the Remote Service 130. In some cases, the user profile information is stored on a remote device accessible by the Remote Service 130 or on a remote server providing the Remote Service 130. Once the request portion has been analyzed by remote service 130, the response can be transmitted back to user equipment 110 for execution. Upon receiving the response, attachment interaction instance 126 can then execute the response. Execution of the response may include delegating one or more elements of the response to other processes on the user equipment or another device, including other user equipment, attachment devices, or remote devices in the residential environment. Following the example shown in Figure 101, execution may include determining the current time and preparing an audio response to be played for the user. Attachment interaction instance 126 may delegate the request to a process on user equipment 110 to provide the current time to attachment interaction instance 126. Attachment interaction instance 126 may include a text-to-speech module that converts the received current time information into an audio response.

[0062] Moving to box 108, user equipment 110 can transmit a response to accessory device 116. For a response requiring an audio response 140 to the user, accessory interaction instance 126 on user equipment 110 can communicate with smart speaker 116 and transmit audio. As shown in Figure 101, audio response 140 is the reply "10:30 PM," corresponding to the current time requested by the user. Other responses may include indications that the user requests execution by another device or that the request cannot be executed. In some embodiments, accessory device 116 may not have the ability to play audio responses (e.g., it does not have speaker output), but may include a visual user interface (e.g., a screen) or other means (e.g., a light) to indicate the response to the user.

[0063] Figure 2 A residential environment 200 including user equipment and accessory devices is illustrated according to some embodiments. The user equipment may include a hub speaker 202, a media player 204, and a smartphone 206. These user devices may correspond to those described above relative to... Figure 1 The user equipment 110 of the described embodiment. Accessory devices may include smart speakers 212, 214, a smartwatch 216, and a thermostat 224. Similarly, these accessory devices may correspond to... Figure 1 The accessory devices 111 are described above. Some or all of these accessory devices may be third-party devices (e.g., not manufactured, programmed, or supplied by the user equipment's manufacturer, programmer, or supplier). Therefore, they may not be automatically and / or initially compatible with the user equipment. Each user equipment in residential environment 200 may be associated with zero, one, or more accessory devices. As shown by the long dashed lines, hub speaker 202 is associated with smart speakers 212, 214, and smartwatch 216, while media player 204 is associated with thermostat 224. Smartphone 206 is not associated with any accessory device. Devices within residential environment 200 may be configured to communicate using one or more network protocols over one or more networks associated with residential environment 200. For example, residential environment 200 may be associated with a local area network (“LAN”), WAN, cellular network, or other networks, and these devices may communicate using WiFi connectivity, Bluetooth connectivity, Thread connectivity, Zigbee connectivity, or other communication methods.

[0064] The arrangement of associations between accessory devices and user equipment can include various combinations and can be modified by the user equipment. For example, smartphone 206 can receive information about accessory devices to be associated with the smartphone. Accessory devices may include one or more accessory devices currently associated with other user equipment in the residence, and may include new accessory devices added to the residence. Smartphone 206 then creates an accessory interaction instance for each accessory association. In some embodiments, a user equipment in residential environment 200 can communicate with another user equipment to transfer one or more accessory devices associated with a first user equipment to the second user equipment. This transfer may occur automatically based on information received by the user equipment about residential environment 200, including but not limited to information that another user equipment may be more suitable to be associated with one or more accessories or that an accessory has been added to or removed from residential environment 200. The suitability of any particular user equipment associated with an accessory may be based at least in part on the capabilities of the user equipment, the capabilities of the accessory device, the current processing load experienced by the user equipment, the location of the device within the residential environment, and the state of communication between devices on the network. Many other criteria are envisioned for rearranging device associations in a residential environment.

[0065] In some implementations, accessory devices and non-resident user equipment may also leave the residential environment or lose network connectivity with the residential environment. Accessory devices leaving the residential environment can be unassociated with previously associated user equipment, causing the user equipment to remove the corresponding accessory interaction instance from its memory. Accessory devices associated with user equipment that has lost network connectivity with the residential environment can be reassigned by another user equipment that maintains network connectivity. Some implementations may have a user equipment designated as a leader device to manage the allocation of accessory devices among user equipment within the residential environment. In other implementations, if a user equipment and an accessory device are associated and leave the residential environment and lose network connectivity, the user equipment may maintain its association with the accessory device and perform the embodied methods described herein.

[0066] Return to Figure 2As an example of some implementations described above, the hub speaker can communicate with the smartphone 206 to transfer its association with the smartwatch 216. A user 230 wearing the smartwatch 216 can collect the smartphone 206 and move it to different rooms in the residential environment, making the smartphone 206 a more suitable user device associated with the smartwatch 216. The user 230 can then leave the residential environment 200 with both the smartphone 206 and the smartwatch 216. Even if the smartwatch 216 loses network connectivity with other devices in the residential environment, the smartphone 206 can maintain its association with the smartwatch 216 as an accessory device. Furthermore, the user 230 can take an additional accessory device outside the residential environment, and this accessory can also be associated with the smartphone 206, allowing the smartphone 206 to be associated with both accessory devices when outside the residential environment 200. As another example, the media player 204 can begin processing and playing media files and becomes unsuitable for association with the thermostat 224 due to load. The media player 204 can transfer the association of its accessory thermostat 224 to the smartphone 206 or the hub speaker 202.

[0067] continue Figure 2 The residential environment 200 may have multiple users 230 and 234, who issue multiple audio requests 232 and 236 to the attachment. Requests 232 and 236 correspond to the above reference. Figure 1 The audio input 120. Requests 232 and 236 may occur separately or simultaneously and may be received by multiple accessory devices, as shown by the short dashed lines. For example, request 232 may be received by smart speaker 214 or smartwatch 216, while request 236 may be received by smart speaker 212 and thermostat 224. As previously mentioned, the arrangement of accessory devices and their associations can take various forms and may change over time. Therefore, user requests may be received by multiple accessory devices associated with different user devices. For example, user request 236 may be received by both thermostat 224 associated with media player 204 and smart speaker 212 associated with hub speaker 202.

[0068] In some implementations, the accessory device can coordinate with other accessory devices within the residential environment 200 to determine which accessory device should respond to a user request received by one or more accessory devices. See below for reference. Figure 8In more detail, the selection of an accessory device in response to audio input can occur through an election process among the accessory devices. This election may use a criteria-based score, including but not limited to the intensity (e.g., loudness) of the audio input received at the accessory device, the quality of the received audio input, the capabilities of the accessory device, and the capabilities of the user device associated with the accessory device. As an example, both smart speaker 212 and thermostat 224 receive user request 236. Upon receiving the request, both accessories can process the portion of the user request corresponding to the wake word and transmit audio to their respective user device, hub speaker 202, and media player 204. The accessory interaction instance associated with the corresponding accessory on the user device can process the received wake word and determine the accessory's score. The user device can transmit the score back to the accessory before further processing user request 236. The accessories can then open a communication channel between them and exchange scores. Each accessory compares its score with other scores to determine a winner. The winning accessory can report its victory to its user device. Similarly, a lost accessory can report its loss to its user device and will not respond to user requests. In this example, since user 234 is in the same room as thermostat 224, it's possible that the thermostat will be louder and clearer than the smart speaker 212 in another room, thus hearing user request 236 more clearly. Therefore, the accessory election might result in thermostat 224 being the winning accessory. Conversely, smart speaker 212 or hub speaker 202 could have capabilities better suited to responding to user requests, such as when the request requires an audio response and thermostat 224 lacks audio output capabilities. In this scenario, a higher score could be assigned to smart speaker 212 to reflect its greater capability, leading to smart speaker 212 winning the election. Those skilled in the art will recognize the numerous potential scoring criteria for accessory devices.

[0069] Figure 3 An exemplary coordination process 300 for associating one or more user equipment 302 with one or more accessories 304 is illustrated. In some embodiments, user equipment 302 may correspond to the user equipment described herein (e.g., Figure 1 User equipment 110 Figure 2 (e.g., user equipments 202, 204, and 206). The configuration device 306, depicted herein as a smartphone, can be a user equipment within user equipment 302, a hub device, a leader device, or other devices associated with the configuration device. Configuration device 306 can be configured to communicate with user equipment 302 and attachment 304 via one or more networks (including LANs or WANs) described herein. In some embodiments, configuration device 306 is a remote server device configured to communicate with user equipment 302 and attachment 304 via a WAN (e.g., the Internet).

[0070] The coordination process 300 is presented in more detail, encompassing several elements. Configuration device 306 may include an attachment configuration module 310. Attachment configuration module 310 may be a software process running on configuration device 306. Attachment configuration module 310 may be configured to store, update, receive, and transmit information relating to the association of attachments 304 and user equipment 302. This information may include attachment management settings 312 and attachment settings 314. Attachment management settings 312 may include information identifying which attachments 304 are assigned to any specific device in user equipment 302. Attachment configuration module 310 may also include attachment settings 314, which provide association information for each attachment regarding the assigned user equipment.

[0071] Each user equipment in user equipment 302 may include an accessory management module 320, which may be a software process running on user equipment 302. In some embodiments, accessory management module 320 may receive, process, store, update, and transmit accessory management settings 322. Accessory management settings 322 correspond to information in accessory management settings 312 associated with a particular user equipment 302. For a particular user equipment, its accessory management settings 322 may include a list of all accessories assigned to that user equipment and other information relating to the capabilities of those assigned accessories. Accessory management module 320 may also include an accessory interaction instance 324, which may correspond to Figure 1 Attachment interaction instances 122, 124, and 126 are provided. Attachment interaction instance 324 can be created by user equipment 302 based on attachment management settings 322 received from configuration device 306. In this way, configuration device 306 can update the association between attachment 304 and user equipment 302 by sending the updated attachment management settings 322 to each user equipment based on the updated attachment management settings 312. Then, the attachment management module 320 on each user equipment can create or remove attachment interaction instances 324 as needed, so that it has attachment interaction instances 324 corresponding to its currently associated attachments.

[0072] Each accessory may also store accessory settings 334. Accessory settings 334 may include information identifying the user device currently associated with the accessory. Accessory settings 334 may also include information related to the processing of triggers or wake words at the accessory. For example, a user may change the wake word they want to activate (e.g., “wake up”) the device, which may be different from a generic wake word and may be identified with that specific user. Configuration device 306 may transmit correspondence information about custom wake words (e.g., an audio pattern of a wake word used for comparison with received audio input) to accessory device 304. Accessory settings 334 may be configured to include information about multiple wake word or trigger configurations corresponding to different wake words or triggers that may be detected at accessory device 304.

[0073] Finish Figure 3 The detailed elements of the process indicators 330 and 340 represent data transmissions between the configuration device 306 and the user device 302, and between the user device 302 and the accessory 304, respectively. Process indicators 330 and 340 can indicate communication between various devices as described herein via one or more networks, including but not limited to WiFi LAN or Internet WAN. Process indicator 330 indicates the transmission of data within accessory management settings 312 and accessory settings 314 to the user device 302. Correspondingly, the user device 302 can transmit data within its accessory management settings 322 or other data to the configuration device 306. This data can identify that one or more accessories are no longer communicating with their assigned user device. Similarly, process indicator 340 indicates the transmission of data including accessory settings 334 between the user device 302 and the accessory 304. In some embodiments, the configuration device 306 may also communicate directly with the accessory to transmit accessory settings 314 and receive information in return.

[0074] As a specific example of the foregoing description of several implementation schemes, consider a scenario where a new accessory device is introduced into a residential environment. The residential environment may correspond to... Figure 2The residential environment 200. Initially, no device in the residential environment has any information related to the new accessory, and no user equipment is associated with the new accessory. Configuration device 306 may be a user's smartphone and may have access to information about the new accessory. This information may be obtained through user input, for example, through an application running on the smartphone, to configure and provide devices within the residential environment. This information may also include the identifier of the new accessory from a list of identified accessories that may be added to the residential environment, or obtained by the smartphone communicating with the new accessory through a network within the residential environment's network. The smartphone may then communicate with user equipment 302 within the residential environment to receive information about user equipment 302 and their associated current accessories 304. Once the smartphone has the appropriate information about the new accessory and existing devices, it may update accessory management settings 312 and accessory settings 314 in its accessory configuration module 310, and then assign the new accessory to one of the user equipment 302. This assignment may include transmitting data corresponding to accessory management settings 322 to the selected user equipment and data corresponding to accessory settings 334 to the new accessory.

[0075] In a specific example, an accessory may be removed from a residential environment. This may occur if the accessory is a non-residential device and capable of leaving the residential environment (e.g., a smartwatch), or if the accessory has lost network communication with its assigned user device. In this case, the user device may transmit the updated accessory management settings 322 to the configuration device 306, which may then update its accessory management settings 312 and accessory settings 314 and transmit the updated information back to the user device. Updating the accessory management settings 312 may include reconfiguring the arrangement of the accessory 304 that still exists in the residential environment with the user device 302. A similar update may occur if the user device is removed (e.g., if the user device is a non-residential device such as a smartphone or has lost network connectivity). The accessory previously associated with the removed user device may communicate with the configuration device 306 and report the updated accessory settings 334, including the inability to communicate with the assigned user device. The configuration device 306 may then select a new user device for the accessory and transmit the updated accessory management settings 322 and accessory settings 334 to the appropriate device.

[0076] It should be understood that the various scenarios described in Reference Procedure 300 are representative. One or more aspects of each scenario may be changed, but the implementation scheme as described herein will still be performed, including but not limited to, the number and type of attachments, the number and type of user equipment, the type of network through which the configuration device communicates with other devices, and the way the configuration device obtains information about new attachment devices.

[0077] Figure 4 This is a block diagram 400 illustrating at least some techniques used for communication between accessory device 401 and user equipment 402 to process audio input and thus create a response. Diagram 400 includes some detailed architecture of representative devices and process flow arrows providing general indications of the transfer of data or information. These process flow arrows are not intended to suggest any specific architectural connection between the elements detailed herein. Figure 4 Each element depicted in Figure 400 may resemble one or more elements depicted in other figures described herein. For example, accessory device 401 may correspond to one or more of the accessories and accessory devices described herein, etc. In some embodiments, at least some of the elements in Figure 400 may be similar to... Figure 2 200 Residential Environment Context Operations.

[0078] Turning more specifically to each element, accessory device 401 may have audio input and output capabilities, including an accessory microphone input 404 and an accessory speaker output 406. The accessory microphone input 404 may include both the hardware and software / firmware necessary to provide the audio input functionality. Similarly, the accessory speaker output 406 may include both hardware and software to provide its functionality. Accessory device 401 also has an accessory audio module 412 that may interface with one or both of the accessory microphone input 404 and the accessory speaker output 406, and receive audio from other devices. Accessory device 401 also includes an accessory development kit (“ADK”) 408. The ADK may be an SDK stored on accessory device 401 and configured to be executed or processed on that accessory device. As used herein, the SDK may include application programming interfaces and associated software libraries sufficient to implement the operation of other software within or associated with the SDK. In some embodiments, the ADK may be provided by an entity associated with user device 402 (e.g., its manufacturer). ADK 408 may include a wake-word detection module 410 that performs first processing on a portion of the audio input corresponding to a trigger or wake-word. The wake-word detection module 410 itself may contain information about the wake-word and trigger, including, for example, trigger criteria and an audio pattern corresponding to a specific wake-word. As indicated by the process flow arrows, accessory device 401 may receive audio input at its microphone 404 and then process a portion of that audio in the wake-word detection module 410. This processing may be performed at a first level to determine the presence of a wake-word. The first-level processing may determine when the wake-word is likely to be present and in a resource-efficient manner. If a wake-word is detected, the received audio input may be transmitted to user device 402. In some embodiments, this transmission involves establishing a streaming audio connection from accessory device 401 to user device 402.

[0079] User equipment 402 may include a management module 414, which can correspond to Figure 3 The accessory management module 320. The management module 414 can provide one or more audio input repeaters 416, each associated with an accessory device. In this way, the user equipment 402 can manage multiple inbound audio inputs received from multiple accessory devices, including multiple simultaneous streaming audio connections. The audio input repeaters 416 can send the received audio input to the voice processing module 420, where the audio input is received by the accessory input plug-in 422. In some embodiments, the voice processing module 420 does not have an operational instance for each accessory device associated with the user equipment. In these cases, the voice processing module may have separate instances for each accessory device, including the accessory input plug-in and the wake word detection module 424.

[0080] As mentioned above, in contrast to Figure 1 The user equipment can process audio received from the accessory device. This may include processing a portion of the received audio to detect the presence of a wake word. The voice processing module 420 may include multiple instances of the wake word detection module 424, and in some embodiments, includes a wake word detection module 426 configured to process audio input directly received by the user equipment. The wake word detection module 424 may process the wake word audio at a second level where the presence of a wake word can be confirmed with a higher probability than that of the wake word detection module 410 at the accessory device 401. If the voice processing module 420 does not detect a wake word, the user equipment 402 may ignore the audio input. If a wake word is detected, the audio input may be further processed at the accessory interaction instance 430.

[0081] Attachment interaction instance 430 may include a virtual device assistant and other processes such as server interaction module 432, delegation process 434, and text-to-speech module 436. To process audio requested by the user or other audio input via wake word detection module 424, attachment interaction instance 430 may connect to remote service 450 and transmit a portion of the audio input to remote service 450. Each attachment interaction instance associated with an attachment may connect to remote service 450 via the server interaction module 432 corresponding to that attachment interaction instance. Remote service 450 may be hosted on a remote server or other remote device and may be implemented via a network (e.g., an Internet WAN) to which the user device can connect. NLP and other services for processing audio input may include remote service 450. Furthermore, remote service may include information related to user profiles, user language, and user authorization regarding voice interaction between the user and attachments and user devices within the residential environment. In some embodiments, user information or components of the user profile may be stored locally on the user device and accessible by attachment interaction instance 430 via a local service in local service 452. In this way, the remote service can identify the user making the user request, process the audio input in the language corresponding to the identified user, and determine whether the user is authorized to have the request executed by one or more devices. Since each accessory interaction instance 430 can interact with the remote service 450 separately and individually, multiple user requests can be processed and executed simultaneously or nearly simultaneously by a single user device 402.

[0082] Once the request has been processed, the attachment interaction instance 430 may receive a response from the remote service 450 for execution. The response may contain data corresponding to an audio message to be played for the requesting user at the attachment device 401. The audio response may be generated by the text-to-speech module 436. In some embodiments, the received response may require the attachment interaction instance 430 to communicate with the local service 452 to receive additional information to complete the request. For example, if the request inquires about the current time, the attachment interaction instance 430 may obtain the current time from a clock process or other service located elsewhere on the user device 402. In other embodiments, the response may require execution by another device or an action by the user device 402 or the attachment device 401 without expected further output. In those cases, the attachment interaction instance 430 may delegate the response to the local service 452 via a delegation process 434. The local service 452 may include communication processes for transmitting response instructions to other user devices for execution on those user devices or their associated attachments. The local service 452 may also include a music service. A user request may include a response to play music or other audio content at the attachment. The music service can then communicate with media module 470 to transmit music or other audio to accessory device 401, as described below. In yet another embodiment, the response may include a delay, making the delegation to local service 452 temporary. For example, the request may be to set a timer, which can be delegated to a local timer process. The accessory interaction instance that delegates the response can then freely handle additional request processing from its associated accessory until the delegating process 434 receives an indication that the timer process has completed, at which point the accessory interaction instance will complete the execution of the timer response by sending an appropriate indicator to the accessory.

[0083] Continuing with the details of user equipment 402, an audio response may be sent to accessory device 401 via media module 470, including accessory audio repeater 472. The accessory audio repeater may negotiate an audio connection with accessory device 401. The audio connection may be established through one of the networks to which user equipment 402 and accessory device 401 are connected, and may use any number of methods or protocols, including but not limited to AirPlay, Real-Time Transport Protocol (“RTP”), Real-Time Streaming Protocol (“RTSP”), etc. In some embodiments, the response is translated into a phrase or sentence of audible speech to be played at accessory device 401. In other embodiments, the response may be an audio stream of music or similar media to be played at accessory device 401. In yet another embodiment, the response may be an instruction to accessory device 401 to play a segment of audio stored locally at accessory device 401, such as a notification chime specific to the accessory.

[0084] Depending on its capabilities, accessory device 401 may not be configured to store the received audio response, which could be attributed to space limitations, the streaming nature of the audio response, or other reasons. As follows relative to... Figure 7 More specifically, as part of a response to a user request, the accessory interaction instance 430 may provide the accessory device 401 with instructions for processing received audio in the event that the accessory device 401 receives multiple input audio responses or other audio inputs simultaneously or nearly simultaneously. These instructions may include rules for attenuating or muting portions of the output audio to favor other portions of the audio, or other rules for mixing or balancing the level of the output audio generated based on the received audio responses. For example, a user may request a timer at the accessory device 401 and then later request the accessory to play a music stream. When the timer is off, the accessory interaction instance on the user device 402 may send a response corresponding to the timer prompt and instructions for the accessory device 401 to attenuate the streaming audio at the output, making the timer prompt audible louder than the music.

[0085] Return to Figure 4 In some implementations, the audio input repeater 416, accessory input plug-in 422, wake word detection module 424, and accessory audio repeater 472 represent multiple instances of the same or similar processes or modules running on user equipment 402. Instances of these elements correspond to an associated accessory device in the same way that accessory interaction instance 430 corresponds to an associated device. Depending on the software architecture, accessory interaction instance 430 may also encompass one or more of these instance elements, such that accessory interaction instance 430 includes one or more audio input repeaters 416, accessory input plug-in 422, wake word detection module 424, and accessory audio repeater 472. Various ways of presenting the device architecture to provide per accessory instance of various processes are envisioned.

[0086] continue Figure 4User equipment 402 may have its own microphone and speaker input and output capabilities, represented by microphone input 474 and speaker output 476 within media module 470. In several embodiments, user equipment 402 is capable of receiving audio input and directly processing audio through wake-word detection module 426 and device interaction instance 440, which may include digital virtual assistant and server interaction module 442, delegation process 444, and text-to-speech module 446. Other embodiments are characterized by user equipment that cannot directly receive audio input but can still process audio data transmitted from accessory devices to accessory interaction instances. For example, a media player user equipment (e.g., a TV media player) may not have a microphone input and therefore cannot “hear” any user audio input. The media player may still be associated with one or more accessory devices and include one or more corresponding accessory interaction instances. Processing of user requests made directly through user equipment 402 is performed in a manner similar to the processing of audio received by accessory devices. Because the device interaction instance 440 and the accessory interaction instance 430 are separate, the user equipment 402 can simultaneously process the second user request directly received by the user equipment 402 and the user request received from the accessory.

[0087] Figure 5 This is a flowchart illustrating a specific exemplary process 500 in which an accessory device and a user device detect a user request and take action based on the user request. Figure 5 Each of the elements and operations depicted herein may be similar to one or more elements depicted in other figures described herein. For example, user equipment 502 may be similar to other user equipment, and so on. In some embodiments, process 500 may be in a residential environment (e.g., Figure 2 The process is carried out within the residential environment (200). Process 500 and... Figure 8 and Figure 9 Processes 800 and 900 (described below) are shown as logic flowcharts, each operation of which represents a series of operations that can be implemented by hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media, which performs the operation when executed by one or more processors. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the operations may be implemented in any order and / or in parallel combinations.

[0088] Furthermore, some, any, or all of these processes can be executed under the control of one or more computer systems configured to have executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that is executed on one or more processors, by hardware, or a combination thereof. As described above, the code can be stored on a computer-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable storage medium is non-transitory.

[0089] At frame 504, accessory device 501 can receive user speech 503. User speech can correspond to... Figure 1 Audio input 120. User speech 503 may include a portion corresponding to a trigger or wake word (e.g., “computer”) and a portion corresponding to a user request (e.g., “What time is it?”).

[0090] At box 506, accessory device 501 may process the portion of user utterance 503 corresponding to the wake word in a first pass to determine the presence of the wake word. The first pass can determine the timing and resource-efficient manner in which the wake word may exist. At decision 508, based on the first pass, accessory device 501 determines whether the wake word exists. If not, the process can terminate at endpoint 510 by ignoring the user utterance. If, according to the first pass, a wake word exists, the process continues to box 512.

[0091] At box 512, accessory device 501 can establish a streaming audio connection with user equipment 502. This connection can occur via one of the networks to which accessory device 501 and user equipment 502 are connected (e.g., via WiFi LAN). Streaming audio can be performed using any number of methods or protocols, including but not limited to AirPlay, Real-time Transport Protocol (“RTP”), Real-time Streaming Protocol (RTSP), etc. User speech 503 is then transmitted to user equipment 502 via streaming audio. In some embodiments, portions of user speech 503 corresponding to the wake word and user request can be received by accessory device 501 at intervals. Due to the initial processing of the wake word portion, this portion can be sent separately to user equipment 502 via WiFi LAN through a buffered audio connection using Transmission Control Protocol (“TCP”) or other suitable methods for sending recorded audio data.

[0092] At box 514, user equipment 502 receives user speech 503 and any other streaming audio transmitted from the accessory device, which may include a portion of a longer user request transmitted after the accessory device processes the wake word and opens the streaming audio connection. At box 516, user equipment 502 may process the wake word a second time. This processing may occur at a second level where the presence of the wake word can be confirmed with a higher probability than the first processing at accessory device 501 at box 506. At decision 518, if user equipment 502 does not confirm the presence of the wake word, the process moves to box 520 and terminates the streaming audio connection. The process then moves to endpoint 522 and ignores the user speech. If user equipment 502 does confirm the presence of the wake word, the process moves to box 524 and begins processing other portions of the user speech received from the streaming audio connection or transmitted along with the wake word portion.

[0093] Box 524 may include supplementary boxes 526-534. These boxes represent portions of a process relating to speech processing by user equipment 502 or a remote server communicating with user equipment 502. Boxes 526-534 are not necessarily arranged in a specific implied order, and processing audio may require the execution of one, some, or all of these boxes. At box 526, user equipment 502 may connect to a remote service located at a remote server. Speech processing may include NLP or other speech analysis services provided as part of the remote service. At box 528, the user equipment, or in some embodiments, one or more remote services within the remote service, may determine the identity of the user uttering user utterance 503. This identification may be based on user information or user profile data stored at user equipment 502 or at a remote device with access rights to the remote service. Similarly, at box 530, the user equipment or a remote service may determine the language of the user utterance. This determination may be based on the user information associated with box 528 or according to NLP analysis or similar analysis. At box 532, the user's request is determined as a result of NLP or other speech processing analysis. This step includes parsing the request to determine the appropriate response and which device(s) should execute that response. In some implementations, at box 534, the process may determine whether the identified user is authorized to make the request contained in the user's utterance 503. For example, the user may not be authorized to access a streaming music service that user device 502 can access and transmit to an accessory device. In this case, the response to the request could be to play a message to the user indicating that they lack the appropriate authorization. Other authorizations may cover device-level authorization. For example, the process at box 534 may determine that an accessory device is not authorized to interact with an associated accessory interaction instance, request a specific response, or be delegated to execute a specific response. Other authorizations may cover more advanced functions, such as the user not being authorized to make a specific type of voice request to one or more devices in the residential environment or being fully capable of making any voice request.

[0094] Once the audio has been processed and the response has been determined, process 500 moves to decision 536. Depending on the nature of the request, the response may partially or fully require the audio response to be played to the user at accessory device 501. If the request requires an audio response, the process moves to box 538, and the user equipment may synthesize an audio message for the user. This synthesis may be performed by a text-to-speech module on user equipment 502. The synthesis may also include selecting from a plurality of previously prepared responses corresponding to a typically issued request or situation that user equipment 502 can respond to. At box 540, the prepared response is transmitted to accessory device 501, which plays the response at endpoint 542.

[0095] At endpoint 544, the user equipment executes the request based on a response determined from the audio processing. Execution of the response may include delegating one or more elements of the response to the user equipment or other processes on another device, including other user equipment or accessory devices or remote devices in a residential environment. Some examples include delegation to a music streaming process or voice communication service, which may then connect user equipment 502 to the accessory device and transmit streaming music or voice communication to accessory device 501.

[0096] Figure 6 This is a simplified block diagram 600 illustrating an exemplary architecture of a system for detecting user requests and taking actions based on user requests, according to some implementation schemes. The diagram includes a representative user device 602, one or more accessory devices 604, a representative accessory device 606, one or more networks 608, and a server device. Figure 6 Each of these elements depicted may be similar to one or more elements depicted in other figures described herein. In some embodiments, at least some of the elements of Figure 600 may be in a residential environment (e.g., Figure 2 Operating within the context of the residential environment (200).

[0097] Accessory device 604 and representative accessory device 606 can be any suitable computing device (e.g., smart speaker, smartwatch, smart thermostat, camera, etc.). In some embodiments, the accessory device can perform any one or more of the operations described herein. Depending on the type of accessory device and / or its location (e.g., within or outside a residential environment), the accessory device may be enabled to communicate via network 608 (e.g., including LAN or WAN) using one or more network protocols (e.g., Bluetooth connectivity, Thread connectivity, ZigBee connectivity, WiFi connectivity, etc.) and network paths, as further described herein.

[0098] In some embodiments, server device 610 may be a computer system including at least one memory, one or more processing units (or processors), storage units, communication devices, and I / O devices. In some embodiments, server device 610 may perform any one or more of the operations of the server device described herein. In some embodiments, these elements may be implemented in a similar manner (or differently) to those described with similar elements of reference user equipment 602.

[0099] In some implementations, representative user equipment 602 may correspond to any one or more user equipments described herein. For example, user equipment 602 may correspond to... Figure 2One or more user devices in the residential environment 200. Representative user devices can be any suitable computing device (e.g., mobile phone, tablet, smart hub speaker device, smart media player communicatively connected to TV, etc.).

[0100] In some implementations, one or more networks 608 may include an Internet WAN and a LAN. As described herein, a residential environment may be associated with a LAN, wherein devices within the residential environment can communicate with each other via the LAN. As described herein, a WAN may be external to the residential environment. For example, a router associated with a LAN (and therefore with the residential environment) may enable traffic from the LAN to be routed to the WAN, and vice versa. In some implementations, server device 610 may be external to the residential environment and therefore communicate with other devices via the WAN.

[0101] As described herein, user equipment 602 may represent one or more user equipments connected to one or more networks in network 608. User equipment 602 has at least one memory 612, a communication interface 614, one or more processing units (or processors) 616, a storage unit 618, and one or more input / output (I / O) devices 620.

[0102] Turning further to each element of user equipment 602 in greater detail, processor 616 may be implemented, as appropriate, in hardware, computer-executable instructions, firmware, or a combination thereof. The specific implementation of the computer-executable instructions or firmware of processor 616 may include computer-executable instructions or machine-executable instructions written in any suitable programming language to perform the various functions described above.

[0103] Memory 612 may store program instructions that can be loaded and executed on processor 616, as well as data generated during the execution of these programs. Depending on the configuration and type of user equipment 602, memory 612 may be volatile (such as random access memory (“RAM”)) or non-volatile (such as read-only memory (“ROM”), flash memory, etc.). In some embodiments, memory 612 may include various types of memory, such as static random access memory (“SRAM”), dynamic random access memory (“DRAM”), or ROM. User equipment 602 may also include additional storage devices 618, such as removable or non-removable storage devices, including but not limited to magnetic storage devices, optical disc and / or magnetic tape storage devices. Disk drives and their associated computer-readable media may provide non-volatile storage devices for computer-readable instructions, data structures, program modules, and other data to the computing device. In some embodiments, storage device 618 may be used to store data content received from one or more other devices (e.g., server device 610, other user equipment, accessory device 604, or representative accessory device 606). For example, storage device 618 can store accessory management settings, accessory settings, and user data associated with users belonging to the residential environment.

[0104] User equipment 602 may also include a communication interface 614 that allows user equipment 602 to communicate via network 608 with a storage database, another computing device or server, a user terminal, or other devices. User equipment 602 may also include I / O devices 620, such as those for connecting to a keyboard, mouse, pen, voice input device, touch input device, display, speaker, printer, etc. In some embodiments, I / O device 620 may be used to output an audio response or as part of executing other instructions in response to a user request.

[0105] The memory 612 may include an operating system 622 and one or more applications or services for implementing the features disclosed herein, including a communication module 624, a user interface module 626, a voice processing module 630, an accessory interaction instance 632, and a management module 634. The voice processing module also includes a wake-up word module 636, and the accessory interaction instance 632 also includes a digital assistant 638.

[0106] Communication module 624 may include code that enables processor 616 to generate instructions and messages, transmit messages, or otherwise communicate with other entities. For example, communication module 624 may be combined with management module 634 to transmit and receive data associated with accessory settings, accessory management settings, and accessory ratings from accessory devices 604, 606, other user devices, or server device 610. As described herein, communication module 624 may transmit messages via one or more network paths of network 608 (e.g., via a LAN or Internet WAN associated with a residential environment).

[0107] The user interface module 626 may include code that causes the processor 616 to present information corresponding to accessory devices and user devices present in the residential environment. For example, the user interface module 626 may present a graphical representation of the user device and accessory device currently associated with each accessory device. In some embodiments, the user interface module 626 may allow the user to provide configuration information about a new accessory device to be added to the residential environment, or allow the user to select a user device or accessory device to remove from the residential environment.

[0108] The voice processing module 630 may include code that enables the processor 616 to receive and process audio input corresponding to voice or other sounds that can be analyzed by the techniques described herein. In some embodiments, one or more operations of the voice processing module 630 may be similar to those described in the references. Figure 5 The operations described in block 524. The wake word module 636 may include code that causes the processor 616 to receive and process audio input corresponding to a trigger or wake word. In some embodiments, one or more operations of the wake word module 636 may be similar to those described in reference [reference]. Figure 5 box 516 or Figure 4 The wake word detection module 424 performs the operations described above. For example, the wake word module 636 may analyze a portion of the audio input to determine the presence of a wake word. In some embodiments, the speech processing module may also determine the language corresponding to the audio input and use that language to inform the analysis of the wake word portion.

[0109] Attachment interaction instance 632 may include code that causes processor 616 to receive and process audio input corresponding to a user request. In some embodiments, one or more operations of attachment interaction instance 632 may be similar to those described in the referenced document. Figure 4The operations described in accessory interaction example 430 are as follows. For example, accessory interaction example 632 may include multiple processes or services that enable processor 616 to send and receive data to and receive data from a remote service, delegate the execution of a response to another process or service, or synthesize an audio response based on speech analysis of that portion of the audio input. Accessory interaction example 632 may include a digital assistant 638 that can perform one or more of these exemplary operations, as well as additional operations related to the interaction between accessory devices 604, 606 and user device 602 as described herein. Accessory interaction example 632 may also include an election scoring module 644. In some embodiments, multiple accessory devices 604, 606 may receive the same audio input. In those cases, election scoring module 644 may include code that enables a processor to calculate a score for a wake word received and processed at the user device (e.g., by wake word module 636) and then transmit that score to the associated accessory device.

[0110] The management module 634 may include code that causes the processor 616 to send information to and receive information from one or more accessory devices 604, 606, or other user equipment. In some embodiments, one or more operations of the management module may be similar to those described in the references. Figure 3 The attachment management module 320 and Figure 4 The management module 414 performs the operations described therein. For example, the management module may be combined with the communication module 624 to transmit and receive information corresponding to accessory devices 604, 606 associated with user equipment 602. The management module 634 may include accessory management settings 640 and a new accessory setting module 642. The accessory management settings 640 may include information corresponding to one or more accessory devices 604, 606 and their association with one or more user equipments within the residential environment. The accessory management settings 640 may also include information corresponding to the characteristics and capabilities of accessory devices 604, 606, user equipment 602, or other user equipment. In some embodiments, user equipment 602 may be a configuration device capable of sending and receiving accessory information to add a new accessory device to the residential environment. The new accessory setting module 642 may perform one or more procedures included in configuring the association between the new accessory device and the selected user equipment.

[0111] Turning now to the details of representative accessory device 606, in some embodiments, accessory device 606 may have at least one memory 650, communication interface 652, processor 654, storage unit 656, and I / O device 658. As described herein with respect to user equipment 602, these elements of the accessory device may have the same suitable hardware implementation as their counterparts on user equipment 602.

[0112] The memory 650 of the accessory device 606 may include an operating system 664 and one or more applications or services for implementing the features disclosed herein, including a communication module 660, an audio module 662, and an ADK 670. As described herein with respect to user equipment 602, the communication module 660 may have suitable functionality similar to its corresponding communication module 624.

[0113] The audio module 662 may include code that enables the processor 654 to engage with the I / O device 658 to receive, process, and transmit audio signals. In some embodiments, one or more operations of the audio module may be similar to those described in the reference. Figure 4 The operations described in the attached audio module 412. For example, audio module 662 may receive user speech or other audio input at a microphone using I / O device 658, and transmit the audio data to user equipment 602 via a streaming audio channel or other suitable connection. Audio module 662 may also receive response audio from user equipment 602 and play the audio at a speaker within I / O device 658.

[0114] ADK 670 may include code that causes processor 654 to receive and process a portion of audio input corresponding to a trigger or wake-up word. In some embodiments, one or more operations of ADK 670 may be similar to those described in the reference document. Figure 4 The operations described in ADK 408. ADK 670 may include a voice detection module 672 and a wake word module 674. The wake word module 674 may include code that causes the processor 654 to receive and process a wake word. In some embodiments, one or more operations of the wake word module 674 may be similar to those described in the reference. Figure 5 box 506 or Figure 4 The same operations described in the wake word detection module 410. For example, the wake word module 674 can analyze a portion of the audio input to determine the presence of a wake word.

[0115] In some implementations, ADK 670 may also include an accessory election module 676. The accessory election module 676 may include code that causes processor 654 to send and receive scores to and from accessory devices 606 and user equipment 602, and that causes processor 654 to compare scores received from other accessories to determine a winning score. For example, when the wake-up word module receives a wake-up word for processing, the accessory election module 676 may communicate with other accessory devices 604 to conduct an election to determine which accessory device should respond to the wake-up word. References below... Figure 8 Describe the process in detail.

[0116] Figure 7This is another simplified block diagram 700 illustrating an exemplary architecture of an accessory device 702 that receives and processes multiple communications from a user equipment according to some embodiments. The accessory device 702 and each of the depicted elements therein may be similar to other accessory devices and similar elements depicted in the other figures described herein, including... Figure 6 Accessory device 606. In some embodiments, accessory device 702 may perform any one or more of the operations of the accessory device described herein. Accessory device 702 may be any suitable computing device (e.g., smart speaker, smartwatch, etc.) and may include memory 710, processor 724, storage unit 726, I / O device 728, and communication interface 730. Each of these elements may have a suitable embodiment in hardware, firmware, computer-executable instructions, or combinations thereof, and have similar characteristics to those described above relative to... Figure 6 The functions of the computing device are described in detail for user equipment 602 and accessory equipment 606.

[0117] Turning more specifically to the elements of memory 710, memory 710 may include a communication module 712, an audio module 714, an operating system 716, and an ADK 720. Each of these elements may be similar to the referenced element. Figure 6 The accessories 702 include the elements described in accessory device 606. In some embodiments, ADK 720 may include audio mixing logic 722. The audio mixing logic may include code that combines communication module 712 and audio module 714 to enable processor 724 to receive, process, combine, mix, and output one or more audio sources. Audio sources may include one or more streaming audio inputs 754 or request responses 750, 752. Furthermore, audio mixing logic 722 may also receive mixing rules 756. Mixing rules 756 may instruct accessory device 702 to perform one or more audio mixing processes on received audio inputs via its audio mixing logic 722 or other elements of ADK 720. Mixing rules 756 may be transmitted from associated user equipment to accessory device 702 and may be part of request responses 750, 752, streaming audio inputs 754, or separate communications. Mixing rule 756 can provide instructions for the desired volume of the incoming audio response, the desired volume of any other audio response to be output simultaneously with the incoming audio response, whether the current output audio stream should be attenuated or muted during the output of the incoming audio response or other audio responses, and so on.

[0118] As an example of the aforementioned implementation, consider a scenario where accessory device 702 is currently playing a music stream on its speaker in response to a previous user request. The user then issues another request to the accessory device (e.g., “Computer, what time is it?”). This request can be processed at the associated user device, and accessory device 702 can receive an audio response (e.g., “The time is 10:30 PM”). This audio response may be accompanied by mixing rules generated by the user device, instructing the audio response to be played at a specific volume at the speaker that is greater than the high-frequency portion of the music stream, and that the music stream should be attenuated to a lower volume. The accessory device can apply the mixing rules and play the audio response at a volume greater than the music stream. As another example, consider adding a second request corresponding to an alarm clock (e.g., “Computer, set the alarm for 10:30 PM”). In this case, the response to the time request and the response to the alarm clock may arrive at accessory device 702 simultaneously or nearly simultaneously. The user device can generate mixing rules to instruct that both the music stream and the alarm clock indication should be muted until the time response is announced, subsequently restoring the music stream volume to a reduced level and playing the alarm clock indication at a volume greater than the music stream. Many other rules, parameters, or combinations thereof are envisioned. In some embodiments, the audio mixing logic 722 may store mixing rules 756, such that these rules persist and are applied to future request responses and other received audio. Subsequent mixing rules may modify or update the stored set of rules. In other embodiments, the mixing rules 756 are transient and apply only to one or more request responses or audio inputs currently received or played at the accessory device 702.

[0119] Figure 8 This is a flowchart illustrating a process 800 in which an accessory device 801, according to some embodiments, determines which of a plurality of accessory devices will respond to a user request. The accessory device 801 may correspond to any one or more accessory devices described herein. In some embodiments, some or all of these processes may be performed by one or more user devices or another device (e.g., a server device), which may correspond to either the user device or the server device / server computer described herein, respectively.

[0120] At box 802, accessory device 801 may receive a wake word. In some embodiments, the wake word may correspond to a portion of the audio input received at accessory device 801, including user speech. Similar to Figure 4 The wake word detection module 410, box 802 can also be included in the first level of wake word processing.

[0121] At block 804, accessory device 801 can establish an audio channel with user equipment. Accessory device 801 can transmit a wake-up word to user equipment via the audio channel. At block 806, accessory device can receive an accessory response score from user equipment. The accessory response score can be based on analysis of the wake-up word sent to user equipment. In some embodiments, the score is based on criteria including, but not limited to, the intensity (e.g., loudness) of the audio input received at accessory device, the quality of the received audio input, the capabilities of accessory device, and the capabilities of user equipment associated with accessory device.

[0122] At block 808, accessory device 801 may open an election communication channel with one or more accessory devices, which may have already received audio input from the same user speech or other audio source. The election communication channel may also be configured to communicate with one or more user devices, such that participants in the election include both accessory devices and user devices. In some embodiments, the communication channel may reside on one or more networks with which the accessory device can communicate. In other embodiments, for election purposes, a self-organizing network or other small area network or LAN may be established. For example, accessory devices may use Bluetooth to establish anonymous election connections to send and receive election scores.

[0123] At box 810, the attachment device 801 may transmit its attachment response score to other attachments connected to the election communication channel. At box 812, the attachment device receives competition scores from one or more other attachments. In some scenarios, no other attachment devices or user equipment transmit scores to attachment device 801, in which case the attachment device may continue as if it had won the election. Once scores have been received from all participating devices, the process may move to box 814, where attachment device 801 compares its score with all other received scores. Meanwhile, in some embodiments, other participating devices are performing the same or similar comparisons between their scores and those transmitted by attachment device 801. This comparison may include a simple comparison between numerical scores. In some embodiments, a method can be used to ensure that there is a single winner (i.e., no tie) in the score generation. As shown here... Figure 8 As described in the text, a "better" score is indicated as greater than a less desirable score, resulting in the election winner having a higher score. Other scoring systems are possible that alter the comparison hierarchy without changing the outcome of the election process as described herein.

[0124] At decision 816, if the accessory device receives a score from another accessory device that exceeds its own score, the process may proceed to endpoint 818 and the accessory device may ignore the wake word. Ignoring the wake word may include terminating the audio channel with the associated user equipment. If the response score of accessory device 801 is greater than all other received scores from other participating devices, accessory device 801 has won the election. The process moves to endpoint 820, where accessory device 801 reports its victory to its associated user equipment. This election determines which of the multiple devices receiving audio input is the preferred device for responding to that input. The winning device may continue with other operations related to receiving or performing a response to the audio input.

[0125] Figure 9 This is another simplified flowchart illustrating an exemplary process 900 in which a user device coordinates interactions with multiple accessory devices. In some embodiments, one or more operations of process 900 may be similar to those described in the reference. Figure 1 and Figure 4 The operations described above.

[0126] At block 902, the user equipment may receive information identifying one or more accessories capable of communicating with the user equipment. In some embodiments, one or more operations in the operation of block 902 may be similar to those described in the reference. Figure 3 One or more operations described for process indicator 330.

[0127] At box 904, the user equipment can implement an accessory interaction instance corresponding to each of the accessories identified in box 902. In this way, the user equipment can have an accessory interaction instance associated with each identified accessory, such that operations performed by the user equipment to interact with accessory devices can be managed by a single accessory interaction instance without affecting the user equipment's interactions with other accessory devices. In some implementations, one or more operations in box 904 may be similar to... Figure 1 One or more operations of box 102.

[0128] At box 906, the user equipment may receive a first audio input from a first attachment and a second audio input from a second attachment, wherein the first and second attachments are attachments among those previously identified in box 902 and associated with the attachment interaction instance in box 904. The first and second audio inputs may correspond to user speech or other audio sources received at the first and second attachments. In some embodiments, the first and second audio inputs may have the same audio source.

[0129] At block 908, a first accessory interaction instance of the user equipment may process at least a portion of the first audio input received in block 906. The first accessory interaction instance may be an accessory interaction instance corresponding to a first accessory. This portion of the first audio input may correspond to a user request, such that processing of the accessory interaction instance may parse or otherwise analyze the request and determine a response. Processing this portion of the first audio input may include transmitting the portion to a server computer for analysis. The first audio input does not need to contain a request, and processing may determine an appropriate response based on the analyzed portion. In some embodiments, one or more operations in block 908 may be similar to... Figure 5 One or more operations of box 524.

[0130] At box 910, the first accessory interaction instance may receive a first response from the server computer corresponding to the processed portion of the first audio input. In some embodiments, the server computer performs analysis of that portion of the audio input, causing the server computer to parse any request or determine the first response corresponding to the processed portion. This analysis may include techniques such as NLP or other speech processing. In some embodiments, one or more operations in the operation of box 910 may be similar to... Figure 5 One or more operations of boxes 524 and 532.

[0131] At block 912, the user equipment may transmit a first response to the first attachment. In some embodiments, one or more operations in the operation of block 904 may be similar to... Figure 5 One or more operations of box 540.

[0132] The foregoing describes exemplary techniques for coordinating interactions between a user equipment and multiple accessory devices. Some or all of these techniques can be at least partially implemented through architectures such as those described above. Figures 1 to 9 The architectures shown herein can be used to implement, but are not required to implement, the techniques disclosed herein. While numerous implementations have been described above with reference to server devices, accessory devices, and user devices, it should be understood that other types of computing devices may be suitable for implementing the techniques disclosed herein. Furthermore, various non-limiting examples have been described in the foregoing description. For purposes of explanation, many specific configurations and details have been set forth to provide a thorough understanding of the examples. However, it will also be apparent to those skilled in the art that some examples can be implemented without these specific details. In addition, well-known features have sometimes been omitted or simplified to prevent confusion with the examples described herein.

[0133] A technology for load balancing hub devices and multiple endpoints.

[0134] Additional embodiments of this disclosure provide techniques for managing the association of accessory devices and hub devices. As a first example, consider a residential environment corresponding to a residence. A person in the residence may want to know the current time. This person can use a verbal request (e.g., “What time is it?”) to query an accessory device (e.g., a nearby smart speaker) within the residential environment. The accessory device can determine that the request is directed to that device and then transmit the received audio information to a hub device (e.g., a hub speaker). The hub device can process the audio information to determine the nature of the request and prepare a corresponding response (e.g., “It is 10:30 PM”). Alternatively, or in part in combination with the above, a user device can transmit some or all of the verbal request to a server computer (e.g., a service provider), where the service provider can determine the nature of the request and / or prepare a corresponding response. The hub device can then transmit the response back to the accessory device for playback to the user. To provide coordination between the accessory device and the hub device, the user device can be configured to assign the accessory device to that particular hub device. The user equipment can monitor the hub device and other hub devices in the residential environment, and reassign the accessory device to another hub device if that hub device would be better suited to handle user requests or other interactions with the accessory. Additionally, the user equipment can be configured to assign newly added accessory devices to one of the hub devices, at least in part, based on various factors such as accessory capabilities, hub capabilities, etc.

[0135] As an example of the above examples, a residential environment can include numerous “smart” devices, such as electronic devices with features that allow them to interact and operate autonomously to some extent. Smart devices (which can have various functions) can be cameras, speakers, thermostats, headphones and headsets, telephones, or media players. Smart devices can also have various network communication capabilities, including WiFi, Ethernet, Bluetooth, Zigbee, cellular, etc. These devices can be manufactured by different companies. In some cases, smart devices can be categorized as hub devices and accessory devices. Hub devices can be residential devices (e.g., smart speakers, smart digital media players configured to control a television (TV), mobile phones, etc.). While not always the case, in some examples, it is expected that residential devices reside within the residence and are not frequently moved (e.g., moved within or outside the residence). Hub devices can have capabilities equal to or greater than those of accessory devices. For example, a hub device can be a mobile phone that can include wireless (e.g., WiFi) and cellular communication capabilities, multimedia capabilities, and device assistants. In the same example, the accessory device may be a smart speaker that may include audio media and wireless communication capabilities but lacks a device assistant. The device assistant may be a virtual assistant program configured to interact with a user. In these examples, depending on its capabilities, the smart speaker may be a hub device or an accessory device. In some examples, if the accessory is manufactured by a different entity than the entity that manufactures the user equipment, the accessory may not initially be configured to have the ability to communicate with the user equipment. In some cases, the user equipment manufacturer may provide an accessory development kit (“ADK”) for installation on the accessory, which enables such communication after the accessory has been manufactured, sold, supplied, or used. The user equipment may be a hub device as described herein and may include user interface features. In some embodiments, the user equipment is a leader device selected from hub devices in a residential environment. As used herein, the terms hub device, user equipment, and leader device may refer to one or more similar devices distinct from accessory devices.

[0136] In some implementations, the user equipment (UE) may obtain information about accessory devices and hub devices present in the residential environment. This information may be obtained by the UE that communicates directly with the accessory devices and hub devices sharing the same network within the residential environment. In other implementations, the information about accessory devices and hub devices may be sent to the UE by a second UE, a leader device, or a remote server device. For example, a user in a residence may add a new accessory device to the residential environment. As part of this process, the user may interact with a second UE (e.g., a mobile phone) to configure the new accessory device and send the new accessory device information to a first UE. As another example, a leader device in the residential environment may have information about multiple accessory devices and hub devices in the residential environment and report information about some or all of these devices to the UE. The UE can then use this information to assign the accessory device to the appropriate hub device. The accessory information and hub device information may be stored by the UE.

[0137] Information received by the user equipment from the hub device and accessory device can correspond to the characteristics, attributes, or capabilities of the hub device and accessory device. For example, the hub device could be a hub speaker with a microphone input, speaker output, and a device assistant supporting multiple spoken languages. The attributes of the hub speaker would then identify that the hub speaker can receive audio input (via microphone), produce audio output (via speaker), and handle user requests (via device assistant). Furthermore, as an attribute, the hub speaker could support multiple language processing, enabling it to handle user requests in various spoken languages. Continuing this example, the accessory device could be a smart thermostat with a touchscreen, microphone input, no speaker output, and connection to a stove in a residential environment. The attributes of the smart thermostat would then identify that the thermostat can interact with the user at the touchscreen, present visual information or other instructions to the user at the touchscreen, receive audio input, not produce audio output, and operate the stove to control the ambient conditions in the residence. However, those skilled in the art will understand that this is merely an example, and most accessories, if not all, will have a speaker for output. In some implementations, the attribute information may also include information corresponding to the device's computing capabilities, including the device's current processing load. Furthermore, each hub device may have a limited number of connection slots corresponding to the maximum number of associated accessories that the hub device can support. Each accessory associated with the hub occupies one slot, leaving unoccupied slots available for association with other accessories. The attribute information may also include the device's network communication capabilities. Any particular capability or function of the device may be an identifiable attribute, and many combinations of capabilities and device functions are envisioned and described in several examples herein.

[0138] Once the user device has received attribute information from the hub device and accessory devices, it can score the hub devices to determine which is the "best" hub device associated with the accessory device. This score can be based on whether the accessory device requires specific attributes of the hub device. For example, a camera accessory might require any hub device it will associate with to have a hardware video decoder, causing any hub device without the required decoder to receive a low score or 0 from the user device. As another example, an accessory smart speaker might require specific language support from any hub device associated with it. One hub device might partially implement that language support, while a second hub device might implement it under high-quality conditions. The first hub might receive a low score, while the second hub might receive a higher score, becoming the best hub associated with the smart speaker.

[0139] In some implementations, the hub device may receive a base score corresponding to its general computing power. For example, a tablet or smartphone hub device may have a higher base score than a hub speaker due to its greater computing power. However, in other examples, a smartphone hub device may have a lower base score due to its transient nature (e.g., smartphones may move around and / or leave a residence more frequently than other devices). Transient devices may not be optimal hubs because when they move, they may discard associated accessory devices and force those accessory devices to re-associate with other hubs (which can introduce latency within the system). The base score can be modified by comparing accessory and hub attributes. The final connectivity score can then be calculated by multiplying the modified score by the number of available connection slots at each hub. The hub device with the highest connectivity score is considered the optimal hub device for accessories, and the user device can transmit information to this optimal hub device to associate the accessory with it.

[0140] Figure 10This is a simplified block diagram 1001 of an exemplary implementation. Process 1000 is an exemplary high-level process flow of a system including accessory device 1010, which requests allocation to a hub device via user device 1012. Figure 1001 illustrates the state of the system corresponding to the block of process 1000. Process 1000 may be performed in a residential environment containing multiple accessory devices, user devices, and hub devices. As depicted herein, accessory device 1010 may be a smart speaker, and user device 1012 may be a hub speaker. Although described as specific devices, it will be apparent that accessory device 1010, user device 1012, or hub devices 1014, 1016 may be several types of smart devices in various combinations and numbers. For example, a smartphone, media device (e.g., smart TV), or tablet computer (connected to a cellular network, connected to a local area network via WiFi from a residential network, or connected to a wide area network (“WAN”)) may perform one or more operations of process 1000.

[0141] Referring more specifically to process 1000, at block 1002, user equipment 1012 may receive an allocation request from accessory device 1010. This allocation request may occur when accessory device 1010 recognizes that it is not currently associated with a hub device. For example, accessory device 1010 may be a new accessory device introduced into a residential environment. As another example, a hub device previously associated with accessory device 1010 may have lost network connectivity with devices in the residential environment, making it necessary for accessory device 1010 to associate with a new hub device to properly relay user requests or other functions. In other cases, accessory device 1010 may be discarded by its currently associated hub device. This may occur when the currently associated hub device reduces the total number of its connection slots (e.g., instructed to do so by user equipment or another device due to increased computing load).

[0142] At box 1004, user equipment 1012 can receive information from accessory device 1010, first hub device 1014, and second hub device 1016. This information may include attribute 1020 from accessory device 1010 and attribute information 1024 and 1026 from hub devices 1014 and 1016. (The following text is related to...) Figure 12In more detail, accessory attribute 1020 may include any characteristics of accessory device 1010 related to its association with the hub device. Some accessory attributes may be categorized as required attributes. Similarly, hub information 1024, 1026 may include any attributes or characteristics of hub devices 1014, 1016 related to the selection of the optimal hub device. In some embodiments, user equipment 1012 may receive hub attribute information 1024, 1026 upon request. User equipment 1012 may store the received hub and accessory attribute information. The stored attribute information may be stored locally at the user equipment or at a remote device such as a server computer. In other embodiments, hub devices 1014, 1016 may periodically update the stored attribute information 1024, 1026 without a request from user equipment 1012. For example, hub devices 1014 and 1016 may update their attribute information stored on a server device whenever these attributes change (e.g., if the number of accessory slots decreases) or at regular intervals (e.g., every minute). User device 1012 can then access the stored data to receive hub attribute information 1024 and 1026 without directly querying each hub in the residential environment.

[0143] Moving down to box 1006, user equipment 1012 can compare accessory attribute 1020 with first hub information 1024 and second hub information 1026. This comparison can produce a score or other metric to determine which of the first hub device 1014 and the second hub device 1016 should be assigned to accessory device 1010. In some embodiments, each of hub devices 1014, 1016 can be assigned a base score corresponding to the general computing power of each device. User equipment 1012 can then modify the base score by combining it with the comparison result. The connectivity score of each hub device is calculated by multiplying the modified score by the number of available slots at hub devices 1014 and 1016. User equipment 1012 can then assign accessory device 1010 to the hub device with the highest connectivity score, as shown in box 1008.

[0144] As an example of the aforementioned implementation scheme, consider Figure 10The scenario depicted in the diagram shows that accessory device 1010 is a smart speaker, hub device 1014 is a hub speaker, and hub device 1016 is a tablet computer. Because the tablet computer can have greater general computing power than the hub speaker, hub device 1016 can be assigned a higher base score than hub device 1014. The smart speaker may only require basic language support for one language in the residential environment, and both the hub speaker and the tablet computer can provide this support. Therefore, when the required attributes of accessory device 1010 are compared with hub information 1024, 1026, each hub receives the same non-zero score. Regarding other accessory features, the tablet computer can provide both a microphone and a speaker, as well as a touchscreen, while the hub speaker can only provide a microphone and a speaker. The smart speaker accessory attributes can indicate that it is beneficial for the hub device to visually present information (e.g., via speech-to-text). All other attributes being equal, the tablet computer can receive a higher comparison score than the hub speaker. User equipment 1012 can then calculate a modified score for the second hub device 1016, which is significantly higher than the modified score for the first hub device 1014. However, the tablet may have only one accessory connection slot, which is already occupied by another accessory, while the hub speaker may have two open slots. User equipment 1012 can then calculate a connection score for each hub device by multiplying the modified score of the hub speaker by 2 and the modified score of the tablet by 0. The result would be that the hub speaker is the winning device, and user equipment 1012 can assign accessory device 1010 to hub device 1014, as depicted. The above description should be considered only as one example of the result of the disclosed process. Those skilled in the art will recognize that residential environments with many devices can produce a variety of different accessory / hub allocation results depending on the dynamic nature of hub attributes, the available connection slots, and the migration of hub devices and accessory devices into or out of the residential network.

[0145] Figure 11 This is a schematic diagram of a residential environment 1100 including hub devices and accessory devices according to some embodiments. The hub devices may include a hub speaker 1102, a media player 1104, and a smartphone 1106. These hub devices may correspond to those described above relative to... Figure 10 The hub devices 1014 and 1016 of the described embodiment. Accessory devices may include smart speakers 1112 and 1114, a smartwatch 1116, and a thermostat 1124. Similarly, these accessory devices may correspond to... Figure 10The accessory devices 1010 are described above. Some or all of these accessory devices may be third-party devices (e.g., not manufactured, programmed, or supplied by the manufacturer, programmer, or supplier of the hub device or user equipment). Therefore, they may not be automatically and / or initially compatible with the user equipment. Each hub device in the residential environment 1100 may be associated with zero, one, or more accessory devices. As shown by the long dashed line, hub speaker 1102 is associated with smart speakers 1112, 1114, and smartwatch 1116, while media player 1104 is associated with thermostat 1124. Smartphone 1106 is not associated with any accessory device. Devices within the residential environment 1100 may be configured to communicate via one or more networks associated with the residential environment 1100 using one or more network protocols. For example, the residential environment 1100 may be associated with a local area network (“LAN”), WAN, cellular network, personal area network, or other networks, and these devices may communicate using WiFi connectivity, Bluetooth connectivity, Thread connectivity, Zigbee connectivity, or other communication methods.

[0146] The arrangement of the association between accessory devices and hub devices can include various combinations and can be modified by the user equipment. For example, the user equipment may receive an assignment request from an accessory device within the residential environment. The accessory device may be one of many accessory devices previously associated with the hub device in the residence, or it may be a new accessory device added to the residence. The user equipment then obtains attribute information from the accessory device and the hub speaker 1102, media player 1104, and smartphone 1106, and then scores these hub devices. Because the smartphone 1106 is not currently associated with an accessory device, it may have the highest score and receive the assignment of the new accessory. In some embodiments, the user equipment in the residential environment 1100 may communicate with the hub device to transfer one or more accessory devices associated with a first hub device to a second hub device. This transfer may occur automatically based on information received by the user equipment regarding the residential environment 1100, including but not limited to information that another hub device may be more suitable to be associated with one or more accessories, or that an accessory has been added to or removed from the residential environment 1100. The suitability of any particular hub device associated with an accessory may be based at least in part on the hub device's capabilities, the accessory device's capabilities, the current processing load experienced by the hub device, the location of the devices within the residential environment, and the state of communication between devices on the network. Many other criteria are envisioned for rearranging device associations within a residential environment.

[0147] In some implementations, accessory devices and non-resident hub devices may also leave the residential environment or lose network connectivity with the residential environment. Accessory devices leaving the residential environment can be unassociated from previously associated hub devices, causing the hub device to update its accessory slots to indicate the new available slots. Accessory devices associated with hub devices that have lost network connectivity with the residential environment can be reassigned by user equipment that maintains network connectivity. In this case, the user equipment may receive information that the hub device is no longer able to communicate with the accessory device and reassign the accessory device. Some implementations may have a user equipment designated as a leader device to manage the allocation of accessory devices among hub devices within the residential environment. In other implementations, if a user equipment and an accessory device are associated and leave the residential environment and lose network connectivity, the user equipment may maintain its association with the accessory device and perform the embodied methods described herein.

[0148] In another implementation, the user equipment acting as the leader device can monitor hub devices within the residential environment to determine whether each hub device can respond to its attachments within an acceptable time frame. In some cases, a hub device may be affected by increased processing load, causing it to be unable to respond to one or more of its assigned attachments within a threshold time amount. The user equipment can receive information about the hub device indicating a delay in its response to attachment requests and instruct the hub device to discard one or more of its attachment devices. The user equipment can then determine which other hubs within the residential environment should reassign the discarded attachment devices.

[0149] Return to Figure 11As an example of some implementations described above, hub speaker 1102 may discard its association with smartwatch 1116. This may occur due to increased processing load at hub speaker 1102, for example, if the hub speaker is currently streaming separate music channels to smart speakers 1112, 1114. Because smartwatch 1116 does not have an associated hub, it may request assignment from a user device, which may be smartphone 1106. Smartphone 1106 may then receive attribute information from the hub device in the residential environment 1100. This attribute information may include attribute information about smartphone 1106 itself, as in some implementations, the user device may operate similarly to the hub device. Smartwatch 1116 then scores each hub device based on a comparison of the attribute information and accessory attributes. Because hub speaker 1102 recently discarded the accessory smartwatch 1116 due to increased demand, the hub speaker may have a reduced total number of accessory connection slots and no available slots. The smartphone 1106 can then determine the winning hub device between itself and the media player 1104, and appropriately assign the smartwatch 1116.

[0150] continue Figure 11 The residential environment 1100 may have multiple users 1130, 1134 issuing audio requests 1132, 1136 to accessories. Requests 1132, 1136 may occur separately or simultaneously and may be received by multiple accessory devices, as shown by the short dashed lines. For example, request 1132 may be received by a smart speaker 1114 or a smartwatch 1116, while request 1136 may be received by a smart speaker 1112 and a thermostat 1124. As previously described, the arrangement of accessory devices and their associated devices can take various forms and may change over time. Therefore, user requests may be received by multiple accessory devices associated with different hub devices. For example, user request 1136 may be received by both the thermostat 1124 associated with media player 1104 and the smart speaker 1112 associated with hub speaker 1102. Because user interaction with devices in the residential environment 1100 can occur in several arrangements, the ability of a user device or leader device to assign accessories to the optimal hub device prevents missed user requests. If accessory devices are discarded by their hub devices, as described in the previous examples, or if the hub devices lose network connectivity, the accessory devices may be unable to process user requests unless these accessory devices are quickly reassigned to the best available hub device.

[0151] Hub device management by the user equipment or leader equipment may also allow the hub devices to receive and process audio requests 1132, 1136 that require a response at one or more accessory devices other than the accessory device that initially received the request. In these embodiments, the accessory devices may not have information about other accessories within the residential environment and may not have mechanisms for establishing trust or permission with other accessories on one or more networks within the residential network. For example, the smart thermostat 1124 and the smartwatch 1116 may be third-party accessories manufactured by two different entities and therefore may not have the inherent ability or permission to directly connect to each other to transmit or exchange data and information. The user equipment may manage the allocation of accessories to the hub devices such that each hub can establish a connection with each accessory in its array and transmit information received from other hubs to those accessories, without the accessory being aware of any other devices in the residential environment besides the hub to which it was assigned.

[0152] As a specific example of the aforementioned implementation, consider the following scenario: when user 1134 is about to leave the residence, user 1130 wants to send a notification to user 1130 reminding them to buy milk at the grocery store. The smart thermostat 1124 assigned to media player 1104 can hear the user's request 1136 more clearly because user 1134 is in the same room as the thermostat 1124. In processing the user's request, media player 1104 can transmit the response to hub speaker 1102 and smartphone 1106 (e.g., other known hub devices in the residential environment). The notification can be played on every capable device in the residence, including smartphone 1106, hub speaker 1102, smart speakers 1112 and 1114, and smartwatch 1116, to ensure it is heard by user 1130. The notification can also be selectively transmitted by hub speaker 1102 to smartwatch 1116, so that it is presented by the most appropriate device. In this way, a user request can be received at one third-party device (thermostat) and executed at a second third-party device (smartwatch), without either third-party device needing to communicate directly with the other. Consider another example where user 1130 takes smartphone 1106 and leaves the residential environment before user 1134 issues a notification request. Since smartwatch 1116 has left the residential environment, it may lose network connectivity with hub speaker 1102. When this happens, smartwatch 1116 can request allocation to another hub device. Smartphone 1106 or other user devices can allocate smartwatch 1116 to another suitable hub device, which could be smartphone 1106, since the smartphone remains in close proximity to smartwatch 1116. Smartphone 1106 can also maintain network connectivity with the residential environment via its cellular network connection to the Internet WAN. When user 1134 sends a notification that milk has been purchased from the grocery store, the request can again be processed by media player 1104 and transmitted to hub speaker 1102 and smartphone 1106 for playback. Since hub speaker 1102 is no longer assigned to smartwatch 1116, it will not take any further action to fulfill the notification request. Smartphone 1106 can receive the notification and transmit it to its accessory smartwatch 1116 for audio playback to user 1130.

[0153] Figure 12 This is a schematic diagram illustrating a process 1200 in which user equipment 1201 assigns accessory device 1206 to one of the hub devices 1202. In some embodiments, user equipment 1201 may correspond to the user equipment described herein (e.g., Figure 10User equipment 1012). Similarly, annex 1204 and annex device 1206 may correspond to other annex devices, while hub device 1202 may correspond to other similar hub devices described herein. User equipment 1201, depicted herein as a hub speaker, may be a hub device, leader device, configuration device, or other device used to determine device allocation and association within hub device 1202. User equipment 1201 may be configured to communicate with hub device 1202, annex 1204, and annex device 1206 via one or more networks (including LANs or WANs) described herein. In some embodiments, user equipment 1201 is a remote server device configured to communicate with hub device 1202, annex 1204, and annex device 1206 via a WAN.

[0154] The allocation process 1200 is presented in more detail for several elements. Hub devices 1202 may each include hub attribute information, which includes hub attributes 1210 and slots 1216. This information may be stored in memory or storage units at the hub device. In some embodiments, hub information may be stored at a remote device such as a server computer or cloud device. In other embodiments, hub information is stored at user equipment 1201 and is updated periodically. Hub attributes 1210 may include any attributes or characteristics of hub devices 1202 associated with selecting the optimal hub device. This may include language processing 1212, hardware decoder 1214, supported network connections (e.g., WiFi, cellular, or Thread), the ability to act as an edge router for a personal area network (e.g., a Thread border router), and the types of other accessories currently associated with the hub device. Slots 1216 may include a number of available accessory connection slots 1218 and a number of currently allocated slots 1220. The total number of slots 1216 is a dynamic quantity and can be changed or updated by the hub device based on changes in the residential environment. If the hub device no longer has enough slots 1216 available to support accessories, a change in the total number of slots 1216 may result in the removal of assigned accessories from the hub device. In some embodiments, if the number of slots 1216 becomes less than the number of assigned accessories, an accessory assigned to the hub based on matching requirement characteristic 1226 is only de-associated from the hub if no other accessory device can be de-associated first. As depicted herein, an exemplary hub device may have a total of four slots, with two slots available and two slots assigned to “Accessory 1” (thermostat) and “Accessory 2” (camera).

[0155] Accessory device 1206 may be a representative accessory in accessory 1204 and may include information corresponding to accessory status 1222 and accessory characteristics 1224. As used herein, the terms accessory characteristics and accessory attributes are used interchangeably. Accessory status 1222 is a message corresponding to whether accessory 1206 is currently assigned to a hub. If the accessory is assigned, accessory status 1222 identifies the assigned hub. If the accessory is not assigned (e.g., the accessory is a new accessory in a residential environment), accessory status 1222 may indicate that accessory 1206 needs to be assigned, thereby instructing the accessory to request assignment from user equipment 1201. In some embodiments, accessory devices may not be assigned to a hub and may not request assignment from the hub. In these cases, unassigned accessories may transmit their status information to user equipment 1201 but do not request assignment from the hub device. Accessory characteristics 1224 may include any features of accessory device 1206 related to its association with the hub device, including but not limited to language support requirements, hardware encoding / decoding requirements, existing input / output (“I / O”) devices (e.g., speakers and microphones or screens), types of supported network connections (e.g., WiFi and Bluetooth), and external device control (e.g., control of lighting switches or household stoves). Some accessory attributes may be categorized as required attributes.

[0156] Finish Figure 12 The detailed elements, process indicators 1230 and 1240 respectively represent data transmission between accessory user equipment 1201 and accessory device 1206, and between user equipment 1201 and hub device 1202. Process indicators 1230 and 1240 can indicate communication between various devices as described herein via one or more networks, including but not limited to WiFi LAN or Internet WAN. Process indicator 1230 indicates the transmission of data to user equipment 1201 corresponding to the allocation request and accessory attribute 1224. This data may include accessory device 1206 no longer communicating with its assigned hub device. Similarly, process indicator 1240 indicates the transmission of data including hub attributes 1210 and slot 1216 between hub device 1202 and user equipment 1201.

[0157] Process 1200 provides the above relative to Figure 10A more detailed picture of the scoring process described in box 1004. Once user equipment 1201 has received accessory feature 1224, hub attribute 1210, and slot 1216 from hub device 1202, the user equipment can score hub device 1202 to determine the optimal hub. This scoring begins by determining a base score for each hub device. User equipment 1201 can then compare accessory feature 1224 with hub attribute 1210. This comparison begins by comparing the required feature 1226 of accessory device 1206 with hub attribute 1210. Since accessory device 1206 has indicated in some way that these attributes are required, if the comparison does not match the required feature 1226 with the corresponding attribute 1210 of the hub device, the hub device will receive a score of 0. This result of 0 will occur regardless of the scores of the hub device relative to the other attributes of accessory device 1206. In some cases, a hub device may only partially meet the required characteristic 1226 (e.g., by providing the required characteristic at a lower quality). User equipment 1201 may assign a non-zero score to the hub device for partially meeting the requirement. Once the required characteristic 1226 has been compared, user equipment 1201 compares other ancillary characteristics 1228 with each characteristic in hub attributes 1210. The results are then combined with the base score to obtain a modified score. The connectivity score for each hub device can be calculated by multiplying the modified score by the number of available slots at each hub device in hub device 1202.

[0158] Figure 13 This is a simplified block diagram 1300 illustrating an exemplary architecture of a system for assigning accessory devices to hub devices according to some embodiments. The diagram includes user device 1302 (e.g., a leader device), one or more accessory devices 1304, representative accessory devices 1306, one or more networks 1310, and server device 1312. In some examples, user device 1302 may be one of a plurality of hub devices that has been elected as the leader of the hub devices. Figure 13 Each of these elements depicted may be similar to one or more elements depicted in other figures described herein. In some embodiments, at least some of the elements of Figure 1300 may be in a residential environment (e.g., Figure 11 Operating within the context of the residential environment (1100).

[0159] Accessory device 1304 and representative accessory device 1306 can be any suitable computing device (e.g., smart speaker, smartwatch, smart thermostat, camera, etc.). In some embodiments, the accessory device can perform any one or more of the operations described herein. Depending on the type of accessory device and / or its location (e.g., within or outside a residential environment), the accessory device may be enabled to communicate via network 1310 (e.g., including LAN or WAN) using one or more network protocols (e.g., Bluetooth connectivity, Thread connectivity, ZigBee connectivity, WiFi connectivity, etc.) and network paths, as further described herein.

[0160] In some embodiments, server device 1312 may be a computer system including at least one memory, one or more processing units (or processors), storage units, communication devices, and I / O devices. In some embodiments, server device 1312 may perform any one or more of the operations of the server device described herein. In some embodiments, these elements may be implemented in a similar manner (or differently) to those described with similar elements of reference user equipment 1302.

[0161] In some implementations, user equipment 1302 may correspond to any one or more user equipments described herein. For example, user equipment 1302 may correspond to... Figure 11 One or more user devices in the residential environment 1100. A representative user device can be any suitable computing device (e.g., mobile phone, tablet, smart hub speaker device, smart media player communicatively connected to a TV, etc.). Similarly, hub device 1308 can be any device capable of performing the functions of a user device.

[0162] In some embodiments, one or more networks 1310 may include an Internet WAN and a LAN. As described herein, a residential environment may be associated with a LAN, wherein devices within the residential environment can communicate with each other via the LAN. As described herein, a WAN may be external to the residential environment. For example, a router associated with a LAN (and therefore with the residential environment) may enable traffic from the LAN to be routed to the WAN, and vice versa. In some embodiments, server device 1312 may be external to the residential environment and therefore communicate with other devices via the WAN.

[0163] As described herein, user equipment 1302 may represent one or more user equipments connected to one or more networks in network 1310. User equipment 1302 has at least one memory 1314, a communication interface 1316, one or more processing units (or processors) 1318, a storage unit 1320, and one or more input / output (I / O) devices 1322.

[0164] Turning further to each element of user equipment 1302 in greater detail, processor 1318 may be implemented, as appropriate, in hardware, computer-executable instructions, firmware, or a combination thereof. The specific implementation of the computer-executable instructions or firmware of processor 1318 includes computer-executable instructions or machine-executable instructions written in any suitable programming language to perform the various functions described above.

[0165] Memory 1314 may store program instructions that can be loaded and executed on processor 1318, as well as data generated during the execution of these programs. Depending on the configuration and type of user equipment 1302, memory 1314 may be volatile (such as random access memory (“RAM”)) or non-volatile (such as read-only memory (“ROM”), flash memory, etc.). In some implementations, memory 1314 may include various different types of memory, such as static random access memory (“SRAM”), dynamic random access memory (“DRAM”), or ROM. User equipment 1302 may also include additional storage devices 1320, such as removable or non-removable storage devices, including but not limited to magnetic storage devices, optical disk and / or magnetic tape storage devices. Disk drives and their associated computer-readable media may provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data to the computing device. In some implementations, storage device 1320 may be used to store data received from one or more other devices (e.g., server device 1312, other user devices, hub device 1308, accessory device 1304, or representative accessory device 1306). For example, storage device 1320 may store accessory management settings, accessory attributes, hub attribute information, and user data associated with users belonging to a residential environment.

[0166] User equipment 1302 may also include a communication interface 1316 that allows user equipment 1302 to communicate with a storage database, another computing device or server, user terminal or other devices via network 1310. User equipment 1302 may also include I / O devices 1322, such as those for connecting to a keyboard, mouse, pen, voice input device, touch input device, display, speaker, printer, etc.

[0167] The memory 1314 may include an operating system 1324 and one or more applications or services for implementing the features disclosed herein, including a communication module 1326, a user interface module 1328, a digital assistant 1330, and a management module 1332. The management module 1332 also includes a rating module 1334 and device attributes 1336.

[0168] Communication module 1326 may include code that causes processor 1318 to generate instructions and messages, transmit messages, or otherwise communicate with other entities. For example, communication module 1326 may, in conjunction with management module 1332, transmit and receive data associated with accessory allocation requests, accessory characteristics, and hub device attributes from accessory devices 1304, 1306, hub device 1308, other user devices, or server device 1312. As described herein, communication module 1326 may transmit messages via one or more network paths of network 1310 (e.g., via a LAN or Internet WAN associated with a residential environment).

[0169] The user interface module 1328 may include code that causes the processor 1318 to present information corresponding to the accessory devices 1304 and hub devices 1308 present in the residential environment. For example, the user interface module 1328 may present a graphical representation of the hub devices 1308 and the accessory devices 1304 currently associated with each hub device. In some embodiments, the user interface module 1328 may allow the user to provide configuration information about a new accessory device to be added to the residential environment, or allow the user to select a hub device 1308 or an accessory device 1304 to remove from the residential environment.

[0170] Digital assistant 1330 may include code that enables processor 1318 to receive and process user requests. User requests may be transmitted from accessory device 1304 to user device. Digital assistant 1330 may include voice processing capabilities and language support. The presence of digital assistant 1330 and its features may include one or more attributes considered when comparing accessory features 1358 with hub attributes to score hub device 1308.

[0171] Management module 1332 may include code that causes processor 1318 to send and receive information to or from one or more accessory devices 1304, 1306, and to send and receive information to or from one or more hub devices 1308 or other user equipment. For example, management module 1332 may receive allocation requests and accessory characteristics from accessory device 1306, or hub attributes from hub device 1308, in conjunction with communication module 1326. Management module 1332 may also transmit information indicating allocation to the optimal hub to one of the hub devices 1306 and 1308. In some embodiments, management module 1332 may store information corresponding to the accessory status of each accessory managed by user equipment 1302 within the residential environment. The accessory status identifies which hub among the plurality of accessory devices 1304 has been assigned to.

[0172] Management module 1332 may include scoring module 1334 and device attributes 1336. Device attributes 1336 may include accessory characteristics 1358 received from accessory device 1306 and hub attributes received from hub device 1308. Device attributes 1336 may be stored in memory 1314 or storage device 1320 at the user equipment. In some embodiments, device attributes may be stored at another device including server device 1312 and received in memory 1314 when user equipment 1302 processes an allocation request. Scoring module 1334 may include code that causes processor 1318 to compare accessory characteristics and hub attributes including device attributes 1336 to calculate a score for each hub device in hub device 1308.

[0173] Turning now to the details of representative accessory device 1306, in some embodiments, accessory device 1306 may have at least one memory 1342, communication interface 1344, processor 1346, storage unit 1348, and I / O device 1350. As described herein with respect to user equipment 1302, these elements of the accessory device may have the same suitable hardware implementation as their counterparts on user equipment 1302.

[0174] The memory 1342 of accessory device 1306 may include an operating system 1354 and one or more applications or services for implementing the features disclosed herein, including a communication module 1352 and an accessory development kit (“ADK”) 1356. The ADK may be a software development kit (“SDK”) stored on accessory device 1306 and configured to be executed or processed on that accessory device. As used herein, the SDK may include application programming interfaces and associated software libraries sufficient to implement the operation of other software within or associated with the SDK. In some embodiments, the ADK may be provided by an entity associated with hub device 1308 or user equipment 1302 (e.g., their manufacturer). As described herein with respect to user equipment 1302, communication module 1352 may have suitable functionality similar to its corresponding communication module 1326.

[0175] ADK 1356 may include code that causes processor 1346 to determine the allocation status of accessory device 1306 and, if accessory device 1306 is not currently allocated to a hub device, transmit an allocation request to user equipment 1302. ADK 1356 may include accessory characteristics, including required characteristics of accessory device 1306 and other characteristics. In some embodiments, accessory characteristics may be similar to those referenced in the reference... Figure 12 The accessory features described in accessory feature 1224. For example, accessory device 1306 may require specific language support for any hub device to which it is assigned in hub device 1308.

[0176] Figure 14 This is a flowchart illustrating a specific exemplary process 1400 for assigning Attachment 1402 to a selected hub device. Figure 14 Each of the elements and operations depicted herein may be similar to one or more elements depicted in other figures described herein. For example, user equipment 1401 may be similar to other user equipment (e.g., a leader device), and so on. In some embodiments, process 1400 may be in a residential environment (e.g., Figure 11 The process is carried out within the residential environment (1100). Process 1400 and... Figure 15 , Figure 16 , Figure 17 and Figure 18Processes 1500, 1600, 1700, and 1800 (described below) are shown as logic flowcharts, each operation of which represents a series of operations that can be implemented by hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media, which performs the operation when executed by one or more processors. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the operations may be implemented in any order and / or in parallel combinations.

[0177] Furthermore, some, any, or all of these processes can be executed under the control of one or more computer systems configured to have executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that is executed on one or more processors, by hardware, or a combination thereof. As described above, the code can be stored on a computer-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable storage medium is non-transitory.

[0178] At box 1408, attachment 1402 may request allocation to a hub device. This request may be initiated based on the attachment device recognizing that it is not currently allocated to a hub device. This could be because the attachment device is new to the residential environment or because a previously allocated hub device discarded the attachment device. At box 1410, user equipment 1401 may receive the allocation request. In some embodiments, user equipment 1401 may be a leader hub device among hub devices in a residential environment. The leader hub device may be selected as leader by another process, including a process that occurs at a server device or at other devices determining leadership among hub devices.

[0179] At box 1412, user equipment 1401 can obtain the accessory characteristics reported by accessory 1402 at box 1414. The accessory characteristics may be similar to those in the reference... Figure 12The attachment feature 1224 is described above. In some embodiments, user equipment 1401 may query attachment 1402 to obtain those features. In other embodiments, the attachment device transmits the attachment features along with the allocation request. In yet another embodiment, attachment 1402 may periodically report its attachment features to the leader device (or one or more other devices (including other leader devices)) or server device, so that the leader device, other devices, or server device maintains the latest storage of the attachment features. Similarly, at block 1416, user equipment 1401 may obtain hub information from one or more hub devices. Hub information may be similar to that described above relative to... Figure 12 The hub properties 1210 and slot 1216 are described in detail.

[0180] In some examples, the controller device may enable or disable hub capabilities, such that only devices with hub capabilities enabled can act as hubs for annex 1402. Similarly or alternatively, the device may enable or disable digital and / or virtual assistants, which can be used to determine whether the device can act as a hub. User device 1401 may maintain a list of all devices with hub capabilities (or assistant capabilities) enabled. Therefore, when user device 1401 obtains hub information at box 1416, the user device may only query devices (e.g., first hub device 1404 and second hub device 1406) because these devices have already been listed as having hub or assistant capabilities enabled. Alternatively, when user device 1401 is updated to have information identifying devices with hub or assistant capabilities enabled, hub information may have already been obtained from the hub device (e.g., when the user device learns which devices have hub and / or assistant capabilities enabled, user device 1401 may obtain hub information for each device simultaneously). Additionally, in some examples, block 1416 is actually executable when user device 1401 becomes the leader and is thus configured to poll all other devices in the mesh for their ability. As mentioned above, it is possible that user device 1401 obtains the hub information at this time, rather than in response to the allocation request at attachment 1402 in block 1408. In this case, at block 1416, user device 1401 can obtain the hub information by retrieving it from its own memory, since the user device may have already received the hub information from the first hub device 1404 and / or the second hub device 1406 when the mesh was created or when user device 1401 became the leader. In some examples, the hub devices may periodically update their corresponding hub information (e.g., as features are added / removed). When this occurs, user device 1401 may need to obtain the updated hub information from each of the first hub device 1404 and the second hub device 1406.

[0181] like Figure 14 As depicted, the hub devices may include a first hub device 1404 and a second hub device 1406. Process 1400 is not limited to only two hub devices and can be performed for any number of hub devices in a residential environment. Similar to receiving the attachment feature, in some embodiments, user equipment 1401 may query hub devices 1404 and 1406 at blocks 1418 and 1420 to report hub information. Hub devices 1404 and 1406 may also periodically report hub information to user equipment 1401 or a server device to maintain up-to-date storage of hub information. In these embodiments, user equipment 1401 may receive hub information by accessing a hub information repository stored on itself or on a server device.

[0182] At box 1422, user device 1401 may calculate a baseline metric for the first hub device 1404 and the second hub device 1406. This baseline metric may be based on the general computing power of the hub devices 1404 and 1406. For example, the first hub device 1404 may be a tablet computer with a powerful processor and large onboard memory, while the second hub device 1406 may be a hub speaker with a less powerful processor and less memory. The tablet computer's baseline score may be higher than the hub speaker's baseline score. The baseline score may also depend on the current processing load experienced by the hub device. In the previous example, although the tablet computer may have greater computing power than the hub speaker, it may be continuously used to perform other computing tasks (e.g., running applications or playing media) for users in a residential environment. Therefore, the calculated baseline score may be different for the same hub device depending on the hub device's current processing power when the score is determined.

[0183] At block 1424, user equipment 1401 may compare the received attachment characteristics with hub information received from the first hub device 1404 and the second hub device 1406. In some embodiments, one or more operations in the operation of block 1424 may be similar to those for... Figure 10 Box 1006 or reference Figure 12 The process 1200 involves one or more operations. For example, annex 1402 may have required characteristics and other characteristics. Hub information can identify the features and functions of hub devices 1404, 1406 that can meet the requirements of the annex's required characteristics. User equipment 1401 can compare the required characteristics with the attributes of the first hub device 1404 and the second hub device 1406, and calculate a modified score for each hub device. The modified score is obtained by modifying the base score using the comparison results. Hub devices that do not meet the required characteristics of the annex devices are given a modified score of 0.

[0184] Numerous methods have been envisioned for quantifying the comparison between accessory characteristics and hub attributes, and the scoring algorithm used by user equipment 1401 can be updated or modified over time to provide different quantifications of the best hub device score. Consider, for example, a residential environment where the hub device frequently experiences significant load due to activities unrelated to the interaction between the hub device and accessory devices. This load might be due to many hub devices receiving user interactions for performing other tasks, such as a tablet or media player streaming video content from the internet. In these cases, hub devices may frequently indicate that they cannot support the accessories they have been assigned, which could lead to frequent accessory reassignment. The connection stability between the assigned accessory devices and the hub device can be improved by changing the scoring algorithm so that hub devices with attributes indicating that the hub device may suffer from frequent load will receive lower scores.

[0185] Returning to process 1400, at box 1426, user equipment 1401 can calculate the final scores for the first hub device 1404 and the second hub device 1406. The final score can be a first connectivity score for the first hub device 1404 and a second connectivity score for the second hub device 1406. The final score is calculated by multiplying the modified score obtained at box 1424 by the number of available connectivity slots for each hub device. If a hub device has no available slots, its final score can be 0. Similarly, if a hub device receives a modified score of 0 because it does not meet the requirements of the accessory feature, its final score can also be 0 regardless of the number of available slots the hub device has or its general computing power.

[0186] At decision 1428, the final scores from hub devices 1404 and 1406 are compared, and the highest score wins, thus indicating the best hub device for attachment allocation. If the first hub device 1404 has a higher connectivity score, the process moves to box 1430, and user device 1401 allocates attachment 1402 to the first hub device 1404. Allocation may include the first hub device 1404 receiving an allocation instruction identifying attachment 1402 and allowing the first hub device to associate with attachment 1402. Association may include, for example, creating an interactive instance including software modules for communicating with attachment 1402 and processing user requests transmitted from attachment 1402 to the first hub device 1404. The association may also include the first hub device 1404 establishing a connection with attachment 1402. At endpoint 1434, due to the newly associated attachment 1402, the first hub device 1404 may update its slots to reflect one less available connectivity slot. Conversely, if the second hub device 1406 has a higher connectivity score, the process can be moved to box 1436, and user device 1401 can assign attachment 1402 to the second hub device 1406. Box 1438 and endpoint 1440 are similar to boxes 1432 and endpoint 1434, but the operation is performed by the second hub device 1406. For example, upon receiving an assignment instruction at box 1438, the second hub device can associate itself with attachment 1402 (e.g., establish a connection with attachment 1402). In some examples, when attachment 1402 is first connected to a network (e.g., the same network to which each of user device 1401, first hub device 1404, and second hub device 1406 is connected), attachment 1402 can provide its information (e.g., identifiers, etc.) so that each device on the network will know about the device and how to connect to it. In this way, once the first hub device 1404 or the second hub device 1406 is assigned as a hub for accessory 1402, they are configured to connect.

[0187] Once the optimal hub device has been identified and assigned, the process can move to box 1442 to assign attachment 1402 to the optimal hub device. The assignment of attachment 1402 may include transmitting information to attachment 1402 identifying which of the first hub device 1404 and the second hub device 1406 received the higher connection score. At endpoint 1444, attachment 1402 may update its attachment status to reflect the current hub assignment. However, in some cases, box 1442 is skipped, and user equipment 1401 does not report any information about the hub assignment back to attachment 1402. In this case, at endpoint 1444, attachment 1402 may instead update its attachment status as needed (e.g., as described above) based on being contacted (and / or connected) by either the first hub device 1404 or the second hub device 1406.

[0188] Figure 15 This is another flowchart illustrating an exemplary process 1500 for reassigning an accessory device from one hub device to another hub device according to an implementation scheme. Figure 15 Each of the elements and operations depicted herein may be similar to one or more elements depicted in other figures described herein. For example, user equipment 1501 may be similar to other user equipment (e.g., a leader device), while the first hub device 1504, the second hub device 1506, and the third hub device 1508 may be similar to other hub devices, and so on. As with process 1400 previously described, in some embodiments, process 1500 may be performed in a residential environment (e.g., Figure 11 It can be performed in a residential environment (1100) or in any type of network environment (e.g., office network, school network, general network, etc.).

[0189] At block 1510, the first hub device 1504 may discard the currently assigned accessory 1502 (e.g., by closing the socket, etc., to end the session with accessory 1502), for example, because the first hub device 1504 begins to experience increased processing load due to other activity at the first hub device 1504. At block 1512, the first hub device 1504 may indicate to accessory 1502 that the accessory has been discarded and unassociated. The first hub device may then update its slot information to reflect that accessory 1502 is no longer associated. In some embodiments, the first hub device 1504 may update its slot information in response to increased processing load before reporting to accessory 1502 that the accessory is being discarded. In these embodiments, the operations of blocks 1510, 1512, and 1514 may occur in a different order than that depicted in the illustration of process 1500. Slot information may be similar to that relative to Figure 12 Slot 1216 is described in detail.

[0190] At box 1516, after receiving information from the first hub device 1504 that it is being discarded, attachment 1502 may update its status to reflect that it is no longer assigned to the first hub device. At box 1518, the attachment may request assignment from user equipment 1501. Boxes 1520-1526 may include similar... Figure 14 One or more operations in blocks 1410-1416. At block 1526, user equipment 1501 may receive second hub information from second hub device 1506 and third hub information from third hub device 1508. Second hub device 1506 may report hub information at block 1528, while third hub device 1508 may report hub information at block 1530. In some embodiments, although not in... Figure 15As described, however, user equipment 1501 can also obtain first hub information and hub information corresponding to its own attributes from first hub device 1504. In these embodiments, first hub device 1504 may not report updated hub information to user equipment 1501, making user equipment 1501 unaware that first hub device 1504 recently discarded attachment 1502, and potentially losing subsequent scoring processes between hub devices. Such embodiments can represent instances where the scoring algorithm and communication between hub devices and user equipment 1501 are more efficient if the specific reason for user equipment 1501 issuing an allocation request for attachment 1502 is unknown.

[0191] At box 1532, user equipment 1501 scores the hub based on received accessory characteristics and hub information. Boxes 1532-1536 may include similar... Figure 14 One or more operations in boxes 1422-1436. At box 1536, user equipment 1501 may assign accessory 1502 to either the second hub device 1506 or the third hub device 1508 based on their connection scores. If the second hub device 1506 has a higher connection score, it may receive an assignment instruction at box 1542, update its slot accordingly at endpoint 1544, and connect to accessory device 1502 at endpoint 1548. Similarly, if the third hub device 1508 has a higher connection score, it may receive an assignment instruction at box 1538, update its slot at endpoint 1540, and connect to accessory 1502 at endpoint 1548. The operations at endpoints 1544, 1540, and / or 1548 may be performed in any order. In other words, the second hub device 1506 can be connected to the accessory 1502 at the endpoint 1548 before or after updating its slot information at the endpoint 1544, and the same applies to the endpoints 1548 and 1540.

[0192] As with several previous boxes, one or more operations in the operations of box 1546 and endpoint 1548 may be similar to those relative to... Figure 14 One or more operations of the aforementioned frame 1442 and endpoint 1444.

[0193] Figure 16 This is another flowchart illustrating an exemplary process 1600 for reassigning an accessory device from one hub device to another hub device according to an implementation scheme. Figure 16Each of the elements and operations depicted herein may be similar to one or more elements depicted in other figures described herein. For example, user equipment 1601 may be similar to other user equipment (e.g., a leader device), while the first hub device 1604, the second hub device 1606, and the third hub device 1608 may be similar to other hub devices, and so on. As with process 1400 previously described, in some embodiments, process 1600 may be performed in a residential environment (e.g., Figure 11 It can be performed in a residential environment (1100) or in any type of network environment (e.g., office network, school network, general network, etc.).

[0194] At box 1610, the first hub device 1604 may lose connectivity to the network or lose connection to attachment 1602, for example, because the first hub device is turned off or moved. In some examples, the hub device (e.g., the first hub device 1604) acting as a hub for the attachment (e.g., attachment 1602) may be configured to ping the attachment at regular intervals (e.g., every 15 to 30 seconds, etc.), thereby notifying the attachment that the hub device is still acting as a hub. If attachment 1602 does not detect a ping within a threshold time period, at box 1616, attachment device 1602 may enter a discarded state. This ping may be a "keep-alive" message sent to each connected attachment. In some examples, the "keep-alive" message may be a request from the first hub device 1604 to read the "ping" property of attachment 1602. When accessory device 1602 detects that the first hub device 1604 has read the "ping" feature, the detection of the read "ping" feature notifies accessory device 1602 that the first hub device 1604 is still acting as a hub and the session is still active. Alternatively, instead of the hub device sending a "ping" message (and reading the accessory feature) to the accessory, in some cases, accessory devices may be configured to initiate pings to their assigned hubs to verify connectivity to each assigned hub. If the accessory pings its respective hub and cannot verify the connection, accessory device 1602 may enter a discarded state at box 1616.

[0195] In some examples, Attachment 1602 may remain in a discarded state at box 1616 for a period of time (e.g., a "grace period") before requesting a new hub. If the first device 1604 comes back online and attempts to connect to Attachment 1602 (e.g., establish a new socket and / or session), Attachment 1602 may be reassigned to the first hub device 1604 (e.g., reconnected to the first hub device 1604) without requesting a new hub from the leader (e.g., user device 1601). However, if the grace period expires, Attachment 1602 will instead transition to box 1618, where Attachment 1602 will request the allocation of a new hub from the leader (e.g., user device 1601).

[0196] Boxes 1620-1626 may include similar to Figure 14 Box 1410-1416 or Figure 15 One or more operations in blocks 1520-1526. At block 1626, user equipment 1601 may receive second hub information from second hub device 1606 and third hub information from third hub device 1608. Second hub device 1606 may report hub information at block 1628, while third hub device 1608 may report hub information at block 1630. In some embodiments, although not in... Figure 16 As described, however, user equipment 1601 can also obtain first hub information and hub information corresponding to its own attributes from first hub device 1604. In these embodiments, first hub device 1604 may not report updated hub information to user equipment 1601, making user equipment 1601 unaware that first hub device 1604 recently discarded attachment 1602, and potentially losing subsequent scoring processes between hub devices. Such embodiments can represent instances where the scoring algorithm and communication between hub devices and user equipment 1601 are more efficient if the specific reason for user equipment 1601 issuing an allocation request for attachment 1602 is unknown.

[0197] At box 1632, user equipment 1601 scores the hub based on received accessory characteristics and hub information. Boxes 1632-1636 may include similar... Figure 14One or more operations in boxes 1422-1436. At box 1636, user equipment 1601 may assign accessory 1602 to either the second hub device 1606 or the third hub device 1608 based on their connection scores. If the second hub device 1606 has a higher connection score, it may receive an assignment instruction at box 1642, update its slot accordingly at endpoint 1644, and connect to accessory device 1602 at endpoint 1648. Similarly, if the third hub device 1608 has a higher connection score, it may receive an assignment instruction at box 1638, update its slot at endpoint 1640, and connect to accessory 1602 at endpoint 1648. The operations at endpoints 1644, 1640, and / or 1648 may be performed in any order. In other words, the second hub device 1606 can connect to the accessory 1602 at the endpoint 1648 before or after updating its slot information at the endpoint 1644, and the same applies to the endpoints 1648 and 1640.

[0198] As with several previous boxes, one or more operations in the operations of box 1646 and endpoint 1648 can be analogous to those relative to... Figure 14 One or more operations of the aforementioned frame 1442 and endpoint 1444.

[0199] Figure 17 An exemplary process 1700 for transferring hub management from one user equipment to another, according to an embodiment, is illustrated. A server device 1703 may participate in the transfer of hub management responsibility from a first user equipment 1701 to a second user equipment 1702. The first user equipment 1701, the second user equipment 1702, and the server device 1703 may correspond to any or more of the user equipment, hub device, and server device described herein. In some embodiments, process 1700 may be performed in a residential environment (e.g., Figure 11 The residential environment (1100) shall be implemented.

[0200] At block 1704, the first user equipment 1701 may receive information or instructions to transfer hub leadership to another hub device within the residential environment. In some embodiments, user equipment 1701 may determine that it is no longer suitable to act as the leader device for hub devices in the residential environment. For example, user equipment 1701 may be a tablet computer subjected to significant processing loads due to playing streaming media for users in the residence. Upon determining that it is no longer suitable to provide management for other hub devices, the tablet computer may transmit a request to transfer leadership to another device. In other embodiments, another user equipment or server device 1703 may instruct the first user equipment 1701 to relinquish its leadership position. This may occur when a user reconfigures devices within the residential environment and selects a different device as the default leader device. The first user equipment 1701 may then transmit a leader reassignment request to server device 1703.

[0201] At box 1708, server device 1703 can identify a new leader device. This operation is similar to a user device reassigning an accessory device to another hub device. The server device can select a new leader device based on several criteria, including but not limited to: the processing power of the new leader device, the number of hub devices and accessory devices present in the residence, whether the new leader device is a resident device in the residence, and whether the user has indicated that a particular device should be the new leader device. In some embodiments, another user device or hub device in the residence can perform one or more of the operations of server device 1703, including selecting a new leader device based on the aforementioned criteria.

[0202] Once a suitable device is selected as the new leader, at box 1710, the first user device 1701 can assign leadership capabilities to the selected device. For example... Figure 17As shown, the new leader device can be a second user device 1702. As part of the assignment operation, the first user device 1701 can send information to the second user device 1702 about the hub device, accessory devices, and their relationships. This information may include current hub attributes and accessory characteristics. In some embodiments, the first user device 1701 only sends instructions to the second user device 1702 to accept the leadership role for the hub device and accessories in the residential environment. In yet another embodiment, the server device 1703 can communicate directly with the second user device 1702 to provide leadership assignment. For example, in some cases, the first user device 1701 may lose network connectivity with other devices in the residential environment. If no leader assignment is received from the first user device 1701, the server device can receive information that the first user device is disconnected from the network and is no longer able to act as the leader device. The server device 1703 can then instruct the second user device 1702 to act as the hub leader.

[0203] At box 1712, the second user equipment 1702 receives leadership capability assignment. At decision 1714, the second user equipment determines whether it has received hub and accessory information from the first user equipment 1701. If not, the process proceeds to box 1716, and the second user equipment can query the hubs for their current hub attributes and accessory assignment information. Once the second user equipment 1702 has the current hub information, it can store the information at endpoint 1718, or at the second user equipment or another device.

[0204] Figure 18 This is a flowchart illustrating an exemplary process 1800 in which a user equipment assigns an accessory device to a hub device selected from a plurality of hub devices. In some embodiments, one or more operations of process 1800 may be similar to those described in reference to [reference]. Figure 10 and Figure 14 The operations described above.

[0205] At box 1802, the user equipment may receive first information about a first hub device and second information about a second hub device. The first and second information may correspond to hub attribute information as described herein, and may be similar to... Figure 12 Hub properties 1210 and slot 1216 and Figure 13 Device attribute 1336. In some implementations, one or more operations in the operation of block 1802 may be similar to those referenced. Figure 12 As described for process indicator 1240 or for Figure 14 One or more operations as described in box 1416.

[0206] At block 1804, the user equipment may receive a connection request from an accessory. The connection request may be a request to connect to a hub device to which the user equipment assigns the accessory. In some embodiments, one or more operations in the operation of block 1804 may be similar to those for... Figure 14 One or more operations as described in box 1412.

[0207] At block 1806, the accessory device may send accessory information to the user equipment for determining accessory assignment. The accessory information may include information about accessory attributes or characteristics and the requirements of the assigned hub device. In some embodiments, one or more operations in the operation of block 1806 may be similar to those for... Figure 12 Process indicator 1230 or Figure 14 One or more operations described in box 1414.

[0208] At block 1808, the user equipment can determine the scores of the first hub device and the second hub device by comparing the attachment attribute information received at block 1806 with the first information and the second information received from the first hub device and the second hub device at block 1802. In some embodiments, one or more operations in the operation of block 1808 may be similar to those for... Figure 14 One or more operations as described in box 1424.

[0209] At box 1810, the user equipment can compare the score determined at box 1808 to determine whether to assign the attachment to the first hub device or the second device. This determination may be based on which hub device has a higher score. In some embodiments, one or more operations in the operation of box 1810 may be similar to those for... Figure 14 The decision 1428 refers to one or more operations.

[0210] At box 1812, the selected hub device may receive an instruction to connect to an accessory. In some embodiments, connecting to an accessory may include creating an accessory interaction instance at the hub device corresponding to the assigned accessory device. In some embodiments, one or more operations in the operation of box 1802 may be similar to those for... Figure 14 Box 1430 or Figure 15 One or more operations as described in box 1536.

[0211] The foregoing describes exemplary techniques for load balancing among multiple hub devices and accessories. Some or all of these techniques can be implemented, at least partially, through architectures such as those described above. Figures 10 to 18The architectures shown herein can be used to implement, but are not required to implement, the techniques disclosed herein. While numerous implementations have been described above with reference to server devices, accessory devices, user devices, and hub devices, it should be understood that other types of computing devices may be suitable for implementing the techniques disclosed herein. Furthermore, various non-limiting examples have been described in the foregoing description. For purposes of explanation, many specific configurations and details have been set forth to provide a thorough understanding of the examples. However, it will also be apparent to those skilled in the art that some examples can be implemented without these specific details. In addition, well-known features have sometimes been omitted or simplified to prevent confusion with the examples described herein.

[0212] While specific exemplary embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also covered within the scope of this disclosure. The embodiments are not limited to operating within certain specific data processing environments, but are freely operable within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, it will be apparent to those skilled in the art that the scope of this disclosure is not limited to said series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0213] Examples of the described technology can be illustrated by the following clauses.

[0214] Clause 1. A method comprising:

[0215] The user equipment receives first information about a first hub device among a plurality of hub devices and second information about a second hub device among the plurality of hub devices;

[0216] The user equipment receives the connection request from the attachment;

[0217] Receive attachment attribute information from the attachment;

[0218] The user equipment compares the accessory attribute information of the accessory with the first information about the first hub device and the second information about the second hub device;

[0219] The determination of whether the accessory is connected to the first hub device or the second hub device is based at least in part on the comparison; and

[0220] Based on the determination that the accessory is connected to the first hub device, an instruction is provided to the first hub device to connect to the accessory.

[0221] Clause 2. The method according to Clause 1, wherein the first information regarding the first hub device includes at least one of a set of capabilities of the first hub device or the number of available connection slots of the first hub device, and wherein the second information regarding the second hub device includes at least one of a set of capabilities of the second hub device or the number of available connection slots of the second hub device.

[0222] Clause 3. The method according to Clause 2, wherein comparing the attachment attribute information with the first information regarding the first hub device and the second information regarding the second hub device includes:

[0223] A first metric for the potential connection of the accessory to the first hub device is generated, based at least in part on the accessory attribute information and the set of capabilities of the first hub device.

[0224] Multiply the first metric by the number of available connection slots of the first hub device to generate a first connectivity score;

[0225] A second metric for the potential connection of the accessory to the second hub device is generated, based at least in part on the accessory attribute information and the set of capabilities of the second hub device; and

[0226] The second metric is multiplied by the number of available connection slots of the second hub device to generate a second connectivity score.

[0227] Clause 4. The method according to Clause 3, wherein determining whether the attachment is connected to the first hub device or the second hub device includes selecting which of the first connection score or the second connection score is higher.

[0228] Clause 5. The method described in Clause 1 further includes:

[0229] The user equipment receives hub connection information about the first hub device; and

[0230] The determination that the first hub device is no longer connected to the accessory is based at least in part on the hub connection information.

[0231] Clause 6. The method described in Clause 5 further includes: determining, based on the determination, that the first hub device is no longer connected to the accessory:

[0232] The user equipment compares the accessory attribute information with the second information about the second hub device and the third information about the third hub device;

[0233] The determination of whether the accessory is connected to the second hub device or the third hub device is based at least in part on the comparison; and

[0234] Based on the determination, an instruction to connect to the accessory is provided to the second hub device or the third hub device.

[0235] Clause 7. The method described pursuant to Clause 1 further includes:

[0236] The user equipment receives information identifying it as a leader hub device configured to manage connections between multiple hub devices and multiple accessory devices, wherein the first hub device and the second hub device are hub devices among the multiple hub devices, and the accessory device is an accessory device among the multiple accessory devices.

[0237] Clause 8. The method according to Clause 7, wherein the user equipment is one of the plurality of hub devices.

[0238] Clause 9. The method described pursuant to Clause 8 further includes:

[0239] The user equipment determines that it is no longer suitable to serve as the leader hub device; and

[0240] Based on the determination, the hub leader responsibility is assigned to another hub device among the plurality of hub devices.

[0241] Clause 10. The method according to Clause 9, wherein assigning the hub leader responsibility to the other hub device comprises:

[0242] The user equipment transmits a hub leader reassignment request to the server equipment, the server equipment being configured to provide second information to identify the second user equipment as the leader hub equipment.

[0243] Clause 11. The method described pursuant to Clause 1 further includes:

[0244] The user equipment receives hub processor usage information about the first hub device; and

[0245] The determination is made, at least in part, based on the hub processor usage information, that the first hub device is no longer able to respond to the attachment within a threshold time period.

[0246] Clause 12. The method according to Clause 11 further comprises: determining that the first hub device is no longer able to respond to the appendix within a threshold time amount:

[0247] The user equipment compares the accessory attribute information with the second information about the second hub device and the third information about the third hub device;

[0248] The determination of whether the accessory is connected to the second hub device or the third hub device is based at least in part on the comparison; and

[0249] Based on the determination, an instruction to connect to the accessory is provided to the second hub device or the third hub device.

[0250] Clause 13. The method described pursuant to Clause 1 further includes:

[0251] The user equipment receives functional information about the first hub device; and determines, at least in part based on the functional information, that the first hub device is no longer the optimal hub for the accessory.

[0252] Clause 14. The method according to Clause 13 further comprises: determining that the first hub device is no longer the optimal hub for the accessory:

[0253] The user equipment compares the accessory attribute information with the second information about the second hub device and the third information about the third hub device;

[0254] The determination of whether the accessory is connected to the second hub device or the third hub device is based at least in part on the comparison; and

[0255] Based on the determination, an instruction to connect to the accessory is provided to the second hub device or the third hub device.

[0256] Clause 15. The method described in Clause 1 further includes:

[0257] The user equipment receives the second connection request from the second accessory;

[0258] Receive the attribute information of the second attachment from the second attachment;

[0259] The user equipment compares the second accessory attribute information with the first information about the first hub device and the second information about the second hub device;

[0260] The determination of whether the second accessory is connected to the first hub device or the second hub device is based at least in part on the comparison; and

[0261] Based on the determination that the second accessory is connected to the first hub device, an instruction is provided to the first hub device to connect to the second accessory.

[0262] Clause 16. The method described pursuant to Clause 1 further includes:

[0263] The user equipment receives a second connection request from a second accessory among a plurality of accessory devices, including at least the first accessory and the second accessory;

[0264] Receive the attribute information of the second attachment from the second attachment;

[0265] The user equipment compares the second accessory attribute information with the first information about the first hub device and the second information about the second hub device;

[0266] The determination of whether the second accessory is connected to the first hub device or the second hub device is based at least in part on the comparison.

[0267] Based on the determination that the second accessory is connected to the second hub device, an instruction is provided to the second hub device to connect to the second accessory; and

[0268] Based on the determination that the second accessory is connected to the first hub device, an instruction is provided to the first hub device to connect to the second accessory.

[0269] Clause 17. The method described pursuant to Clause 16 further includes:

[0270] The user equipment manages a first state of the accessory and a second state of the second accessory, wherein the first state indicates that the accessory is connected to the first hub device, and

[0271] The second status indicates that the attachment is connected to either the first attachment or the second attachment.

[0272] Clause 18. The method described pursuant to Clause 17 further includes:

[0273] The user equipment manages a third state of a third accessory among the plurality of accessories, the third state indicating that the third accessory is not connected to any of the hubs among the plurality of hub devices.

[0274] Clause 19. A user equipment comprising:

[0275] Memory, the memory being configured to store computer-executable instructions; and

[0276] A processor configured to be connected to the memory and to execute the computer-executable instructions to perform at least the method described in any one of clauses 1 to 18.

[0277] Clause 20. A computer-readable storage medium configured to store computer-executable instructions that, when executed by a user equipment, cause the user equipment to perform the method according to any one of Clauses 1 to 18.

[0278] Technology used to establish communication with third-party attachments

[0279] The embodiments of this disclosure provide techniques for establishing communication between an accessory device and a cellular-enabled device. As a first example, consider a residential environment corresponding to a dwelling. A person in the dwelling may want to make a telephone call using voice commands. This person may issue a verbal request (e.g., “Computer, call Mom”) to an accessory device (e.g., a third-party accessory not manufactured or designed by the same entity that manufactures or designs the residential device (e.g., a hub) or the cellular-enabled device (e.g., a smartphone). The accessory device may determine that the request is directed to it and then transmit the received audio information to a hub device (e.g., a hub speaker). The hub device may process the audio information to determine the nature of the request and prepare a corresponding response (e.g., connecting to a cellular phone to initiate a call). Alternatively, or in part in combination with the foregoing, the hub device may transmit some or all of the verbal request to a server computer (e.g., a service provider), where the service provider may determine the nature of the request and / or prepare a corresponding response. The hub device may then communicate with the cellular-enabled device to instruct the cellular-enabled device to initiate a call and establish a separate audio communication channel with the accessory. The hub device can then enter a listening state to listen for requests from users to end a phone call (e.g., "Computer, hang up"). When the hub device receives a hang-up request, it can send a call termination instruction to an accessory device. The accessory device can then transmit the call termination information to a cellular-enabled device.

[0280] As an example of the above examples, a residential environment can include numerous “smart” devices, such as electronic devices with features that allow them to interact and operate autonomously to some extent. Smart devices can have a variety of functions, including cameras, speakers, thermostats, headphones and headsets, telephones, or media players. Smart devices can also have various network communication capabilities, including WiFi, Ethernet, Bluetooth, Zigbee, cellular, etc. These devices can be manufactured by different companies. In some cases, smart devices can be categorized as hub devices and accessory devices. Hub devices can be residential-resident devices (e.g., smart speakers, smart digital media players configured to control a television (TV), mobile phones, etc.). While not always the case, in some examples, it is expected that residential devices reside within the residence and are not frequently moved (e.g., moved within or outside the residence). Hub devices can have capabilities equal to or greater than those of accessory devices. For example, a hub device can be a mobile phone that may include wireless (e.g., WiFi) and cellular communication capabilities, multimedia capabilities, and device assistants. In the same example, the accessory device may be a smart speaker that may include audio media and wireless communication capabilities but lacks a device assistant. The device assistant may be a virtual assistant program configured to interact with a user. In these examples, depending on its capabilities, the smart speaker may be a hub device or an accessory device. In some examples, if the accessory is manufactured by a different entity than the entity that manufactures the hub device, the accessory may not initially be configured to have the ability to communicate with a user device. In some cases, the hub device manufacturer may provide an accessory development kit (“ADK”) for installation on the accessory, which enables such communication after the accessory is manufactured, sold, supplied, or used. The controller device may be a hub device as described herein and may include user interface features. In some embodiments, the controller device is a leader device selected from hub devices in a residential environment. As used herein, the terms hub device, user device, leader device, and controller device may refer to one or more similar devices distinct from an accessory device. A cellular-enabled device may be any device associated with a residential environment capable of connecting to a cellular network. In some embodiments, a cellular-enabled device may be a hub device or an accessory device.

[0281] In some implementations, the hub device may obtain information about accessory devices present in a residential environment. This information may be obtained by the hub device directly communicating with accessory devices sharing the same network within the residential environment. In other implementations, information about accessory devices may be sent to the hub device by a second hub device, a user device configured as the leader device, or a remote server device (e.g., a service provider). For example, a user in a residence may add a new accessory device to the residential environment. As part of this process, the user may interact with a second hub device (e.g., a mobile phone) to configure the new accessory device and send information about the new accessory device to a first hub device. As another example, the leader device in the residential environment may have information about multiple accessory devices in the residential environment and report information about some or all of the accessory devices to the hub device. The hub device can then use this information to form associations with the corresponding accessory devices. The accessory information may be stored by the hub device.

[0282] A hub device can be associated with multiple accessory devices by creating an accessory interaction instance for each accessory device. An interaction instance can be a software module or process configured to perform tasks at the hub device. In some embodiments, the interaction instance may each implement a device assistant and / or communicate with the device assistant. For example, the hub device may receive information about accessory smart speakers and smart thermostats located in a residential environment. The hub device may create two interaction instances corresponding to the device assistant, one for each of the smart speaker and smart thermostat. In some embodiments, the interaction instance may be a copy of the device assistant, while in other embodiments, the instance may be a collection of modules including the device assistant and other processes for performing tasks on the hub device. Depending on the associated accessory and its capabilities, the interaction instance may include different modules or processes. It should be understood that any suitable combination of processes running on the hub device may be included in the interaction instance corresponding to the accessory device.

[0283] Continuing with the first example above, a user can make a request to an accessory. For example, a user could speak into the microphone of a nearby smart speaker (or thermostat, light bulb, etc.) and say, “Computer, call Mom?” In this example, the request (“call Mom”) corresponds to a portion of the user’s audio input to the smart speaker. The start phrase (“computer”) can correspond to a different portion of the user’s audio input and can be a trigger or wake word. In some implementations, the smart speaker can perform speech recognition processing on the wake word. Based on this processing, the smart speaker can determine whether the user’s voice is intended to be a request or command that the speaker should respond to. If so, the smart speaker can then transmit the user’s audio to a hub device running an accessory interaction instance corresponding to the smart speaker. In some implementations, the accessory device can temporarily store a copy of the audio input for transmission to the user device after processing the wake word portion. In other implementations, while processing the wake word portion of the audio input, the accessory device can establish a streaming audio connection with the hub device to relay the portion of the user’s audio input after the wake word. In these implementations, the accessory device can transmit a stored copy of the wake word portion of the audio input to the hub device for additional processing.

[0284] When receiving audio input from a smart speaker, the hub device may perform additional processing on both the wake word portion of the audio input and the portion corresponding to the request or command. For example, the hub device may perform natural language processing (“NLP”) on the wake word. Based on this wake word processing, the hub device may then process the portion of the audio corresponding to the request. If the hub device determines that the wake word portion is not actually the accurate wake word, it may ignore the rest of the audio or terminate the audio stream from the attachment. In some implementations, the voice processing module on the hub device may be part of an attachment interaction instance. The interaction instance may also transmit all or part of the audio input to another device for analysis (e.g., to a service provider device). The service provider device may be a remote server computer or cloud device that performs voice processing and parses the request to provide an appropriate response. In some cases, the hub device performs wake word processing while remotely processing the rest of the audio. However, in other examples, the hub device may handle all processing. Parsing the request includes determining the content and context of the user-spoken audio and providing a response so that the hub device can take action. In the current example, the response could be communicating with a cellular-enabled device to initiate a call to Mom. This could be performed by the hub device using an appropriate process, by a remote server device, or by a combination of both. In some implementations, the hub device could relay instructions for establishing a call to an accessory device, including the identity of the selected cellular-enabled device. The accessory device could then establish communication with the cellular-enabled device and send instructions to initiate the call. In another example, the hub device could relay instructions for establishing a call to a cellular-enabled device, including the identity of the accessory device, which could then establish communication with the accessory device and subsequently initiate the call. Calling Mom could include, for example, accessing user information identifying "Mom" in the context of the requesting user by identifying the user and subsequently accessing that user's contact information. Mom's phone number could then be obtained and sent to the cellular-enabled device.

[0285] Once a response has been determined, the hub device can execute that response. This may include identifying the cellular-enabled device and sending it an instruction to initiate a call. The instruction may include information identifying the call recipient, including the phone number to be dialed or a recipient identity stored locally at the cellular-enabled device to provide the phone number at the cellular-enabled device. For example, in one embodiment, as part of processing an audio request, the hub device may obtain the mother's phone number and then instruct the cellular-enabled device to dial that number. In another embodiment, the hub device may instruct the cellular-enabled device to call "Mom" and have the cellular-enabled device use its own information about the mother's identity to obtain the number. This latter embodiment applies to instances where the hub device can identify the cellular-enabled device corresponding to the user making the call request but cannot access the user's contact information, for example, where contacts are only stored at the cellular-enabled device. Response preparation and execution may occur within the corresponding interaction instance. Responses requiring specific actions (e.g., initiating a phone call) may, as appropriate, be delegated from the interaction instance to another process on the hub device or to another device with which the hub device can communicate.

[0286] The hub device can also communicate with an accessory device to identify a cellular-enabled device initiating a call and instruct the accessory to establish a communication channel with that cellular-enabled device. The communication channel can be a real-time audio connection using a Real-Time Transport Protocol (“RTP”) or other suitable method. The audio channel can be used to send and receive audio between the accessory device and the cellular-enabled device. In some embodiments, the accessory device can establish a second communication channel with the cellular-enabled device to send telephone control commands to the cellular-enabled device. These telephone control commands can be commands to terminate a call based on information received from the hub device by the accessory device that the user has requested to terminate the call. In some embodiments, where the hub device does not directly transmit call commands to the cellular-enabled device, the second communication channel can also be used by the accessory device to send a call initiation command to the cellular-enabled device.

[0287] Once a call is established, the hub device can enter call listening state at the accessory interaction instance corresponding to the accessory device. While in call listening state, the hub device only listens for "hang up" or "end call" or other similar commands from the accessory device that indicate the user wishes to terminate the call. In this way, device assistants and other processes associated with this accessory interaction instance do not inadvertently capture, record, or process audio information from the call. For example, at the end of a call, the user might say "Computer, hang up." As with the call request described above, the "Computer" part of this command could correspond to a wake word that indicates to the accessory device that the audio following the wake word can be a user command rather than part of the call session. The phrase "hang up" could be the end word. While in call listening state, the accessory can receive the audio corresponding to the wake word and end word. The wake word will be processed like any other wake word described herein. If the wake word is detected, the accessory interaction instance can process the end word. Because the hub device is in call listening state, end word processing may be more restricted compared to audio processing for other user requests received. For example, the attachment interaction instance can perform end-word detection locally without transmitting the audio to a remote service provider for NLP. In some implementations, when in call listening mode, the attachment interaction instance at the hub device can process only a limited portion of the audio input following the wake word, such that the attachment interaction instance only receives a short audio segment sufficient to contain an end-word such as "hang up" or "end call." In this way, the attachment interaction instance does not capture user audio that does not correspond to an end-word.

[0288] In some implementations, call listening state can be the state of a specific accessory interaction instance associated with an accessory device connected to the call. Other accessory interaction instances present at the hub device can function normally and are not limited by call listening state. For example, continuing the user call described above, the user can use a smart speaker to make a phone call to their mother. A second user in the home can make a request to a smart thermostat also associated with the hub device (e.g., “Computer, turn the heating up to 72℉”). The accessory interaction instance corresponding to the smart thermostat may not be in call listening state and can process the second user's request normally, instructing the thermostat to change its temperature setpoint according to the request. This allows the hub device to retain its accessory management functions while ensuring the data privacy of the first user and the external call recipient.

[0289] Figure 19This is a simplified block diagram of an exemplary implementation. Process 1900 is an exemplary high-level process flow of a system including an accessory device 1912 that can receive a call request and establish communication with a cellular-enabled device 1930 via a controller device 1920. Figure 1901 illustrates the state of the system corresponding to the block of process 1900. Process 1900 can be performed in a residential environment containing multiple accessory devices, hub devices, and cellular-enabled devices. As depicted herein, accessory device 1912 may be a smart speaker, and controller device 1920 may be a hub speaker. Although described as specific devices, it will be apparent that accessory device 1912 and controller device 1920 may be several types of smart devices in various combinations and numbers. For example, a smartphone, media device (e.g., a smart TV), or tablet computer (connected to a cellular network, connected to a local area network via WiFi from a residential network, or connected to a wide area network (“WAN”)) may perform one or more of the operations of process 1900.

[0290] Turning more specifically to process 1900, at block 1902, accessory device 1912 may receive call request 1916 from user 1914. In some embodiments, call request 1916 may include an audio portion corresponding to the request (e.g., “Initiate a call”) and a second portion corresponding to a wake word (e.g., “Computer”). The wake word need not be a single word and may be a word or phrase that signals to the system that the user has or will speak a request, command, or other audible interaction to the system. Upon receiving input containing the wake word, accessory device 1912 may process this portion of call request 1916 at a first level to determine the presence of the wake word. The first-level processing may be performed in a time- and resource-efficient manner to determine the possible presence of the wake word. For example, accessory device 1912 may use a stored voice pattern corresponding to a user speaking a wake word to perform voice pattern matching. The stored pattern may be associated with a user in the residential environment containing the system, or it may be a general pattern applicable to a large number of possible users. In this way, accessory device 1912 does not bear the burden of a complex voice detection process, nor does it respond to every external audio input received by the user or other sources in its vicinity.

[0291] Moving down to box 1904, upon detecting a wake word, accessory device 1912 can transmit the received call request 1916 to controller device 1920, where the call request will be processed. As shown, smart speaker 1916 has a corresponding accessory interaction instance 1922 on controller device 1920, such that accessory interaction instance 1922 manages the processing of call request 1916 received from smart speaker 1912. Accessory interaction instance 1922 may include modules configured to process call request 1916. For example, accessory interaction instance 1922 may include a voice detection module that analyzes the portion of call request 1916 corresponding to a wake word. This analysis can be performed at a second level where the presence of a wake word can be confirmed with a higher probability than wake word detection at smart speaker 1912. Furthermore, in some embodiments, the voice detection module can determine the user's language and perform wake word detection based on the determined language. If the voice detection module of accessory interaction instance 1922 does not detect a wake word, controller device 1920 may ignore the audio input.

[0292] Controller device 1920 may also access user profiles 1926 and 1928. User profiles 1926 and 1928 may be stored on controller device 1920 or another device, such as a server device. User profiles 1926 and 1928 may correspond to users within a residential environment and include information that can be used to identify one or more cellular-enabled devices associated with the user. For example, user profile 1926 may correspond to user 1914 and may identify cellular-enabled device 1930, described as a smartphone, associated with user 1914. When processing call request 1916, Attachment Interaction Instance 1922 may identify user 1914 as having made call request 1916 and access user profile 1926 to determine the appropriate cellular-enabled device to execute the call. Furthermore, user profile 1926 may also include information related to potential recipients of the call. For example, user profile 1926 may include a list of contacts for user 1914. Appendix Interaction Example 1922 may use contact information, for example, to determine the outgoing phone number to call when executing a cellular-enabled device 1930 during call request parsing 1916. In some implementations, user profiles 1926, 1928 may be stored on a remote server or other device and accessible to a remote service provider used to process the call request.

[0293] Moving to box 1906, controller device 1920 can process call requests to identify a cellular-enabled device 1930 to initiate a call. As described above with reference to box 1904, controller device 1920 can access user profiles 1926, 1928 to determine the appropriate cellular-enabled device 1930.

[0294] At block 1908, controller device 1920 may instruct cellular-enabled device 1930 to initiate a call corresponding to a call request. In some embodiments, this may include determining the outgoing number to be dialed by cellular-enabled device 1930 when making the call. In other embodiments, controller device 1920 may instruct cellular-enabled device 1930 to initiate the call based on a tag or other identifier (e.g., “Mom,” “Office,” etc.) included in call request 1916. In addition to sending instructions to cellular-enabled device 1930, controller device 1920 may also instruct accessory device 1912 to establish a communication channel with cellular-enabled device 1930. This communication channel may be a real-time audio channel for transmitting and receiving audio during a call.

[0295] At box 1910, the attachment interaction instance 1922 at controller device 1920 can enter a call listening state to listen for hang-up commands (e.g., "Computer, end call"). The hang-up command can consist of a portion corresponding to a wake-up word (e.g., "Computer") and a portion corresponding to an end word (e.g., "end call" or "hang up"). Like the wake-up word, the end word does not need to be a single word and can be any word or phrase recognized as indicating the end of a telephone call. When user 1914 issues a hang-up command, attachment 1912 can process the wake-up word at a first level as described above relative to box 1902. If the wake-up word is detected, attachment 1912 can send the audio of the hang-up command to controller device 1920. Controller device 1920 can process the wake-up word at a second level as described relative to 1904. Upon confirmation of the wake-up word's presence, attachment interaction instance 1922 can process the end word portion of the hang-up command. The processing of the end-of-call word can be performed in a restricted manner, such that the controller device 1920 does not receive or process additional audio information potentially captured from an ongoing telephone call at the accessory device 1912. If the end-of-call word is detected, the controller device 1920 can transmit an instruction to the accessory device 1912 to terminate the call. The accessory device 1912 can then issue a hang-up command to the cellular-enabled device 1930 to close the cellular connection at the cellular-enabled device 1930. Alternatively, in some embodiments, the controller device can instruct the cellular-enabled device 1930 to directly terminate the call and transmit an indication to the accessory device 1912 that the call has been successfully terminated.

[0296] Figure 20 This is a schematic diagram of a residential environment including hub devices, accessory devices, and cellular-enabled devices according to some implementation schemes. The hub devices may include a hub speaker 2002 and a media player 2004. These hub devices may correspond to those described above relative to... Figure 19The controller device 1920 of the described embodiment. The smartphone 2006 can be a cellular-enabled device, such as... Figure 19 Cellular-enabled device 1930. In several embodiments, smartphone 2006 may act as a hub device. Accessory devices may include smart speakers 2012, 2014, smartwatches 2016, and thermostats 2024. Similarly, these accessory devices may correspond to... Figure 19 The accessory device 1912 is described above. Some or all of these accessory devices may be third-party devices (e.g., not manufactured, programmed, or supplied by the manufacturer, programmer, or supplier of the hub device or user equipment). Therefore, they may not be automatically and / or initially compatible with the user equipment. Each hub device in the residential environment 2000 may be associated with zero, one, or more accessory devices. As shown by the long dashed line, hub speaker 2002 is associated with smart speakers 2012, 2014, and smartwatch 2016, while media player 2004 is associated with thermostat 2024. Smartphone 2006 is not associated with any accessory device. Devices within the residential environment 2000 may be configured to communicate using one or more network protocols through one or more networks associated with the residential environment 2000. For example, the residential environment 2000 may be associated with a local area network (“LAN”), WAN, cellular network, personal area network, or other networks, and these devices may communicate using WiFi connectivity, Bluetooth connectivity, Thread connectivity, Zigbee connectivity, or other communication methods.

[0297] The arrangement of accessory devices associated with hub devices can include various combinations and can be modified by another device associated with the residential environment (e.g., a hub device within a hub device or a user device). For example, a user device can associate a new accessory device with a hub device in the residence. In some implementations, assigning accessory devices to hub devices can be based on a scoring algorithm applied by the user device. The user device can also use the scoring algorithm to transfer existing accessory devices from one hub device to another. This transfer can occur automatically based on information received by the hub device about the residential environment 2000, including but not limited to information that another user device may be more suitable to be associated with one or more accessories or that an accessory has been added to or removed from the residential environment 2000. When an accessory is assigned to or associated with a hub device, the hub device can create an accessory interaction instance corresponding to each assigned accessory. Therefore, the hub device can include a unique software ecosystem for each assigned accessory. The suitability of any particular hub device associated with an accessory may be based at least in part on the hub device's capabilities, the accessory device's capabilities, the current processing load experienced by the hub device, the location of the devices within the residential environment, and the state of communication between devices on the network. Many other criteria are envisioned for rearranging device associations within a residential environment.

[0298] In some implementations, non-resident accessory devices and hub devices may also leave the residential environment or lose network connectivity with the residential environment. Accessory devices leaving the residential environment can be unassociated from previously associated hub devices. Accessory devices associated with hub devices that have lost network connectivity with the residential environment can be reassigned by another hub device that maintains network connectivity. In this case, the other hub device may receive information that the hub device is no longer able to communicate with the accessory device and reassign the accessory device. Some implementations may have a hub device designated as a leader device to manage the allocation of accessory devices among hub devices within the residential environment. In other implementations, if a hub device and accessory device are associated and leave the residential environment and lose network connectivity, the hub device may maintain its association with the accessory device and perform the embodied methods described herein.

[0299] As a hub device, smartphone 2006 can communicate with other hub devices within the residential environment, including receiving accessory assignments from user equipment or leader devices. Thus, other hub devices can communicate with smartphone 2006 to instruct it to initiate a call via the cellular network in response to a user call request. In some embodiments, smartphone 2006 may not act as a hub device but remain known to hub devices, user equipment, or leader devices within the residence, allowing call requests to be transmitted to smartphone 2006 as a cellular-enabled device. In other embodiments, smartphone 2006 may be identifiable by a remote device (e.g., a server device communicating with one or more networks in the network associated with the residential environment).

[0300] continue Figure 20 The residential environment 2000 may have multiple users 2030 and 2034 issuing audio requests 2032 and 2036 to accessories. Requests 2032 and 2036 may occur separately or simultaneously and may be received by multiple accessory devices, as shown by the short dashed lines. For example, request 2032 may be received by a smart speaker 2014 or a smartwatch 2016, while request 2036 may be received by a smart speaker 2012 and a thermostat 2024. As previously described, the arrangement of accessory devices and their associated devices can take various forms and may change over time. Therefore, user requests may be received by multiple accessory devices associated with different hub devices. For example, user request 2036 may be received by both the thermostat 2024 associated with the media player 2004 and the smart speaker 2012 associated with the hub speaker 2002. As depicted, depending on the location of users 2030 and 2034 in the residence, these users may not have direct access to cellular-enabled devices to make telephone calls. Furthermore, each hub device can be assigned to more than one accessory device. The ability to pair accessory devices with cellular-enabled devices for calls via corresponding accessory interaction instances allows the hub device to continue managing the functionality of other accessory devices while maintaining data privacy regarding ongoing calls.

[0301] As a specific example of the aforementioned implementation scheme, consider the scenario where user 2030 issues call request 2032. The receiving accessory, smart speaker 2014, and smartwatch 2016 may lack cellular communication capabilities. Similarly, hub speaker 2002 may not be a cellular-enabled device. In some implementations, the accessory device may coordinate with other accessory devices within the residential environment 2000 to determine which accessory device should respond to a user request received by one or more accessory devices. For the purposes of this example, consider the scenario where smart speaker 2014 is selected to handle call request 2032. Upon receiving request 2032 and detecting a wake word, the smart speaker may transmit request 2032 to hub speaker 2002. Hub speaker 2002 may process call request 2032 and identify smartphone 2006 as the appropriate cellular-enabled device for performing the call. Hub speaker 2002 may then instruct smartphone 2006 to initiate the call and establish a communication channel with smart speaker 2014 for relaying call audio. Alternatively, in some implementations, hub speaker 2002 may instruct smart speaker 2014 to communicate with smartphone 2006 to establish a call. Then, an accessory interaction instance associated with smart speaker 2014 at hub speaker 2002 may enter call monitoring mode and listen for user 2030 to say a closing word at smart speaker 2014.

[0302] Continuing the example above, consider a scenario where, during the time user 2030 is making a phone call at smart speaker 2014, user 2034 issues user request 2036 to inquire about the current time (e.g., “Computer, what time is it?”). Request 2036 can be received by smart speaker 2012 associated with hub speaker 2002. Although the accessory interaction instance associated with smart speaker 2014 at hub speaker 2002 is in a call listening state, the accessory interaction instance associated with smart speaker 2012 is not in a restricted state and can process request 2036 normally, relaying the response to smart speaker 2012 (e.g., “It is 10:30 PM”).

[0303] Figure 21 This is another simplified block diagram illustrating a process 2100 for establishing communication between an accessory device and a cellular-enabled device according to some embodiments. In some embodiments, accessory device 2102 may correspond to the accessory device described herein (e.g., Figure 19(The accessory device 1912). Similarly, controller device 2104 may correspond to other controller devices or hub devices, and cellular-enabled device 2106 may correspond to other cellular-enabled devices as described herein. Controller device 2101 (described herein as a hub speaker) may be a hub device, leader device, configuration device, or other device used to determine the appropriate cellular-enabled device 2106 to initiate a call and direct accessory device 2102 to connect to cellular-enabled device 2106. Controller device 2101 may be configured to communicate with other hub devices, accessory devices 2102, other accessories, and cellular-enabled devices 2106 via one or more networks (including LANs or WANs) described herein.

[0304] The connection process 2100 is presented in more detail for several elements. The accessory device 2102 may include an ADK 2108. The ADK 2108 may be a software development kit (“SDK”) stored on the accessory device 2102 and configured to be executed or processed on that accessory device. As used herein, the SDK may include application programming interfaces and associated software libraries sufficient to implement the operation of other software within or associated with the SDK. In some embodiments, the ADK 2108 may be provided by an entity associated with the controller device 2104 (e.g., its manufacturer). The ADK 2108 may include a telephone audio module 2110 and a telephone control module 2112. The telephone audio module 2110 may establish a real-time audio connection with the cellular-enabled device 2106 to send and receive audio during a telephone call. The telephone control module 2112 may send instructions and indications corresponding to device control of the telephone connection to the cellular-enabled device 2106 and receive instructions and indications corresponding to device control of the telephone connection from the cellular-enabled device. In some implementations, the telephone control module 2112 may send a signal to the cellular-enabled device 2106 to terminate the cellular connection and thus end the call when it receives a hang-up command from the controller device 2104.

[0305] Controller device 2104 may include accessory management module 2114, which may be a software process running on controller device 2104. In some embodiments, accessory management module 2114 may receive, process, store, update, and transmit accessory management settings. For a particular user device, its accessory management settings may include a list of all accessories assigned to that controller device and other information relating to the capabilities of those assigned accessories. The accessory management module may include user profiles 2116. These user profiles 2116 may correspond to one or more users within a residential environment and may contain information associating each user with one or more cellular-enabled devices (including cellular-enabled device 2106). User profiles 2116 may also include information identifying one or more contacts or other information that may be used by controller device 2104 in response to a call request and to guide the establishment of a call. Accessory management module 2120 may also include accessory interaction instances 2118. Accessory interaction instances 2118 may be created by controller device 2104 for each accessory assigned to controller device 2104. Each accessory interaction instance in accessory interaction instances 2118 may represent one or more different software ecosystems on the controller device. For example, an accessory interaction instance corresponding to accessory device 2102 can represent a first software ecosystem of the controller device, while another accessory interaction instance corresponding to another accessory device can represent a second software ecosystem of the controller device.

[0306] The cellular-enabled device 2106 may include a media module 2120 and a telephone control module 2122. In some embodiments, the media module 2120 may transmit and receive media data, including telephone audio, through one or more cellular networks to which the cellular-enabled device 2106 can connect. The media module 2120 may also be connected to the accessory device 2106 via a real-time audio channel or other channels through which the cellular-enabled device 2106 may transmit and receive audio data corresponding to the telephone audio. The telephone control module 2122 may send instructions and instructions corresponding to device control connected to the telephone to the accessory device 2102 and receive instructions and instructions corresponding to device control connected to the telephone from the accessory device. In some embodiments, upon receiving an instruction from the controller device 2104, the telephone control module 2122 may initiate a telephone call by dialing a selected telephone number or by accessing contact information locally stored on the cellular-enabled device and dialing a telephone number associated with that contact information.

[0307] Finish Figure 21Detailed elements of the process indicators 2130, 2140, and 2150 respectively represent data transmission between the accessory device 2102 and the controller device 2104, between the controller device 2104 and the cellular-enabled device 2106, and between the accessory device 2102 and the cellular-enabled device 2106. Process indicators 2130, 2140, and 2150 can indicate communication between various devices as described herein over one or more networks, including but not limited to WiFi LAN or Internet WAN. In some embodiments, process indicator 2130 indicates the transmission of data corresponding to a subsequent response to a user request initiating a call at the accessory device 2102 and an indication that the request has been successfully processed from the controller device 2104. Similarly, process indicator 2140 indicates the transmission of data including instructions to the cellular-enabled device 2106 to initiate a call in response to a user request. In some embodiments, the cellular-enabled device 2106 does not transmit any corresponding data or information back to the controller device 2104. The process indicator 2150 indicates the transmission of data including real-time audio of a telephone call and telephone control instructions or indications, which include instructions to establish or terminate a call according to some implementation scheme.

[0308] Figure 22 This is a simplified block diagram 2200 illustrating at least some technologies used for communication between an accessory device 2201 and a cellular-enabled device 2203 to initiate a telephone call at the cellular-enabled device 2203. Diagram 2200 includes some detailed architecture of representative devices and process flow arrows providing general indications of the transfer of data or information. These process flow arrows are not intended to suggest any specific architectural connection between the elements detailed herein. Figure 22 Each element depicted herein may resemble one or more elements depicted in other figures herein. For example, accessory device 2201 may correspond to one or more of the accessories and accessory devices described herein, etc. In some embodiments, at least some of the elements in Figure 2200 may be similar to... Figure 20 Residential environment 2000 Contextual operation of residential environment.

[0309] Turning more specifically to each element, accessory device 2201 may have audio input and output capabilities, including accessory audio input / output (“I / O”) 2204. Accessory audio I / O 2204 may include both the hardware (e.g., speakers and microphones) and software / firmware necessary to provide audio input and output capabilities. Accessory device 2201 also includes ADK 2206. ADK 2206 may be similar to the above description relative to… Figure 21The ADK 2108 is described above. The ADK may include a wake-word detection module 2208, an audio module 2210, and a telephone control module 2212. The wake-word detection module 2208 can perform first processing on a portion of the audio input corresponding to a trigger or wake-word. The wake-word detection module itself may contain information about the wake-word and the trigger, including, for example, trigger criteria and an audio pattern corresponding to a specific wake-word. The audio module 2210 and the telephone control module 2212 may be similar to those described in reference [reference missing]. Figure 21 The aforementioned telephone audio module 2110 and telephone control module 2112. As shown by the process flow arrows, audio input received at the accessory device 2201 can be processed by the wake-word detection module 2208. If a wake-word is detected, the accessory device can transmit the audio input to the controller device 2202 for further processing. The accessory device 2201 can continue to listen for wake-words during a telephone call. Because the user will speak during the call, and the spoken audio is transmitted from the audio module 2210 to the cellular-enabled device 2203, the wake-word detection module 2208 allows the accessory device to distinguish the user's telephone audio from audio intended to convey requests or commands to the accessory device.

[0310] The controller device may include a voice processing module 2214, which includes a wake-word detection module 2216. For example... Figure 22 As depicted, the wake word detection module 2216 may have instances corresponding to each of the accessory interaction instances 2218, such that each instance of the wake word detection module 2216 is part of a separate software ecosystem at the controller device. The wake word detection module 2216 may process the wake word audio at a second level where the presence of a wake word can be confirmed with a higher probability than the wake word detection module 2208 at the accessory device 2201. If the speech processing module 2214 does not detect a wake word, the controller device 2202 may ignore the audio input. If a wake word is detected, the audio input may be further processed at the accessory interaction instance 2218.

[0311] Attachment interaction instance 2218 may include virtual device assistant 2220. During normal operation, controller device 2202 may process audio from user requests or other audio input via wake word detection module 2216 by connecting attachment interaction instance 2218 to a remote service and transmitting a portion of the audio input to the remote service. NLP and other services used to process the audio input may include the remote service. However, during a call, the attachment interaction instance corresponding to attachment device 2201 may be in a call listening state. In this state, device assistant 2220 may not process any other user audio except for the end word at end word detection module 2222. In some embodiments, the end word detection module may operate entirely within the corresponding attachment interaction instance, such that device assistant 2220 does not transmit any telephone call audio to a remote service or other device if wake word detection module 2216 indicates that a wake word has been heard. In other embodiments, device assistant 2220 may process only a limited portion of the audio input received after the wake word is detected. For example, end word detection may process a short audio portion sufficient to cover the end words "hang up" and "end call". If an end-of-call word is detected, the attached interaction instance can transmit a hang-up command 2224 to the attached device 2201. The attached device 2201 can then signal the cellular-enabled device 2203 to terminate the call and the audio between the devices.

[0312] The cellular-enabled device 2203 may include a media module 2230 configured to transmit, receive, and process audio and video data. The media module may include a call module 2232. The call module 2232 is configured to transmit, receive, and process audio data for telephone calls made via a cellular network 2250 to which the cellular-enabled device is connected. The call module 2232 can transmit audio data to and receive audio data from the accessory device via an audio channel 2226, which may be a real-time audio channel using RTP or a similar communication protocol. The cellular-enabled device 2203 may also include a call service module 2234 configured to perform processes including negotiating telephone calls (e.g., making an outgoing call) and receiving call instructions (e.g., terminating a call) from the accessory device. The call service module 2234 may include a telephone control module 2236 and an accessory discovery module 2238. The telephone control module 2236 may be similar to... Figure 21The telephone control module 2122. The accessory discovery module 2238 allows the cellular-enabled device 2203 to discover and negotiate communication channels with accessory devices within the residential environment. For example, in some embodiments, the cellular-enabled device 2203 may receive instructions from the controller device 2202 to initiate a call and connect to the accessory device 2201. The accessory discovery module 2238 can locate the accessory device 2201 within a network of the residential environment and, in conjunction with the telephone control module 2236, establish a communication connection with the accessory device 2201, including call control channel 2228.

[0313] Figure 23 This is a simplified block diagram 2300 illustrating an exemplary architecture of a system for establishing communication between an accessory device and a cellular-enabled device according to an embodiment. The diagram 2300 includes a controller device 2302, one or more accessory devices 2304, a representative accessory device 2306, one or more networks 2308, a cellular-enabled device 2310, and a cellular network 2312. Figure 23 Each of these elements depicted may be similar to one or more elements depicted in other figures described herein. In some embodiments, at least some of the elements of Figure 2300 may be in a residential environment (e.g., Figure 20 Operating within the context of the residential environment (2000).

[0314] Accessory device 2304 and representative accessory device 2306 can be any suitable computing device (e.g., smart speaker, smartwatch, smart thermostat, camera, etc.). In some embodiments, accessory devices 2304 and 2306 can perform any one or more of the operations of the accessory devices described herein. Depending on the type of accessory device and / or the location of the accessory device (e.g., within or outside a residential environment), accessory device 2306 may be enabled to communicate via network 2308 (e.g., including LAN or WAN) using one or more network protocols (e.g., Bluetooth connectivity, Thread connectivity, ZigBee connectivity, WiFi connectivity, etc.) and network paths, as further described herein.

[0315] In some implementations, controller device 2302 may correspond to any one or more of the controller device or hub device described herein. For example, controller device 2302 may correspond to Figure 20 One or more hub devices in a residential environment 2000. The controller device can be any suitable computing device (e.g., mobile phone, tablet, smart hub speaker device, smart media player communicatively connected to TV, etc.).

[0316] In some implementations, one or more networks 2308 may include an Internet WAN and a LAN. As described herein, a residential environment may be associated with a LAN, wherein devices within the residential environment can communicate with each other via the LAN. As described herein, a WAN may be outside the residential environment. For example, a router associated with a LAN (and therefore with the residential environment) may enable traffic from the LAN to be routed to the WAN, and vice versa.

[0317] As described herein, controller device 2302 may represent one or more controller devices or hub devices connected to one or more networks in network 2308. Controller device 2302 has at least one memory 2314, a communication interface 2316, one or more processing units (or processors) 2318, a storage unit 2320, and one or more input / output (I / O) devices 2322.

[0318] Turning further to each element of controller device 2302 in greater detail, processor 2318 may be implemented, as appropriate, in hardware, computer-executable instructions, firmware, or a combination thereof. The specific implementation of the computer-executable instructions or firmware of processor 2318 includes computer-executable instructions or machine-executable instructions written in any suitable programming language to perform the various functions described.

[0319] Memory 2314 may store program instructions that can be loaded and executed on processor 2318, as well as data generated during the execution of these programs. Depending on the configuration and type of controller device 2302, memory 2314 may be volatile (such as random access memory (“RAM”)) or non-volatile (such as read-only memory (“ROM”), flash memory, etc.). In some implementations, memory 2314 may include various types of memory, such as static random access memory (“SRAM”), dynamic random access memory (“DRAM”), or ROM. Controller device 2302 may also include additional storage devices 2320, such as removable or non-removable storage devices, including but not limited to magnetic storage devices, optical disk and / or magnetic tape storage devices. Disk drives and their associated computer-readable media may provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data to the computing device. In some implementations, storage device 2320 may be used to store data received from one or more other devices (e.g., other controller devices, cellular-enabled device 2310, accessory device 2304, or representative accessory device 2306).

[0320] The controller device 2302 may also include a communication interface 2316, which allows the controller device 2302 to communicate with a storage database, another computing device or server, a user terminal, or other devices via a network 2308. The controller device 2302 may also include I / O devices 2322, such as those for connecting to a keyboard, mouse, pen, voice input device, touch input device, display, speaker, printer, etc.

[0321] The memory 2314 may include an operating system 2324 and one or more applications or services for implementing the features disclosed herein, including a communication module 2326, a voice processing module 2328, and an accessory interaction instance 2330. The voice processing module 2328 also includes a wake-up word module 2332, and the accessory interaction instance 2330 also includes a digital assistant 2334 and a closing word module 2336.

[0322] Communication module 2326 may include code that causes processor 2318 to generate instructions and messages, transmit messages, or otherwise communicate with other entities. For example, communication module 2326 may, in conjunction with digital assistant 2334, transmit data associated with establishing a telephone call to accessory device 2306 and cellular-enabled device 2310, and receive data associated with establishing a telephone call from accessory device and cellular-enabled device. As described herein, communication module 2326 may transmit messages via one or more network paths of network 2308 (e.g., via a LAN or Internet WAN associated with a residential environment).

[0323] According to some embodiments, the voice processing module 2328 may include code that causes the processor 2318 to receive and process audio input corresponding to a dictated request to initiate or terminate a call. Processing the dictated audio may include, for example, NLP or audio pattern matching. In some embodiments, one or more operations of the voice processing module 2328 may be similar to those described in the references. Figure 22 The operations described in the voice processing module 2214. The wake word module 2332 may include code that causes the processor 2318 to receive and process a portion of the audio input corresponding to a trigger or wake word. In some embodiments, one or more operations of the wake word module 2332 may be similar to those described in the reference. Figure 22 The operations described in the wake word detection module 2216. For example, the wake word module 2332 may analyze a portion of the audio input to determine the presence of a wake word.

[0324] Attachment interaction instance 2330 may include code corresponding to a user request that causes processor 2318 to receive and process audio input. In some embodiments, one or more operations of attachment interaction instance 2330 may be similar to those described in the reference. Figure 22The operations described in Attachment Interaction Instance 2218 are as follows. For example, Attachment Interaction Instance 2330 may include multiple processes or services that enable processor 2318 to send and receive data to and from a remote service, identify an appropriate cellular-enabled device for initiating a call upon user request, or enter a call monitoring state that restricts the functionality of digital assistant 2334 while a call is in progress. Attachment Interaction Instance 2330 may include digital assistant 2334 that can perform one or more of these exemplary operations, as well as additional operations related to the interaction between attachment devices 2304, 2306 as described herein and cellular-enabled device 2310. Attachment Interaction Instance 2330 may also include end-of-call module 2336. When in call monitoring state, the Attachment Interaction Instance corresponding to the attachment device in a call may restrict the voice processing functionality of its digital assistant 2334 to only detecting end-of-call words (e.g., "hang up"). In those cases, end-of-call module 2336 may receive audio input after processing at wake-up word module 2332. The end-of-call module 2336 processes the audio input to detect the end-of-call word. In some implementations, the end-of-call module 2336 performs speech analysis similarly to the wake-up word module 2332. When the end-of-call word is detected, the digital assistant 2334, in conjunction with the communication module 2326, can send a command to the accessory device to end the call.

[0325] Turning now to the details of representative accessory device 2306, in some embodiments, accessory device 2306 may have at least one memory 2340, communication interface 2342, processor 2344, storage unit 2346, and I / O device 2348. As described herein with respect to controller device 2302, these elements of the accessory device may have the same suitable hardware implementation as their counterparts on controller device 2302.

[0326] The memory 2340 of the accessory device 2306 may include an operating system 2350 and one or more applications or services for implementing the features disclosed herein, including a communication module 2352, an audio module 2354, and an ADK 2356. As described herein with respect to the controller device 2302, the communication module 2352 may have suitable functionality similar to its corresponding communication module 2326.

[0327] The audio module 2354 may include code that enables the processor 2344 to receive, process, and transmit audio signals in conjunction with the I / O device 2348. In some embodiments, one or more operations of the audio module may be similar to those described in the reference. Figure 22The audio module 2354 can perform the operations described in the attached audio module 2210. For example, the audio module 2354 can receive user speech or other audio input at a microphone using I / O device 2348 and transmit the audio data to controller device 2302 via a streaming audio channel or other suitable connection. The audio input may correspond to a wake-up word combined with a closing word. The audio module 2354 can also receive user audio at a microphone and relay the audio to a cellular-enabled device as part of a telephone call. Similarly, the audio module can receive audio from a cellular-enabled device and play the audio at a speaker.

[0328] ADK 2356 may include code that causes processor 2344 to receive and process a portion of audio input corresponding to a trigger or wake-up word. In some embodiments, one or more operations of ADK 2356 may be similar to those described in the reference document. Figure 22 The operations described in ADK 2206. ADK 2356 may include a wake word module 2358. The wake word module 2358 may include code that causes the processor 2344 to receive and process a wake word. In some embodiments, one or more operations of the wake word module 2358 may be similar to those described in the reference. Figure 22 The operations described in the wake word detection module 2208. For example, the wake word module 2358 can analyze a portion of the audio input to determine the presence of a wake word.

[0329] In some implementations, ADK 2356 may also include a telephone control module 2360. The telephone control module 2360 may include code that causes the processor 2344 to send commands and instructions to and receive commands and instructions from the cellular-enabled device 2310. For example, upon receiving audio input containing an end-of-call word, the controller device 2302 may transmit an instruction to the accessory device 2306 to terminate the call. The accessory device 2306 may then signal via its telephone control module 2360 to the cellular-enabled device 2310 to terminate the call and close the audio connection between the two devices.

[0330] Now let's turn to the details of the cellular-enabled device 2310, which is similar to... Figure 23 Other device architectures illustrated in the diagram, in some embodiments, may include a cellular-enabled device 2310 having at least one memory 2362, a communication interface 2364, a processor 2366, a storage unit 2368, and I / O devices. As described herein with respect to controller device 2302, these elements of the accessory device may have the same suitable hardware implementation as their counterparts on controller device 2302.

[0331] The memory 2362 of the cellular-enabled device 2310 may include an operating system 2372 and one or more applications or services for implementing the features disclosed herein, including a media module 2374, a telephone control module 2376, and an accessory discovery module 2376. As described herein with respect to accessory device 2306, the telephone control module 2376 may have suitable functionality similar to its corresponding telephone control module 2360.

[0332] Media module 2374 may include code that causes processor 2366 to send, receive, and process data contained in a telephone call. This data can be received from and transmitted to a server device or other device connected to cellular network 2312. Media module 2374 may also send, receive, and process data corresponding to real-time audio of a telephone call sent to accessory device 2306 via one of the networks in network 2308. In some embodiments, one or more operations of media module 2374 may be similar to those described in reference to [reference missing]. Figure 22 The operations described in media module 2230.

[0333] The accessory discovery module 2376 may include code that enables the processor 2366 to receive information about a selected accessory device 2306 on one of the networks 2308 within the residential environment, to establish a communication channel from the accessory device 2306 for direct dialing a call at a cellular-enabled device 2310. In some embodiments, one or more operations of the accessory discovery module 2376 may be similar to those described in the references. Figure 22 The operations described in the attachment discovery module 2238.

[0334] Figure 24 This is a flowchart illustrating a specific exemplary process 2400 according to an embodiment for requesting a telephone call at Annex 2401 and initiating the telephone call at a cellular-enabled device 2403.

[0335] Figure 24 Each of the elements and operations depicted herein may be similar to one or more elements depicted in other figures described herein. For example, user equipment 2402 may be similar to other user equipment, controller equipment, or hub equipment, etc. In some embodiments, process 2400 may be in a residential environment (e.g., Figure 20 The residential environment (2000) shall be implemented.

[0336] Process 2400 and Figure 25 and Figure 26Processes 2500 and 2600 (described below) are shown as logic flowcharts, each operation of which represents a series of operations that can be implemented by hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the operations may be implemented in any order and / or in parallel combinations.

[0337] Furthermore, some, any, or all of these processes can be executed under the control of one or more computer systems configured to have executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that is executed on one or more processors, by hardware, or a combination thereof. As described above, the code can be stored on a computer-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable storage medium is non-transitory.

[0338] At box 2404, attachment 2401 can receive audio input containing a wake word and a call request. For example, the audio input could be the user's words "Computer, call Mom," where "Computer" includes the wake word and "call Mom" ​​includes the request.

[0339] At box 2406, appendix 2401 can process the wake word in the first pass to determine its presence. The first pass can determine when the wake word might exist and how resources are available. At decision 2408, based on the first pass, appendix 2401 determines whether the wake word exists. If not, the process can terminate at endpoint 2410 by ignoring the user's utterance. If, according to the first pass, the wake word exists, the process continues to box 2412.

[0340] At box 2412, accessory 2401 can transmit audio input to user equipment 2402 via a streaming audio connection. This connection can occur through one of the networks to which accessory 2401 and user equipment 2402 are connected (e.g., via WiFi LAN). Streaming audio can use any number of methods or protocols, including but not limited to AirPlay, Real-Time Transport Protocol (“RTP”), Real-Time Streaming Protocol (RTSP), etc.

[0341] At box 2414, user equipment 2402 receives a wake-up word and may process it a second time. This processing may occur at a second level where the presence of the wake-up word can be confirmed with a higher probability than the first processing at box 2406 at attachment 2401. At decision 2416, if user equipment 2402 does not confirm the presence of the wake-up word, the process moves to endpoint 2418 and ignores the audio input by terminating the streaming audio connection with attachment 2401. If user equipment 2402 confirms the presence of the wake-up word, the process moves to box 2420 and processes the call request. While processing the call request, user equipment 2402 may transmit some or all of the call request to a remote server device for speech analysis using NLP or other technologies. This analysis may determine the user who made the call request. When identifying a user, user equipment 2402 or the remote server device may access user profiles associated with the user in the residential environment. These user profiles may be stored at user equipment 2402 or at the remote server device or other devices accessible by the device performing the speech analysis. Processing a call request can ultimately lead to the identification of the call recipient (e.g., a mother). The call recipient can be a phone number associated with the name or other label spoken by the user making the call request. In some implementations, the call recipient can be identifiable by the cellular-enabled device 2403, such that the portion of the call request identifying the recipient is sent to the cellular-enabled device 2403 along with the instruction to initiate the call.

[0342] At box 2422, user equipment 2402 may determine an appropriate cellular-enabled device for initiating the call. This determination may be based on information about the requesting user obtained from a user profile. In some embodiments, the appropriate cellular-enabled device 2403 may be the user's personal cellular phone. At box 2424, user equipment 2402 may instruct the cellular-enabled device to establish the call. This may include sending information to the cellular-enabled device 2403 identifying the call recipient and identifying that the cellular-enabled device 2403 should connect to an accessory 2401 to relay call audio. Although Figure 24Not depicted, but in some embodiments, user equipment 2402 may optionally instruct attachment 2401 to communicate with a selected cellular-enabled device 2403 and instruct the cellular-enabled device 2403 to initiate a call. This scenario may occur in environments where user equipment 2402 can identify an appropriate cellular-enabled device 2403 but cannot communicate with it, for example, because the two devices do not have a trusted connection to each other. User equipment 2402 may then enter a call listening state at endpoint 2426. While in call listening state, the attachment interaction instance corresponding to attachment 2401 at user equipment 2402 may have limited voice processing regarding the analysis of user requests received from attachment 2401. Other attachment interaction instances corresponding to different attachments associated with user equipment 2402 may process user requests at those attachments normally.

[0343] At box 2428, cellular-enabled device 2403 can initiate a call to a call recipient via cellular network 2430. At box 2432, cellular-enabled device 2403 can establish an audio channel with attachment 2401. Attachment 2401 can then begin relaying audio to and from cellular-enabled device 2403 to form a telephone session. While a call is in progress, user equipment 2402 can remain in a listening state at box 2426 to listen for the "end" word to terminate the call. However, in other examples, attachment 2401 can be configured to also (or instead of user equipment 2402) be in a listening state to listen for the "end" word to terminate the call, or to listen for other instructions (e.g., potentially unrelated to the telephone call). Although both devices (e.g., attachment 2501 and user equipment 2502) are capable of listening for the "end" word, user equipment 2502 is better suited for this task due to the fact that the detector on attachment 2501 may be inferior to the detector on user equipment 2502.

[0344] Figure 25 This is another flowchart illustrating an exemplary process 2500 for requesting termination of a telephone call at an accessory device 2501 and ending the call at a cellular-enabled device 2503, according to an embodiment. Figure 25 Each of the elements and operations depicted herein may be similar to one or more elements depicted in other figures described herein. In some embodiments, process 2500 may be as described above. Figure 24 The process 2400 described herein continues. Therefore, Annex 2501 may correspond to Annex 2401, User Equipment 2502 may correspond to User Equipment 2402, and Cellular Supporting Device 2503 may correspond to Cellular Supporting Device 2403.

[0345] When a call is in progress between a cellular-enabled device 2503 and one of its associated accessories (e.g., accessory 2501), user equipment 2502 initiates process 2500 in call listening state 2504. In some embodiments, the call listening state may be similar to that for... Figure 24 The listening state described at endpoint 2426. Alternatively, annex 2501 may also be in a listening state (for both the wake word and / or the "end" word). At block 2506, annex 2501 may receive user audio input. This audio input may consist of a wake word and an end word, such as "computer, hang up". annex 2501 may process the wake word and transmit the audio input to user equipment 2502 for further processing in blocks 2508-2520. In some embodiments, one or more operations in blocks 2508–2520 may be similar to those described in the reference. Figure 24 For one or more operations described in blocks 2406–2418.

[0346] At box 2518, if user equipment 2502 detects a wake-up word, then at box 2522, the user equipment may proceed to process the end-of-call portion of the audio input. Processing of the end-of-call word may be limited to detecting and processing only specific end-of-call words (e.g., “hang up”) or a small set of equivalent words (e.g., “end call,” “end,” etc.). At decision 2524, if no end-of-call word is detected, user equipment 2502 will ignore the audio input, and the process will end at endpoint 2526. Because user equipment 2502 is in call listening mode (e.g., at box 2504), the user equipment will ignore all audio input that does not contain an end-of-call word, even if such input otherwise contains a valid and definitive request. In many embodiments, if no end-of-call word is detected in a portion sufficient to contain the desired end-of-call word, user equipment 2502 may not process any portion of the audio input beyond that sufficient portion. In this way, user equipment 2502 cannot listen to the call and / or record any portion of the call.

[0347] At box 2528, if an end-of-call word is detected, user equipment 2502 may instruct attachment 2501 to terminate the call. At box 2529, the attachment receives the instruction and communicates with cellular-enabled device 2503 to terminate the call. Alternatively, in some examples, user equipment 2502 may instead instruct cellular-enabled device 2503 to terminate the call at box 2528. In this example, box 2529 is skipped, and cellular-enabled device 2503 is instructed to terminate the call without going through attachment 2501. At box 2530, cellular-enabled device 2503 may terminate the call connection with cellular network 2532 and then close the communication channel with attachment 2501 at endpoint 2534.

[0348] Figure 26 This is a flowchart illustrating a process 2600 in which a controller device establishes a connection between an accessory device and a cellular-enabled device according to some embodiments. In some embodiments, one or more operations of process 2600 may be similar to those described in the referenced document. Figure 24 and Figure 25 The operations described above.

[0349] At block 2602, the controller device may establish a first network connection with a cellular-enabled device and a second network connection with an accessory device. The network connections may occur on one or more networks associated with a residential environment. With respect to the accessory device, the second network connection may be a network connection through which the controller device communicates with the accessory device in response to a user request at the accessory device. In some embodiments, one or more operations in the operation of block 2602 may be similar to those described in the references. Figure 21 One or more operations described for process indicators 2130 and 2140.

[0350] At box 2604, the controller device can listen for audio input from an accessory device via a second network connection. This listening behavior is typical of a controller device acting as a hub device for one or more associated accessories. When an accessory receives audio input containing a trigger or wake word, it can transmit the audio input to its associated hub device for further processing.

[0351] At box 2606, the controller device can receive audio input from an accessory device via a second network connection. The audio input may include a wake-up word and a call request. The call request may correspond to a request to make a telephone call to a cellular-enabled device. In some embodiments, one or more operations of box 2606 may be similar to those described in the reference section. Figure 24 For one or more operations described in block 2420.

[0352] At block 2608, upon receiving a call request, the controller device can transmit instructions to the cellular-enabled device to establish a third network connection with the auxiliary device and initiate a call via a first network connection used by the cellular-enabled device. In some embodiments, one or more operations in the operation of block 2608 may be similar to those described in the reference. Figure 24 For one or more operations described in block 2424.

[0353] At block 2610, the controller device can enter a call listening state. While in this state, the controller device can listen for a second audio input from an accessory device. In some embodiments, one or more operations of block 2610 may be similar to those described in the reference. Figure 25 For one or more operations described in blocks 2516 and 2522.

[0354] At block 2612, if the controller device recognizes an end word in the second audio input, the controller device may transmit a command to terminate the call and close the connection with the accessory device to the cellular-enabled device. In some embodiments, one or more operations in block 2612 may be similar to those described in the reference. Figure 21 One or more operations described for process indicator 2140.

[0355] The foregoing describes exemplary techniques for communication between an accessory device and a cellular-enabled device. Some or all of these techniques may be at least partially implemented through an architecture such as at least the above. Figures 19 to 8 The architectures shown herein can be used to implement, but are not required to implement, the techniques disclosed herein. While numerous implementations have been described above with reference to server devices, accessory devices, user devices, and hub devices, it should be understood that other types of computing devices may be suitable for implementing the techniques disclosed herein. Furthermore, various non-limiting examples have been described in the foregoing description. For purposes of explanation, many specific configurations and details have been set forth to provide a thorough understanding of the examples. However, it will also be apparent to those skilled in the art that some examples can be implemented without these specific details. In addition, well-known features have sometimes been omitted or simplified to prevent confusion with the examples described herein.

[0356] While specific exemplary embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also covered within the scope of this disclosure. The embodiments are not limited to operating within certain specific data processing environments, but are freely operable within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, it will be apparent to those skilled in the art that the scope of this disclosure is not limited to said series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0357] As described above, one aspect of this technology involves collecting and using data from specific and legitimate sources to improve the delivery of inspirational or other content that may be of interest to users when updating firmware. This disclosure envisions that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, and exercise information), date of birth, or any other personal information.

[0358] This disclosure recognizes that the use of such personal information data in the present invention can benefit users. For example, personal information data can be used to deliver targeted content that a user may be interested in based on their preferences. Therefore, the use of such personal information data enables users to have greater control over the delivered content. Furthermore, this disclosure also anticipates other uses of personal information data that benefit users.

[0359] This disclosure assumes that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, it is expected that such entities will implement and consistently apply privacy practices generally recognized as meeting or exceeding industry or governmental requirements for protecting user privacy. Such information regarding the use of personal data should be highlighted and easily accessible to users, and should be updated as data collection and / or use change. Users' personal information should be collected only for lawful use. Furthermore, such collection / sharing should only occur after receiving user consent or other lawful grounds provided for in applicable law. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and processes. Additionally, such entities may be subject to third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be tailored to the specific types of personal information data collected and / or accessed, and made applicable to applicable laws and standards, including jurisdiction-specific considerations that may be used to impose higher standards. For example, in the United States, the collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0360] Regardless of the foregoing, this disclosure also anticipates implementation schemes for users to selectively prevent the use or access to personal information data. That is, this disclosure anticipates providing hardware and / or software components to prevent or block access to such personal information data. For example, in the case of advertising delivery services, the technology can be configured to allow users to opt-in or opt-out at any time during or after service registration to participate in the collection of personal information data. In another example, users can choose not to provide emotion-related data for targeted content delivery services. Furthermore, users can choose to limit the duration for which emotion-related data is retained, or completely prevent the development of underlying emotional states. In addition to providing opt-in and opt-out options, this disclosure also envisions providing notifications related to access to or use of personal information. For example, users can be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.

[0361] Furthermore, the purpose of this disclosure is to manage and process personal information data to minimize the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by limiting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods such as differentiated privacy.

[0362] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not be rendered inoperable due to the absence of all or part of such personal information data. For example, content can be selected and delivered to the user based on aggregated non-personal information data or an absolute minimum amount of personal information, such as content processed only on the user's device or other non-personal information that can be used for content delivery services.

[0363] Various implementation schemes can be implemented in a wide variety of operating environments, in some cases of which may include one or more user computers, computing devices, or processing devices that can be used to operate any of a number of applications. User devices or client devices may include any of a variety of different types of computers, such as desktop or laptop computers running standard operating systems, and cellular, wireless, and handheld devices running mobile software and capable of supporting multiple networking and instant messaging protocols. The system may also include multiple workstations running any of a variety of commercially available operating systems and other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and other devices capable of communicating via a network.

[0364] Most implementations utilize at least one network familiar to those skilled in the art to support communication using any of the various commercial protocols such as TCP / IP, OSI, FTP, UPnP, NFS, CIFS, and AppleTalk. The network can be, for example, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), the Internet, an intranet, an extranet, the public switched telephone network (PSTN), an infrared network, a wireless network, or any combination thereof.

[0365] In implementations utilizing a web server, the web server can run any of a variety of server or middleware applications, including HTTP servers, FTP servers, CGI servers, data servers, Java servers, and business application servers. One or more servers may also be able to execute programs or scripts in response to requests from user devices, such as by executing one or more applications, which can be implemented in any programming language such as... One or more scripts or programs written in C, C#, or C++, or any scripting language such as Perl, Python, or TCL, and combinations thereof. The server may also include a database server, including but not limited to those that can access databases from... and Those obtained through commercial purchases.

[0366] The environment may include various data repositories and other storage media, as described above. These may reside in various locations, such as on storage media local to one or more computers or on storage media of any or all computers on a network (and / or reside within one or more computers). In a particular set of embodiments, information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to a computer, server, or other network device may be stored locally and / or remotely as needed. When the system includes computerized devices, each such device may include hardware elements electrically coupled via a bus, including, for example, at least one central processing unit (CPU), at least one input device (e.g., mouse, keyboard, controller, touchscreen, or keypad), and at least one output device (e.g., display device, printer, or speaker). Such systems may also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices such as RAM or ROM, as well as removable media devices, memory cards, flash memory cards, and so on.

[0367] Such devices may also include computer-readable storage medium readers, communication devices (e.g., modems, network interface cards (wireless or wired), infrared communication devices, etc.), and working memory as described above. Computer-readable storage medium readers may be connected to or configured to receive non-transitory computer-readable storage media representing remote, local, fixed, and / or removable storage devices, as well as storage media for temporarily and / or more permanently containing, storing, transmitting, and retrieving computer-readable information. Systems and various devices will also typically include multiple software applications, modules, services, or other elements, including operating systems and applications such as client applications or browsers, residing within at least one working memory device. It should be understood that alternative embodiments may have many variations as described above. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software such as applets), or both. Furthermore, connections to other computing devices, such as network input / output devices, may be used.

[0368] Non-transitory storage media and computer-readable storage media used for portions containing code may include any suitable media known or used in the art, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data), including RAM, ROM, electrically erasable programmable read-only memory (“EEPROM”), flash memory or other memory technologies, CD-ROM, DVD or other optical memory, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other media that can be used to store desired information and is accessible by system devices. Other ways and / or methods for implementing various embodiments will be recognized by those skilled in the art, at least in part, based on the disclosure and teachings provided herein. However, computer-readable storage media do not include transient media such as carrier waves.

[0369] Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of this disclosure as set forth in the claims.

[0370] Other variations are within the scope of this disclosure. Therefore, although the disclosed technology is susceptible to various modifications and alternative constructions, certain exemplary embodiments are shown in the accompanying drawings and have been described in detail above. However, it should be understood that this disclosure is not intended to be limited to the specific forms disclosed, but rather is intended to cover all modifications, alternative constructions, and equivalents falling within the scope and spirit of this disclosure as defined by the appended claims.

[0371] In the context of describing the disclosed embodiments (particularly in the context of the claims below), the terms “a,” “an,” and “the,” as well as similar indicator words, shall be construed to cover both singular and plural forms, unless otherwise stated or clearly contradicted by the context. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” shall be construed as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” is construed as including, attaching, or joining together, even with intervening elements. The phrase “at least partially based on” shall be understood as open-ended and not in any way limiting, and is intended to be construed or otherwise understood as “at least partially based on” where appropriate. Unless otherwise stated herein, the description of numerical ranges herein is intended merely as a simple way of referring separately to each individual value falling within that range, and each individual value is incorporated into the specification as if separately referenced herein. All methods described herein can be performed in any suitable order, unless otherwise stated or clearly contradicted by the context. Unless otherwise stated, the use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended merely to better illustrate embodiments of this disclosure and does not limit the scope of this disclosure. No language in the specification should be construed as indicating that any unstated element is essential to the practice of this disclosure.

[0372] Unless otherwise specifically stated, parse languages ​​such as the phrase "at least one of X, Y, or Z" are understood in context to generally refer to items, terms, etc., which can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such parse languages ​​are generally not intended and should not imply that certain embodiments require the existence of at least one of X, at least one of Y, or at least one of Z. Additionally, unless otherwise specifically stated, union languages ​​such as the phrase "at least one of X, Y, and Z" should also be understood to mean X, Y, Z, or any combination thereof, including "X, Y, and / or Z".

[0373] This document describes preferred embodiments of the present disclosure, including the best mode. After reading the foregoing description, variations of those preferred embodiments will become apparent to those skilled in the art. It is expected that those skilled in the art will be able to adopt such variations as appropriate and intend to practice this disclosure in ways different from those specifically described herein. Therefore, this disclosure includes all modifications and equivalents of the subject matter recited in the appended claims, as permitted by applicable law. Furthermore, unless otherwise indicated herein or clearly contradicted by the context, this disclosure encompasses any combination of all possible variations of the foregoing elements.

[0374] Examples of the aforementioned technology can be illustrated by the following clauses:

[0375] Clause 1. A method comprising:

[0376] The controller device establishes the first network connection with the cellular-enabled device and with the accessories.

[0377] The device's second network connection;

[0378] The controller device listens for the first audio input from the accessory device via the second network connection;

[0379] During monitoring, the first audio input, including at least a request to initiate a telephone call, is identified from the accessory device via the second network connection;

[0380] Transmit instructions for establishing a call between the cellular support device and the accessory device via a third network connection to the cellular support device via the first network connection or to the accessory device via the second network connection;

[0381] During the call between the cellular-enabled device and the accessory device, a second audio input from the accessory device is monitored via the second network connection;

[0382] Based on the determination that the second audio input is recognized, an instruction to terminate the call with the accessory device is transmitted via the first network connection to the cellular-enabled device or via the second network connection to the accessory device.

[0383] Clause 2. The method according to Clause 1, wherein the second audio input includes an end word configured to recognize that the call will be terminated.

[0384] Clause 3. The method according to Clause 2, wherein during the call, the controller device only listens for the end word.

[0385] Clause 4. The method according to Clause 2, wherein during the call, the controller device does not process any audio received from the accessory device via the second network connection, except for the closing word.

[0386] Clause 5. The method described in Clause 2, wherein the terminator includes at least one of “hang up” or “end call”.

[0387] Clause 6. The method described in Clause 1 further includes:

[0388] Based on the determination that the second audio input is recognized, information is transmitted to the accessory device via the second network connection, and the information notifies the accessory device that the call will be terminated.

[0389] Clause 7. The method described in Clause 1, wherein the first audio input further includes at least a wake word.

[0390] Clause 8. The method according to Clause 2, wherein the wake word instructs a second portion of the first audio input to identify an action to be performed by the controller device.

[0391] Clause 9. The method described in Clause 8, wherein the action includes transmitting the instructions for establishing the call via the third network connection.

[0392] Clause 10. The method according to Clause 2, wherein the wake word corresponds to a first software ecosystem of the controller device and not to a second software ecosystem of the accessory device.

[0393] Clause 11. The method of Clause 1, wherein the accessory device is a third-party device built for a first entity, which is different from the second entity for which the controller device is built.

[0394] Clause 12. The method according to Clause 11, wherein the accessory device is configured to implement a software development kit provided by the second entity associated with the controller device.

[0395] Clause 13. The method described in Clause 1, wherein the accessory device does not support cellular functionality.

[0396] Clause 14. The method of Clause 1, wherein the accessory device is configured to have a speaker and a microphone.

[0397] Clause 15. The method of Clause 1, wherein the cellular-enabled device is configured to relay the call from the service provider to the accessory device.

[0398] Clause 16. The method according to Clause 1, wherein the controller device is one of a plurality of hub devices in a residential environment.

[0399] Clause 17. A controller device comprising:

[0400] Memory, the memory being configured to store computer-executable instructions; and

[0401] A processor configured to access the memory and execute the computer-executable instructions to perform at least the method described in any one of clauses 1 to 16.

[0402] Clause 18. A computer-readable storage medium configured to store computer-executable instructions that, when executed by a controller device, cause the controller device to perform the method according to any one of Clauses 1 to 16.

Claims

1. A method comprising: The user equipment receives information identifying multiple attachments configured to communicate with the user equipment; The user equipment implements a corresponding attachment interaction instance for each of the plurality of attachments, wherein the corresponding attachment interaction instances are configured to be executed separately from each other; Receive a first audio input from a first attachment among the plurality of attachments and a second audio input from a second attachment among the plurality of attachments; At least a portion of the first audio input is processed by the first attachment interaction instance in the corresponding attachment interaction instance; The first attachment interaction instance in the corresponding attachment interaction instance receives a first response from the server computer, the first response corresponding to the processed portion of the first audio input; as well as The user equipment transmits the first response to the first attachment among the plurality of attachments.

2. The method of claim 1, wherein the processing of the portion of the first audio input by the first attachment interaction instance in the corresponding attachment interaction instance includes: The first attachment interaction instance transmits the processed portion of the first audio input to the server computer.

3. The method according to claim 2, wherein the processing of the portion of the first audio input by the first attachment interaction instance in the corresponding attachment interaction instance further includes: The first attachment interaction instance delegates the action to one or more other processes.

4. The method of claim 3, wherein the one or more other processes include at least one of a music service or a voice communication service, and wherein the action includes instructions for the one or more other processes to provide audio content for the first attachment.

5. The method according to claim 1, further comprising: The user equipment determines whether at least a portion of the second audio input matches a wake word; as well as The user equipment determines whether the second attachment is authorized to interact with the second attachment interaction instance in the corresponding attachment interaction instance.

6. The method according to claim 5, further comprising: At least a portion of the second audio input is processed by the second attachment interaction instance in the corresponding attachment interaction instance according to at least one of the following: determining that the portion of the second audio input matches the wake word or determining that the second attachment is authorized to interact with the second attachment interaction instance in the corresponding attachment interaction instance.

7. The method of claim 1, wherein each of the plurality of attachments is configured to implement a software development kit provided by an entity associated with the user equipment.

8. The method of claim 7, wherein each of the corresponding attachment interaction instances is configured to communicate with the corresponding software development kit of the corresponding attachment in the plurality of attachments.

9. The method of claim 1, further comprising the user equipment managing respective accessory settings for each of the plurality of accessories.

10. The method of claim 1, wherein the user equipment is a first user equipment, and wherein the information includes at least one of the following: A request to connect to the first user equipment from at least one of the plurality of attachments; or Instructions from the second user equipment to connect the first user equipment to at least one of the plurality of attachments.

11. A user equipment, comprising: A memory configured to store computer-executable instructions; and A processor, configured to be connected to the memory and execute the computer-executable instructions to at least: Receive information identifying multiple attachments configured to communicate with the user equipment; Implement a corresponding attachment interaction instance for each of the plurality of attachments, wherein the corresponding attachment interaction instances are configured to be executed separately from each other; Receive a first audio input from a first attachment among the plurality of attachments and a second audio input from a second attachment among the plurality of attachments; At least a portion of the first audio input is processed by the first attachment interaction instance in the corresponding attachment interaction instance; The first attachment interaction instance in the corresponding attachment interaction instance receives a first response from the server computer, the first response corresponding to the processed portion of the first audio input; as well as The first response is transmitted to the first attachment among the plurality of attachments.

12. The user equipment of claim 11, wherein the processing of the portion of the first audio input by the first accessory interaction instance in the corresponding accessory interaction instance includes at least one of the following: The first accessory interaction instance transmits the processed portion of the first audio input to the server computer; or The first attachment interaction instance delegates the action to one or more other processes.

13. The user equipment of claim 11, wherein the processor is configured to execute the computer-executable instructions to at least: Determine whether at least a portion of the second audio input matches the wake word; and Determine whether the second attachment is authorized to interact with the second attachment interaction instance in the corresponding attachment interaction instance.

14. The user equipment of claim 13, wherein the processor is configured to execute the computer-executable instructions to at least: At least a portion of the second audio input is processed by the second attachment interaction instance in the corresponding attachment interaction instance according to at least one of the following: determining that the portion of the second audio input matches the wake word or determining that the second attachment is authorized to interact with the second attachment interaction instance in the corresponding attachment interaction instance.

15. The user equipment of claim 11, wherein each of the plurality of accessories is configured to implement a software development kit provided by an entity associated with the user equipment.

16. A computer-readable storage medium configured to store computer-executable instructions, which, when executed by a user equipment, cause the user equipment to perform an operation, the operation comprising: Receive information identifying multiple attachments configured to communicate with the user equipment; Implement a corresponding attachment interaction instance for each of the plurality of attachments, wherein the corresponding attachment interaction instances are configured to be executed separately from each other; Receive a first audio input from a first attachment among the plurality of attachments and a second audio input from a second attachment among the plurality of attachments; At least a portion of the first audio input is processed by the first attachment interaction instance in the corresponding attachment interaction instance; The first attachment interaction instance in the corresponding attachment interaction instance receives a first response from the server computer, the first response corresponding to the processed portion of the first audio input; as well as The first response is transmitted to the first attachment among the plurality of attachments.

17. The computer-readable storage medium of claim 16, wherein the processing of the portion of the first audio input by the first accessory interaction instance in the respective accessory interaction instance comprises at least one of the following: The first accessory interaction instance transmits the processed portion of the first audio input to the server computer; or The first attachment interaction instance delegates the action to one or more other processes.

18. The computer-readable storage medium of claim 16, wherein the operation further comprises: The user equipment determines whether at least a portion of the second audio input matches a wake word; as well as The user equipment determines whether the second attachment is authorized to interact with the second attachment interaction instance in the corresponding attachment interaction instance.

19. The computer-readable storage medium of claim 18, wherein the operation further comprises: At least a portion of the second audio input is processed by the second attachment interaction instance in the corresponding attachment interaction instance according to at least one of the following: determining that the portion of the second audio input matches the wake word or determining that the second attachment is authorized to interact with the second attachment interaction instance in the corresponding attachment interaction instance.

20. The computer-readable storage medium of claim 16, wherein each of the plurality of attachments is configured to implement a software development kit provided by an entity associated with the user equipment.

Images