User interaction and task management using multiple devices

Abstract

The present disclosure provides systems and methods for user interaction and task completion using multiple devices. A group of devices can be used to perform a task so that different devices can perform different steps of the task. A device management service can update the status information of each device in the group, so that a user can interact with any computing device to perform a task. The device management service can also automatically determine which device a user should use based on task or step requirements, device characteristics, and device capabilities, etc. Therefore, the user does not need to continue to perform a task on the same device (even if the device is not suitable for the current step or task), but instead provides the user with options to use different devices in the group, and automatically switches to using different devices in the group in some instances.

Description

Background Art

[0001] As computing devices increase in prominence and diversity, users may have multiple devices that can be used to perform tasks. However, some computing devices may be better suited for certain tasks than others. Furthermore, in some cases, a computing device that is well suited for one step of a task may not be well suited for another step of the task.

[0002] It is with respect to these and other general considerations that the various aspects disclosed herein are made.In addition, although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving specific problems identified elsewhere in the background or this disclosure. Summary of the invention

[0003] The present disclosure provides systems and methods for user interaction and task completion using multiple devices. In an example, a task consists of one or more steps. Therefore, a group of devices can be used to perform a task so that different devices can perform different steps. A device management service can update the state information of each device in the group, so that a user can interact with any computing device when performing the steps of a task. For example, information entered or changed by a user on one device can be used on other devices, so that the user can complete various aspects of a task on any other device.

[0004] The device management service may also automatically determine which device the user should use based on the task or step requirements, device characteristics, device capabilities, etc. As a result, the computing device that receives the initial user input associated with a step may not be the computing device that is used to interact with the user while the user performs the step. Thus, the user need not use only the same device to complete a task (even if the device is not appropriate for the current step or task), but rather the user is also provided with the option to use a different device within the group, and in some instances, automatically transition to using a different device within the group.

[0005] The present summary is provided to introduce a selection of concepts that are further described below in the detailed description in a simplified form. The present summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the subsequent description and will become apparent in part from the description, or may be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0007] Figure 1 An overview of an example system for user interaction and task completion using multiple devices is illustrated.

[0008] Figure 2A An overview of an example method for managing task sessions using multiple devices is illustrated.

[0009] Figure 2B An overview of an example method for selecting devices used to perform steps of a task during a session is illustrated.

[0010] Figure 3A An overview of an example process flow for processing user input associated with a session using multiple devices is illustrated.

[0011] Figure 3B An overview of an example process flow for processing user input associated with a session using multiple devices is illustrated, wherein the user input is processed by a different computing device than the one that originally received the user input.

[0012] Figure 3C An overview of an example process flow for communicating session state information in an example session with multiple devices is illustrated.

[0013] Figure 3D An overview of an example process flow for communicating session state information in an example session with multiple devices is illustrated.

[0014] Figure 4 is a block diagram illustrating example physical components of a computing device that can be used to practice aspects of the present disclosure.

[0015] Figure 5A and Figure 5B is a simplified block diagram of a mobile computing device that can be used to practice aspects of the present disclosure.

[0016] Figure 6 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced.

[0017] Figure 7 A tablet computing device for performing one or more aspects of the present disclosure is illustrated. DETAILED DESCRIPTION

[0018] Various aspects of the present disclosure are described more fully below with reference to the accompanying drawings, which form a part of the present disclosure and illustrate specific example aspects. However, different aspects of the present disclosure can be implemented in many different forms and should not be construed as being limited to the aspects set forth herein; on the contrary, these aspects are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of these aspects to those skilled in the art. Various aspects can be practiced as methods, systems, or devices. Therefore, various aspects can take the form of hardware implementation, complete software implementation, or implementation combining software and hardware aspects. Therefore, the following detailed description should not be considered as limiting.

[0019] Traditionally, a user uses a single computing device to initiate and then perform a task to completion. For example, a user may use a voice command computing device to converse with a voice assistant to obtain information, send a message, set a timer, and so on. During the conversation, the user may answer voice prompts and receive information from the voice assistant in the form of auditory responses. As another example, a user may use a laptop computing device or a tablet computing device to interact with graphical elements while performing a task, such as selecting seats for a movie showing, making reservations for dinner at a geographically convenient and / or well-reviewed restaurant, or purchasing an item from an online retailer.

[0020] However, given that voice input has traditionally been used as the primary or sole input method for controlling voice command computing devices, voice command computing devices may not be well suited to perform tasks that are typically performed using computing devices with displays, touch screens, mice, and / or keyboards. As a result, performing such tasks using voice command computing devices may be frustrating, difficult, and time consuming. Conversely, there may be tasks that are better suited for voice command computing devices, such that using a voice command computing device can avoid user frustration, reduce task complexity, or reduce task duration. In other examples, a subset of steps associated with a task may be better suited for one device, while another subset of steps may be better suited for another device, due to different device characteristics and capabilities, etc. It should be understood that the above combinations of computing devices are provided as examples, and arrangements of devices with different capabilities may be used without departing from the aspects described herein.

[0021] Therefore, the present disclosure provides a system and method for user interaction and task completion using multiple devices. As used herein, a task consists of one or more steps. A task may include a set of discrete steps, or in other examples, the steps of a task may be implicit or inferred. In an example, a group of devices are used to perform a task, wherein a device management service evaluates a given step of a task to determine which device or devices should be used to interact with a user. For example, the device management service evaluates various factors, including but not limited to task or step capability requirements, device characteristics, device capabilities, proximity of the device to the user, user preferences, historical device use or success rate for performing similar steps, and / or the ease of completing steps using a given device. Additionally, when a user interacts with one of the devices in a group, the device management service may generate a status information update provided to each of the other devices. For example, information entered or changed by a user on one device may be used for other devices, thereby enabling the user to perform different aspects of a task on any other device. Therefore, the user does not need to use only the same device to complete a task (even if the device is not suitable for the current step or task), but also provides the user with options to use different devices within the group, and in some instances, automatically converts to using different devices within the group. In addition to using different computing devices, it should be understood that any number of devices can be used to complete a step or task. As an example, the device management service can select a group of devices with the capabilities required to complete a task so that the group of devices aggregate to provide the necessary capabilities, even if no one device has all the necessary capabilities.

[0022] In the example, a group of devices are associated using a task session. The task session can be initialized when the user starts a task and can associate the group of devices when the user completes the task. As an example, each device in the group can be associated with an identifier so that the device management service can identify the devices in the group accordingly. Example identifiers include, but are not limited to, user account identifiers (e.g., user names, email addresses, accounts, etc.), one or more Internet Protocol (IP) or Media Access Control (MAC) addresses, a group of device hardware characteristics and / or device capabilities, a group of device names (e.g., can be defined by the user, automatically generated, etc.) or a globally unique identifier (GUID). It should be understood that the identifier can be pre-associated with the device or can be associated with the device as part of the initialization process. For example, a quick response (QR) code can be used to provide an identifier to the device, which should be part of the group during the initialization process. In addition to or as an alternative to the techniques described herein, other techniques can be used to determine the group of devices, including but not limited to radio device detection (e.g., Bluetooth, Wi-Fi, etc.), network broadcasting / scanning, or using audio and / or light signals. Although the examples are described in the context of session initialization, it should be understood that similar techniques can be used to add devices to existing sessions.

[0023] Before the task session is initialized, the device can provide pre-session context information to the device management service. The device management service can use such information when initializing the session or during the session, etc. For example, context information about a web page accessed by a computing device or a skill accessed by a voice command computing device can be provided to the device management service. Example context information includes, but is not limited to (e.g., web page, skill, etc.) title, description, uniform resource locator (URL) associated with the accessed content, and / or data provided to or received from the user. As a result, when the session is initialized, the device management service can use the context information to identify one or more devices to be included in a set of devices for the session, because the device management service can automatically determine that such a computing device has been used previously in a task-related manner. As another example, the device management service can use context information to identify the task for which the session is initialized and / or pre-fill data within the task. It should be understood that pre-session context information can be received from any number of computing devices so that the device management service can aggregate and process such information accordingly.

[0024] As used herein, a service provider provides functionality or content that can be used by a computing device. As an example, a service provider distributes applications that can be executed by a computing device. In another example, a service provider provides a skill processing engine to enable a smart speaker computing device (which may or may not be manufactured by the service provider) to interact with the skill processing engine in order to access the service provider's content or functionality. Other examples include, but are not limited to, web applications, smart home services, content streaming services, and content creation and / or distribution platforms.

[0025] Figure 1 An overview of an example system 100 for user interaction and task completion using multiple devices is illustrated. As illustrated, the system 100 includes a computing device 102, a computing device 104, a skill processing engine 106, a service provider 108, and a device management service 110. In the example, the various aspects of the system 100 are connected using a computer network such as the Internet, a wired network, or a wireless network, or any combination thereof. It should be understood that although the system 100 is described herein with respect to a computing device 102 and a computing device 104, any number of computing devices may be used. Additionally, although various features are discussed with respect to elements 102-130, it should be understood that in other examples, certain features may be combined or different elements may include different features. For example, the service provider 108 may incorporate various aspects of the functionality discussed herein with respect to the skill processing engine 106 and / or the device management service 110, etc., and so on.

[0026] In an example, according to various aspects described herein, device management service 110 uses computing devices 102 and 104 to facilitate user interaction and task completion. In one example, computing device 104 enables a user to access services provided by service provider 108. As another example, a user uses computing device 102 to interact with skill processing engine 106 to access functionality provided by service provider 108. Traditionally, such user interactions are not associated, so a user who starts a task at computing device 102 will not be able to perform various aspects of the task at computing device 104, and vice versa. In contrast, device management service 110 transmits state information related to the user's interaction with service provider 108 to computing devices 102 and 104, and in some instances, determines which device (e.g., computing device 102 or computing device 104) should be used to interact with the user. In other examples, the user can provide instructions about which device should be used. As another example, the user can interact with either device at his or her discretion. As a result, the user can perform various aspects of the task on either computing device, and in some instances, a device that is well suited for the task can be automatically determined for the user.

[0027] As illustrated, computing device 102 includes microphone 112 and speaker 114. In some examples, computing device 102 is a voice command computing device, where the primary method of user input is using voice commands received using microphone 112, and the primary method of user output is using auditory responses provided using speaker 114. For example, computing device 102 can be a smart speaker computing device, a smart remote control for a television, a controller for a video game console computing device, a smart watch, or any of a variety of other computing devices. In some instances, the voice command computing device includes a display and / or a touch screen, thereby enabling alternative mechanisms to convey information to the user and / or receive user input. Although computing device 102 is illustrated as including one microphone 112 and one speaker 114, it should be understood that any number of such elements and other hardware features may be used.

[0028] The computing device 102 communicates with the skill processing engine 106 to process user input and provide responses. As used herein, a skill enables a user to interact with functionality provided by a service provider. For example, a skill may include a voice user interface (VUI) through which a user can request information and answer prompts, thereby using auditory interactions to control computing operations. In an example, a third-party developer may provide skills using an application programming interface (API) provided by a service provider. In some examples, the skill processing engine 106 communicates with the service provider 108 to access data, perform remote computing, and / or manipulate user data, etc. In other examples, the skill processing engine 106 does not need to communicate with the service provider 108 to process requests from the computing device 102. It should be understood that in some instances, at least a portion of the processing performed by the skill processing engine 106 described herein may be performed instead by different elements of the system 100, such as the computing device 102, the service provider 108, and / or the device management service 110.

[0029] The computing device 104 is illustrated as including a display 116 and a physical input device 118. The computing device 104 may be a mobile device, a tablet device, a desktop computing device, a laptop computing device, a smart home device, or any of a variety of other devices. The computing device 104 is provided as an example of a different computing device having different device characteristics and different capabilities compared to the computing device 102. For example, in the case where the computing device 102 provides a VUI for interacting with a user, the computing device 104 may instead provide a graphical user interface (GUI) as the primary user interaction mechanism. As an example, a user may use a physical input device 118 to interact with the GUI displayed on the display 116 by the computing device 104. Example physical input devices include, but are not limited to, a keyboard, a mouse, a touch screen, one or more hardware buttons, and components for sensing user gestures or other actions (e.g., using computer vision technology, proximity sensors, etc.). In one example, the computing device 104 includes a web browser application that enables a user to access Internet content (e.g., which may be provided by a service provider 108). In another example, the computing device 104 executes a local application, which may be provided by a service provider 108.

[0030] Although computing devices 102 and 104 are discussed with respect to example device types (e.g., smart speaker computing devices, smart watches, laptop computing devices, devices with physical input devices, devices without screens, etc.), it should be appreciated that in other examples, any of a variety of other computing devices, each of which may have different device characteristics and / or capabilities, may be used. Example computing devices include, but are not limited to, mobile devices, tablet devices, desktop computing devices, laptop computing devices, server computing devices, distributed computing devices, smart watch computing devices, smart home or Internet of Things (IoT) computing devices, smart speaker computing devices or smart TVs, video game console computing devices, and the like.

[0031] Furthermore, it should be understood that example device characteristics are described herein, and other examples of device characteristics include, but are not limited to, storage space, amount of random access memory (RAM), availability and / or type of specialized processing hardware (e.g., graphics processing unit, field programmable gate array, neural network hardware, hardware cryptographic engine, video encoding / decoding hardware, etc.), available communication modules (e.g., Wi-Fi, Bluetooth, Ethernet, USB, etc.), display characteristics (e.g., presence or absence of a display, display size, pixel density, type of display technology, etc.), presence or number of available speakers, microphones, and / or cameras, and / or presence or type of available input devices (e.g., physical input device, voice input, etc.). Example device capabilities include, but are not limited to, the ability to discreetly or privately communicate information to a user, the ability to move within a user's environment, the ability to play back video content, the ability to execute machine learning models, the ability to identify hot words or other calls to action and process subsequent input accordingly, or the ability to provide notifications to a user, the ability to control a user's environment (e.g., manage one or more lights, adjust the room temperature, etc.). Thus, in contrast to device characteristics, device capabilities are instead related to the functionality provided by a device, rather than the device characteristics used to provide such functionality. Information regarding device characteristics and / or device capabilities may be stored in a device profile, as described herein.

[0032] The service provider 108 provides services accessed by the computing devices 102 and 104, as described above. As an example, the service provider 108 can provide content and functionality via the Internet, which can be accessed using a web browser, a native application or a VUI (e.g., via the skill processing engine 106), etc. The service provider 108 includes a request processing engine 120 and a user data repository 122. In one example, the request processing engine 120 processes requests from the computing devices 104 and the skill processing engine 106. For example, the request processing engine 120 can process requests from the computing devices 102 and 104 (e.g., in the form of requests for web content, requests from native applications, requests via the skill processing engine 106, etc.), and can communicate with the user data repository 122 (e.g., access, modify, delete, and / or add data stored therein). The user data repository 122 can store user information, as can be received or generated by the request processing engine 120. Examples include, but are not limited to, account information (eg, username, email address, hashed password, etc.), user-generated content, and other data used by the service provider 108 in providing the service.

[0033] As illustrated, the device management service 110 includes a session management engine 124, a session data repository 126, a device selection engine 128, and a device information data repository 130. The session management engine 124 manages a group of computing devices associated with one or more tasks in accordance with the various aspects described herein. As an example, the session management engine 124 receives a state update from the computing device 102 and / or the computing device 104 so that the updated state information is provided to or otherwise available to other devices in the group accordingly. In some examples, context information related to the computing device 102 and / or the computing device 104 may be received prior to initialization of the session so that the context information may later provide context for the session. In another example, the session management engine 124 initializes and manages a task session that is associated with a group of devices (e.g., the computing device 102 and the computing device 104).

[0034] Thus, the session management engine 124 enables the user to use the group of computing devices to perform different steps of a task. As described herein, the device may be selected automatically, selected under the guidance of the user, or the user may select the device, etc. In some examples, the session management engine 124 communicates with the device selection engine 128 to determine the device with which the user interacts to perform the steps of the task, as discussed in more detail below. As another example, the user may be using an initial device to perform a step, and the device management service 110 may determine that a different device should be used. Therefore, an indication may be provided to the user at the initial device so that the user may choose to switch to the different device. In some instances, instead of switching from the initial device to the different device, two devices may be used. If the device management service 110 identifies a type of device that the user does not currently own, a recommendation to purchase such a device may be presented to the user. Although an example task session may be described with respect to a single task, it should be understood in other examples that a task session may be associated with a group of devices for multiple tasks.

[0035] The session data repository 126 may be used by the session management engine 124 to store context information, state information, and information related to potential, active, and / or past task sessions. For example, when a session state update is received from a computing device, the session management engine 124 may store information related to the session state update in the session data repository 126. Such state information may then be used to determine tasks associated with the task session, pre-populate data required for the task, and provide state updates to other computing devices, etc. Example session state information includes, but is not limited to, a session or user account identifier of a service provider (e.g., service provider 108), at least a portion of a user input received at a computing device, at least a portion of a response generated based on the user input or data related to a user action (e.g., recently visited web pages, recently used skills, recently contacted contacts, missed calls, etc.).

[0036] The device selection engine 128 evaluates a set of factors to determine one or more devices used to interact with the user from a set of devices as part of the task. In an example, the device selection engine 128 communicates with the session management engine 124 so that the device determination made by the device selection engine 128 is used by the session management engine 124 to assign a specific computing device to a given step of the task. Therefore, although the examples herein are described with respect to selecting a single device and assigning it to a step of a task, it should be understood that in other examples, multiple devices are selected so that each of the selected devices is used to perform different aspects of the step. Example factors include, but are not limited to, task or step capability requirements (e.g., the availability of a display, the ability to receive voice input, the ability to provide video playback, etc.), computing device characteristics, computing device capabilities, proximity of the device to the user, user preferences, historical device use or success rate for performing similar steps, and / or the ease of completing the step using a given device. In some examples, a task or step may provide an indication of the type, device characteristics, device capabilities, and / or available capabilities of a device that is needed and / or recommended.

[0037] The device information data repository 130 stores information related to the factors described herein. For example, the device information data repository 130 stores user preference information, which may indicate one or more mappings between a task type and a general type of computing device or a computing device with a specific capability. In some examples, the mapping indicated by the user preference may be between a task type and a specific computing device. It should be understood that in other examples, mappings may associate multiple devices with tasks and / or steps. For example, a device associated with a task requiring user input may include a device having a display for presenting information to a user and another device implementing a VUI for receiving user input, so that both devices are used when completing the task. As another example, the device information data repository 130 stores one or more device profiles. As used herein, a device profile includes information related to a computing device, such as device characteristics, device capabilities, proximity to a user, and / or battery charge status (e.g., percentage of charge, charge, exhaustion, absence, etc.), etc.

[0038] In other examples, the device information data store 130 stores historical device usage or success rate information, where the historical device usage information may relate to a specific user and / or may relate to a larger population of users. As an example, population device usage information may be used when user-specific usage information is not available (e.g., as may be the case when the user is a new user, has recently purchased a new device, etc.). Success rate information describes the effectiveness of a type of device for performing a type of task (e.g., based on user satisfaction, completion time, accuracy, etc.). Such information may be collected based on analysis of user survey information and / or usage data provided from computing devices for which the user has opted-in to such data collection.

[0039] Figure 2A An overview of an example method 200 for managing task sessions using multiple devices is illustrated. In an example, aspects of the method 200 are performed by, for example, Figure 1 Method 200 begins at operation 202, where an indication of user input is received at a computing device. In an example, directly from a computing device (e.g., Figure 1The method 200 may be configured to receive an indication from computing devices 102 and 104 in the example, from a service provider (e.g., service provider 108), from a skill processing engine (e.g., skill processing engine 106), or from any other various sources. In an example, the indication includes an identifier associated with the computing device. Example identifiers include, but are not limited to, a user account identifier, one or more IP or MAC addresses, a set of device hardware characteristics and / or device capabilities, a set of device names or GUIDs. The method 200 is described in an example where the identifier is pre-associated with the computing device. In other examples, it should be appreciated that the identifier may be requested or otherwise received from the computing device in a separate step according to the aspects described herein.

[0040] At operation 204, at least one other computing device is determined based on the received identifier. In an example, the pre-session context information received from the computing device may be evaluated to identify an association between the computing device and the identifier received at operation 202. As discussed herein, the pre-session context information may be stored by a session data repository, such as Figure 1 In some instances, the device proximity information may be evaluated such that the determined computing device is a computing device that is close to the user (e.g., in the same room, within arm's reach, within speaking distance, etc.). In other instances, the user may provide explicit indication of one or more devices that should be included in the group. Other evaluations may be used to determine other devices, such as evaluating device profiles (e.g., which may be stored by a device information data store, such as Figure 1 The device information data repository 130 in the device information repository 130), historical device usage information, etc.

[0041] Moving to operation 206, a task session is generated that associates the computing device from operation 202 and the at least one other computing device determined at operation 204 with the task. In an example, information related to the session (e.g., an associated set of devices, one or more tasks associated with the session, etc.) is stored in a session data repository, such as a session data repository. Figure 1 The session data store 126 in the task session. One or more tasks associated with a task session may be completed by communicating with a service provider (e.g., Figure 1 The determination may be made by communicating with the service provider 108 in operation 201, or may be determined based on information in the indication received at operation 202, and so on.

[0042] Flow proceeds to operation 208, where a step of the task associated with the user input is determined. In some examples, the indication received at operation 202 includes an indication of a step associated with the user input. In other instances, the step may be determined by communicating with a service provider, may be determined based on analysis of the user input, may be determined using state information stored in a session data repository, and so on.

[0043] At operation 210, the determined steps are evaluated to generate a set of requirements. As described herein, in some instances, the tasks and / or steps may provide indications of required and / or recommended capabilities. In other examples, the set of requirements is determined based on an evaluation action, potential user input, potential output, and / or other information associated with the step. In such examples, the evaluation may include a mapping between the type of evaluation action, input, output, and other information and preferred or required device capabilities. In other examples, historical device usage information may be evaluated.

[0044] Moving to operation 212, one or more processing computing devices are determined for performing this step. In some examples, aspects of operation 212 are performed by a device selection engine of a device management service, such as Figure 1 , the device selection engine 128 of the device management service 110 in the task session. As described herein, the determination includes analyzing a set of factors associated with the devices of the task session in view of the set of requirements determined at operation 210. Example factors include, but are not limited to, computing device characteristics, computing device capabilities, proximity of the device to the user, user preferences, historical device usage or success rates for performing similar steps, and / or the ease with which a step is completed using a given device. It should be appreciated that while operation 212 is described with respect to determining a device from a set of devices associated with the session, in other examples, the set of devices may be ranked based on the suitability of each device in view of various requirements and factors, or the top n ranked devices may be determined, and so on.

[0045] At operation 214, an instruction is provided to the determined device (or in some instances, multiple determined devices) to perform the step. In some instances, the instruction includes at least a portion of the information received at operation 202 (e.g., user input, task, step, etc.). The instruction can be provided directly to the computing device or can be provided via an intermediary (e.g., via a service provider, a skill processing engine, etc.), etc. It should be appreciated that in some examples, multiple instructions can be provided. For example, if it is determined that a browser on a computing device should be used to process a step, an instruction to open the browser to a web page can be provided to the computing device, and a second instruction can be provided to the service provider so that the service provider has information related to the step, task, and / or session. For example, a web page can be populated with data previously received from a user. As another example, a skill processing engine can receive state information so that a user using a voice command computing device can refer to the state information when talking to the voice command computing device.

[0046] Finally, the process may proceed to operation 216, where another indication of user input is received, as is the case when the user performs another step in the task session. Thus, the process returns to operation 208, where the step associated with the user input is determined, as discussed above. Thus, as the user performs the steps of the task using a set of devices associated with the session, the process loops through operations 208, 210, 212, 214, and 216. Ultimately, the process terminates at operation 214.

[0047] Figure 2B An overview of an example method 220 for selecting a device to use to perform steps of a task during a session is illustrated. In an example, aspects of the method 220 are provided by, for example, Figure 1 For example, a device management service such as the device management service 110 in Figure 1 The method 220 begins at operation 222, where a set of step requirements are received. In one example, the set of step requirements can be performed by executing the above Figure 2A In another example, the set of step requirements are specified by steps and / or tasks.

[0048] At operation 224, a device profile is accessed. For example, a device profile for each device in a set of devices associated with the session may be accessed. Figure 1The method 220 may be used to access the device profile from a device information data store such as the device information data store 130 in the device information data store 130. In other examples, the device profile may be associated with a user account for a user of the computing device. Although the method 220 is described with respect to using a device profile, it should be appreciated that in other examples, at least a portion of the information contained in the device profile may instead be stored and accessed separately.

[0049] At operation 226, device hardware characteristics are evaluated. Example device hardware characteristics include, but are not limited to, storage space, amount of RAM, availability and / or type of specialized processing hardware, available communication modules, displays, the presence or number of available speakers, microphones, and / or cameras, and / or the presence or type of available input devices. Such hardware characteristics are evaluated in view of the step requirements received at operation 222 to determine the extent to which available devices satisfy the set of step requirements.

[0050] The process proceeds to operation 228, where device capabilities are evaluated. Example device functions include, but are not limited to, the ability to discreetly or privately transmit information to the user, the ability to move within the user's environment, the ability to play back video content, the ability to execute machine learning models, the ability to identify hot words or other call actions and process subsequent input accordingly, the ability to provide notifications to the user, the ability to control the user's environment (e.g., manage one or more lights, adjust the room temperature, etc.). Therefore, in contrast to hardware characteristics, device capabilities are related to the functionality provided by the device rather than the hardware characteristics used to provide such functionality. Such device capabilities are evaluated in view of the step requirements received at operation 222 to determine the extent to which the available devices meet the set of step requirements.

[0051] At operation 230, historical device usage information is evaluated. In an example, from data such as Figure 1Historical device usage information may be accessed from a device information data repository such as the device information data repository 130 in . For example, such historical device usage information may relate to a specific user and / or may relate to a larger population of users. As an example, when user-specific usage information is not available, population device usage information may be used (e.g., when the user is a new user, has recently purchased a new device, etc.). Success rate information may describe the effectiveness of a type of device for performing a type of task having a set of step requirements (e.g., based on user satisfaction, completion time, accuracy, etc.). Historical device usage information may map the step requirements to one or more device hardware characteristics and / or device capabilities. Therefore, the evaluation at operations 226 and / or 228 may be used in conjunction with the historical device usage information to identify one or more devices having characteristics and / or capabilities that meet the set of task requirements indicated by the historical device usage information. In some instances, such information is used to generate and provide recommendations to a user, wherein the recommendation indicates one or more devices that the user should consider purchasing in order to have a set of more powerful devices available to complete the task.

[0052] Moving to operation 232, a ranking is generated for the group of devices based on the factors evaluated at operations 226-230. In an example, when ranking the group of devices, the factors may be weighted or prioritized. It should be appreciated that while example factors and evaluations have been described with respect to method 220, one or more evaluations may be omitted, or alternative or additional evaluations may be used. At operation 234, a device is selected to perform a step based on the generated ranking. In some examples, a group of devices is selected, wherein the group includes devices ranked above a predetermined threshold. The process terminates at operation 234.

[0053] Figure 3A An overview of an example process flow 300 for processing user input associated with a session using multiple devices is illustrated. As illustrated, process flow 300 occurs between computing device 302, computing device 304, service provider 308, and management service 310, each of which can be similar to such elements discussed above. For example, computing device 302 and computing device 304 can have different characteristics and capabilities (e.g., Figure 1 In an example, the service provider 308 may include a skills processing engine and / or a request processing engine, such as Figure 1 The skill processing engine 106 and the request processing engine 120 in.

[0054] Process flow 300 begins at operation 312, where user input related to a step of a task is received at computing device 302. As an example, computing device 302 may receive user input in the form of a user interacting with a physical input device or issuing a voice command, etc. It should be appreciated that although Figure 3A-3D 312 is described with respect to user input, but other types of input may also be received. For example, the input may include the arrival of a message, the completion of a timer, the delivery of a packet, and / or an event in a video stream. The input may be any of a variety of asynchronous events (e.g., may be detected by a sensor, received from a server, generated by an application, etc.). The arrow from operation 312 to operation 314 indicates that information related to the user input is provided to the service provider 308. As an example, the user input may be the submission of a web page for processing using a request processing engine, or may be a voice utterance provided to the service provider 308 for processing using a skill processing engine. It should be appreciated that in some examples, the user input may alternatively or additionally be processed at the computing device 302 or by any of a variety of other computing devices and / or processing engines.

[0055] At operation 314, an indication of the user input is generated and provided to the management service 310. In an example, the indication may include an identifier associated with the computing device 302. As discussed herein, the identifier may be associated with the ongoing session. Thus, using the identifier, the management service 310 identifies the set of session devices associated with the task at operation 316.

[0056] Move to operation 318, determine the step requirements for the step of the task being performed by the user. In the example, this determination includes performing the above Figure 2A Aspects of operation 210 discussed. For example, the tasks and / or steps may provide indications of required and / or recommended capabilities. In other examples, the set of requirements is determined based on evaluating actions, potential user inputs, potential outputs, and / or other information associated with the steps. As another example, management service 310 may communicate with service provider 308 to determine the requirements, or service provider 308 may have provided the requirements as part of the indication generated at operation 314.

[0057] At operation 320, a processing computing device for performing the step is determined from the set of devices associated with the session. In one example, the determination includes performing Figure 2A Various aspects of operation 212 discussed in connection with method 200 and / or with respect to Figure 2BAspects of the method 220 discussed. For example, the requirement determined at operation 318 is evaluated based on device capabilities, proximity of the device to the user, user preferences, historical device usage or success rates for performing similar steps, and / or the ease with which a step can be completed using a given device, etc. Accordingly, at operation 322, instructions for processing the user input are provided to the determined device.

[0058] As illustrated, the indication is provided to the service provider 308 so that the service provider 308 processes the user input at operation 324. In an example, the indication includes session state information and / or at least a portion of the user input, among other things. As an example, processing the user input may include manipulating data, generating a web page, and / or generating a voice response, among other things. Thus, the process flow 300 is illustrated as providing the indication to the service provider 308 rather than the computing device 302, as the computing device 302 provides the user input to the service provider 308 at operation 312, enabling the service provider 308 to generate a response and thereby respond. It should be appreciated that in other examples, the indication is provided to the computing device so that the computing device takes action instead of or in addition to the service provider 308. For example, sensitive data (e.g., camera input, financial information, passwords, etc.) may be stored locally by the computing device so that only the computing device can process the sensitive data. Such an example is described below with respect to Figure 3B to discuss.

[0059] At operation 326, a response to the user input is provided to computing device 302, which is received by computing device 302 at operation 328. As an example, receiving the response may include processing received web page data in a web browser, playing a received voice response using a speaker, or updating a state of a native application, etc.

[0060] At operation 330, an indication of the current session state (e.g., which may have changed as a result of processing user input at operation 324) is provided to management service 310. Accordingly, at operation 332, management service 310 generates a session state update notification, which is provided to other devices of process flow 300, which, as illustrated, is computing device 304. In some examples, management service 310 updates a session data store, such as Figure 1 The computing device 304 stores the session update at operation 334, which can be used by the computing device 304 when processing subsequent user requests. The process flow 300 terminates at operation 334.

[0061] Figure 3BAn overview of an example process flow 340 for processing user input associated with a session using multiple devices is illustrated, wherein a computing device different from the computing device that originally received the user input processes the user input. Figure 3B The multiple elements depicted in Figure 3A Operations 312-322 are similar to those described above with respect to Figure 3A discussed above, but the instruction to process the user input at operation 322 is provided to the computing device 304 rather than the service provider 308, which is Figure 3B This is because the computing device 304 is the device determined at operation 320.

[0062] Thus, computing device 304 processes the user input at operation 342. In an example, the indication received from management service 310 includes session state information and / or at least a portion of the user input, etc. Thus, computing device 304 can process the user input taking into account the current state of the session and information related to past steps previously performed by the user.

[0063] At operation 344, computing device 304 outputs a response to the user input. As an example, outputting a response may include updating the state of a local application, outputting a voice response using a speaker, or providing tactile feedback, etc.

[0064] At operation 346, an indication of the current session state (e.g., which may have changed as a result of processing the user input at operation 342) is provided to the management service 310. Accordingly, at operation 348, the management service 310 generates a session state update notification, which is provided to other devices of the process flow 340, as illustrated, the other device is the service provider 308. In some examples, the management service 310 updates a session data repository, such as Figure 1 302. The service provider 308 stores the session state update at operation 350, which can be used by the service provider 308 when processing subsequent user requests. The service provider 308 is depicted as receiving the session state update to illustrate that the updated state information need not be transmitted directly to a computing device (e.g., computing device 302), but can be transmitted to a computing device that performs operations on behalf of the computing device 302. Such instances may include a web service or instance in which a skill processing engine is used, etc. However, it should be appreciated that in other instances, in addition to or as an alternative to transmitting the updated state information to the service provider 308, the updated state information can be transmitted to the computing device 302. The process flow 340 terminates at operation 350.

[0065] Figure 3CAn overview of an example process flow 360 for transferring session state information in an example session with multiple devices is illustrated. As a result of implementing aspects of process flow 360, a user can perform subsequent steps of a task using a different device within a set of devices associated with the session. Figure 3C The multiple elements depicted in Figure 3A-3B The context of is described above and therefore no further description is necessary.

[0066] Process flow 360 begins at operation 362, where user input is received. As an example, computing device 302 may receive user input in the form of a user interacting with a physical input device or issuing a voice command, etc. The arrow from operation 362 to operation 364 indicates that information related to the user input is provided to service provider 308. As an example, the user input may be a submission of a web page for processing using a request processing engine, or may be a voice utterance provided to service provider 308 for processing using a skill processing engine. It should be appreciated that in some examples, the user input is processed at least in part by computing device 302 before being transmitted to service provider 308.

[0067] At operation 364, the service provider 308 processes the user input. In an example, the service provider 308 uses session state information (e.g., which may be received from a device management service in accordance with aspects described herein). As an example, processing the user input may include manipulating data, generating a web page, and / or generating a voice response, etc. Thus, at operation 366, a response to the user input is provided to the computing device 302, which is received by the computing device 302 at operation 368. As an example, receiving the response may include processing the received web page data in a web browser, playing the received voice response using a speaker, or updating the state of a native application, etc.

[0068] At operation 370, an indication of the current session state (e.g., which may have been changed as a result of processing user input at operation 364) is provided to the management service 310. Accordingly, at operation 372, the management service 310 identifies the session devices to which updated state information should be provided. According to aspects described herein, the management service 310 can use an identifier to identify a set of devices associated with the session, such as a user account identifier, one or more IP or MAC addresses, a set of device hardware characteristics and / or device capabilities, a set of device names or GUIDs, and the like.

[0069] Moving to operation 374, the management service 310 generates a session state update notification, which is provided to other devices of the process flow 300, as illustrated, the other device is the computing device 304. In some examples, the management service 310 updates the session data store, such as Figure 1 The computing device 304 stores the session update at operation 376, which can be used by the computing device 304 when processing subsequent user requests. The process flow 360 terminates at operation 376.

[0070] Figure 3D An overview of an example process flow 380 for communicating session state information in an example session with multiple devices is illustrated. As a result of implementing aspects of process flow 380, a user can perform aspects of a task using different devices other than the set of devices associated with the session. Figure 3D The multiple elements depicted in Figure 3A-3C The context of is described above and therefore no further description is necessary.

[0071] Process flow 380 begins at operation 382, ​​where user input is received. As an example, computing device 304 may receive user input in the form of a user interacting with a physical input device or issuing a voice command, etc. At operation 384, computing device 304 processes the user input. In an example, computing device 304 uses session state information (e.g., which may be received from a device management service in accordance with aspects described herein).

[0072] Therefore, at operation 386, computing device 304 outputs a response to the user input. For example, outputting a response may include updating the state of a local application, outputting a voice response using a speaker, or providing tactile feedback, etc.

[0073] At operation 388, an indication of the current session state (e.g., which may have been changed as a result of processing the user input at operation 384) is provided to the management service 310. Accordingly, at operation 390, the management service 310 identifies the session devices to which updated state information should be provided. According to aspects described herein, the management service 310 can use an identifier to identify a set of devices associated with the session, such as a user identifier, an IP or MAC address, a set of device hardware characteristics and / or device capabilities, a device name or GUID, and the like.

[0074] Moving to operation 392, the management service 310 generates a session state update notification, which is provided to other devices of the process flow 300, as illustrated, the other device is the service provider 308. In some examples, the management service 310 updates the session data repository, such as Figure 1394. The service provider 308 stores the session update at operation 394, which can be used by the service provider 308 when processing subsequent user requests. The service provider 308 is depicted as receiving the session state update to illustrate that the updated state information need not be transmitted directly to a computing device (e.g., computing device 302), but can be transmitted to a computing device that performs operations on behalf of the computing device 302. Such instances may include a web service or instance in which a skill processing engine is used, etc. However, it should be appreciated that in other instances, in addition to or as an alternative to transmitting the updated state information to the service provider 308, the updated state information can be transmitted to the computing device 302. The process flow 380 terminates at operation 394.

[0075] Figure 4 is a block diagram illustrating the physical components (e.g., hardware) of a computing device 400 that can be used to practice various aspects of the present disclosure. The computing device components described below may be applicable to the computing devices described above. In a basic configuration, the computing device 400 may include at least one processing unit 402 and a system memory 404. Depending on the configuration and type of the computing device, the system memory 404 may include, but is not limited to, volatile storage devices (e.g., random access memory), non-volatile storage devices (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 404 may include an operating system 405 and one or more program modules 406 suitable for performing various aspects disclosed herein, such as a session management engine 424 and a device selection engine 426. For example, the operating system 405 may be suitable for controlling the operation of the computing device 400. In addition, embodiments of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application, and is not limited to any particular application or system. The basic configuration is Figure 4 408. Computing device 400 may have additional features or functionality. For example, computing device 400 may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data or other data. Figure 4 This is illustrated by a removable storage device 409 and a non-removable storage device 410.

[0076] As described above, a number of program modules and data files may be stored in system memory 404. When executed on processing unit 402, program modules 406 (e.g., applications 420) may perform, including but not limited to, various aspects, as described herein. Other program modules that may be used in accordance with various aspects of the present disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-assisted applications, and the like.

[0077] In addition, embodiments of the present disclosure may be practiced in circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or on a single chip containing electronic components or microprocessors. For example, embodiments of the present disclosure may be practiced via a system on a chip (SOC), wherein Figure 4 Each component or many components illustrated in the figure can be integrated into a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functionalities, all of which are integrated (or "burned") into a chip substrate as a single integrated circuit. When operated via the SOC, the functionality described herein regarding the ability to switch protocols for the client can be operated via dedicated logic integrated with other components of the computing device 400 on a single integrated circuit (chip). Embodiments of the present disclosure may also be practiced using other technologies capable of performing logical operations such as, for example, "AND", "OR", and "NOT", including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present disclosure may be practiced within a general-purpose computer or in any other circuit or system.

[0078] The computing device 400 may also have (multiple) input devices 412, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or sliding input device, etc. (Multiple) output devices 414, such as a display, a speaker, a printer, etc., may also be included. The foregoing devices are examples and other devices may be used. The computing device 400 may include one or more communication connections 416 that allow communication with other computing devices 450. Examples of suitable communication connections 416 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuits; universal serial bus (USB), parallel and / or serial ports.

[0079] As used herein, the term "computer-readable medium" may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, or program modules. System memory 404, removable storage device 409, and non-removable storage device 410 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other article that can be used to store information and can be accessed by computing device 400. Any such computer storage media may be part of computing device 400. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0080] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0081] Figure 5A and Figure 5B A mobile computing device 500 is illustrated, which can be used to practice embodiments of the present disclosure, such as a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc. In some aspects, the client can be a mobile computing device. Figure 5A, illustrates one aspect of a mobile computing device 500 for implementing various aspects. In a basic configuration, the mobile computing device 500 is a handheld computer having both input elements and output elements. The mobile computing device 500 typically includes a display 505 and one or more input buttons 510 that allow a user to input information into the mobile computing device 500. The display 505 of the mobile computing device 500 may also be used as an input device (e.g., a touch screen display). If included, an optional side input element 515 allows additional user input. The side input element 515 may be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 500 may incorporate more or fewer input elements. For example, in some embodiments, the display 505 may not be a touch screen. In yet another alternative embodiment, the mobile computing device 500 is a portable telephone system, such as a cellular phone. The mobile computing device 500 may also include an optional keypad 535. The optional keypad 535 may be a physical keypad or a "soft" keypad generated on the touch screen display. In various embodiments, output elements include a display 505 for showing a graphical user interface (GUI), a visual indicator 520 (e.g., a light emitting diode), and / or an audio transducer 525 (e.g., a speaker). In some aspects, the mobile computing device 500 incorporates a vibration transducer for providing tactile feedback to the user. In yet another aspect, the mobile computing device 500 incorporates input and / or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port), for sending signals to or receiving signals from external devices.

[0082] Figure 5B 5 is a block diagram illustrating the architecture of one aspect of a mobile computing device. That is, a mobile computing device 500 can incorporate a system (e.g., architecture) 502 to implement some aspects. In one embodiment, the system 502 is implemented as a "smartphone" capable of running one or more applications (e.g., a browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 502 is integrated into a computing device, such as an integrated personal digital assistant (PDA) and wireless phone.

[0083] One or more applications 566 may be loaded into the memory 562 and run on or in association with the operating system 564. Examples of applications include a phone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 502 also includes a non-volatile storage area 568 within the memory 562. The non-volatile storage area 568 may be used to store persistent information that should not be lost when the system 502 is powered off. The application program 566 may use information and store it in the non-volatile storage area 568, such as emails or other messages used by the email application, and the like. A synchronization application (not shown) also resides on the system 502 and is programmed to interact with a corresponding synchronization application residing on a host computer to keep the information stored in the non-volatile storage area 568 synchronized with the corresponding information stored at the host computer. It should be appreciated that other applications may be loaded into the memory 562 and run on the mobile computing device 500 described herein.

[0084] System 502 has a power source 570, which can be implemented as one or more batteries. Power source 570 can also include an external power source, such as an AC adapter or a powered docking cradle to replenish or recharge the batteries.

[0085] The system 502 may also include a radio interface layer 572 that performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 572 facilitates wireless connectivity between the system 502 and the "outside world" via a communications carrier or service provider. Transmissions to and from the radio interface layer 572 are implemented under the control of the operating system 564. In other words, communications received by the radio interface layer 572 can be propagated to the application 566 via the operating system 564, and vice versa.

[0086] The visual indicator 520 can be used to provide visual notifications, and / or the audio interface 574 can be used to generate auditory notifications via the audio transducer 525. In the illustrated embodiment, the visual indicator 520 is a light emitting diode (LED) and the audio transducer 525 is a speaker. These devices can be directly coupled to the power supply 570 so that when activated, they will remain on for the duration specified by the notification mechanism even if the processor 560 and other components may be turned off to save battery power. The LED can be programmed to remain lit indefinitely until the user takes action to indicate the power-on state of the device. The audio interface 574 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 525, the audio interface 574 can also be coupled to a microphone to receive audible input to facilitate telephone conversations. According to an embodiment of the present disclosure, the microphone can also be used as an audio sensor to facilitate the control of notifications, as will be described below. The system 502 can also include a video interface 576, which enables the operation of the onboard camera 530 to record still images, video streams, etc.

[0087] The mobile computing device 500 implementing the system 502 may have additional features or functionalities. For example, the mobile computing device 500 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data in a manner that is convenient for the user. Figure 5B This is illustrated by non-volatile storage area 568.

[0088] Data / information generated or captured by the mobile computing device 500 and stored via the system 502 may be stored locally on the mobile computing device 500, as described above, or the data may be stored on any number of storage media that may be accessed by the device via the radio interface layer 572 or via the mobile computing device 500 and a separate computing device associated with the mobile computing device 500 (e.g., a server computer in a distributed computing network such as the Internet). It should be appreciated that such data / information may be accessed via the mobile computing device 500 via the radio interface layer 572 or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use in accordance with well-known data / information transfer and storage means, including electronic mail and collaborative data / information sharing systems.

[0089] Figure 6One aspect of the architecture of a system for processing data received from a remote source at a computing system is illustrated, the remote source being such as a personal computer 604, a tablet computing device 606, or a mobile computing device 608, as described above. The content displayed at the server device 602 can be stored in different communication channels or other storage types. For example, a directory service 622, a web portal 624, a mailbox service 626, an instant messaging repository 628, or a social networking site 630 can be used to store various documents. A request processing engine 621 can be employed by a client communicating with the server device 602, and / or a device management service 620 can be employed by the server device 602. The server device 602 can provide data to and from a client computing device such as a personal computer 604, a tablet computing device 606, and / or a mobile computing device 608 (e.g., a smart phone) through a network 615. For example, the above-mentioned computer system can be embodied in a personal computer 604, a tablet computing device 606, and / or a mobile computing device 608 (e.g., a smart phone). In addition to receiving graphics data that may be used for pre-processing at a graphics originating system or post-processing at a receiving computing system, any of these embodiments of the computing device may obtain content from repository 616 .

[0090] Figure 7 An exemplary tablet computing device 700 that can perform one or more aspects disclosed herein is illustrated. In addition, the various aspects and functionalities described herein can be operated on a distributed system (e.g., a cloud-based computing system), where application functionality, memory, data storage and retrieval, and various processing functions can be remotely operated from each other through a distributed computing network such as the Internet or an intranet. Various types of user interfaces and information can be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, various types of user interfaces and information can be displayed on a wall on which various types of user interfaces and information are projected and interacted with. Interaction with a variety of computing systems that can practice embodiments of the present invention includes keystroke input, touch screen input, voice or other audio input, gesture input (where the associated computing device is equipped with detection (e.g., camera) functionality for capturing and interpreting user gestures for the functionality of controlling the computing device), etc.

[0091] As can be understood from the above disclosure, one aspect of the present technology relates to a system, including: at least one processor; and a memory storing instructions, which, when executed by the at least one processor, causes the system to perform a set of operations, the set of operations including: receiving an indication of a user input at a first computing device associated with a step of a task session, wherein the indication includes an identifier associated with the task session; using the identifier to determine a set of computing devices associated with the task session, wherein the set of computing devices includes a first computing device and a second computing device; determining a processing computing device for performing the step of the task session from the set of computing devices; and providing an indication for processing the user input associated with the step of the task session to the processing computing device. In one example, the set of operations also includes: receiving a session state indication including session state information from the processing computing device; when the processing computing device is the first computing device, providing a state update indication to the second computing device based on the session state information; and when the processing computing device is the second computing device, providing a state update indication to the first computing device based on the session state information. In another example, determining the processing computing device includes: evaluating at least one device characteristic of the first computing device and at least one device characteristic of the second computing device based on the requirements associated with the step of automatically determining the processing computing device. In another example, determining the processing computing device includes: evaluating at least one device capability of the first computing device and at least one device capability of the second computing device based on the requirements associated with the step of automatically determining the processing computing device. In yet another example, providing an indication for processing a user input associated with a step of a task session also includes: providing an indication to a third computing device that the processing computing device is processing the user input. In yet another example, the processing computing device is a second computing device, so that the user interacts with a computing device different from the one that received the user input to perform the step. In another example, the group of computing devices also includes a third computing device, and the group of operations also includes: determining that the third computing device performs the step of the task session, so that the user interacts with the second computing device and the third computing device to perform the step.

[0092] In another aspect, the technology relates to a method for initializing a task session associated with a group of computing devices. The method includes: receiving an indication of an input associated with a step of a task from a first computing device of a user, wherein the indication includes an identifier; identifying a second computing device associated with the user based on the identifier; generating a task session associated with a group of computing devices, wherein the group of computing devices includes a first computing device and a second computing device; determining a processing computing device for executing the step from the group of computing devices to process the input; and providing an indication to the processing computing device for processing the input associated with the step of the task session. In one example, the method also includes: receiving a session state indication including session state information from the processing computing device; providing a state update indication to the second computing device based on the session state information when the processing computing device is the first computing device; and providing a state update indication to the first computing device based on the session state information when the processing computing device is the second computing device. In another example, the method also includes: receiving pre-session context information from the second computing device, wherein the pre-session context information includes an identifier; and storing the pre-session context information. In yet another example, the second computing device is identified based on the stored pre-session context information, and generating the task session also includes analyzing the pre-session context information. In yet another example, the second computing device is identified by receiving input including an identifier from the second computing device after receiving the indication from the first computing device. In yet another example, determining the processing computing device includes: evaluating a set of factors for evaluating each computing device based on the requirement associated with the step, wherein the set of factors is selected from a group including: computing device characteristics; computing device capabilities; proximity of the computing device to a user; user preference information; and historical device usage information.

[0093] In another aspect, the technology relates to another method for determining a computing device for performing a step of a task session. The method includes: receiving an indication of a user input at a first computing device associated with the step of the task session, wherein the indication includes an identifier associated with the task session; using the identifier to determine a group of computing devices associated with the task session, wherein the group of computing devices includes a first computing device and a second computing device; determining a processing computing device for performing the step of the task session from the group of computing devices; and providing an indication to the processing computing device for processing the user input associated with the step of the task session. In one example, the method also includes: receiving a session state indication including session state information from the processing computing device; when the processing computing device is the first computing device, providing a state update indication to the second computing device based on the session state information; and when the processing computing device is the second computing device, providing a state update indication to the first computing device based on the session state information. In another example, determining the processing computing device includes: evaluating at least one device characteristic of the first computing device and at least one device characteristic of the second computing device based on a requirement associated with the step of automatically determining the processing computing device. In another example, determining the processing computing device includes: evaluating at least one device capability of the first computing device and at least one device capability of the second computing device based on a requirement associated with the step of automatically determining the processing computing device. In yet another example, providing an indication for processing a user input associated with a step of the task session further comprises: providing an indication to a third computing device that the processing computing device is processing the user input. In yet another example, the processing computing device is a second computing device, such that the user interacts with a computing device different from the one that received the user input to perform the step. In another example, the processing device is determined based on at least one of historical device usage information and user preference information.

[0094] For example, various aspects of the present disclosure are described above with reference to the block diagrams and / or operational descriptions of the methods, systems, and computer program products according to various aspects of the present disclosure. The functions / actions noted in the blocks may not occur in the order shown in any flow chart. For example, depending on the functionality / actions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order.

[0095] The description and explanation of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure claimed in any way. The various aspects, examples and details provided in this application are considered to be sufficient to convey ownership and enable others to make and use the best mode of the disclosure claimed. The disclosure claimed should not be interpreted as being limited to any aspect, example or detail provided in this application. Whether it is shown and described in combination or separately, various features (structures and methods) are intended to be selectively included or omitted to produce an embodiment with a specific feature group. In the case of providing a description and explanation of the application, those skilled in the art can envision the changes, modifications and alternative aspects that fall within the spirit of the broader aspects of the general inventive concept embodied in this application but do not depart from the broader scope of the disclosure claimed.

Claims

1. A system for user interaction and task management using multiple devices, comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations comprising: receiving, at a first computing device, an indication of a user input associated with a step of a task session, wherein the indication includes an identifier associated with the task session; determining, using the identifier, a set of computing devices associated with the task session, wherein the set of computing devices includes the first computing device and a second computing device; determining, from the set of computing devices, a processing computing device for performing the step of the task session, wherein determining the processing computing device comprises evaluating a set of factors of one or more computing devices based on a user input or output requirement associated with the step of the task session, and wherein the processing computing device is determined based on an ability of the processing computing device to perform the user input or output requirement and a historical success rate of the processing computing device with respect to similar task steps performing the step of the task session; and Instructions are provided to the processing computing device for processing the user input associated with the step of the task session.

2. The system of claim 1, wherein the set of operations further comprises: receiving a session state indication including session state information from the processing computing device; When the processing computing device is the first computing device, providing a status update indication to the second computing device based on the session state information; as well as When the processing computing device is the second computing device, providing a status update indication to the first computing device based on the session state information.

3. The system of claim 1 , wherein determining the processing computing device comprises: At least one device characteristic of the first computing device and at least one device characteristic of the second computing device are evaluated to automatically determine the processing computing device based on a requirement associated with the step of the task session.

4. The system of claim 1 , wherein determining the processing computing device comprises: At least one device capability of the first computing device and at least one device capability of the second computing device are evaluated to automatically determine the processing computing device based on a requirement associated with the step of the task session.

5. The system of claim 1 , wherein providing the instructions for processing the user input associated with the step of the task session further comprises: An indication that the processing computing device is processing the user input is provided to a third computing device. The system of claim 1 , wherein the processing computing device is the second computing device.

7. The system of claim 6, wherein the set of computing devices further comprises a third computing device, and wherein the set of operations further comprises: The third computing device is determined to perform the steps of the task session and is operable to interact with the user.

8. A method for initializing a task session associated with a group of computing devices, comprising: receiving, from a first computing device of a user, an indication of an input associated with a step of a task, wherein the indication includes an identifier; identifying a second computing device associated with the user based on the identifier; generating a task session associated with a set of computing devices, wherein the set of computing devices includes the first computing device and the second computing device; determining a processing computing device from the set of computing devices for performing the step of the task to process the input, wherein determining the processing computing device comprises evaluating a set of factors of one or more computing devices based on an input requirement associated with the step of the task session, and wherein the processing computing device is determined based on an ability of the processing computing device to perform the user input requirement and a historical success rate of the processing computing device with respect to similar task steps performing the step of the task session; and Instructions are provided to the processing computing device for processing the input associated with the step of the task session.

9. The method according to claim 8, further comprising: receiving a session state indication including session state information from the processing computing device; When the processing computing device is the first computing device, providing a status update indication to the second computing device based on the session state information; as well as When the processing computing device is the second computing device, providing a status update indication to the first computing device based on the session state information.

10. The method according to claim 8, further comprising: receiving pre-session context information from the second computing device, wherein the pre-session context information includes the identifier; as well as The pre-session context information is stored. 11 . The method of claim 10 , wherein the second computing device is identified based on the stored pre-session context information, and wherein generating the task session further comprises analyzing the pre-session context information.

12. The method of claim 8, wherein after receiving the indication from the first computing device, the second computing device is identified by receiving input from the second computing device including the identifier.

13. The method of claim 8, wherein determining the processing computing device comprises: A set of factors is evaluated for each computing device based on the requirements associated with the step of the task, wherein the set of factors is selected from the group consisting of: Computing device characteristics; Computing device capabilities; proximity of the computing device to the user; User preference information; as well as Historical device usage information.

14. A method for determining a computing device for performing steps of a task session, comprising: receiving, at a first computing device, an indication of a user input associated with the step of the task session, wherein the indication includes an identifier associated with the task session; determining, using the identifier, a set of computing devices associated with the task session, wherein the set of computing devices includes the first computing device and a second computing device; determining, from the set of computing devices, a processing computing device for performing the step of the task session, wherein determining the processing computing device comprises evaluating a set of factors of one or more computing devices based on a user input or output requirement associated with the step of the task session, and wherein the processing computing device is determined based on an ability of the processing computing device to perform the user input or output requirement and a historical success rate of the processing computing device with respect to similar task steps performing the step of the task session; and Instructions are provided to the processing computing device for processing the user input associated with the step of the task session.

15. The method according to claim 14, further comprising: receiving a session state indication including session state information from the processing computing device; When the processing computing device is the first computing device, providing a status update indication to the second computing device based on the session state information; as well as When the processing computing device is the second computing device, providing a status update indication to the first computing device based on the session state information.

16. The method of claim 14, wherein determining the processing computing device comprises: At least one device characteristic of the first computing device and at least one device characteristic of the second computing device are evaluated to automatically determine the processing computing device based on a requirement associated with the step of the task session.

17. The method of claim 14, wherein determining the processing computing device comprises: At least one device capability of the first computing device and at least one device capability of the second computing device are evaluated to automatically determine the processing computing device based on a requirement associated with the step of the task session.

18. The method of claim 14, wherein providing the instructions for processing the user input associated with the step of the task session further comprises: An indication that the processing computing device is processing the user input is provided to a third computing device.

19. The method of claim 14, wherein the processing computing device is the second computing device.

20. The method of claim 14, wherein the processing computing device is determined based on at least one of historical device usage information and user preference information.

Images