Unified Interface for User Computing Devices for Pairing

Abstract

Techniques for providing a unified user interface between actively paired devices, including: determining that the first device is physically proximate to the second device when both the first device and the second device are authenticated for the same user service account, determining that the first device and the second device are actively paired at a first time; receiving an indication of actuation of a user interface control associated with a user activity and presented by the first device at the first time; and in response to receiving the indication and determining that the first device and the second device are actively paired together at the first time, causing the second device to present a user interface associated with the user activity.

Description

Technical Field

[0001] The present application generally relates to interactions with computing devices, and more particularly to systems and methods for providing a unified user interface between actively paired user computing devices. Background Art

[0002] Many user computing devices are portable, such as but not limited to: laptop computers, notebook computers, smartphones, and tablet computers. For many people, work may involve using and interacting with multiple different computing devices, including multitasking across devices. Additionally, the work environment has become more dynamic. For example, an increasing amount of work is done across home, coffee shops, airports, conference rooms, and personal workspaces, and on-demand workspace allocation (where an individual may work at different locations each day) is becoming more common. New and improved solutions are needed to enhance user interactions with these different devices and work environments and to more fully realize the capabilities provided by these different devices and work environments. Summary of the Invention

[0003] According to a first aspect of the present disclosure, a system adapted to provide a unified user interface between actively paired user computing devices includes one or more processors and a machine-readable medium including instructions that, when executed by the one or more processors, may cause the one or more processors to: determine that a first user computing device is eligible to be actively paired with a second user computing device. The instructions may also cause the one or more processors to: determine that the first user computing device is physically proximate to the second user computing device at a first time when both the first user computing device and the second user computing device are authenticated for the same user service account. Further, the instructions may cause the one or more processors to: based on determining that the first user computing device is eligible to be actively paired with the second user computing device and determining that the first user computing device is physically proximate to the second user computing device, determine that the first user computing device and the second user computing device are included in a first active pairing set at the first time. The instructions also cause the one or more processors to receive a first indication of an actuation of a first user interface control associated with a first user activity and presented by the first user computing device at the first time. Similarly, the instructions cause the one or more processors to: in response to receiving the first indication and determining that the first user computing device and the second user computing device are included in the first active pairing set at the first time, cause the second user computing device to present a second user interface associated with the first user activity.

[0004] According to a second aspect of the present disclosure, a method for providing a unified user interface between actively paired user computing devices includes: determining that a first user computing device is eligible for active pairing with a second user computing device. The method may also include: determining that the first user computing device is physically proximate to the second user computing device at a first time when both the first user computing device and the second user computing device are authenticated for the same user service account. The method may further include: based on determining that the first user computing device is eligible for active pairing with the second user computing device and determining that the first user computing device is physically proximate to the second user computing device, determining that the first user computing device and the second user computing device are included in a first active pairing set together at the first time. Additionally, the method includes: receiving, at the first time, a first indication of actuation of a first user interface control associated with a first user activity and presented by the first user computing device. Similarly, the method includes: in response to receiving the first indication and determining that the first user computing device and the second user computing device are included in the first active pairing set together at the first time, causing the second user computing device to present a second user interface associated with the first user activity.

[0005] The present invention content is provided to introduce a selection of concepts in a simplified form, which will be further described in the detailed description below. The present invention content is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additionally, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The drawings illustrate one or more implementations in accordance with the teachings of the present disclosure by way of example and not limitation. In the drawings, like reference numerals refer to the same or similar elements. Additionally, it should be understood that the drawings are not necessarily drawn to scale.

[0007] Figure 1 Illustrates an example of a system configured to provide a unified user interface between paired user computing devices;

[0008] Figure 2A Illustrates in Figure 1 Exemplary details of aspects of the system shown in are incorporated in Figure 1 Various examples of operations at a first location shown in Figure 2B Illustrates in Figure 1 and Figure 2A Exemplary details of aspects of the pairing service shown in

[0009] Figure 3A, Figure 3B and Figure 3C illustrate a first example of the system shown in Figure 1-2B which performs unified user interface (UI) interactions for user activities across multiple devices included in an active pairing set, where using a UI control presented by a first device causes a second device to join as an endpoint of a conference call session;

[0010] Figure 4 illustrate a second example of the system shown in Figure 1-2B which performs unified UI interactions for user activities across multiple devices included in an active pairing set, where while maintaining the second device as the primary audio-visual endpoint for an active session, a projected portion of the active session initiated by a remote participant is presented via the first device;

[0011] Figure 5A , Figure 5B and Figure 5C illustrate a third example of the system shown in Figure 1-2B which performs unified UI interactions for user activities across multiple devices included in an active pairing set, where a first user initiates a projected portion of the active session from the first device using a UI control on the second device;

[0012] Figure 6A , Figure 6B and Figure 6C illustrate a fourth example of the system shown in Figure 1-2B which performs unified UI interactions for user activities across multiple devices included in an active pairing set, where capture of a real-time media stream is selectively performed by the first device and the second device based on detection of the first user's attention;

[0013] Figure 7A , Figure 7B , Figure 7C and Figure 7D illustrate a fifth example of the system shown in Figure 1-2B which performs unified UI interactions for user activities across multiple devices included in an active pairing set, where media presentation is selectively performed via the first device and the second device based on detection of the first user's attention;

[0014] Figure 8A and Figure 8B illustrate a sixth example of the system shown in Figure 1-2BA sixth example of the system shown in , which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein, while participating in a teleconference session, a first user selects a file for projection and initiates the projection portion of the session via a handheld third device using the selected file;

[0015] Figure 9A 、 Figure 9B and Figure 9C The diagram shows Figure 1-2B A seventh example of the system shown in , which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein a presentation of a whiteboard interface for a teleconference session is automatically transferred between devices based on device state;

[0016] Figure 10A and Figure 10B The diagram shows Figure 1-2B An eighth example of the system shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein the first user uses the second device to initiate transmission of content generated in a teleconference session to other participants via a UI on the first device;

[0017] Figure 11A 、 Figure 11B 、 Figure 11C 、 Figure 11D and Figure 11E The diagram shows Figure 1-2B A ninth example of the system shown in , which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein, while participating in a teleconference session, the first user invites additional participants via a handheld third device;

[0018] Figure 12A 、 Figure 12B and Figure 12C The diagram shows Figure 1-2B A tenth example of the system shown in , which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein the handheld third device is temporarily recruited as a media input device for a virtual whiteboard presented by a second device;

[0019] Figure 13A 、 Figure 13B and Figure 13C The diagram shows Figure 1-2B An eleventh example of the system shown in , which performs a unified UI interaction for user activity across multiple devices included in an active pairing set, wherein a file opened for viewing on a second device is opened on a first device by actuating a UI control on the second device;

[0020] Figure 14A and Figure 14B illustrates a twelfth example of system 100 as shown in Figure 1-2B which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein content elements included in a file being viewed on a second device are opened for inspection on the first device by actuating a UI control on the second device;

[0021] Figure 15A 、 Figure 15B and Figure 15C illustrates a thirteenth example of the system as shown in Figure 1-2B which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein the first device presents a UI control to provide a geographical location identified in the content presented by the first device to a third device;

[0022] Figure 16A 、 Figure 16B and Figure 16C illustrates a fourteenth example of the system as shown in Figure 1-2B which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein the first device initiates a user input operation on the third device, causing the user input to be delivered from the third device to the initiating first device;

[0023] Figure 17 is a flowchart illustrating an implementation of an exemplary process for providing a unified user interface between actively paired user computing devices;

[0024] Figure 18 is a flowchart illustrating an implementation of an exemplary process for providing a unified user interface between actively paired user computing devices;

[0025] Figure 19 is a block diagram illustrating an exemplary software architecture, various parts of which can be used in combination with the various hardware architectures described herein, and these hardware architectures can implement any of the features described herein;

[0026] Figure 20 is a block diagram illustrating components of an exemplary machine configured to read instructions from a machine-readable medium and perform any of the features described herein. Detailed Description

[0027] In the following detailed description, numerous specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. It should be appreciated, however, that the present teachings may be practiced without such details. In other instances, well-known methods, procedures, components, and / or circuits have been described at a relatively high level without detail in order to avoid unnecessarily obscuring aspects of the present teachings. In the following material, directional indicators (such as "top" or "left") are provided only for reference during the following discussion and are not intended to indicate the required, desired, or expected direction of the items described.

[0028] Figure 1 An example of a system 100 configured to provide a unified user interface ("UI") between paired user computing devices is illustrated. Note that the term "user interface" (or "UI") is not limited to a graphical user interface ("GUI") but may also include other user interfaces such as, but not limited to, voice input and rendered output. In this example, a first user 120 is shown using a first user computing device 130 (which may be more simply referred to as a "computing device" or "device") at a first location 110a during a first time period t1, at a second location 110b during a second time period t2, and at a third location 110c during a third time period t3. During the first time period t1, the first user 120 also uses a second user computing device 132 at the first location 110a. During the second time period t2, a third user computing device 134 associated with the first user 120 is located at the second location 110b. During the third time period t3, the first user 120 also uses a fourth user computing device 136 at the third location 110c. In Figure 1 the specific example shown, the first device 130 is a portable computing device in the form of a notebook or laptop computing device, the second device 132 is a desktop computing device including a touch screen, a camera, a microphone, and a speaker, the first device 134 is in the form of a smart phone, and the fourth device 136 is in the form of a desktop computing device (represented by the Figure 1 monitor in). However, it can be understood that the first device 130, the second device 132, the third device 134, and / or the fourth device 136 may be implemented in other forms. In some examples, a user computing device provides capabilities not provided by another user computing device. For example, in Figure 1 the first device 130 includes a physical keyboard while the second device 132 does not include a physical keyboard. As another example, the second device 132 includes a touch screen while the first device 130 does not include a touch screen.

[0029] The term "user computing device" refers to a computing device that is oriented towards an end user and typically assumes that a single user uses the device at a given time. For example, a software program running on a user computing device may require providing a password or other credentials to access resources associated with a user account. In some examples, a user computing device may support multiple such user accounts, thereby allowing different users to "log in" to the device at their respective times. In some examples, a user computing device may support a single user account; for example, most smartphone devices are assumed to be used in conjunction with a single individual.

[0030] A first software program instance 131 (which may be referred to as a "software instance", "application", or "app" in some examples) is run by a first device 130 during a first time period t1, a second time period t2, and a third time period t3. In addition, the first software program instance 131 is configured to interact with a pairing service 150 via one or more networks 140 in conjunction with a first user service account 152 associated with a first user 120. For example, the first software program instance 131 may be configured to provide credentials to the pairing service 150 to authenticate the first software program instance 131 with respect to the first user service account 152. The interaction may include, for example, commands and / or data sent and received between the first software program instance 131 and the pairing service 150 to implement aspects of the techniques described herein. These interactions may be achieved via, for example, persistent connections, asynchronous communication, and / or polling for transmitting messages between the first software program instance 131 running on the first device 130 and the pairing service 150. It should be understood that references to an interaction between a user computing device and the pairing service 150 or being configured to interact with the pairing service 150 occur via a correspondingly configured software program example run by the user computing device.

[0031] During a first time period t1, a second device 132 is running a second software program instance 133. Very similar to that described for the first software program instance 131, the second software program instance 133 is configured to interact with a pairing service 150 via one or more networks 140 in conjunction with a first user service account 152. Thus, during the first time period t1, both the first device 130 and the second device 132 are configured to interact with the pairing service 150, and do interact with the pairing service 150 in conjunction with the first user service account 152. In some examples, the second software program instance 133 may be configured to interact with the pairing service 150 in conjunction with the first user service account 152 during a second time period t2 and / or a third time period t3. In some implementations, the use of the same first user service account 152 by different software program instances provides a consistent and robust security mechanism between the software program instances, such as by distributing and managing access tokens via the pairing service 150 to enable secure data access and exchange.

[0032] During a second time period t2, a third device 134 is running a third software program instance 135. Very similar to that described for software program instances 131 and 133, the third software program instance 135 is configured to interact with a pairing service 150 via one or more networks 140 in conjunction with a first user service account 152. Thus, during the second time period t2, both the first device 130 and the third device 134 are configured to interact with the pairing service 150, and may interact with the pairing service 150 in conjunction with the first user service account 152. In some examples, the third software program instance 135 may be configured to interact with the pairing service 150 in conjunction with the first user service account 152 during the first time period t1 and / or the third time period t3 while remaining at a first location 110a. In some examples, the third device 134 is with the first user 120 during the first time period t1 and / or the third time period t3.

[0033] During a third time period t3, a fourth software program instance 137 is being run by a fourth device 136. Very similar to that described for software program instances 131, 133, and 135, the third software program instance 135 is configured to interact with a pairing service 150 via one or more networks 140 in conjunction with a first user service account 152. Thus, during the third time period t3, both the first device 130 and the fourth device 136 are configured to interact with the pairing service 150 and do, in fact, interact with the pairing service 150 in conjunction with the first user service account 152. In some examples, the fourth software program instance 137 may be configured to interact with the pairing service 150 in conjunction with the first user service account 152 during the first time period t1 and / or the second time period t2 while remaining at the third location 110c.

[0034] The system 100 is configured to automatically identify a time period (such as the first time period t1) for two or more computing devices (such as the first device 130 and the second device 132 during the first time period t1), during which each user computing device in each set interacts with the pairing service 150 in conjunction with the same user service account and the user computing devices in each set are close to or likely to be close to each other and / or a common user or in the vicinity thereof. The set of user computing devices identified by the system 100 may be referred to as an "active pairing" and are included together in an active pairing set during the time period identified for the set of devices. Note that in some examples, three or more devices may be included in the active pairing set. For example, in some cases, when the first user 120 is at the first location 110a, the first device 130, the second device 132, and the third device 134 may be included in the active pairing set.

[0035] In some examples, the system 100 is configured to adjust the inclusion of user computing devices in the identified sets based on whether the user has identified the user computing device as eligible for pairing (e.g., by explicitly identifying the device as eligible or ineligible for pairing). In an example where the system 100 is so configured and the first device 130 and the second device 132 have been identified as eligible for pairing while the third device 134 has not been identified as eligible for pairing, the system 100 may identify the first device 130 and the second device 132 as being included in the active pairing set during the first time period t1 (based on both devices having been identified as eligible for pairing), even though the first device 130 and the third device 134 are close to each other during the second time period t2, but does not identify the two devices as being included in the active pairing set during that time (due to the third device 134 not being identified as eligible for pairing).

[0036] In some examples, system 100 is configured to regulate the inclusion of two user computing devices in the identified set based on whether the user has identified the two user computing devices as eligible to be paired with each other (e.g., by explicitly identifying the devices as eligible or ineligible to be paired with each other). In some implementations, the user may perform a pairing process to identify the devices as eligible to be paired with each other. In an example where system 100 is configured in this way, the first device 130 and the second device 132 have been identified as eligible to be paired together, the first device 130 and the fourth device 136 have been identified as suitable to be paired together, but the second device 132 and the fourth device 136 have not been identified as eligible to be paired together. In the case where the second device 132 and the fourth device 136 are in proximity to each other, system 100 will not identify the devices as being included in the active pairing set together, although each device is capable of actively pairing with other devices.

[0037] In Figure 1 the example shown, system 100 identifies the first device 131 and the second device 132 as a first active pairing set of user computing devices during a first time period t1 (during which they are close enough to perform active pairing), but not during a second time period t2 or a third time period t3 (during which they are not close enough). Similarly, system 100 identifies the first device 131 and the fourth device 136 as a second active pairing set of user computing devices during the third time period t3 (during which they are close enough), but not during the first time period t1 or the second time period t2 (during which they are not close enough).

[0038] System 100 is also configured to control the presentation of the user interface for various activities by the user computing devices based on whether the user computing devices involved are actively paired during the associated time period. In some examples, in response to system 100 identifying that a user computing device has transitioned between being not actively paired and actively paired, the user interface presented by the user computing device may be modified (including but not limited to: adding displayed user interface (UI) elements, removing displayed UI elements, and / or changing the appearance of the displayed UI elements). System 100 is also configured to control the behavior of the software program instance based on whether the user computing device running the software program instance is actively paired during the associated time period.

[0039] Various mechanisms can be employed to identify periods of time when user computing devices are in close proximity to each other. For example, the first device 130 and the second device 132 can be configured to send and / or receive short-range radio signals (such as but not limited to according to the Bluetooth Low Energy (BLE) and / or Near Field Communication (NFC) communication protocol standards) and / or sound signals. By another example, the first device 130 and the second device 132 can each be configured to detect the presence of a particular user or device (such as but not limited to: through RFID tag detection, facial recognition, and / or voice recognition technologies).

[0040] Figure 1 Also illustrated are a second user 160a and corresponding computing device 170a, a third user 160b and corresponding computing device 170b, a fourth user 160c and corresponding computing device 170c, and a fifth user 160d and corresponding computing device 170d. In some examples discussed below, the second user 120 participates in a conference call session with some or all of the users 160a, 160b, 160c, and 160d via a conference call service 180. In some examples, the conference call session involves capturing and transmitting video media streams and audio media streams for distribution to other participants, and receiving and presenting the video media streams and audio media streams for other participants. The conference call service 180 can provide additional capabilities via the conference call session, such as but not limited to: projecting display and / or application content to and / or from other participants, live transcription of spoken words, and / or whiteboard media streams and / or services.

[0041] Figure 2A Illustrated in Figure 1 are exemplary details of aspects of the system 100 shown in Figure 1 in operation at a first location 110a shown in Figure 1 However, it can be understood that the described details can also be applied in conjunction with other user computing devices (such as but not limited to the third device 134 and the fourth device 136 shown in

[0042] As mentioned above, the first user computing device 130 is in the form of a notebook or laptop computer. As shown in Figure 2AAs shown in FIG. 0, the first device 130 includes a first display device 210, such as but not limited to an LCD-based display panel, which can be used to present various graphical user interfaces (“GUIs”). The first device 130 also includes a first local storage device 212, such as volatile and / or non-volatile storage devices, which is shown storing first local data 213, such as electronic files. The first device 130 also includes a first runtime environment 214 in which a first software program instance 131 is run by the first device 130. Other software program instances can be run by the first device 130 within the first runtime environment 214 and / or other runtime environments provided by the first device 130. The first device 130 also includes various input devices 216, such as but not limited to: a first camera 217, a first microphone 218, and / or a keyboard 219. The first device 130 also includes various output devices 220, such as but not limited to a first speaker 221. Other examples of input devices and output devices will be discussed in conjunction with Figure 20 for discussion.

[0043] As mentioned above, the second user computing device 132 is in the form of a desktop computing device configured to receive touch-based input and serve as a voice-driven interface for a digital assistant (e.g., Microsoft Cortana TM ). As shown in Figure 2A FIG. 8, the second device 132 includes a second display device 230, which can be used to present various GUIs. The second device 132 also includes a second local storage device 232 shown storing second local data 233. The second device 132 also includes a second runtime environment 234 in which a second software program instance 133 is run by the second device 132. Other software program instances can be run by the second device 132 within the second runtime environment 234 and / or other runtime environments provided by the second device 132. The second device 132 also includes various input devices 236, such as but not limited to: a second camera 237, a second microphone 238, and / or a touchscreen 239. The second device 132 also includes various output devices 240, such as but not limited to a second speaker 241.

[0044] Figure 2A Including the Figure 1 conference call service 180 shown in FIG. 15. In some examples, as shown in Figure 2AAs shown in, the conference call service 180 can maintain a session record 251 for a conference call session. In some examples, the session record 251 can include a scheduled session 252 for a conference call session that has been scheduled but has not yet started. In some examples, the session record 251 can include an active session 253 for a conference call session with one or more active participants. The active session 253 may be from a previous scheduled session or an ad-hoc session. In Figure 2A In the example shown in, the system 100 can include a storage service 254 configured to receive, store, and make accessible storage data 255, including storage data items 256. For example, the storage data item 256 can be an electronic document generated by a content creation application such as, but not limited to, a word processor application.

[0045] In Figure 2A In the example shown in, the system 100 can include a control service 250 configured to interact with the pairing service 150 as a client service of the pairing service, allowing it to receive notifications of changes in the active pairing set from the pairing service 150, send messages or exchange messages with devices, and / or obtain device information. Additionally, the control service 250 is configured to interact with the conference call service 180 to obtain information from the conference call service 180 and issue commands to the conference call service 180. Via interaction with the pairing service 150, the conference call service 180, and the user computing device, the control service 250 is configured to enable pairing-aware features for conference call sessions without having to modify the conference call service 180 itself. In some examples, the control service 250 is configured to interact with the storage service 254 to obtain information about the storage data 255 (e.g., file metadata) and / or access or make accessible items included in the storage data 255. In some examples, the functions described for the control service can be partially or wholly incorporated into the pairing service 150 and / or the conference call service 180. In Figure 2A In the example shown in, the system 100 can include a third-party client service 258 configured to interact with the pairing service 150 as a client service to enable incorporating pairing-aware features into third-party services and / or applications. In some implementations, the control service 250 and / or the third-party service 258 can obtain pairing-related information via the device instead of directly interacting with the pairing service 150 as a client service. Note that in some implementations, the program software instances 131, 133, 135, 137, and / or other pairing-aware program software instances described herein can be configured to interact with the pairing service 150 at least partially via a client service such as the control service 250 or the third-party client service 258.

[0046] Figure 2B illustrates exemplary details of various aspects of the pairing service 150 shown in Figure 1 and Figure 2A The pairing service 150 includes a user authenticator 260 configured to identify and authenticate a user service account associated with a communication session between a user computing device and the pairing service 150. For example, the user authenticator 260 may be configured to receive and process credentials in the form of a username and password hash. In some implementations, the user authenticator 260 supports the use of credentials obtained from an external authentication service (e.g., a token issued by a single sign-on (SSO) service) and may associate the user account identifier provided by such a service with its respective user service account. In some implementations, the user authenticator 260 is configured to issue an access token to a device that has been successfully authenticated, and the pairing service 150 is configured to receive and process the access token in subsequent communications with the device for authentication and authorization operations.

[0047] In some implementations, the pairing service 150 includes a client service interface 270 for interacting between the pairing service 150 and "client" services configured to receive information about the set of active pairings for one or more selected user service accounts and to interact with a device based on the status of the device as a paired or unpaired device and / or the devices with which it is actively paired. Exemplary client services include the control service 250 and the third-party client service 258 shown in Figure 2A In some examples, the client service interface 270 is configured to perform authentication and / or authorization of the client service in conjunction with the selected user service account, which may include the use of the authentication techniques described for the user authenticator 260 and acceptance of a token issued by the user authenticator 260 to the device. In some implementations, the client service may obtain information about the user-associated device 286, the available device 288, and / or the active pairing record 292 discussed below via the client service interface 270. In some examples, a service may be designed to obtain pairing-related information via a device rather than interacting with the pairing server 150 as a client service.

[0048] In some implementations, the pairing service 150 maintains a user service account record 280 for a user service account established with the pairing service 150, including a user service account record 282 for the user service account 152 of the first user 120. In some examples, the user service account record 282 is stored in multiple separate records. For example, an access token 284 associated with the user service account may be maintained by the user authenticator 260. In some implementations, the user service account record 282 includes an access token 284 issued for the user service account 152.

[0049] In some implementations, the user service account record 282 includes user - associated device 286 information of devices that have been authenticated using the pairing service 150 for the user service account 152. In some implementations, the pairing service 150 is configured to add a device to the user - associated devices 286 in response to successful user authentication (or “login”) of the device with the pairing service 150. In some implementations, the pairing service 150 is configured to remove a device from the user - associated devices 286 in response to “check - out” of the device with the pairing service 150. In some examples, a device supported for use by multiple different users is configured to “check - out” of the pairing service when the user finishes using the device.

[0050] In some implementations, the user service account record 282 includes available device 288 information that identifies devices currently authenticated with the pairing service 150 for the user service account 152, and the devices may currently be included in the active pairing set of devices. In some examples, devices may be added to and removed from the available devices 288 in response to the establishment and termination of a persistent network connection, but other schemes may be used to identify the available devices 288. In some examples, a device is configured to send a registration message to the pairing service 150 to indicate its availability for pairing with other devices, and to send an unregistration message to the pairing service 150 to indicate its unavailability for pairing, and the pairing service 150 is configured to add and remove the device from the available devices 288 respectively. In some examples, a device is configured to send a registration message to the pairing service 150 in response to an event via a software program instance running on the device, such as but not limited to: user login to the device, start of the software program instance, and / or re - establishment of a network connection. In some implementations, the pairing service 150 is configured to add / maintain and remove the device from the available devices 288 respectively based on receipt of a periodic keep - alive message from the device and / or expiration of a timeout for the most recently received keep - alive message from the device.

[0051] In some implementations, the user service account record 282 includes a device pairing eligibility record 290 that records the eligibility of a recording device to be paired with other devices, and the pairing service 150 is configured to identify devices eligible to be included in the active pairing set based on the device pairing eligibility record 290. In such an implementation, the pairing service 150 includes a device pairing eligibility acceptor 262 that is configured to receive a device pairing eligibility message (e.g., from a user computing device) indicating a change to the device pairing eligibility and modify the device pairing eligibility record 290 accordingly. In some implementations, some or all devices may be defaulted to eligible, and the user may "opt out" of device pairing, where the "opt out" is recorded in the device pairing eligibility record 290. In some implementations, some or all devices may be defaulted to ineligible, and the user may "opt in" a device to be paired with other eligible devices, where the "opt in" is recorded in the device pairing eligibility record 290. In some implementations, for a given device, the user is able to identify specific other devices that can definitely be paired with and / or definitely cannot be paired with it, where this information is recorded in the device pairing eligibility record 290. For example, the system 100 may implement a device pairing process where the user must definitely identify a device as being allowed to be paired with another device, resulting in the device pairing eligibility record 290 indicating that the two devices can be included together in the active pairing set. In some implementations, a set of three or more devices may be identified as a pairing group, and all devices must be physically close to each other to establish an active pairing set, where the pairing group is recorded in the device pairing eligibility record 290.

[0052] The pairing service 150 includes an active pairing detector 264, which is configured to: instantiate an active pairing set based on information received from a device associated with a user service account, add the device to the currently instantiated active pairing set, remove the device from the currently instantiated active pairing set (e.g., in response to the active pairing set including two or more devices after the removal), and / or cancel the instantiation of the active pairing set (e.g., in response to fewer than two devices remaining in the active pairing set). In some implementations, in response to detecting an event indicating that a device has become available to be included in an active pairing set (such as the various events described above for adding a device to the available devices 288) and / or an event that may reflect a change in the physical proximity of the device to another device (e.g., a detected change in the device's location that removes it from the physical proximity of another device), the active pairing detector 264 is configured to determine whether the device is to be included in the active pairing set. The active pairing detector 264 is configured to perform proximity-based pairing of devices, wherein it is configured to: determine whether a device is (or may be) physically proximate to one or more other devices, include the device in an active pairing set with the other devices based on an affirmative determination that the device is physically proximate to one or more other devices, and not include the device in the active pairing set or remove the device from the active pairing set based on a negative determination that the device is not physically proximate to other devices. In some examples, to determine whether a device is physically proximate to one or more other devices, the active pairing detector 264 is configured to issue a proximity detection command to the device, the device is configured to perform a proximity detection action (e.g., in response to the proximity detection command), the device is configured to send device proximity information to the active pairing detector 264 based on the proximity detection action, and / or the active pairing detector 264 is configured to make a proximity determination based at least on the received device proximity information. In implementations that utilize a device pairing eligibility record 290, the active pairing detector 264 is configured to identify an active pairing set that complies with the device pairing eligibility record 290. In some implementations, a single device may be included in two different active pairing sets simultaneously. For example, in such an implementation, if device A is eligible to actively pair with device B, device A is eligible to actively pair with device C, device B is not eligible to actively pair with device C, and devices A, B, and C are physically proximate to each other, the first active pairing set will include devices A and B, while a different second active pairing set will include devices A and C. In some implementations, the pairing service 150 is configured to use the active pairing detector 264 and / or the device pairing eligibility acceptor 262 to implement an initial pairing process between devices (e.g., it may modify the device pairing eligibility record 290 to indicate that the devices are eligible to actively pair with each other).

[0053] The user service account record 282 includes an active pairing record 292 that records the currently instantiated set of active pairings and the devices included in each set of active pairings. The active pairing detector 264 is configured to update the active pairing record 292 based on instantiating a set of active pairings, de-instantiating a set of active pairings (including the devices in the set of active pairings), and removing a device from a set of active pairings.

[0054] The pairing service 150 includes a device communicator 268 that is configured to deliver (or attempt to deliver) messages (such as, but not limited to, command or request messages) to a device and, in some implementations, also receive and route messages (such as, but not limited to, response messages) from the device. In some implementations, this allows the pairing service 150 to encapsulate the use of different message delivery techniques for different devices. For example, for some devices, it may be preferable to utilize a platform-specific notification framework rather than maintaining a persistent network connection. In some implementations, the pairing service 150 is configured to receive messages from a device or a client service to be delivered by the device communicator 268 to a selected device, each device included in the selected set of active pairings, and / or each active paired device.

[0055] The pairing service 150 includes an active pairing notifier 266 that is configured to notify a device and / or a client service of a change to a set of active pairings. The active pairing notifier 266 is configured to notify, via the device communicator 268, a device of a pairing notification recipient 294 of a change to a set of active pairings that includes the device, such as, but not limited to, adding the device to the set of active pairings or removing the device from the set of active pairings, and another device being added to the set of active pairings or removed from the set of active pairings. In some examples, a device is configured to modify the UI and behavior based on whether it is included in a set of active pairings and / or whether other devices are included in the set of active pairings. In some implementations, the active pairing notifier 266 may notify a client service of a change to a set of active pairings. In some examples, the client service is configured to modify commands issued to a device and client service behavior based on a notification provided by the active pairing notifier 266.

[0056] Figure 3A , Figure 3B and Figure 3C illustrates a first example of the system 100 shown in Figure 1 , Figure 2A and Figure 2B that performs unified UI interactions for user activities across multiple devices included in a set of active pairings, where using a UI control presented by a first device 130 causes a second device 132 to join as an endpoint of a conference call session. In Figure 3A-3CThe first example in Figure 4-16C can be combined with any of the examples in the example in Figure 3A As shown in Figure 1 this first example occurs during a first time period t1 described in Figure 3A during which the first user 120, the first device 130, and the second device 132 are at the first location 110a. At the time shown in

[0057] and via interaction with the pairing service 150, the first device 130 and the second device 132 are actively paired (meaning they are included in the same active pairing set). TM At this time, a first software program instance 131 (labeled "First APP") configured to interact with the pairing service 150 and / or the control service 250 is run by the first device 130 in the first runtime environment 214. Additionally, a fifth software program instance 310 (labeled "Fifth APP") is also run by the first device 130 in the first runtime environment 214. In some implementations, some or all of the features described for the fifth software program instance 310 may alternatively be incorporated into the first software program instance 131. In some implementations, the fifth software program instance 310 is configured to interact with the first software program instance 131 for all or some of the interactions with the pairing service 150 and / or the control service 250. In some implementations, the fifth software program instance 310 is configured to communicate directly with the pairing service 150 for all or some of the interactions with the pairing service 150 and / or the control service 250. The fifth software program instance 310 is configured to interact with the teleconference service 180, such as accessing the session record 251 and participating in a teleconference session. For example, the fifth software program instance 310 may incorporate various features of the Microsoft Teams 10 operating system for the Microsoft Windows TM software.

[0058] Similarly, at this time, a second software program instance 133 (labeled "Second APP") configured to interact with the pairing service 150 and / or the control service 250 is run by the second device 132 in the second runtime environment 234. Additionally, a sixth software program instance 320 (labeled "Sixth APP") is also run by the second device 132 in the second runtime environment 234. In some implementations, some or all of the features described for the sixth software program instance 320 may alternatively be incorporated into the second software program instance 133. In some implementations, the sixth software program instance 320 is configured to interact with the second software program instance 133 for all or some of the interactions with the pairing service 150 and / or the control service 250. In some implementations, the sixth software program instance 320 is configured to communicate directly with the pairing service 150 for all or some of the interactions with the pairing service 150 and / or the control service 250. The sixth software program instance 320 is configured to interact with the conference call service 180. In this example, the sixth software program instance 320 is configured to provide a "home screen" interface for the second device 132.

[0059] In Figure 3A it, a first UI 312 is presented on the first display 210 for the fifth software program instance 310 and is displaying an agenda for a scheduled conference call session for the first user 120. The first UI 312 includes a scheduled conference call session UI element 314 corresponding to the active session 253 in Figure 2A it. The UI element 314 includes a first session join UI control 316 (labeled "Join") that can be used to join the first user 120 as a participant in the active session 253, in which users 160a, 160b, 160c, and 160d are already participating. Additionally, a second UI 322 is presented on the second display 230 for the sixth software program instance 320 to provide a "home screen" interface. In this particular example, the second UI 322 includes a second session join UI control 324 (labeled "Join") that can also be used to join the first user 120 as a participant in the active session 253.

[0060] Figure 3B Illustrated is that the first user 120 has joined by using the one in Figure 3AThe pointer 302 shown in [description] actuates the result of the first session join UI control 316. The system 100 is configured to process the actuation of the first session join UI control 316 at least based on whether the first device 130 is included in the active pairing set. In response to the first device 130 being included in the active pairing set, the system 100 has caused, for example via the second software program instance 133, the seventh software program instance 340 running in the second runtime environment 234 to operate as an audiovisual endpoint to enable the first user 120 to participate in the active session 253. The seventh software program instance 340 is configured to transmit the outgoing session media 368 (labeled "media output") for the active session 253 via the network(s) 140 for presentation by the devices used by the session participants, such as Figure 1 the computing devices 170a, 170b, 170c, and 170d in [description]. For example, the outgoing session media 368 may include the live video of the first user 120 captured via the second camera 237 and / or the live audio of the first user 120 captured via the second microphone 238.

[0061] Note: The parts in square brackets [description] need to be filled with the corresponding Chinese context if there is any specific reference in the original text. Since it's not clear from the provided content, they are left as placeholders for now.The seventh software program instance 340 is also configured to receive and present incoming session media 366 for the active session 253. The seventh software program instance 340 is configured to present the live video included in the incoming live media 366 in respective regions of a third UI 342 on the second display 230. In this example, the third UI 342 includes a first region 352 presenting a first video stream 353 of user 160a, a second region 354 presenting a second video stream 355 of user 106b, a third region 356 presenting a third video stream 357 of user 160c, and a fourth region presenting a fourth video stream 359 of user 160d. The seventh software program instance 340 is also configured to present the live audio included in the incoming live media 366 via the second speaker 241. Thus, although the first session join UI control 316 presented by the first device 130 is used to initiate joining the active session 253, the visual and audio presentation of the remote participants and the video and audio capture of the first user 120 are performed by the second device 312 based on its active pairing with the first device 130. Additionally, the third UI 342 includes a first session UI control 360 that includes, for example, a first camera enable / disable UI control 361 (to toggle the video transmission of the first user 120), a first microphone mute UI control 362 (to toggle the audio transmission of the first user 120), a first volume UI control 363 (to adjust the volume for presenting audio to the first user 120), and / or a first session termination UI control 364 (for ending the participation of the first user 120 in the active session 253). Thus, the second device 132 operates as a focal center for the active session 253 while leaving the first device 130 idle for other activities, such as a third UI interface for an email program running on the first device 130).

[0062] In various implementations, the first device 130, the second device 132, and / or the control service 250 may make an affirmative determination that the first device 130 and the second device 132 are included in the active pairing set. The first device 130, the second device 132, and / or the control service 250 may select the second device 132 as the audio-visual endpoint to operate from among the devices included in the active pairing set at least based on the affirmative determination, and cause the selected device to operate as the audio-visual endpoint (e.g., via the second message 328 (MSG328) sent by the system 100 to the second device 132). As an example, in response to the actuation of the first session join UI control 316, the fifth software program 310 may transmit a first message 318 (MSG318) indicating the actuation of the first session join UI control 316 to the control service 250. Then, in response to obtaining information that the first device 130 and the second device 132 are actively paired (e.g., through a notification from the pairing service 150) and receiving the first message 318, the control service 250 may select the second device 132 and transmit the second message 328 as a request for the second device 132 to operate as the audio-visual endpoint. As another example, in response to receiving information that the first device 130 is included in the active pairing set, the fifth software program 310 may operate in the pairing mode, and in response to being in the pairing mode, transmit the first message 318 to the second device 132 via the pairing service 150 (and received by the second device 132 as the second message 328), where the first message 318 requests the second device 132 to operate as the audio-visual endpoint. In some examples, the second device 132 may reject the request in response to receiving the second message 328 (e.g., because the second device 132 is already being used for another purpose), and in response, the system 100 will cause the first device 130 to be used as the audio-visual endpoint. It should be understood that other schemes involving messages exchanged between the first device 130, the second device 132, and the control service 250 and determined by the first device 130, the second device 132, and the control service 250 may be applied to achieve the same effect.

[0063] In some examples, the system 100 is configured to obtain the capabilities and / or configuration information for the devices included in the active pairing set and select a device based on the obtained capabilities information and / or configuration information. For example, the capabilities and / or configuration information for the second device 132 may indicate that it is a preferred audio-visual endpoint when available, and the system 100 may select the second device 132 based on this indication. The configuration information may include, for example, user settings, group settings, or system settings.

[0064] Note that the first device 130 and the second device 132 are each configured to operate as an audiovisual endpoint when unpaired. Thus, based on the determination by the first device 130 and / or the control service 250 that the first device 130 is unpaired, actuation of the first session join UI control 316 will instead cause the first device 130 to operate as an audiovisual endpoint for the active session 253, much like that shown for the second device 132 in Figure 3B . Similarly, based on the determination by the second device 132 and / or the control service 250 that the second device 132 is unpaired, actuation of the second session join UI control 316 will cause the second device 132 to operate as an audiovisual endpoint for the active session 253.

[0065] As shown in Figure 3C , in some implementations, more than one device included in the active pairing set can be used simultaneously to present the video of other participants in the active session 253. In the example shown in Figure 3C , the first user 120 joins the active session 253 with four other participants, where audio and video capture and presentation are performed as described in Figure 3B . After that, two additional participants with their respective real-time video streams 375 and 377 have joined the active session 253. The system 100 continues to use the second device 132 to present the audio for the active session 253, present the real-time video streams 353, 355, 357, and 359, and capture the real-time video and audio of the first user 120, where the seventh software program instance 340 receives the inbound media 366 and sends the outbound media 368, as described in Figure 3B .

[0066] However, instead of presenting all six video streams 353, 355, 357, 359, 375, and 377 together on the second device 132, the system 100 is configured to have multiple devices included in the active pairing set (and, in some examples, determined to be capable of video presentation based on the capabilities and / or configuration information obtained for the devices) each present a portion of the video stream for the active session 253. This allows for a larger presentation size than can be achieved using a single device. In some implementations, the system 100 is configured to select a primary video presentation device and, based on information about the video stream and the devices included in the active pairing set, select and use one or more additional video presentation devices to present corresponding portions of the video stream. In some examples, the use of a device as an additional video presentation device is based on the type of activity and / or the level of activity; for example, if the first device 130 has not been actively used recently, the system 100 is more likely to select and use the first device 130 as an additional video presentation device. In some examples, the use of additional presentation devices can be based on the number of video streams to be presented; for example, the system 100 can be configured to limit the second device 100 to presenting four video streams unless there are more than six video streams (in which case, a different presentation scheme can be used).

[0067] In Figure 3C , based on the number of video streams and the first device 130 and the second device 132 being in the active pairing set, the system 100 is presenting the live video streams 375 and 377 on the first device 130. As a result, the fifth software program instance 310 is configured to receive and present the incoming session media 386 (labeled "media input") for the active session 253. The fifth software program instance 310 is configured to present the live video included in the incoming session media 386 in respective regions of the fourth UI 372 on the first display 210. In this example, the fourth UI 372 includes a fifth region 374 that presents the video stream 375 and a sixth region 376 that presents the video stream 377. In this example, the second device 132 is still used to capture the video and audio of the first user 120 and send the outgoing session media 368, and presents the audio for the active session 253, as described in Figure 3B . In some implementations, the system 100 is configured to present audio for a participant on the same device that is used to present the video stream for the participant.

[0068] Additionally, in some implementations, the fifth software program instance 310 is configured to also present a second session UI control 380 on the first display 210, which is similar to and presented concurrently with the first UI control 360. In this example, the second session UI control includes UI controls 381, 382, 383, and 384, which have the same functionality as the corresponding UI controls 361, 362, 363, and 364 shown on the second display 230. Note that actuating UI controls 381, 382, and 383 causes the first device 130 to transmit corresponding messages (e.g., to the control service 250 and / or the second device 132), which causes the system 100 to cause the seventh software program instance 340 to switch video transmission, switch audio transmission, and adjust the volume for audio presentation, respectively. The UI control 384 on the first device 130 and the UI control 364 on the second device 132 can both be used to end participation in the active session 253. Thus, the first user 120 can easily submit commands for the active session 253, regardless of whether their attention is directed to the first device 130 or the second device 132 at that time. In some examples, the fifth software program instance 310 can be configured to present the second session UI control 380 on the first display 210 even when the fourth UI 372 is not presented.

[0069] Regarding Figure 3B and Figure 3C examples, the devices used by the system 100 to present and capture audio and video for a conference call session can change dynamically in response to a device being added to or removed from the active pairing set. For example, if the second device 132 is to be turned off or loses its network connection, the system 100 can be configured to alternatively use the first device 130 to present and capture all audio and video for the active session 253.

[0070] Figure 4 Illustrates a second example of the system 100 shown in Figure 1 , Figure 2A and Figure 2B that performs unified UI interactions for user activities across multiple devices included in the active pairing set, where, while keeping the second device 132 as the primary audio-visual endpoint for the active session 253, the projected portion of the active session 253 initiated by a remote participant is presented via the first device 130. Figure 4 Continuing from the state shown in Figure 3B and can be combined with any of the examples described in Figure 3A-3C and Figure 5A-16C .

[0071] In Figure 4In this case, one of the remote participants in the active session 253 has initiated the projection portion of the active session 253. The projection portion can include, for example, a portion of the display of the "screen projection" remote participant, but other projection techniques can be used. The system 100 is configured to: in response to determining that the first device 130 and the second device 132 are included in the active pairing set and a notification of the projection portion, continue to use the second device 132 as the Figure 3B audiovisual endpoint described in, and at the same time cause the first device 130 to operate as a projection endpoint, where the projection content is presented on the first display 210. Various schemes involving messages exchanged between and determined by the first device 130, the second device 132, and the control service 250 can be used. For example, the control service 250 can be configured to identify these conditions through interaction with the pairing service 150 and the conference call service 180, and send a message to the first device 130 requesting it to start operating as a projection endpoint for the projection portion.

[0072] In this example, the fifth software program instance 310 is configured to receive and respond to projection data 410 for the projection portion of the active session 253 (e.g., by presenting visual projection content 424 on the first display 210). The fifth software program instance 310 (although different software program instances can be used) is configured to present a fifth UI 420 including a content display area 422, in which the visual projection content 424 included in the projection data 410 is presented. Additionally, in some implementations, the fifth software program instance 310 is configured to also present a third session UI control 430 on the first display 210, which is very similar to that previously described for Figure 3C the second session UI control 380 in. In this example, the third session UI control 430 includes the UI controls 381, 382, and 384 described above. In some examples, the third session UI control 430 also includes a projection view UI control 440, which controls the presentation of the projection data 410 by the fifth software program instance 310. In this example, the projection view UI control 440 includes a back navigation UI control 441 and a forward navigation UI control 442, which allow the first user 120 to asynchronously move backward or forward through the Figure 4 document being projected in. Thus, the first user 120 can easily submit commands for the active session 253, regardless of whether their attention is directed to the first device 130 or the second device 132 at that time.

[0073] Note that system 100 is configured to: in response to the second device 132 not being in the active pairing set and being used to join an active session 253 including a projection portion, receive projection data 410 using the second device 132 and present visual projection content 424 on the second display 230. It is similar and independent for system 100 to use the first device 130 in response to the first device 130 not being in the active pairing set. However, as shown in Figure 4 when both the first device 130 and the second device 132 are in the active pairing set during the active session 253, system 100 provides an improved experience to the first user 120.

[0074] Figure 5A 、 Figure 5B and Figure 5C illustrate a third example of system 100 as shown in Figure 1 、 Figure 2A and Figure 2B which performs unified UI interactions for user activities across multiple devices included in the active pairing set, where the first user 120 initiates the projection portion of the active session 253 from the first device 130 using UI controls on the second device 132. The third example in Figure 5A-5C continues from the state shown in Figure 3B and can be combined with any of the examples described in Figure 3A-4 and Figure 6A-16C . In Figure 5A the first user 120 is initiating the projection portion of the active session 253 via the second device 132. Since the second device 132 may already be the area of user focus for the active session 253, it is more efficient to initiate the projection from the second device 132. To this end, using the hand 122 to provide input via the touch screen, the first user 120 actuates the UI control 510 to present a list of projection devices 512. System 100 is configured to identify devices included in the active pairing set with the second device 132 that are capable of projecting content (e.g., based on the capabilities and / or configuration information obtained for the device), and present the identified devices as available projection devices with corresponding projection device selector UI controls 514. In this example, the first device 130 (which has been given the human-readable identifier "PC") is the only identified projection device, and thus only one projection device selector UI control 514 (with the text "Project to PC") is included in the projection source list 512. Here, the first user 120 actuates the projection device selector UI control 514 and selects the first device 130.

[0075] Figure 5BIllustrated is a projection source selection operation that can be performed by system 100 in response to determining that the selected device provides multiple projection sources (which, in some examples, can be identified by the selected device in response to actuation of device selector UI control 514). In this example, the fifth software program instance 310 is configured to identify projection sources available from the first device 130. However, if the selected device does not provide multiple projection sources, system 100 can skip this projection source selection operation. In Figure 5B In response to determining that the selected first device 130 provides multiple projection sources, system 100 displays projection source selection UI 530 to the first user 120. In this example, system 100 is configured to present projection source selection UI 530 via the selected first device 130 (in this example, via the fifth software program instance 310 on the first display 210), but in some examples, system 100 can alternatively display projection source selection UI 530 via the second device 132 on which the device selector UI control 514 has been actuated. Projection source selection UI 530 includes projection source selector UI controls for each projection source available on the selected device. As illustrated in Figure 5B In, the first device 130 is capable of providing window-specific projection sources, and thus, projection source selection UI 530 includes projection source selector UI controls 532, 534, and 536 for the respective windows 522, 524, and 526. Many devices are capable of providing the full display area as a projection source, as indicated by projection source selector UI control 538 for the first device 130 included in projection source selection UI 530. In Figure 5B In the example shown in, the first user 120 uses pointer 502 to actuate projection source selector UI control 532 for window 522.

[0076] Figure 5C Continuing with the initiation of the projection portion. System 100 is configured to configure the selected first device 130 to begin transmitting projection data 540 for presentation to remote participants in response to actuation of the projection device selector UI control 514 and / or the projection source selector UI control 532. In this example, system 100 causes the fifth software program instance 310 to generate and transmit projection data 504. Additionally, in some implementations, the fifth software program instance 310 is configured to present fourth session UI control 540 on the first display 210, just as previously for in Figure 3Cas described by the second session UI control 380 in [the above]. In this example, the fourth session UI control 540 includes the UI controls 381, 382, 383, and 384 described above. In some examples, the fourth session UI control 540 also includes a projection source UI control 550 that controls the projection portion of the active session 253. In this example, the projection source UI control 550 includes a projection indicator 551 (indicating that the first device 130 is projecting) and a projection termination UI control 552 (which causes the system 100 to terminate the projection portion). In some implementations, the sixth program software instance 540 is configured to include a similar projection source UI control 560 in the first session UI control 360. In this example, the projection source UI control 560 includes a projection indicator 561 (indicating that the first device 130 is projecting) and a projection termination UI control 562 (which causes the system 100 to terminate the projection portion).

[0077] Figure 6A , Figure 6B and Figure 6C illustrates a fourth example of the system 100 shown in Figure 1 , Figure 2A and Figure 2B which performs unified UI interactions for user activities across multiple devices included in an active pairing set, where the capture of a real-time media stream is selectively performed by the first device 130 and the second device 132 based on the detection of the attention of the first user 120. In Figure 6A-6C the fourth example in [the above] continues from the example shown in Figure 5A and can be combined with any of the examples described in Figure 3A-5C and Figure 7A-16C in [the above].

[0078] In Figure 6A the system 100 is configured to, in response to the second device 132 being in the active pairing set with the first device 130, identify which devices among the devices included in the active pairing set and capable of real-time video capture are being attended to by the first user 120, and configure the identified devices to capture and transmit a video stream of the first user 120 for the active session 253. In some implementations, the system is configured to identify devices capable of real-time video capture based on the capabilities and / or configuration information obtained for the devices (which may reflect a user-initiated temporary camera disable). In some implementations, the system 100 is configured to evaluate user attention based on, for example, the detected head and / or eye direction (e.g., an image captured via the camera 217 and / or the camera 237), the detected user proximity (e.g., an image captured via the camera 217 and / or the camera 237), and / or user input activity (e.g., via a keyboard, pointer, or touchscreen). In Figure 6AIn the specific example shown, both the first device 130 and the second device 132 are capable of video capture via their respective cameras 217 and 237, and based on the detected line of sight direction 610 towards the second device 132, the system 100 has determined that the attention of the first user 120 is directed at the second device 132. Based on this determination, the system 100 configures the second device 132 to capture and transmit an outbound real-time video stream 620 using the second camera 237, capture and transmit an outbound real-time audio stream 630 using the second microphone 238, and present an inbound real-time audio stream 640 using the second speaker 241. Similarly, based on this determination, the system 100 configures the first device 130 not to use the first camera 217, the first microphone 218, or the first speaker 221 for generating or presenting media for the active session 253.

[0079] In Figure 6B continuing the example in Figure 6A based on the detected line of sight direction 612 towards the first device 130, the system 100 has determined that the attention of the first user 120 is newly directed at the first device 130. Based on this determination, the system 100 configures the second device 132 to stop transmitting the outbound real-time video stream 620, and configures the first device 130 to capture and transmit an outbound real-time video stream 622 using the first camera 217 in place of the outbound real-time video stream 620. As a result, the real-time video provided to the remote participant by the active pairing set better captures the first user 120.

[0080] As shown in Figure 6C the system 100 may also be configured to identify which devices among the devices included in the active pairing set and capable of real-time audio capture are being attended to by the first user 120, and configure the identified devices to capture and transmit an audio stream of the first user 120 for the active session. In some implementations, the system is configured to identify devices capable of real-time audio capture based on the capabilities and / or configuration information obtained for the devices. In Figure 6CIn response to the detected line of sight direction 614 towards the first device 130, the system 100 has determined that the attention of the first user 120 is directed towards the first device 130. Based on this determination, the system 100 configures the second device 132 not to transmit the in-station live video stream 620 or the out-station audio stream 630, and configures the first device 130 to capture and transmit the out-station live video stream 622 and use the first microphone 218 to capture and transmit the live audio stream 632 to replace the out-station live audio stream 630. As a result, the live audio provided to the remote participant by the active pairing set better captures the first user 120. In some examples, based on the capabilities and / or configuration information obtained for the devices, the system 100 may identify the second device 100 as the primary audio capture device and thus not switch from using the second microphone 238 (resulting in the configuration shown in Figure 6B ). This may allow the advanced microphone 238 or audio processing pipeline in the second device 132 to continue to be used to maintain audio quality. In Figure 6A-6C , the system 100 has identified the second device 132 as the primary audio presentation device (e.g., based on the capabilities and / or configuration information obtained for the device), and thus configures the second device 132 to present the live audio stream 640. This may allow the advanced speaker 241 to continue to be used to maintain audio quality.

[0081] Figure 7A , Figure 7B , Figure 7C and Figure 7D illustrate a fifth example of the system 100 shown in Figure 1 , Figure 2A and Figure 2B , which performs unified UI interactions for user activities across multiple devices included in the active pairing set, where media presentation is selectively performed based on the detection of the attention of the first user 120 via the first device 130 and the second device 132. The fifth example in Figure 7A-7D continues from the state shown in Figure 6A and may be combined with any of the examples described in Figure 3A-6C and Figure 8A-16C . In Figure 7A-7D , the system 100 is configured to: in response to the second device 132 being in the active pairing set with the first device 130, identify which devices among the devices included in the active pairing set and capable of video presentation are being attended to by the first user 120, and configure the identified devices to present the selected video content for the active session 253. In some implementations, the system is configured to identify the devices capable of video presentation based on the capabilities and / or configuration information obtained for the devices (which may reflect that the device is in a locked state where it cannot present video on the display).

[0082] In Figure 7A-7C , system 100 is configured to identify the most recent active speaker 751 from among the remote participants in the active session 253, and configure the identified device to present a live video stream corresponding to the most recent active speaker 751. In Figure 7A-7C the specific example shown, the most recent active speaker 751 is user 160a corresponding to the live video stream 353. In Figure 7A , as in Figure 6A , based on the detected line of sight direction 710 towards the second device 132, system 100 has determined that the attention of the first user 120 is directed towards the second device 132. Based on this determination, system 100 configures the second device 132 to present the live video stream 353 corresponding to the most recently active speaker 751 in the first region 352 (as shown in Figure 3B ), and configures the other devices in the active pairing set (in this example, the first device 130) not to present the live video stream 353. In Figure 7A , the live video streams 353, 355, 357, and 359 are presented by the second device 132 in the respective regions 352, 354, 356, and 358 of the third UI 342.

[0083] In Figure 7B , system 100 is configured to present the live video stream corresponding to the most recent active speaker 751 on only one of the devices included in the active pairing set at a time. Continuing with the example in Figure 7A , based on the detected line of sight direction 712 towards the first device 130, system 100 has determined that the attention of the first user 120 has newly been directed towards the first device 130, as in Figure 6B . Based on this determination, system 100 configures the second device 132 to stop presenting the live video stream 353 (and system 100 may stop transmitting the live video stream 353 to the second device 132), and configures the fifth software program instance 310 running on the first device 130 to receive and present the live video stream 353 on the first display 210 in the region 722 of the sixth UI 720. As a result, the first user 120 is more involved in the active session 253 when participating in activities on the first device 130. In some implementations, the sixth UI 720 includes compact fifth session UI controls 724, which can be actuated to display controls similar to the second session UI controls 380 in Figure 3C . In some implementations, the third UI 342 is also rearranged, and new content may be included in the third UI 342 to more effectively utilize the space previously used to present the video of the most recent active speaker 751. In Figure 7BIn the specific example shown, the live video streams 355, 357, and 359 are moved to the corresponding regions 352, 354, and 356, and the region 358 is used to display the content 726 (in this example, a live transcription of the active session 253 is shown). Note that in response to the most recently active speaker 751 changing to a different remote participant, the system 100 will correspondingly modify the video streams presented by the first device 130 and the second device 132.

[0084] Figure 7C is shown Figure 7B an alternative where the system 100 is configured to present the live video stream corresponding to the most recently active speaker 751 on the second device 132 (e.g., based on identifying the second device 132 as the primary audio-visual device), and if different from the second device 132, the device included in the active pairing set and capable of video presentation is determined to be under the attention of the first user 120. As shown in Figure 7B Based on the detected line-of-sight direction 714 towards the first device 130, the system 100 has determined that the attention of the first user 120 is directed towards the first device 130. The live video streams 353, 355, 357, and 359 continue to be presented in the regions 352, 354, 356, and 358 of the third UI 342, as shown in Figure 7A As shown in Figure 7B Based on this determination, the system 100 configures the fifth software program instance 310 executed on the first device 130 to receive and present the live video stream 353 on the first display 210 in the region 722 of the sixth UI 720 described in Figure 7B The sixth UI 720 may also include Figure 7B the compact fifth session UI controls 724 described in Figure 7B and 7C illustrate an example where the presentation of one of the live video streams follows the attention of the first user 120, but in some examples, multiple live video streams may follow the attention of the first user 120; for example, in Figure 7B all four live video streams may be presented by the first device 130 in a manner that allows the first user 130 to utilize other regions of the first display 210.

[0085] Figure 7D Continue Figure 7C the example of Figure 7DIn the specific example shown, the compact fifth session UI control 724 includes one or more UI controls (not shown) that allow the first user 120 to select one of multiple video content items for an active session 253 for presentation via the sixth UI 720 (or otherwise by the device identified as being watched). Exemplary video content items include, but are not limited to: a live video stream associated with the most recent active speaker 751 (as shown in Figure 7C ), a live transcription of the active session 253 (as shown in the region 358 in Figure 7B ), a projection portion (as shown in the region 422 in Figure 4 ), and a whiteboard (virtual or physical). In Figure 7D , the local user 120 has selected the live transcription as the selected video content 746 shown in the region 722 of the sixth UI 720. In some embodiments, the selected video content item is stored and used in a later conference call session involving the active pairing set. In some embodiments, Figure 7C 's example can be modified to show a live transcription, a projection portion, or a whiteboard instead of the most recent active speaker 751.

[0086] Figure 8A and Figure 8B illustrate a sixth example of the system 100 shown in Figure 1 , Figure 2A and Figure 2B that performs unified UI interactions for user activities across multiple devices included in the active pairing set, where, when participating in a conference call session, the first user 120 selects a file for projection and initiates the projection portion of the session with the selected file via the handheld third device 134. In Figure 8A and [[ID= 's sixth example continues from the state shown in ​ and also includes a third device 134 in the active pairing set that includes the first device 130 and the second device 132. The sixth example in ​ and ​ can be combined with any of the examples described in ​ and ​ .

[0087] In ​In [the system], a third software program instance 135 running on a third device 134 in a runtime environment 812 is configured to present a seventh UI 812 on a display 810 of the third device 134. The system 100 is configured to identify files most recently opened by a first user 120 and present some or all of the identified files in a file list 820 included in the seventh UI 812. In some implementations, the system 100 is configured to identify and include a first file maintained by a storage service 254 and most recently opened by the first user 120; for example, a control service 250 may be configured to interact with the storage service 254 to identify the first file and report the identified first file to the third software program instance 135. In some implementations, the system 100 is configured to identify and include a second file locally stored in the third device 134 for selecting a file to be projected. In some implementations, the system 100 is configured to identify and include a third file locally stored in a device included in an active pairing set; for example, a file locally stored on a first device 130 and most recently edited by the first user 120 may be reported to the third software program instance 135 via the first device 130 and the system 100. In ​ [the system], a hand 212 of the first user 120 is used to select a file 822 from the file list 820 for projection.

[0088] In some implementations, the system 100 is configured to identify, as candidate projection devices, devices that are included in an active pairing set with a second device 132 and capable of projecting content (e.g., based on capabilities and / or configuration information obtained for the devices), and identify a projection device from the candidate devices. In some examples, in cases where multiple candidate projection devices are identified, the system 100 is configured to select a default projection device from the candidate projection devices. The default projection device may be selected based on capabilities and / or configuration information obtained for the candidate projection devices; for example, the system 100 may select the default projection device based on various factors such as, but not limited to, display resolution, display size, network connection type, network bandwidth, the ability to present the selected file (e.g., having a spreadsheet installed to open a spreadsheet file), and / or input interface (e.g., a device with a keyboard and / or mouse may be preferred over a touchscreen device). In ​ [a] specific example shown [in the figure], the first device 130, the second device 132, and the third device 134 are candidate projection devices, where the first device 130 is selected as the default projection device. In some examples where multiple candidate projection devices are identified, the system 100 is configured to provide, via the third software program instance 135, a UI control (not shown [in the figure]) that allows the first user 120 to select one of the identified candidate projection devices as the projection device. ​ [the figure]]

[0089] As mentioned above, in this example, the first device 130 is selected to project the document 822. As shown in ​ , in response, the system 100 causes the first device 130 to initiate the projection portion of the active session 253 for the document 822. For example, the third device 134 and / or the control system 250 may send a message to the first device 130 such that the first device 130 opens the selected document in the corresponding application and transmits the corresponding projection data for presentation to the remote participants. In the example shown in ​ , the first display 210 initially displays the eighth UI 832, and then for the projection portion, the fifth software program instance 310 opens the selected document to display the projection content 844 of the document from the area 842 in the ninth UI 840 for the fifth software program instance 310, very similar to that described in ​ .

[0090] ​ , ​ and ​ illustrate a seventh example of the system 100 shown in ​ , ​ and ​ , which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein the presentation of a whiteboard interface for a teleconference session is automatically transferred between devices based on the device state. The seventh example in ​ can be combined with any of the examples described in ​ and ​ .

[0091] In ​ , the first device 130 is not included in the active pairing set with the second device 132, and the second device 132 operates as the primary audio-visual endpoint for the active session 253, very similar to that described in the previous examples. As shown in ​ , the second device 132 is presenting the tenth UI 910 for the active session 253, wherein the tenth UI 910 includes a remote participant display area 950 that includes vertically arranged portions 952, 954, and 956 in which the corresponding live video streams 953, 955, and 957 are presented. Similarly, in response to enabling a whiteboard (in this example, a virtual whiteboard that can be collaboratively edited) in the active session 253, the tenth UI 910 includes a whiteboard interface 912 that shows the first whiteboard content 914. For the purposes of this example, the second device 132 has limited or no ability to edit via the whiteboard interface 912.

[0092] Interested in an improved interface for interacting with a whiteboard, a first user 120 has picked up a fifth user computing device 930 (which is embodied as a tablet computing device in this example), as indicated on a fourth display 932 of the fifth device 930. The fifth user computing device 930 is in ​ the locked state at the moment shown. In some examples, a device in the locked state can be included in the active pairing set, but different capabilities are provided based on whether it is locked or unlocked. For example, in the locked state, the device can be used for audio presentation or notifications but not for video presentation; while in the unlocked state, the device can be used for video presentation. In some examples, a device will not be included in the active pairing set when it is locked. In some examples, a specific application configured to interact with the pairing service 150 and / or the control service 250 must be running on the device to be included in the active pairing set.

[0093] In ​ , the fifth device 930 has been unlocked by the first user 120, an eighth software program instance 960 is running in a runtime environment 962 on the fifth device 930, and is included in the active pairing set with a second device 132. In response to at least the fifth device 930 being unlocked, the system 100 automatically determines that the fifth device 930 currently provides a better whiteboard interface than the second device 132, and in response to this determination, supports the fifth device 960 to stop using the second device 132 for the whiteboard. Accordingly, the eighth software program instance 962 presents an eleventh UI 964, where the first user 120 can use a stylus 124 to view a first whiteboard content 914 and add a second whiteboard content 966. In response to the whiteboard not being presented on the second device 132, the tenth UI 910 is rearranged to provide a larger view of remote participants in a four-panel format as shown, for example, in ​ , where live video streams 953, 955, and 957 are presented in respective portions 942, 944, and 946 of the tenth UI 910, and additional conference call session content 949 (in ​ , a chat session for the active session 253) is presented in a portion 948 of the tenth UI 910.

[0094] In ​ , the first user 120 has finished using the fifth device 930 to add the second whiteboard content 966 to the virtual whiteboard and has returned the fifth device 930 to the locked state, as shown in the display 932. The system 100 is configured to: in response to the fifth device 930 being locked and / or removed from the active pairing set with the second device 132, cause the tenth UI 910 to return to the state in ​The arrangement shown in includes a remote participant display area 950 presenting respective real-time video streams 953, 955, and 957 and now includes a whiteboard interface 912 including both the first whiteboard content 914 shown in and a second whiteboard content 966 added by the first device 120 using the fifth device 930. Thus, as shown in, the first user 120 can quickly and dynamically add and remove the fifth device 930 for performing a specific activity for the active session 253. ​ and the whiteboard interface 912 including both the first whiteboard content 914 shown in and a second whiteboard content 966 added by the first device 120 using the fifth device 930. Thus, as shown in, the first user 120 can quickly and dynamically add and remove the fifth device 930 for performing a specific activity for the active session 253. ​ As shown in, the first user 120 is able to quickly and dynamically add and remove the fifth device 930 for performing a specific activity for the active session 253.

[0095] ​ and ​ illustrate an eighth example of the system 100 shown in, which performs unified UI interactions for user activities across multiple devices included in an active pairing set, where the first user 120 uses the second device 132 to initiate the transmission of content generated in a conference call session to other participants via the UI on the first device 130. The eighth example in and continues from the example shown in and can be combined with any of the examples described in and. ​ , ​ and ​ illustrate an eighth example of the system 100 shown in, which performs unified UI interactions for user activities across multiple devices included in an active pairing set, where the first user 120 uses the second device 132 to initiate the transmission of content generated in a conference call session to other participants via the UI on the first device 130. The eighth example in and continues from the example shown in and can be combined with any of the examples described in and. ​ and ​ The eighth example in and continues from the example shown in and can be combined with any of the examples described in and. ​ and can be combined with any of the examples described in and. ​ and ​ The eighth example in and continues from the example shown in and can be combined with any of the examples described in and.

[0096] In, the first user 120 wishes to provide a copy of the content created in the active session 253 (in this example, the third whiteboard content 1014 seen in the whiteboard interface 912 of the tenth UI 910) to the participants and / or invitees of the active session 253. To this end, using the hand 122 to provide input via the touch screen 239, the first user 120 actuates the UI control 1022 to expose and then actuates the content sharing UI control 1024. ​ In, the first user 120 wishes to provide a copy of the content created in the active session 253 (in this example, the third whiteboard content 1014 seen in the whiteboard interface 912 of the tenth UI 910) to the participants and / or invitees of the active session 253. To this end, using the hand 122 to provide input via the touch screen 239, the first user 120 actuates the UI control 1022 to expose and then actuates the content sharing UI control 1024.

[0097] As shown in, the system 100 is configured to: in response to the actuation of the content sharing UI control 1024 and the second device 132 being included in the active pairing set with the first device 130, automatically generate an email application (e.g., Microsoft Outlook running on the first device 130 ​ As shown in, the system 100 is configured to: in response to the actuation of the content sharing UI control 1024 and the second device 132 being included in the active pairing set with the first device 130, automatically generate an email application (e.g., Microsoft Outlook running on the first device 130 TM)A draft email opened in the twelfth UI 1030 presented on the first display 910. In some implementations, the system 100 may be configured to respond similarly to the termination of the active session 253. The draft email includes a first attachment 1032 containing whiteboard content 1014. In some examples, the draft email may include a second attachment 1033 containing a machine-generated transcription of the active session 253. Additionally, as shown in ​ , the recipient 1034 of the email has been automatically populated with the participants of the active session 253, and the subject 1036 of the email has been automatically generated. In some examples, initial text may be generated for the body 1038 of the email. By processing the draft email on the first device 130, the first user 120 benefits from, among other things: a keyboard 219 for more easily entering text for the body 1038, spell checking, access to contacts, automatically suggested additional contacts for the email, contact auto-completion, and a file browser for identifying additional attachments that can be added via "drag and drop" interactions.

[0098] Note that the system 100 is configured to: in response to the second device 132 not being in the active pairing set, respond to the actuation of the UI control 1024 in "standalone mode" by generating and sending a similar email to the participants of the active session 253, which may include using the associated UI on the second device 132. However, a more efficient, full-featured, and familiar experience can be provided to the first user 120 by completing the draft and sending the email via the first device 130, which allows the first device 130 to automatically enter an aspect of activity where it can provide an improved experience for the first user 120.

[0099] ​ , ​ , ​ , ​ and ​ illustrate a ninth example of the system 100 shown in ​ , ​ and ​ that performs unified UI interactions for user activities across multiple devices included in the active pairing set, where, when participating in a conference call session, the first user 120 invites additional participants via the handheld third device 134. In ​ the ninth example can be combined with any of the examples described in ​ and ​ . ​ Continuing from the example shown in ​ , but the first device 130 is not active and is not included in the active pairing set that includes the second device 132 and the third device 134.

[0100] In ​ , the second device 132 operates as an audio-visual endpoint for the first user 120 to participate in the active session 253, as described in ​ As described in ​ , the sixth software program instance 340 presents a third UI 342 on the second display 230, where real-time video streams 353, 355, 357, and 359 are presented in corresponding portions 342, 344, 346, and 348. Additionally, the third UI 342 includes an invitation UI control 1102, which can be actuated by the first user 120 to present an invitee selection UI (not shown) on the second display 230 for selecting and requesting participants to join the current active session 253.

[0101] However, in this example, the first user 120 would prefer to use the third device 134 to select and request participants to join the current active session 253. For example, the first user 120 may not want the invitee selection UI to block the current content of the second display 230, may want to be more discreet in sending requests to participants, or may find the UI provided by the third device 134 to be more efficient or familiar than the invitee selection UI. At the time shown in ​ , the third device 134 is included in the active pairing set with the second device 132. The eighth software program instance 1110 (labeled "Eighth APP") is a contact list application running in the runtime environment 812 on the third device 134 and presents a thirteenth UI 1112 with a contact list on the display 810, including the first contact 1114 selected by the first user 120 in ​ .

[0102] In ​In response to the selection of the first contact 1114, the eighth software program instance 1110 presents on the display 810 a fourteenth UI 1120 that displays details of the first contact 1114. Additionally, the fourteenth UI 1120 includes a contact action section 1122 that includes a plurality of UI controls for initiating actions related to the first contact 1114 by the third device 134 and / or the system 100, such as, but not limited to: making a call, sending a message, and / or sending an email. The third software program instance 135 configured to interact with the control service 250 is configured to interact with elements of the service 100, such as the control service 250, the pairing service 150, and / or the second device 132. Additionally, the third software program instance 135 is configured to interact with the eighth software program instance 1110; for example, as a plug-in component invoked by the eighth software program instance 1110 and / or a callback registered with the eighth software program instance 1110 and / or the runtime environment 812. In response to the third software program instance 135 obtaining from the service 100 information that the third device 134 is included in the active pairing set and the first device 120 is currently participating in the active session 253, the action section 1122 of the fourteenth UI 1120 includes a contact invitation UI control 1124 that indicates to the first user 120 to issue a request via the fourteenth UI 1120 to the first contact 1114 to join the current active session 253. In ​ In response to the first user 120 having actuated the contact invitation UI control 1124, the third device 134 sends a message (not shown) to the service 100, and in response, the service 100 will issue a request to the first contact 1114 to join the active session 253. For example, the control service 250 can be configured to request the conferencing service 180 to issue an invitation. Additionally, a confirmation 1128 is presented on the display 810.

[0103] In ​ In response to the first contact 1114 having received from the system 100 a request to join the active session 253 and having accepted the request (such as, but not limited to, by clicking on a link in an invitation email), the first contact 1114 is added as a participant in the active session 253. In response, the third UI 342 is rearranged from the arrangement having four sections 352, 354, 356, and 357 shown in ​ to the new arrangement shown in ​ which has a section 1142 for presenting a live video stream 1143 of the first contact 1144 and sections 1144, 1145, 1146, and 1147 for presenting live video streams 353, 355, 357, and 359, respectively. ​ Shows ​An alternative, where a request to join the active session 253 is sent to the first contact 1114 by initiating a real-time call via the conference call system 180. Similar to that shown in ​ The third UI 342 is arranged in a five-panel layout. Initially, the section 1142 indicates that a call to the first contact 1114 is pending. Once the first contact 1114 accepts the call, the section 1142 presents a real-time video stream 1143 for the first contact 1114, as shown in ​ shown.

[0104] ​ ,, ​ and ​ illustrate the tenth example of the system 100 shown in ​ , ​ and ​ which performs unified UI interactions for user activities across multiple devices included in the active pairing set, where the handheld third device 134 is temporarily recruited as a media input device for a virtual whiteboard presented by the second device 132. The tenth example in Figures 12A - 12C continues from the example shown in Figure 11A and can be combined with any of the examples described in Figures 3A - 11E and Figures 13A - 16C .

[0105] In Figure 12A , the first user 120 is using the third device 134 to capture a photo of a portion 1212 (in this example, a portion of a notebook page) of the physical object 1210. In some examples, in response to the system 100 determining that the third device 134 is actively paired with the second device 132, a user interface control (not shown) that can be presented by the second device 132 initiates the use of the third device 134 in this way. The second device 132 can present an indication that the third device 134 is available for this purpose to facilitate discovery of this function by the first user 120.

[0106] Figure 12BShows the fifteenth UI 1220 presented on the display 810, which allows the first user 120 to select the captured photo 1222, and in this example, identify the cropped image 1224 from the photo 1222. Similarly, the fifteenth UI 1220 includes an image action section 1225, which includes a plurality of UI controls for initiating actions related to the image 1224 by the third device 134 and / or the system 100, such as but not limited to, copying the image 1224 to the clipboard and / or sending the image 1224 by email. Additionally, in response to the third software program instance 135 obtaining information from the service 100 that the third device 134 is included in the active pairing set and the first user 120 is currently participating in the active session 253, and in some examples, obtaining information that the virtual whiteboard is active in the active session 253, the image action section 1224 of the fourteenth UI 1120 includes a whiteboard upload UI control 1226 for adding the image 1224 to the virtual whiteboard. The presentation of the whiteboard upload UI control 1226 can be achieved through interaction with the third software program instance 135, which is very similar to that described in Figure 11B and Figure 11C as described.

[0107] In Figure 12C as shown, the second display 230 is showing the tenth UI 910 as shown in Figure 9A which includes the first whiteboard content 914 in the whiteboard interface 912. The system 100 is configured to: in response to the actuation of the whiteboard upload UI control 1226, obtain the image 1224 from the third device 134 and add the new whiteboard image 1230 generated based on the image 1224 to the virtual whiteboard. In various implementations, the control service 250 and / or the second device 132 can be configured to obtain the image 1224 and add the whiteboard image 1230 (e.g., via interaction with the teleconference service 180). In some implementations, as shown in Figure 12C the first user 120 can use the hand 122 to move, rotate, resize, or otherwise adjust the whiteboard image 1230 via the whiteboard interface 912 on the second device 132.

[0108] Therefore, as shown in Figures 12A - 12CAs shown, the first user 120 is able to quickly and dynamically add and remove the third device 134 to be able to take advantage of its unique characteristics (in this example, a small form factor that facilitates casual photo capture). It should be understood that the system 100 can similarly be configured to use the third device 134 to capture images for other devices included in the active pairing set with the third device 134 and / or pairing-aware applications running on such devices. For example, the described techniques can be performed via the third party service 258 to copy such images to a clipboard provided by the operating environment 214 of the first device 130, thereby allowing the images to be pasted into various applications.

[0109] Figure 13A 、 Figure 13B and Figure 13C The diagram shows Figure 1 、 Figure 2A and Figure 2B An eleventh example of the system 100 shown in FIG. 1 is for performing a unified UI interaction for a user activity across multiple devices included in an active pairing set, wherein a file opened for viewing on a second device 132 is opened on a first device 130 by actuating a UI control on the second device 132. Figures 13A - 13C The eleventh example can be compared with Figures 3A - 12C and Figures 14A - 16C Combination of any of the examples described in .

[0110] exist Figure 13A In the example, the first device 130 and the second device 132 are included in the active pairing set. The ninth software program instance 1310 (labeled as "ninth APP") runs on the second device 132 in the execution environment 234 and is configured to display on the second display 230 similar to the one in the example. Figure 3A The sixteenth UI 1312 of the second UI 322 shown in FIG. Figures 3A - 12C In contrast to the example shown in , in this particular example, the first user 120 is not participating in a conference call session. The first user 120 is interested in opening a file to be viewed on the second device 132. Using hand 122, the first user 120 first actuates the file open UI control 1314, and in response, the system 100 identifies files recently opened by the first user 120 and presents some or all of the identified files in a file list 1316, as in Figure 8A The file list 820 described in is very similar. Figure 13A , the hand 212 of the first user 120 is used to select a file 1318 from a list of files 1316 for opening by the second device 132. In another example, the file 1318 may be sent by another user as an attachment to a message and selected using a corresponding UI control included in the UI displaying the message.

[0111] In Figure 13B , in response to the selection of file 1318, the second device 132 is configured to open the selected file 1318 using the associated tenth software program instance 1320 (in this example, the tenth software program instance 1320 is capable of opening word processing files such as file 1318). The tenth software program instance 1320 is a pairing-aware application that is configured to interact with the second software program instance 133 to interact with the pairing service 150 (which may occur via a client service such as third-party service 258). As shown in Figure 13B , the tenth software program instance 1320 presents a seventeenth UI 1322 on the second display 230, which includes a file content portion 1324 for displaying the file content 1326 stored in the file 1318.

[0112] The system 100 is configured to determine whether the active pairing set including the second device 132 includes one or more supporting editing devices other than the second device 132 that can be used to edit the file 1318 (e.g., using a content creation application suitable for the content of file 1318). The determination may be based on the capabilities and / or configuration information obtained for the devices included in the active pairing set. In this example, the first device 130 is identified as a supporting editing device. As shown in Figure 13B , in response to identifying a supporting editing device, the system 100 is configured to present a file editing UI control 1328 in the seventeenth UI 1322 to allow the first user 120 to edit the file 1318 on the supporting editing device. In some implementations, if no supporting editing device is included in the active pairing set but a paired (and currently not actively paired) supporting editing device is identified, the system 100 is configured to indicate (e.g., by displaying an indication in the seventeenth UI 1322) that the paired device can effectively edit the file 1318 to make the first user 120 aware of this possibility. In Figure 13B , the hand 212 of the first user 120 is used to actuate the file editing UI control 1328.

[0113] In Figure 13C , in response to the actuation of the file editing UI control 1328, the system 100 is configured to send a message to the first device 130 (identified as a supporting editing device) that requests the first device 130 to open the file 1318 for editing. In response to receiving the message (e.g., by the first software program instance 131), the first device 130 is configured to open the file 1318 for editing using the associated eleventh software program instance 1330 (in this example, a word processing application such as Microsoft Word TM ) Figure 13CAs shown in, the eleventh software program instance 1330 presents the eighteenth UI 1332 on the first display 210, which includes a file content section 1334 for displaying and editing file content 1326 stored in a file 1318. Thus, as in Figures 13A - 13C As shown in, the first user 120 can conveniently and efficiently directly open a file on the second device 132 and be confident that the first user 120 can quickly and dynamically open a file on the active paired device, thereby providing a more capable application or a convenient user input device (such as the keyboard 219 of the first device 130).

[0114] Figure 14A and Figure 14B illustrate the twelfth example of the system 100 shown in Figure 1 , Figure 2A and Figure 2B which performs unified UI interactions for user activities across multiple devices included in the active paired set, where content elements included in a file being viewed on the second device 132 are opened for inspection on the first device 130 by actuating a UI control on the second device 132. The twelfth example in Figure 14A and Figure 14B can be combined with any of the examples described in Figures 3A - 13C and Figures 15A - 16C .

[0115] In Figure 14A , a different file has been opened on the second device 132 in the manner described for opening the file 1318 in Figure 13B , where the tenth software program instance 1320 presents the seventeenth UI 1322, and a file content section 1324 is displaying file content 1410 stored in the already opened file. The displayed file content 1410 includes content elements 1412 (in this example, a chart generated by a spreadsheet program). The system 100 is configured to determine whether the active paired set including the second device 132 includes one or more support inspection devices (e.g., using an application suitable for opening and / or editing the content elements 1412) in addition to the second device 132 that can be used to inspect the content elements 1412, similar to that described for identifying support editing devices for the file 1318 in conjunction with Figure 13B . In this example, the first device 130 is identified as a support inspection device. As in Figure 14AAs shown, in response to identifying a support check device for content element 1412, system 100 is configured to present a content element check UI control 1414 (adjacent to content element 1412 in this example) in the seventeenth UI 1322 to allow a first user 120 to check content element 1412 on the support check device. In some implementations, if the support check device is not included in the active pairing set but a paired (and not actively paired) support check device is identified, system 100 is configured to indicate (e.g., by displaying an indication in the seventeenth UI 1322) that the paired device can effectively check content element 1412 to make the first user 120 aware of this possibility. In Figure 14A the hand 212 of the first user 120 is used to actuate the element check UI control 1414.

[0116] In Figure 14B response to actuation of the content element check UI control 1414, system 100 is configured to send a message to a first device 130 (identified as the support check device) requesting the first device 130 to open content element 1412. In response to receiving this message (e.g., by the first software program instance 131), the first device 130 is configured to use the associated twelfth software program instance 1420 (a spreadsheet application such as Microsoft Excel in this example) TM ) to obtain and open content element 1412 and / or a file associated with content element 1412. In this example, the spreadsheet file used to generate content element 1412 has been opened by the twelfth software program instance 1420, which presents a nineteenth UI 1422 showing the associated file content 1424 for the spreadsheet file. Thus, as shown in Figure 14A and 14B the first user 120 can conveniently and efficiently directly open the file on the second device 132 and be confident that the first user 120 can quickly and dynamically check the content elements of the file on the actively paired device, thereby providing a more capable application or a convenient user input device (such as the keyboard 219 of the first device 130) for checking the content elements. Note that although the UI controls 1328 and 1414 are presented within the UI 1322 in Figure 13B and Figure 14A , in some examples, such UI controls can be presented outside the UI for displaying the file.

[0117] Figure 15A 、 Figure 15B and Figure 15C illustrate in Figure 1 、 Figure 2A and Figure 2BA thirteenth example of system 100 as shown, which performs unified UI interactions for user activities across multiple devices included in an active pairing set. In this example, a first device 130 presents a UI control to provide a geographical location identified in the content presented by the first device 130 to a third device 134. In Figures 15A - 15C The thirteenth example in Figures 3A - 14B can be combined with any of the examples described in Figures 16A - 16C and Figure 15A and Figure 15B In these examples, the first device 130 and the third device 134 are included in the active pairing set.

[0118] In 15A, a first user 120 is accessing a website via a twentieth UI 1512 presented on a first display 210 through a web browser application instance 1510 running on the first device 130 (e.g., in a runtime environment 214). The twentieth UI 1512 is presenting a first web page 1514, which includes a map element 1516 and an associated street address element 1517 (in this example, a text element including the street address). A pairing-aware thirteenth software program instance 1520 is configured to interact with the web browser application instance 1510 and a first software program instance 131. In some implementations, the thirteenth software program instance 1520 can operate as a plug-in component for the web browser application instance 1510 and can be configured to identify the presence of information items of a specific information type (in this example, a geographical location information type) and obtain such information items - in this example, the geographical location corresponding to the map element in the web page (e.g., by identifying the map element generated by a known map service and obtaining the geographical location based on the map element) and / or the street address element (e.g., by identifying HTML <address>Contact address labels and parse the street address included in the label).

[0119] System 100 is configured to determine whether the active pairing set including the first device 130 includes one or more support guidance devices (e.g., a map application supporting guidance using satellite-based geographic services) that can be used for navigation to the indicated geographical location. The determination can be based on the capabilities and / or configuration information obtained for the devices included in the active pairing set. In this example, the third device 134 is identified as a support guidance device. System 100 is configured to present a location guidance UI control 1518 in response to obtaining both the geographical location from a web page and identifying a support guidance device, to allow the first user 120 to use the support guidance device to navigate to the obtained geographical location. In Figure 15A In the example shown, the presentation of the location guidance UI control 1518 includes presenting an information bar 1519 including the location guidance UI control 1518 in the twentieth UI 1512. In some examples, the location guidance UI control 1518 can be presented as a right-click menu item and / or a mouse-over item for the map element 1516 and / or the street address element 1517. In Figure 15A In, the location guidance UI control 1518 is actuated using the pointer 1504. In response, the first device 130 sends a message (which may include the geographical location) indicating that the location guidance UI control 1518 has been actuated to the system 100.

[0120] As shown in Figure 15B System 100 is configured to: in response to the actuation of the location guidance UI control 1518, send a message to the third device 134 (identified as a support navigation device) requesting the third device 134 to provide navigation guidance to the geographical location obtained in Figure 15A In Figure 15B At the time shown, the third device 134 is in a locked state. The third device 134 is configured to: in response to receiving the message when the third device is in the locked state, present a notification 1530 on the display 810 allowing the first user 120 to initiate the requested navigation guidance (once the third device 134 is unlocked). In Figure 15C In, the third device 134 has been unlocked by the first user 120, and in response to receiving the message and the third device 134 being in the unlocked state, performs navigation guidance to the obtained geographical location via a map mapping running on the third device 134 and presented in the twenty-first UI 1540 on the display 810.

[0121] Figure 16A 、 Figure 16B and Figure 16C illustrate in Figure 1 、 Figure 2A and Figure 2B A fourteenth example of the system 100 shown, which performs unified UI interactions for user activities across multiple devices included in an active pairing set, where a first device 130 initiates a user input operation on a third device 134, causing the user input to be delivered from the third device 134 to the initiating first device 130. In Figures 16A - 16C the fourteenth example in Figures 3A - 15C can be combined with any of the examples described in Figures 16A - 16C In

[0122] In Figure 16A a first user 120 accesses a website via a web browser application instance 1610 running on a first device 130 (e.g., in a runtime environment 214) via a twenty-second UI 1612 presented on a first display 210. The twenty-second UI 1612 is presented on a second web page 1614, which includes a signature input element 1616. A fourteenth software program instance 1620 of the awareness pairing is configured to interact with the web browser application instance 1610 and a first software program instance 131. In some implementations, the thirteenth software program instance 1620 can operate as a plug-in component for the web browser application instance 1610 and can be configured to identify the presence of a signature input element in a web page.

[0123] The system 100 is configured to determine whether an active pairing set including the first device 130 includes one or more signature input devices. The determination can be based on capabilities and / or configuration information obtained for the devices included in the active pairing set. In this example, both the second device 132 and the third device 134 are identified as signature input devices. The system 100 is configured to: in response to detecting a signature input element in a web page and identifying a signature input device, present a corresponding signature input UI control 1618 to allow the first user 120 to perform signature input using the signature input device. In Figure 16A the example shown in Figure 15A In [the context], a pointer 1604 is used to activate the signature input UI control 1618bb of the third device 134.

[0124] As shown in Figure 16B The system 100 is configured to: in response to the activation of the signature input UI control 1618ab or 1618bb for the third device 134, send a message to the third device 134 requesting the third device 134 to obtain a signature from the first user 120. The third device 134 is configured to: in response to receiving the message (e.g., via the third software program instance 135), present on the display 810 a twenty-third UI 1632 for a fifteenth software program instance 1630 running on the third device 134. The fifteenth software program instance 1630 is configured to obtain a signature 1634 provided using the hand 122. The twenty-third UI 1632 includes a signature completion UI control 1636 to complete the input of the signature 1634 via the third device 134. In some examples, the second device 132 can be applied to a similar effect in response to the activation of the signature input UI control 1618aa or 1618ba for the second device 132. As shown in ​ The system 100 is configured to: in response to the activation of the signature completion UI control 1636, transfer the signature 1634 from the third device 134 to the first device 130 and input the signature 1634 into the signature input element 1634. Thus, as shown in ​ The first user 120 can quickly and dynamically use a device with a touch screen or other desired input device for another device lacking such an input device in the active pairing set.

[0125] ​ is a flowchart illustrating an implementation of an exemplary process 1700 for providing a unified user interface between actively paired user computing devices. In some examples, some or all of the process 1700 can be combined with any of the features discussed with respect to ​ , ​ and ​ to perform, but it can also be performed using any other features described herein. In ​ In [the method], a first operation 1710 may include determining that a first user computing device is eligible to be actively paired with a second user computing device. In a second operation 1720, a process 1700 may include determining that the first user computing device is physically proximate to the second user computing device at a first time when both the first user computing device and the second user computing device are authenticated for the same user service account. In a third operation 1730, the process 1700 may include determining that the first user computing device and the second user computing device are included in an active pairing set together at the first time, based on determining that the first user computing device is eligible to be actively paired with the second user computing device and determining that the first user computing device is physically proximate to the second user computing device. In a fourth operation 1740, the process 1700 includes receiving an indication of actuation of a user interface control presented by the first user computing device and associated with a user activity at the first time. In a fifth operation 1750, the process 1700 includes: in response to receiving the indication and determining that the first user computing device and the second user computing device are included in the active pairing set together at the first time, causing the second user computing device to present a second user interface associated with the user activity.

[0126] ​ is a flowchart illustrating an implementation of an exemplary process 1800 for providing a unified user interface between actively paired user computing devices. In some examples, some or all of the operations in process 1800 may be performed in combination with any of the features discussed with respect to ​ , ​ and ​ but may also be performed using any other features described herein. In ​ In [the process], a first operation 1810 may include determining that a first user computing device is eligible to be actively paired with a second user computing device. In a second operation 1820, the process 1800 may include determining that the first user computing device is physically proximate to the second user computing device at a first time when both the first user computing device and the second user computing device are authenticated for the same user service account. In a third operation 1830, the process 1800 includes determining that the first user computing device and the second user computing device are included in an active pairing set together at the first time, based on determining that the first user computing device is eligible to be actively paired with the second user computing device and determining that the first user computing device is physically proximate to the second user computing device. In a fourth operation 1840, the process 1800 includes receiving an indication of an actuation of a user interface control associated with a conference call session and presented by the first user computing device at the first time. In a fifth operation 1850, the process 1800 includes, in response to receiving the indication and determining that the first user computing device and the second user computing device are included in the active pairing set together at the first time, causing the second user computing device to present a real-time component of the conference call session when a first user associated with the user service account joins the conference call session as a first participant.

[0127] In combination with ​ Detailed examples of the systems, devices, and techniques described herein will be presented herein for purposes of illustration of the present disclosure and its benefits. Such usage examples should not be construed as limiting the logical process embodiments of the present disclosure, nor should variations of the user interface methods from those described herein be considered outside the scope of the present disclosure. In some embodiments, the various features described in ​ are implemented in corresponding modules, which may also be referred to as and / or include logic units, components, units, and / or mechanisms. A module may constitute a software module (e.g., code embodied on a machine-readable medium) or a hardware module.

[0128] In some examples, a hardware module may be implemented mechanically, electronically, or using any suitable combination thereof. For example, a hardware module may include dedicated circuitry or logic units configured to perform specific operations. For example, a hardware module may include a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware module may also include programmable logic or circuitry temporarily configured by software to perform specific operations, and may include a portion of machine-readable media data and / or instructions for such configuration. For example, a hardware module may include software embodied within a programmable processor configured to run a set of software instructions. It should be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost, time, support, and engineering considerations.

[0129] Accordingly, the phrase "hardware module" should be understood to encompass a tangible entity capable of performing specific operations and configured or arranged in some physical manner, i.e., a physical structure, a permanently configured entity (e.g., hardwired), and / or a temporarily configured entity (e.g., programmed) to operate or perform the specific operations described herein in some manner. As used herein, "hardware-implemented module" refers to a hardware module. Considering examples where a hardware module is temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, in the case where a hardware module includes a programmable processor configured by software to be a dedicated processor, the programmable processor may be configured to be different dedicated processors (e.g., including different hardware modules) at different times, respectively. Software may accordingly configure one or more particular processors, e.g., to constitute a particular hardware module at one instance in time and different hardware modules at different instances in time. A hardware module implemented using one or more processors may be referred to as "processor-implemented" or "computer-implemented".

[0130] A hardware module is capable of providing information to and receiving information from other hardware modules. Accordingly, the described hardware modules may be considered to be communicatively coupled. In cases where multiple hardware modules are present simultaneously, communication may be achieved via signal transmission between two or more of the hardware modules (e.g., via appropriate circuitry and buses). In embodiments where multiple hardware modules are configured or instantiated at different times, communication between such hardware modules may be achieved, for example, by storing and retrieving information in a memory device accessible to the multiple hardware modules. For example, one hardware module may perform an operation and store the output in a memory device, and then another hardware module may access the memory device to retrieve and process the stored output.

[0131] In some examples, at least some of the operations of the method can be performed by one or more processors or modules implemented by processors. In addition, the one or more processors can also be operative to support the performance of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some of the operations can be performed by multiple computers (as an example of machines including processors) and / or among multiple computers, where these operations can be accessed via a network (e.g., the Internet) and / or via one or more software interfaces (e.g., application programming interface (API)). The execution of a particular operation among the operations can be distributed among processors, deployed not only within a single machine but also across multiple machines. The processor or the module implemented by the processor can be located in a single geographical location (e.g., in a home or office environment, or within a server farm), or can be distributed across multiple geographical locations.

[0132] ​ FIG. 1900 is a block diagram illustrating an exemplary software architecture 1902, various parts of which can be used in conjunction with the various hardware architectures described herein, which can implement any of the features described above. ​ FIG. 1902 is a non-limiting example of a software architecture, and it will be appreciated that many other architectures can be implemented to facilitate the functions described herein. The software architecture 1902 can run on hardware such as ​ a machine 2000, which includes a processor 2010, a memory 2030, and input / output (I / O) components 2050, etc. A representative hardware layer 1904 is illustrated and can represent, for example, ​ a machine 2000. The representative hardware layer 1904 includes a processing unit 1906 and associated executable instructions 1908. The executable instructions 1908 represent the executable instructions of the software architecture 1902, including the implementation of the methods, modules, etc. described herein. The hardware layer 1904 also includes a memory / storage device 1910, which also includes the executable instructions 1908 and accompanying data. The hardware layer 1904 can also include other hardware modules 1912. The instructions 1908 held by the processing unit 1908 can be part of the instructions 1908 held by the memory / storage device 1910.

[0133] The exemplary software architecture 1902 can be conceptualized as layers, with each layer providing various functions. For example, the software architecture 1902 can include layers and components such as an operating system (OS) 1914, libraries 1916, frameworks 1918, applications 1920, and a presentation layer 1944. In operation, the applications 1920 and / or other components within the layers can make API calls 1924 to other layers and receive corresponding results. The illustrated layers are representative in nature, and other software architectures can include additional or different layers. For example, some mobile or specialized operating systems may not provide a framework / middleware 1918.

[0134] The OS 1914 can manage hardware resources and provide common services. The OS 1914 can include, for example, a kernel 1928, services 1930, and drivers 1932. The kernel 1928 can act as an abstraction layer between the hardware layer 1904 and other software layers. For example, the kernel 1928 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The services 1930 can provide other common services for other software layers. The drivers 1932 can be responsible for controlling or interfacing with the underlying hardware layer 1904. For example, the drivers 1932 can include display drivers, camera drivers, memory / storage device drivers, peripheral device drivers (e.g., via Universal Serial Bus (USB)), network and / or wireless communication drivers, audio drivers, etc., depending on the hardware and / or software configuration.

[0135] The libraries 1916 can provide common infrastructure that can be used by the applications 1920 and / or other components and / or layers. The libraries 1916 generally provide functions for other software modules to perform tasks, rather than directly interacting with the OS 1914. The libraries 1916 can include system libraries 1934 (e.g., the C standard library) that provide functions such as memory allocation, string manipulation, file operations. Additionally, the libraries 1916 can include API libraries 1936 such as media libraries (e.g., supporting the rendering and manipulation of image, sound, and / or video data formats), graphics libraries (e.g., the OpenGL library for rendering 2D and 3D graphics on a display), databases (e.g., SQLite or other relational database functions), and web libraries (e.g., WebKit which may provide web browsing functionality). The libraries 1916 can also include a variety of other libraries 1938 to provide many functions for the applications 1920 and other software modules.

[0136] The framework 1918 (sometimes also referred to as middleware) provides a higher-level common infrastructure that can be used by the application 1920 and / or other software modules. For example, the framework 1918 can provide various graphical user interface (GUI) functions, advanced resource management, or advanced location services. The framework 1918 can provide a wide range of other APIs for the application 1920 and / or other software modules.

[0137] The application 1920 includes built-in applications 1940 and / or third-party applications 1942. Examples of built-in applications 1940 can include, but are not limited to: contact applications, browser applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 1942 can include any application developed by an entity other than the vendor of a particular platform. The application 1920 can use the functions available via the OS 1914, libraries 1916, framework 1918, and presentation layer 1944 to create a user interface to interact with the user.

[0138] Some software architectures use virtual machines, as illustrated by the virtual machine 1948. The virtual machine 1948 provides a runtime environment in which an application / module can execute as if it were running on a hardware machine (such as ​ the machine 2000). The virtual machine 1948 can be hosted by a host OS (e.g., OS 1914) or a hypervisor, and can have a virtual machine monitor 1946 that manages the operation of the virtual machine 1948 and its interoperability with the host operating system. A software architecture that may be different from the software architecture 1902 outside the virtual machine runs within the virtual machine 1948, such as the OS 1950, libraries 1952, framework 1954, application 1956, and / or presentation layer 1958.

[0139] ​ FIG. 0 is a block diagram illustrating components of an exemplary machine 2000 configured to read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any of the features described herein. The exemplary machine 2000 is in the form of a computer system within which instructions 2016 (e.g., in the form of software components) can run to cause the machine 2000 to perform any of the features described herein. Thus, the instructions 2016 can be used to implement the modules or components described herein. The instructions 2016 cause the unprogrammed and / or unconfigured machine 2000 to operate as a particular machine configured to perform the described features. The machine 2000 can be configured to operate as a stand-alone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 2000 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a node in a peer-to-peer or distributed network environment. The machine 2000 can be embodied as, for example, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a gaming and / or entertainment system, a smartphone, a mobile device, a wearable device (e.g., a smart watch), and an Internet of Things (IoT) device. Further, although only a single machine 2000 is illustrated, the term "machine" includes a collection of machines that individually or jointly execute the instructions 2016.

[0140] The machine 2000 can include a processor 2010, a memory 2030, and I / O components 2050 that can be communicatively coupled via, for example, a bus 2002. The bus 2002 can include multiple buses that couple various elements of the machine 2000 via various bus technologies and protocols. In an example, the processor 2010 (including, e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, or a suitable combination thereof) can include one or more processors 2012a through 2012n that can run the instructions 2016 and process data. In some examples, one or more of the processors 2010 can run instructions provided or recognized by one or more other processors 2010. The term "processor" includes multi-core processors that include cores that can run instructions concurrently. Although ​ multiple processors are shown, the machine 2000 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors each with a single core, multiple processors each with multiple cores, or any combination thereof. In some examples, the machine 2000 can include multiple processors that are distributed among multiple machines.

[0141] The memory / storage device 2030 may include a main memory 2032, a static memory 2034, or other memories, as well as a storage unit 2036, which the processor 2010 may access via the bus 2002. The storage unit 2036 and the memories 2032, 2034 store instructions 2016 embodying any one or more of the functions described herein. The memory / storage device 2030 may also store temporary, intermediate, and / or long-term data of the processor 2010. The instructions 2016 may also reside, in whole or in part, within the memories 2032, 2034, within the storage unit 2036, within at least one of the processors 2010 (e.g., in a command buffer or cache), within the memory of at least one I / O component 2050, or in any suitable combination thereof. Accordingly, the memories 2032, 2034, the storage unit 2036, the memory within the processors 2010, and the memory within the I / O components 2050 are examples of machine-readable media.

[0142] As used herein, "machine-readable media" refers to devices capable of temporarily or permanently storing instructions and data that cause a machine 2000 to operate in a particular manner, and may include, but are not limited to: random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical storage media, magnetic storage media and devices, caches, network-accessible or cloud storage devices, other types of storage devices, and / or any suitable combination thereof. The term "machine-readable media" applies to a single medium or a combination of multiple media used to store instructions (e.g., instructions 2016) for execution by a machine 2000, such that when the instructions are run by one or more processors 2010 of the machine 2000, the machine 2000 performs one or more of the features described herein. Thus, "machine-readable media" may refer to a single storage device, as well as a "cloud-based" storage system or storage network including multiple storage devices or devices. The term "machine-readable media" does not include signals themselves.

[0143] The I / O components 2050 may include a variety of hardware components suitable for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, and the like. The specific I / O components 2050 included in a particular machine will depend on the type and / or function of the machine. For example, a mobile device (such as a mobile phone) may include a touch input device, while a headless server or IoT device may not include such a touch input device. In ​ The specific examples of the I / O components illustrated herein are in no way limiting, and other types of components may be included in machine 2000. The grouping of the I / O components 2050 is for simplicity of discussion only, and the grouping is in no way limiting. In various examples, the I / O components 2050 may include user output components 2052 and user input components 2054. The user output components 2052 may include, for example, display components for displaying information (e.g., liquid crystal display (LCD) or projector), acoustic components (e.g., speakers), haptic components (e.g., vibration motors or force feedback devices), and / or other signal generators. The user input components 2054 may include, for example, alphanumeric input components (e.g., keyboards or touchscreens), pointing components (e.g., mouse devices, touchpads, or other pointing instruments), and / or haptic input components (e.g., physical buttons or touchscreens that provide position and / or touch force or touch gestures) configured to receive various user inputs, such as user commands and / or selections.

[0144] In some examples, the I / O components 2050 may include biometric components 2056, motion components 2058, environmental components 2060, and / or location components 2062, as well as a broad range of other physical sensor components. The biometric components 2056 may include, for example, components that detect body expressions (e.g., facial expressions, vocal expressions, hand or body gestures, or eye tracking), measure biometric signals (e.g., heart rate or brain waves), and identify people (e.g., via voice-, retina-, fingerprint-, and / or face-based recognition). The motion components 2058 may include, for example, acceleration sensors (e.g., accelerometers) and rotational sensors (e.g., gyroscopes). The environmental components 2060 may include, for example, lighting sensors, temperature sensors, humidity sensors, pressure sensors (e.g., barometers), acoustic sensors (e.g., microphones for detecting ambient noise), physical proximity sensors (e.g., infrared sensing of nearby objects), and / or other components that may provide an indication, measurement, or signal corresponding to the surrounding physical environment. The location components 2062 may include, for example, location sensors (e.g., global positioning system (GPS) receivers), altitude sensors (e.g., barometric pressure sensors from which altitude may be derived), and / or orientation sensors (e.g., magnetometers).

[0145] The I / O component 2050 may include a communication component 2064 that implements various techniques operable to couple the machine 2000 to the network 2070 and / or the device 2080 via respective communication couplers 2072 and 2082. The communication component 2064 may include one or more network interface components or other suitable devices interfacing with the network 2070. The communication component 2064 may include, for example, components adapted to provide wired communication, wireless communication, cellular communication, near field communication (NFC), Bluetooth communication, Wi-Fi, and / or communication via other modalities. The device 2080 may include other machines or various peripheral devices (e.g., coupled via USB).

[0146] In some examples, the communication component 2064 may detect identifiers or include components adapted to detect identifiers. For example, the communication component 2064 may include a radio frequency identification (RFID) tag reader, an NFC detector, an optical sensor (e.g., one-dimensional or multi-dimensional barcodes or other optical codes), and / or an acoustic detector (e.g., a microphone that identifies an audio signal of a tag). In some examples, location information may be determined based on information from the communication component 2062, such as but not limited to: geographical location via an Internet Protocol (IP) address, location via Wi-Fi, cellular, NFC, Bluetooth, or other wireless station identification and / or signal triangulation.

[0147] Although various embodiments have been described, the description is intended to be exemplary and not restrictive, and it should be understood that more embodiments and implementations within the scope of the embodiments are possible. Although many possible feature combinations are illustrated in the drawings and discussed in this detailed description, many other combinations of the disclosed features are also possible. Unless specifically restricted, any feature of any embodiment may be used in combination with or substituted for any other feature or element in any other embodiment. Thus, it should be understood that any of the features shown and / or discussed in this disclosure may be implemented together in any suitable combination. Thus, the embodiments are not restricted except as according to the appended claims and their equivalents. Additionally, various modifications and changes may be made within the scope of the appended claims.

[0148] Although the foregoing has described what is considered to be the best mode and / or other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and these teachings may be applied in many applications, only some of which are described herein. The appended claims are intended to claim any and all applications, modifications, and changes that fall within the true scope of this teaching.

[0149] Unless otherwise specified, all measurements, values, ratings, positions, dimensions, sizes, and other specifications set forth in this specification (including the following claims) are approximate and not exact. They are intended to have a reasonable range consistent with the functions they relate to and the conventions in the fields they relate to.

[0150] The scope of protection is limited only by the following claims. When construed in accordance with this specification and the subsequent prosecution history and including all structural and functional equivalents, the scope is intended to and shall be construed to be consistent with the ordinary meaning of the language used in the claims. Nevertheless, no claim is intended to cover subject matter that fails to satisfy the requirements of the Patent Law, nor should it be construed in such a way. Any inadvertent adoption of such subject matter is hereby disclaimed.

[0151] Except as otherwise stated above, nothing that has been stated or shown is intended or should be construed to dedicate any component, step, feature, object, benefit, advantage, or equivalent to the public, whether or not mentioned in the claims.

[0152] It should be understood that the terms and expressions used herein have the ordinary meanings ascribed to such terms and expressions in their respective corresponding fields of investigation and study, unless a specific meaning is otherwise set forth herein. Relative terms such as "first" and "second" may be used solely to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprising," "including," or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed or elements inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "a" or "an" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes such element.

[0153] The abstract of the disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is understood that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, in the foregoing detailed description, it can be seen that for the purposes of simplifying the disclosure, various features are combined in various examples. This method of disclosure should not be construed as reflecting an intention that the claims require more features than are expressly recited in each claim. On the contrary, as reflected in the following claims, the subject matter of the present invention is not directed to all features of a single disclosed example. Accordingly, the following claims are hereby incorporated into the detailed description, where each claim stands on its own as a separately claimed subject matter.< / address>

Claims

1. A system for providing a unified user interface between paired user computing devices, comprising one or more processors and a machine-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to control the system to operate in two operating modes: a first operating mode that does not pair a first user computing device with a second user computing device; and a second operating mode that pairs the first user computing device with the second user computing device, wherein, The system is configured to: Determine an operation mode of the system by: Determine whether the first user computing device is eligible to be paired with the second user computing device; Determine whether the first user computing device is physically close to the second user computing device; When it is determined that the first user computing device is not eligible to be paired with the second user computing device, or when it is determined that the first user computing device is not physically close to the second user computing device, cause the system to operate in the first operation mode in which the second user computing device is not paired with the first user computing device; And When it is determined that the first user computing device is eligible to be paired with the second user computing device and the first user computing device is physically close to the second user computing device, cause the system to operate in the second operation mode in which the first user computing device and the second user computing device are paired; Receive a first indication to activate a first operation via a first user interface associated with the first user computing device; Execute the first operation to generate an operation result; In response to determining that the operation mode of the system is the first operation mode in which the second user computing device is not paired with the first user computing device, cause the operation result to be exclusively displayed on the first user computing device; And In response to determining that the operation mode of the system is the second operation mode in which the first user computing device is paired with the second user computing device, automatically cause the operation result to be displayed across the first user computing device and the second user computing device by: Causing the first user computing device to display a first portion of the operation result via the first user interface, and Causing the second user computing device to display a second portion of the operation result via a second user interface.

2. The system according to claim 1, wherein, Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and the instructions further cause the one or more processors to: Determine that the first user computing device and the second user computing device have transitioned to be included together in the first active pairing set at a second time prior to the first time; And In response to determining that the first user computing device and the second user computing device have transitioned to be included together in the first active pairing set at the second time, cause the first user computing device to start presenting the first user interface.

3. The system according to claim 1, wherein Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and the instructions further cause the one or more processors to: Determine that a third user computing device is included in a second active pairing set at a second time; Determine that the third user computing device is in a locked state at the second time; In response to determining that the third user computing device is included in the second active pairing set at the second time and determining that the third user computing device is in the locked state at the second time, causing the third user computing device to indicate that the third user computing device is available for a second operation; Determining that the third user computing device is included in the second active pairing set at a third time after the second time; Determining that the third user computing device is not in the locked state at the third time; And In response to determining that the third user computing device is included in the second active pairing set at the second time and determining that the third user computing device is not in the locked state at the third time, causing the third user computing device to present a third user interface associated with the second operation.

4. The system according to claim 1, wherein Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and the instructions further cause the one or more processors to: Determine that the first user computing device, the second user computing device, and a third user computing device are included together in a second active pairing set at a second time; And In response to determining that the first user computing device, the second user computing device, and the third user computing device are included together in the second active pairing set at the second time: Cause the first user computing device to present a third user interface associated with a second operation, Cause the second user computing device to present a fourth user interface associated with the second operation, and Cause the third user computing device to present a fifth user interface associated with the second operation.

5. The system according to claim 1, wherein Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and the instructions further cause the one or more processors to: Determine that the first user computing device and the second user computing device are included together in the first active pairing set at a second time before the first time; Determine that the second user computing device is capable of presenting a first document presented by the first user computing device at the second time; In response to determining that the first user computing device and the second user computing device are included together in the first active pairing set at the second time and determining that the second user computing device is capable of presenting the first document, cause the first user computing device to present an indication of the availability of the second user computing device for presenting the first document and a first user interface control, wherein the first user interface control is for requesting the second user computing device to present the first document; And In response to presenting the indication of the availability of the second user computing device, cause the second user computing device to present the first document via the second user interface.

6. The system according to claim 1, wherein, Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and the instructions further cause the one or more processors to: Determine that the first user computing device and the second user computing device are included together in a second active pairing set at a second time; In response to determining that the first user computing device and the second user computing device are included together in the second active pairing set at the second time, send a request message to the second user computing device requesting the second user computing device to start presenting a third user interface associated with a second operation; Receive a response message to the request message from the second user computing device, the response message indicating that the second user computing device will not display the third user interface because the second user computing device is currently presenting a fourth user interface; And In response to receiving the response message, cause the first user computing device to start presenting a fifth user interface associated with the second operation.

7. The system according to claim 1, wherein Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and the instructions further cause the one or more processors to: Determine that the first user computing device and the second user computing device are included together in the first active pairing set at a second time before the first time; Identify a first information item of a first information type in a document presented by the first user computing device at the second time; In response to the identification of the first information item and the determination that the first user computing device and the second user computing device are included together in the first active pairing set at the second time, cause the first user computing device to present an indication of the availability of the second user computing device associated with the first information item and a first user interface control, wherein the first user interface control is for providing the first information item to the second user computing device; And In response to presenting the indication of the availability of the second user computing device, cause the first information item to be provided to the second user computing device for presenting the second user interface.

8. A method for providing a unified user interface between paired user computing devices by controlling a user computing device to operate in two operating modes: a first operating mode in which the first user computing device is not paired with the second user computing device; And a second operation mode that pairs the first user computing device with the second user computing device, the method including: Determining the operation mode of the user computing device by: Determining whether the first user computing device is eligible to be paired with the second user computing device; Determining whether the first user computing device is physically close to the second user computing device; When it is determined that the first user computing device is not eligible to be paired with the second user computing device, or when it is determined that the first user computing device is not physically close to the second user computing device, cause the user computing device to operate in the first operating mode in which the second user computing device is not paired with the first user computing device; and When it is determined that the first user computing device is eligible to be paired with the second user computing device and the first user computing device is physically close to the second user computing device, cause the user computing device to operate in the second operating mode in which the first user computing device and the second user computing device are paired; Receive a first indication to activate the first operation via a first user interface associated with the first user computing device; Execute the first operation to generate an operation result; In response to determining that the operating mode of the user computing device is the first operating mode in which the second user computing device is not paired with the first user computing device, cause the operation result to be exclusively displayed on the first user computing device; and In response to determining that the operating mode of the user computing device is the second operating mode in which the first user computing device is paired with the second user computing device, automatically cause the operation result to be displayed across the first user computing device and the second user computing device by: Causing the first user computing device to display a first portion of the operation result via the first user interface, and Causing the second user computing device to display a second portion of the operation result via a second user interface.

9. The method according to claim 8, wherein Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and further includes: Determining that the first user computing device and the second user computing device have changed to be included together in the first active pairing set at a second time before the first time; and In response to determining that the first user computing device and the second user computing device have changed to be included together in the first active pairing set at the second time, causing the first user computing device to start presenting the first user interface.

10. The method according to claim 8, wherein, Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and further includes: Determining that the first user computing device and the second user computing device have changed to not be included together in a second active pairing set at a second time before the first time; and In response to determining that the first user computing device and the second user computing device have changed to not be included together in the second active pairing set at the second time: Causing the second user computing device to stop presenting a third user interface associated with a second operation, and Cause the first user computing device to start presenting a fourth user interface associated with the second operation.

11. The method according to claim 8, wherein, Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and further includes: Determining that the first user computing device and the second user computing device are included together in a second active pairing set at a second time; In response to determining that the first user computing device and the second user computing device are included together in the second active pairing set at the second time, sending a request message to the second user computing device requesting the second user computing device to start presenting a third user interface associated with a second operation; Receiving a response message to the request message from the second user computing device, the response message indicating that the second user computing device will not display the third user interface because the second user computing device is currently presenting a fourth user interface; and In response to receiving the response message, causing the first user computing device to start presenting a fifth user interface associated with the second operation.

12. The method according to claim 8, wherein, Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and further includes: Determining that the first user computing device and a third user computing device are included together in a second active pairing set at a second time; In response to determining that the first user computing device and the third user computing device are included together in the second active pairing set at the second time, causing the third user computing device to present a third user interface for obtaining a media item captured by the third user computing device; Receiving a first media item obtained via the third user interface; and In response to receiving the first media item, transmitting the first media item to the first user computing device and causing the first user computing device to present the first media item.

13. The method according to claim 8, wherein Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and further includes: Determining that the first user computing device and the second user computing device are included together in a second active pairing set at a second time; In response to determining that the first user computing device and the second user computing device are included together in the second active pairing set at the second time, sending a request message to the second user computing device requesting the second user computing device to start presenting a third user interface; Receiving a response message from the second user computing device, the response message indicating that the presentation of a fourth user interface associated with a second operation by the second user computing device is stopped to present the third user interface; In response to receiving the response message, cause the first user computing device to present a fifth user interface associated with the second operation.

14. The method according to claim 12, wherein Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and further includes: Determining that the first user computing device and the third user computing device are included together in the second active pairing set at a third time before the second time; and In response to determining that the first user computing device and the third user computing device are included together in the second active pairing set at the third time, cause the first user computing device to present a user interface control for initiating presentation of the third user interface by the third user computing device.

15. The method according to claim 8, wherein Pairing the first user computing device with the second user computing device includes including the first user computing device and the second user computing device together in a first active pairing set at a first time, and further includes: Determining that the first user computing device and the second user computing device are included together in the first active pairing set at a second time before the first time; Determining that the second user computing device is capable of presenting a first document presented by the first user computing device at the second time; In response to determining that the first user computing device and the second user computing device are included together in the first active pairing set at the second time and determining that the second user computing device is capable of presenting the first document, cause the first user computing device to present an indication of the availability of the second user computing device for presenting the first document and a first user interface control, wherein the first user interface control is for requesting the second user computing device to present the first document; and In response to presenting the indication of the availability of the second user computing device, cause the second user computing device to present the first document via the second user interface.

Images