Telephone conference interface and controls for paired user computing devices

Abstract

Technology for providing a unified user interface between actively paired devices, comprising: based on determining that the first device is physically proximate to the second device when both the first device and the second device are authenticated for the same user service account, the first device and the second device are actively paired at a first time; receiving an indication of actuation of a first user interface control associated with a teleconference session and presented by the first device at the first time; and in response to receiving the indication and determining that the first device and the second device are actively paired together at the first time, causing real-time components of the teleconference session to be presented by the second device when a first user associated with the user service account joins the teleconference session as a participant.

Description

Technical Field

[0001] The present application relates generally to interaction of computing devices, and more particularly to systems and methods for providing a unified user interface between actively paired user computing devices. Background Art

[0002] Many user computing devices are portable, such as, but not limited to, laptops, notebooks, smartphones, and tablets. For many people, work may involve using and interacting with multiple different computing devices, including multitasking across devices. In addition, the work environment has become more dynamic. For example, an increasing amount of work is being done across homes, coffee shops, airports, conference rooms, and personal workspaces, and on-demand workspace allocation (where individuals may work in different locations every day) is becoming more and more common. New and improved solutions are needed to improve user interaction with these different devices and work environments, and to more fully realize the capabilities provided by these different devices and work environments. Summary of the invention

[0003] According to a first aspect of the present disclosure, a system suitable for providing a unified user interface between actively paired user computing devices includes one or more processors and a machine-readable medium including instructions, which, when executed by the one or more processors, can cause the one or more processors to: determine that a first user computing device is eligible for active pairing with a second user computing device. The instructions can also cause the one or more processors to: determine that the first user computing device is physically close to the second user computing device at a first time when both the first user computing device and the second user computing device are authenticated for the same user service account. In addition, the instructions can cause the one or more processors to: determine that the first user computing device and the second user computing device are included together in a first active pairing set at the first time based on determining that the first user computing device is eligible for active pairing with the second user computing device and determining that the first user computing device is physically close to the second user computing device. The instructions also cause the one or more processors to receive a first indication of actuation of a first user interface control associated with a teleconference session and presented by the first user computing device at the first time. Similarly, the instructions cause the one or more processors to: in response to receiving the first indication and determining that the first user computing device and the second user computing device are included together in the first active pairing set at the first time, cause a real-time component of the teleconference session to be presented by the second user computing device when a first user associated with the user service account joins the teleconference session as a first participant.

[0004] According to a second aspect of the present disclosure, a method for providing a unified user interface between actively paired user computing devices may include: determining that a first user computing device is eligible for active pairing with a second user computing device. The method may also include: determining that the first user computing device is physically close to the second user computing device at a first time when both the first user computing device and the second user computing device are authenticated for the same user service account. The method may also include: based on determining that the first user computing device is eligible for active pairing with the second user computing device and determining that the first user computing device is physically close to the second user computing device, determining that the first user computing device and the second user computing device are included together in a first active pairing set at the first time. In addition, the method includes: receiving a first indication of actuation of a first user interface control associated with a teleconference session and presented by the first user computing device at the first time. Similarly, the method includes: in response to receiving the first indication and determining that the first user computing device and the second user computing device are included together in the first active pairing set at the first time, so that the real-time component of the teleconference session is presented by the second user computing device when the first user associated with the user service account joins the teleconference session as a first participant.

[0005] This summary is provided to introduce a selection of concepts in a simplified form that are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings depict one or more implementations according to the present teachings by way of example only and not limitation. In the accompanying drawings, like reference numerals refer to the same or similar elements. In addition, it should be understood that the accompanying drawings are not necessarily drawn to scale.

[0007] Figure 1 An example of a system configured to provide a unified user interface between paired user computing devices is illustrated;

[0008] Figure 2A The diagram shows Figure 1 Exemplary details of various aspects of the system shown in Figure 1 Various examples of operation at the first position shown in ; Figure 2B The diagram shows Figure 1 and Figure 2A Exemplary details of various aspects of the matchmaking service shown in;

[0009] Figure 3A , Figure 3B and Figure 3C The diagram shows Figure 1-2B A first example of a system shown in , which performs a unified user interface (UI) interaction for a user activity across multiple devices included in an active pairing set, wherein use of a UI control presented by a first device causes a second device to join as an endpoint of a teleconference session;

[0010] Figure 4 The diagram shows Figure 1-2B A second example of the system shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein a projected portion of the active session initiated by a remote participant is presented via the first device while maintaining the second device as the primary audio-visual endpoint for the active session;

[0011] Figure 5A , Figure 5B and Figure 5C The diagram shows Figure 1-2B A third example of the system shown in , which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein a first user initiates a projection portion of the active session from the first device using a UI control on the second device;

[0012] Fig. 6A , Figure 6B and Figure 6C The diagram shows Figure 1-2B A fourth example of the system shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein capturing of real-time media streams is selectively performed by the first device and the second device based on detection of the first user's attention;

[0013] Fig. 7A , Figure 7B , Figure 7C and Fig.7D The diagram shows Figure 1-2B A fifth example of the system shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein media presentation is selectively performed via the first device and the second device based on detection of the attention of the first user;

[0014] Fig. 8A and Figure 8B The diagram shows Figure 1-2BA sixth example of the system shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein, while participating in a teleconference session, the first user selects a file for projection and initiates the projection portion of the session via a handheld third device using the selected file;

[0015] Fig. 9A , Fig. 9B and Fig. 9C The diagram shows Figure 1-2B A seventh example of the system shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein a presentation of a whiteboard interface for a teleconference session is automatically transferred between devices based on device states;

[0016] Fig. 10A and Fig. 10B The diagram shows Figure 1-2B An eighth example of the system shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein the first user initiates transmission of content generated in a teleconference session to other participants via a UI on the first device using the second device;

[0017] Fig.11A , Fig. 11B , Fig. 11C , Fig.11D and Fig.11E The diagram shows Figure 1-2B A ninth example of the system shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein, while participating in a teleconference session, the first user invites additional participants via a handheld third device;

[0018] Fig. 12A , Fig. 12B and Fig. 12C The diagram shows Figure 1-2B A tenth example of the system shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein the handheld third device is temporarily recruited as a media input device for a virtual whiteboard presented by a second device;

[0019] Fig.13A , Fig. 13B and Fig. 13C The diagram shows Figure 1-2B an eleventh example of the system shown in , which performs a unified UI interaction for a user activity across multiple devices included in an active pairing set, wherein a file opened for viewing on the second device is opened on the first device by actuating a UI control on the second device;

[0020] Fig.14A and Fig. 14B The diagram shows Figure 1-2B A twelfth example of the system 100 shown in , which performs a unified UI interaction for a user activity across multiple devices included in an active pairing set, wherein a content element included in a file being viewed on a second device is opened for inspection on the first device by actuating a UI control on the second device;

[0021] Fig.15A , Fig. 15B and Fig. 15C The diagram shows Figure 1-2B A thirteenth example of the system shown in , which performs a unified UI interaction for a user activity across multiple devices included in an active pairing set, wherein the first device presents a UI control to provide a third device with a geographic location identified in content presented by the first device;

[0022] Fig.16A , Fig. 16B and Fig. 16C The diagram shows Figure 1-2B A fourteenth example of the system shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein the first device initiates a user input operation on the third device, resulting in the user input being delivered from the third device to the initiating first device;

[0023] Fig.17 is a flow chart illustrating an implementation of an exemplary process for providing a unified user interface between actively paired user computing devices;

[0024] Fig.18 is a flow chart illustrating an implementation of an exemplary process for providing a unified user interface between actively paired user computing devices;

[0025] Fig.19 is a block diagram illustrating an exemplary software architecture, portions of which may be used in conjunction with various hardware architectures described herein that may implement any of the features described herein;

[0026] Fig. 20 is a block diagram illustrating components of an exemplary machine configured to read instructions from a machine-readable medium and perform any of the features described herein. DETAILED DESCRIPTION

[0027] In the following detailed description, many specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. However, it should be appreciated that the present teachings may be practiced without such details. In other cases, well-known methods, processes, components and / or circuits have been described at a relatively high level without details to avoid unnecessarily obscuring various aspects of the present teachings. In the following materials, directional indications (such as "top" or "left") are used only to provide a reference frame during the following discussion and are not intended to indicate the desired, desired, or intended direction of the item being described.

[0028] Figure 1 An example of a system 100 configured to provide a unified user interface ("UI") between paired user computing devices is illustrated. Note that the term "user interface" (or "UI") is not limited to a graphical user interface ("GUI"), but may also include other user interfaces, such as, but not limited to, voice input and presentation output. In this example, a first user 120 is shown using a first user computing device 130 (which may be more simply referred to as a "computing device" or "device") at a first location 110a during a first time period t1, at a second location 110b during a second time period t2, and at a third location 110c during a third time period t3. During the first time period t1, the first user 120 also uses a second user computing device 132 at the first location 110a. During the second time period t2, a third user computing device 134 associated with the first user 120 is located at the second location 110b. During the third time period t3, the first user 120 also uses a fourth user computing device 136 at the third location 110c. Figure 1 In the particular example shown in FIG, first device 130 is a portable computing device in the form of a notebook or laptop computing device, second device 132 is a desktop computing device that includes a touch screen, a camera, a microphone, and a speaker, first device 134 is in the form of a smartphone, and fourth device 136 is in the form of a desktop computing device (described by Figure 1 However, it is understood that the first device 130, the second device 132, the third device 134, and / or the fourth device 136 may be implemented in other forms. In some examples, a user computing device provides capabilities that another user computing device does not provide. For example, in Figure 1 In the embodiment, the first device 130 includes a physical keyboard, while the second device 132 does not include a physical keyboard. As another example, the second device 132 includes a touch screen, while the first device 130 does not include a touch screen.

[0029] The term "user computing device" refers to a computing device that is intended for end users and that is generally assumed to be used by a single user at a given time. For example, a software program run by a user computing device may require a password or other credentials to be provided in order to access resources associated with a user account. In some examples, a user computing device may support multiple such user accounts, allowing different users to "log in" to the device at their own time. In some examples, a user computing device may support a single user account; for example, most smartphone devices are assumed to be used in conjunction with a single individual.

[0030] A first software program instance 131 (which may be referred to as a "software instance," "application," or "app" in some examples) is run by the first device 130 during a first time period t1, a second time period t2, and a third time period t3. In addition, the first software program instance 131 is configured to interact with the pairing service 150 via one or more networks 140 in conjunction with a first user service account 152 associated with the first user 120. For example, the first software program instance 131 may be configured to provide credentials to the pairing service 150 to authenticate the first software program instance 131 for the first user service account 152. The interactions may include, for example, commands and / or data sent and received between the first software program instance 131 and the pairing service 150 to implement various aspects of the techniques described herein. These interactions may be implemented via, for example, a persistent connection, asynchronous communication, and / or polling for transmitting messages between the first software program instance 131 run by the first device 130 and the pairing service 150. It should be understood that references to a user computing device interacting with, or being configured to interact with, the pairing service 150 occur via a correspondingly configured software program instance executed by the user computing device.

[0031] During the first time period t1, the second device 132 is running the second software program instance 133. Much like described for the first software program instance 131, the second software program instance 133 is configured to interact with the pairing service 150 via the one or more networks 140 in conjunction with the first user service account 152. Thus, during the first time period t1, both the first device 130 and the second device 132 are configured to interact with the pairing service 150, and do interact with the pairing service 150 in conjunction with the first user service account 152. In some examples, the second software program instance 133 may be configured to interact with the pairing service 150 in conjunction with the first user service account 152 during the second time period t2 and / or the third time period t3. In some implementations, the use of the same first user service account 152 by different software program instances provides a consistent and robust security mechanism between the software program instances, such as by distributing and managing access tokens via the pairing service 150 to enable secure data access and exchange.

[0032] During the second time period t2, the third device 134 is running the third software program instance 135. Much like described for the software program instances 131 and 133, the third software program instance 135 is configured to interact with the pairing service 150 via the one or more networks 140 in conjunction with the first user service account 152. Thus, during the second time period t2, both the first device 130 and the third device 134 are configured to interact with the pairing service 150 and may interact with the pairing service 150 in conjunction with the first user service account 152. In some examples, the third software program instance 135 may be configured to interact with the pairing service 150 in conjunction with the first user service account 152 during the first time period t1 and / or the third time period t3 while remaining at the first location 110a. In some examples, the third device 134 is with the first user 120 during the first time period t1 and / or the third time period t3.

[0033] During the third time period t3, the fourth software program instance 137 is being run by the fourth device 136. Much like described for the software program instances 131, 133, and 135, the third software program instance 135 is configured to interact with the pairing service 150 via the one or more networks 140 in conjunction with the first user service account 152. Thus, during the third time period t3, both the first device 130 and the fourth device 136 are configured to interact with the pairing service 150, and do interact with the pairing service 150 in conjunction with the first user service account 152. In some examples, the fourth software program instance 137 may be configured to interact with the pairing service 150 in conjunction with the first user service account 152 during the first time period t1 and / or the second time period t2 while remaining at the third location 110c.

[0034] The system 100 is configured to automatically identify a time period (such as the first time period t1) for two or more computing devices (such as the first device 130 and the second device 132 during the first time period t1) during which each user computing device in each set interacts with the pairing service 150 in conjunction with the same user service account, and the user computing devices in each set are close to or may be close to each other and / or a common user or are in the vicinity of them. The set of user computing devices identified by the system 100 can be referred to as an "active pairing" and are included together in an active pairing set during the time period identified for the set of devices. Note that in some examples, three or more devices can be included in the active pairing set. For example, in some cases, when the first user 120 is at the first location 110a, the first device 130, the second device 132, and the third device 134 may be included in the active pairing set.

[0035] In some examples, the system 100 is configured to adjust the inclusion of the user computing device in the identified set based on whether the user has identified the user computing device as eligible for pairing (e.g., by explicitly identifying the device as eligible or not eligible for pairing). In an example where the system 100 is so configured, the first device 130 and the second device 132 have been identified as eligible for pairing, while the third device 134 has not yet been identified as eligible for pairing, the system 100 can identify the first device 130 and the second device 132 as being included in the active pairing set during the first time period t1 (based on both devices having been identified as eligible for pairing), even though the first device 130 and the third device 134 are in proximity to each other during the second time period t2, but the two devices are not identified as being included in the active pairing set during that time (because the third device 134 is not identified as eligible for pairing).

[0036] In some examples, the system 100 is configured to adjust the inclusion of two user computing devices in the identified set based on whether the user has identified the two user computing devices as eligible to be paired with each other (e.g., by explicitly identifying the devices as eligible or ineligible to be paired with each other). In some implementations, the user can perform a pairing process to identify the devices as eligible to be paired with each other. In an example of configuring the system 100 in this way, the first device 130 and the second device 132 have been identified as eligible to be paired together, the first device 130 and the fourth device 136 have been identified as suitable for pairing together, but the second device 132 and the fourth device 136 have not yet been identified as eligible to be paired together. In the case where the second device 132 and the fourth device 136 are close to each other, the system 100 will not identify the devices as being included together in the active pairing set, even though each device is capable of actively pairing with other devices.

[0037] exist Figure 1 In the example shown in , the system 100 identifies the first device 131 and the second device 132 as a first active pairing set of user computing devices during the first time period t1 (during which they are close enough for active pairing), but not during the second time period t2 or the third time period t3 (during which they are not close enough). Similarly, the system 100 identifies the first device 131 and the fourth device 136 as a second active pairing set of user computing devices during the third time period t3 (during which they are close enough), but not during the first time period t1 or the second time period t2 (during which they are not close enough).

[0038] The system 100 is also configured to control the presentation of a user interface for various activities by the user computing device based on whether the user computing device involved is actively paired during the associated time period. In some examples, in response to the system 100 identifying the user computing device as having transitioned between an inactive pairing and an active pairing, the user interface presented by the user computing device can be modified (including but not limited to adding a displayed user interface (UI) element, removing a displayed UI element, and / or changing the appearance of a displayed UI element). The system 100 is also configured to control the behavior of the software program instance based on whether the user computing device running the software program instance is actively paired during the associated time period.

[0039] Various mechanisms may be employed to identify time periods when user computing devices are in proximity to one another. For example, the first device 130 and the second device 132 may be configured to send and / or receive short-range radio signals (such as, but not limited to, in accordance with Bluetooth Low Energy (BLE) and / or Near Field Communication (NFC) communication protocol standards) and / or sound signals. By way of another example, the first device 130 and the second device 132 may each be configured to detect the presence of a particular user or device (such as, but not limited to, through RFID tag detection, facial recognition, and / or voice recognition technology).

[0040] Figure 1 Also illustrated are a second user 160a and a corresponding computing device 170a, a third user 160b and a corresponding computing device 170b, a fourth user 160c and a corresponding computing device 170c, and a fifth user 160d and a corresponding computing device 170d. In some examples discussed below, the second user 120 participates in a teleconference session with some or all of the users 160a, 160b, 160c, and 160d via a teleconference service 180. In some examples, the teleconference session involves capturing and transmitting video media streams and audio media streams for distribution to other participants, and receiving and presenting video media streams and audio media streams for other participants. The teleconference service 180 can provide additional capabilities through the teleconference session, such as, but not limited to: projecting display and / or application content to and / or from other participants, live transcription of spoken language, and / or whiteboard media streams and / or services.

[0041] Figure 2A The diagram shows Figure 1 Exemplary details of various aspects of the system 100 shown in Figure 1 110a. However, it will be appreciated that the details described may also be used in conjunction with other user computing devices (such as, but not limited to, Figure 1 Although the present disclosure mainly describes examples involving pairing of user computing devices used by a single first user 120, it should be understood that the system 100 supports providing similar functions related to the user computing devices associated with multiple users.

[0042] As mentioned above, the first user computing device 130 is in the form of a notebook or laptop computer. Figure 2AAs shown in , the first device 130 includes a first display device 210, such as but not limited to an LCD-based display panel, which can be used to present various graphical user interfaces ("GUIs"). The first device 130 also includes a first local storage device 212, such as a volatile and / or non-volatile storage device, which is shown as storing first local data 213, such as an electronic file. The first device 130 also includes a first operating environment 214, in which a first software program instance 131 is executed by the first device 130. Other software program instances can be executed by the first device 130 within the first operating environment 214 and / or other operating environments provided by the first device 130. The first device 130 also includes various input devices 216, such as but not limited to: a first camera 217, a first microphone 218 and / or a keyboard 219. The first device 130 also includes various output devices 220, such as but not limited to a first speaker 221. Other examples of input devices and output devices will be combined with Fig. 20 Have a discussion.

[0043] As mentioned above, the second user computing device 132 is in the form of a desktop computing device that is configured to receive touch-based input and to function as a user interface for a digital assistant (e.g., Microsoft Cortana). TM ) voice-driven interface to provide services. Figure 2A As shown in , the second device 132 includes a second display device 230, which can be used to present various GUIs. The second device 132 also includes a second local storage device 232 shown as storing second local data 233. The second device 132 also includes a second operating environment 234, in which the second software program instance 133 is run by the second device 132. Other software program instances can be run by the second device 132 in the second operating environment 234 and / or other operating environments provided by the second device 132. The second device 132 also includes various input devices 236, such as but not limited to: a second camera 237, a second microphone 238 and / or a touch screen 239. The second device 132 also includes various output devices 240, such as but not limited to a second speaker 241.

[0044] Figure 2A Included in Figure 1 In some examples, such as in Figure 2AAs shown in , teleconference service 180 can maintain session records 251 for teleconference sessions. In some examples, session records 251 can include scheduled sessions 252 for teleconference sessions that have been scheduled but not yet started. In some examples, session records 251 can include active sessions 253 for teleconference sessions with one or more active participants. Active sessions 253 may be from previously scheduled sessions or ad hoc sessions. Figure 2A In the example shown in , system 100 may include storage service 254 configured to receive, store, and make accessible storage data 255, including storage data item 256. For example, storage data item 256 may be an electronic document generated by a content creation application such as, but not limited to, a word processor application.

[0045] exist Figure 2A In the example shown in , the system 100 may include a control service 250 that is configured to interact with the pairing service 150 as a client service of the pairing service, allowing it to receive notifications of changes in the active pairing set from the pairing service 150, send messages or exchange messages with devices, and / or obtain device information. In addition, the control service 250 is configured to interact with the teleconferencing service 180 to obtain information from the teleconferencing service 180 and issue commands to the teleconferencing service 180. Via interaction with the pairing service 150, the teleconferencing service 180, and the user computing device, the control service 250 is configured to enable pairing-aware features to be provided for teleconferencing sessions without having to modify the teleconferencing service 180 itself. In some examples, the control service 250 is configured to interact with the storage service 254 to obtain information about the stored data 255 (e.g., file metadata) and / or to access items included in the stored data 255 or to make the items accessible. In some examples, the functionality described for the control service may be incorporated in part or in whole into the pairing service 150 and / or the teleconferencing service 180. Figure 2A In the example shown in , the system 100 may include a third-party client service 258 that is configured to interact with the pairing service 150 as a client service to enable pairing-aware features to be incorporated into the third-party service and / or application. In some implementations, the control service 250 and / or the third-party service 258 may obtain pairing-related information via the device rather than directly interacting with the pairing service 150 as a client service. Note that in some implementations, the program software instances 131, 133, 135, 137, and / or other pairing-aware program software instances described herein may be configured to interact with the pairing service 150 at least in part via a client service (such as the control service 250 or the third-party client service 258).

[0046] Figure 2B In the diagram Figure 1 and Figure 2A . The pairing service 150 includes a user authenticator 260 that is configured to identify and authenticate a user service account associated with a communication session between a user computing device and the pairing service 150. For example, the user authenticator 260 may be configured to receive and process credentials in the form of a username and a password hash. In some implementations, the user authenticator 260 supports the use of credentials obtained from an external authentication service (e.g., a token issued by a single sign-on (SSO) service), and may associate user account identifiers provided by such a service with their respective user service accounts. In some implementations, the user authenticator 260 is configured to issue an access token to a device that has been successfully authenticated, and the pairing service 150 is configured to receive and process the access token in subsequent communications with the device for authentication and authorization operations.

[0047] In some implementations, the pairing service 150 includes a client service interface 270 for interaction between the pairing service 150 and a "client" service configured to receive information about active pairing sets for one or more selected user service accounts and to interact with devices based on the device's status as a paired or unpaired device and / or the devices with which it is actively paired. Exemplary client services include Figure 2A 250 and the third-party client service 258 shown in . In some examples, the client service interface 270 is configured to perform authentication and / or authorization for the client service in conjunction with the selected user service account, which can include the use of the authentication techniques described for the user authenticator 260 and accepting a token issued to the device by the user authenticator 260. In some implementations, the client service can obtain information about the user's associated devices 286, available devices 288, and / or active pairing records 292 discussed below via the client service interface 270. In some examples, the service can be designed to obtain pairing-related information via the device rather than interacting with the pairing server 150 as a client service.

[0048] In some implementations, the matchmaking service 150 maintains user service account records 280 for user service accounts established with the matchmaking service 150, including a user service account record 282 for the user service account 152 of the first user 120. In some examples, the user service account records 282 are stored in multiple separate records. For example, the access tokens 284 associated with the user service accounts can be maintained by the user authenticator 260. In some implementations, the user service account records 282 include the access tokens 284 issued for the user service accounts 152.

[0049] In some implementations, the user service account record 282 includes user-associated device 286 information for devices that have been authenticated with the pairing service 150 for the user service account 152. In some implementations, the pairing service 150 is configured to add a device to the user-associated devices 286 in response to a successful user authentication (or "login") of the device with the pairing service 150. In some implementations, the pairing service 150 is configured to remove a device from the user-associated devices 286 in response to a "sign out" of the device with the pairing service 150. In some examples, a device that supports use by multiple different users is configured to "sign out" of the pairing service when a user finishes using the device.

[0050] In some implementations, the user service account record 282 includes available device 288 information that identifies devices that are currently authenticated with the pairing service 150 for the user service account 152, which may currently be included in the active pairing set of the device. In some examples, devices can be added and removed from the available devices 288 in response to the establishment and termination of a persistent network connection, but other schemes can be used to identify the available devices 288. In some examples, the device is configured to send a registration message to the pairing service 150 to indicate its availability for pairing with other devices, and to send a deregistration message to the pairing service 150 to indicate that it cannot be paired, and the pairing service 150 is configured to add and remove the device from the available devices 288, respectively. In some examples, the device is configured to send a registration message to the pairing service 150 via a software program instance executed by the device in response to an event, such as, but not limited to: a user login to the device, the launch of a software program instance, and / or the reestablishment of a network connection. In some implementations, the pairing service 150 is configured to add / maintain and remove devices from the available devices 288 based on receipt of periodic keep alive messages from the device and / or expiration of a timeout of a most recently received keep alive message from the device, respectively.

[0051] In some implementations, the user service account record 282 includes a device pairing eligibility record 290 that records the device's eligibility to pair with other devices, and the pairing service 150 is configured to identify devices that are eligible to be included in the active pairing set based on the device pairing eligibility record 290. In such implementations, the pairing service 150 includes a device pairing eligibility acceptor 262 that is configured to receive a device pairing eligibility message (e.g., from a user computing device) indicating a change to the device pairing eligibility and to modify the device pairing eligibility record 290 accordingly. In some implementations, some or all devices may be eligible by default, and the user may “opt out” of device pairing, where the “opt out” is recorded in the device pairing eligibility record 290. In some implementations, some or all devices may be not eligible by default, and the user may “opt in” the device to pair with other eligible devices, where the “opt in” is recorded in the device pairing eligibility record 290. In some implementations, for a given device, a user can identify specific other devices that can be affirmatively paired with it and / or negatively not paired with it, where this information is recorded in the device pairing eligibility record 290. For example, the system 100 can implement a device pairing process in which a user must affirmatively identify a device as being allowed to pair with another device, resulting in the device pairing eligibility record 290 indicating that the two devices can be included together in an active pairing set. In some implementations, a set of three or more devices can be identified as a pairing group, all of which must be physically close to each other to establish an active pairing set, where the pairing group is recorded in the device pairing eligibility record 290.

[0052] The pairing service 150 includes an active pairing detector 264 that is configured to: instantiate an active pairing set based on information received from devices associated with a user service account, add a device to a currently instantiated active pairing set, remove a device from a currently instantiated active pairing set (e.g., in response to the active pairing set including two or more devices after the removal), and / or de-instantiate an active pairing set (e.g., in response to fewer than two devices remaining in the active pairing set). In some implementations, in response to detecting an event indicating that a device has become available for inclusion in an active pairing set (such as the various events described above for adding a device to available devices 288) and / or an event that may reflect a change in the physical proximity of a device to another device (e.g., a detected change in the location of a device may remove it from physical proximity to another device), the active pairing detector 264 is configured to determine whether the device is to be included in the active pairing set. The active pairing detector 264 is configured to perform proximity-based pairing of the device, wherein it is configured to determine whether the device is (or may be) physically proximate to one or more other devices, include the device in an active pairing set with other devices based on a positive determination that the device is physically proximate to the one or more other devices, and not include the device in the active pairing set or remove the device from the active pairing set based on a negative determination that the device is not physically proximate to the other devices. In some examples, to determine whether the device is physically proximate to the one or more other devices, the active pairing detector 264 is configured to issue a proximity detection command to the device, the device is configured to perform a proximity detection action (e.g., in response to the proximity detection command), the device is configured to send device proximity information to the active pairing detector 264 based on the proximity detection action, and / or the active pairing detector 264 is configured to make a proximity determination based at least on the received device proximity information. In an implementation utilizing a device pairing eligibility record 290, the active pairing detector 264 is configured to identify an active pairing set that meets the device pairing eligibility record 290. In some implementations, a single device may be included in two different active pairing sets simultaneously. For example, in such an implementation, if device A is eligible to be actively paired with device B, device A is eligible to be actively paired with device C, device B is not eligible to be actively paired with device C, and devices A, B, and C are physically close to each other, a first active pairing set will include devices A and B, while a different second active pairing set will include devices A and C. In some implementations, the pairing service 150 is configured to use the active pairing detector 264 and / or the device pairing eligibility acceptor 262 to implement an initial pairing process between devices (e.g., which may modify the device pairing eligibility record 290 to indicate that the devices are eligible to be actively paired with each other).

[0053] The user service account record 282 includes an active pairing record 292 that records currently instantiated active pairing sets and the devices included in each active pairing set. The active pairing detector 264 is configured to update the active pairing record 292 based on instantiating an active pairing set, de-instantiating an active pairing set (a device included in the active pairing set), and removing a device from an active pairing set.

[0054] The pairing service 150 includes a device communicator 268 that is configured to deliver (or attempt to deliver) messages (such as, but not limited to, command or request messages) to a device, and in some implementations, also receive and route messages (such as, but not limited to, response messages) from a device. In some implementations, this allows the pairing service 150 to encapsulate the use of different message delivery techniques for different devices. For example, for some devices, it may be preferable to utilize a platform-specific notification framework rather than maintain a persistent network connection. In some implementations, the pairing service 150 is configured to receive messages from a device or client service for delivery by the device communicator 268 to a selected device, each of the devices included in the selected active pairing set, and / or each active paired device.

[0055] The pairing service 150 includes an active pairing notifier 266, which is configured to notify devices and / or client services about changes in active pairing sets. The active pairing notifier 266 is configured to notify the device of the pairing notification recipient 294 via the device communicator 268 about changes in the active pairing set including the device, such as but not limited to the device being added to the active pairing set or removed from the active pairing set, and another device being added to the active pairing set or removed from the active pairing set. In some examples, the device is configured to modify the UI and behavior based on whether it is included in the active pairing set and / or whether other devices are included in the active pairing set. In some implementations, the active pairing notifier 266 can notify the client service about changes in the active pairing set. In some examples, the client service is configured to modify commands issued to the device and client service behavior based on the notification provided by the active pairing notifier 266.

[0056] Figure 3A , Figure 3B and Figure 3C The diagram shows Figure 1 , Figure 2A and Figure 2B A first example of a system 100 is shown in which a unified UI interaction is performed for a user activity across multiple devices included in an active pairing set, wherein use of a UI control presented by a first device 130 causes a second device 132 to join as an endpoint of a conference call session. Figures 3A-3CThe first example in can be compared to Figure 4-16C Any combination of the examples in Figure 3A As shown in the first example, this occurs Figure 1 During the first time period t1 described in FIG. 1 , the first user 120 , the first device 130 , and the second device 132 are at the first location 110 a . Figure 3A At the time shown in and via interaction with the pairing service 150 , the first device 130 and the second device 132 are actively paired (meaning they are included in the same active pairing set).

[0057] At this time, the first software program instance 131 (labeled as "first APP") configured to interact with the pairing service 150 and / or the control service 250 is executed by the first device 130 in the first execution environment 214. In addition, the fifth software program instance 310 (labeled as "fifth APP") is also executed by the first device 130 in the first execution environment 214. In some implementations, some or all of the features described for the fifth software program instance 310 may be incorporated into the first software program instance 131 instead. In some implementations, the fifth software program instance 310 is configured to interact with the first software program instance 131 for all or some interactions with the pairing service 150 and / or the control service 250. In some implementations, the fifth software program instance 310 is configured to communicate directly with the pairing service 150 for all or some interactions with the pairing service 150 and / or the control service 250. The fifth software program instance 310 is configured to interact with the teleconference service 180, such as accessing the session recording 251 and participating in the teleconference session. For example, the fifth software program instance 310 may be incorporated into a computer program for Microsoft Windows TM Microsoft Teams for 10 operating systems TM Various features of the software.

[0058] Likewise, at this time, the second software program instance 133 (labeled as “second APP”) configured to interact with the pairing service 150 and / or the control service 250 is run by the second device 132 in the second operating environment 234. In addition, the sixth software program instance 320 (labeled as “sixth APP”) is also run by the second device 132 in the second operating environment 234. In some implementations, some or all of the features described for the sixth software program instance 320 may be incorporated into the second software program instance 133 instead. In some implementations, the sixth software program instance 320 is configured to interact with the second software program instance 133 for all or some interactions with the pairing service 150 and / or the control service 250. In some implementations, the sixth software program instance 320 is configured to communicate directly with the pairing service 150 for all or some interactions with the pairing service 150 and / or the control service 250. The sixth software program instance 320 is configured to interact with the teleconference service 180. In this example, the sixth software program instance 320 is configured to provide a “home screen” interface for the second device 132 .

[0059] exist Figure 3A , the first UI 312 is presented on the first display 210 for the fifth software program instance 310 and is displaying an agenda for a scheduled telephone conference session for the first user 120. The first UI 312 includes Figure 2A 253 in the active session 253. UI element 314 includes a first session join UI control 316 (labeled "Join"), which can be used to join the first user 120 as a participant in the active session 253, in which users 160a, 160b, 160c, and 160d have participated. In addition, a second UI 322 is presented on the second display 230 for the sixth software program instance 320 to provide a "home screen" interface. In this particular example, the second UI 322 includes a second session join UI control 324 (labeled "Join"), which can also be used to join the first user 120 as a participant in the active session 253.

[0060] Figure 3B The first user 120 has been illustrated by using Figure 3A302 as a result of actuating the first session join UI control 316. The system 100 is configured to process the actuation of the first session join UI control 316 based at least on whether the first device 130 is included in the active pairing set. In response to the first device 130 being included in the active pairing set, the system 100 has caused, for example, via the second software program instance 133, the seventh software program instance 340 running in the second execution environment 234 to operate as an audio-visual endpoint to enable the first user 120 to participate in the active session 253. The seventh software program instance 340 is configured to transmit outgoing session media 368 (labeled "Media Output") for the active session 253 via the network(s) 140 for presentation by devices used by the session participants, such as Figure 1 For example, the outgoing conversation media 368 may include real-time video of the first user 120 captured via the second camera 237 and / or real-time audio of the first user 120 captured via the second microphone 238.

[0061] The seventh software program instance 340 is also configured to receive and present incoming session media 366 for the active session 253. The seventh software program instance 340 is configured to present real-time video included in the incoming real-time media 366 in various areas of the third UI 342 on the second display 230. In this example, the third UI 342 includes a first area 352 presenting a first video stream 353 of user 160a, a second area 354 presenting a second video stream 355 of user 106b, a third area 356 presenting a third video stream 357 of user 160c, and a fourth area presenting a fourth video stream 359 of user 160d. The seventh software program instance 340 is also configured to present real-time audio included in the incoming real-time media 366 via the second speaker 241. Therefore, although the first session join UI control 316 presented by the first device 130 is used to initiate joining the active session 253, the visual and audio presentation of the remote participants and the video and audio capture of the first user 120 are performed by the second device 312 based on its active pairing with the first device 130. In addition, the third UI 342 includes first session UI controls 360, which include, for example, a first camera enable / disable UI control 361 (to switch video transmission of the first user 120), a first microphone mute UI control 362 (to switch audio transmission of the first user 120), a first volume UI control 363 (to adjust the volume for presenting audio to the first user 120), and / or a first session termination UI control 364 (to end the first user 120's participation in the active session 253). Thus, the second device 132 operates as the focus center for the active session 253, while leaving the first device 130 free for other activities, such as a third UI interface for an email program running on the first device 130.

[0062] In various implementations, the first device 130, the second device 132, and / or the control service 250 may make a positive determination that the first device 130 and the second device 132 are included in the active pairing set. The first device 130, the second device 132, and / or the control service 250 may select the second device 132 from the devices included in the active pairing set to operate as an audio-visual endpoint based at least on the positive determination, and cause the selected device to operate as an audio-visual endpoint (e.g., via a second message 328 (MSG328) sent by the system 100 to the second device 132). As an example, in response to actuation of the first session join UI control 316, the fifth software program 310 may transmit a first message 318 (MSG318) indicating actuation of the first session join UI control 316 to the control service 250. Then, in response to obtaining information that the first device 130 and the second device 132 are actively paired (e.g., through a notification from the pairing service 150) and receiving the first message 318, the control service 250 may select the second device 132 and transmit the second message 328 as a request for the second device 132 to operate as an audio-visual endpoint. As another example, in response to receiving information that the first device 130 is included in the active pairing set, the fifth software program 310 may operate in a pairing mode, and in response to being in the pairing mode, transmit the first message 318 to the second device 132 via the pairing service 150 (and be received by the second device 132 as the second message 328), wherein the first message 318 requests the second device 132 to operate as an audio-visual endpoint. In some examples, the second device 132 may deny the request in response to receiving the second message 328 (e.g., because the second device 132 is already used for another purpose), and in response, the system 100 will cause the first device 130 to function as an audio-visual endpoint. It should be understood that other schemes involving messages exchanged between and determined by the first device 130, the second device 132, and the control service 250 may be applied to achieve the same effect.

[0063] In some examples, the system 100 is configured to obtain capabilities and / or configuration information for devices included in the active pairing set, and select a device based on the obtained capabilities and / or configuration information. For example, the capabilities and / or configuration information for the second device 132, when available, may indicate that it is a preferred audiovisual endpoint, and the system 100 may select the second device 132 based on the indication. The configuration information may include, for example, user settings, group settings, or system settings.

[0064] Note that the first device 130 and the second device 132 are each configured to operate as an audio-visual endpoint when unpaired. Therefore, based on a determination by the first device 130 and / or the control service 250 that the first device 130 is unpaired, actuation of the first session join UI control 316 will instead cause the first device 130 to operate as an audio-visual endpoint for the active session 253, much like an actuation of the first session join UI control 316 for the first device 130. Figure 3B Similarly, based on the second device 132 and / or the control service 250 determining that the second device 132 is not paired, actuation of the second session join UI control 316 will cause the second device 132 to operate as an audiovisual endpoint for the active session 253.

[0065] As in Figure 3C As shown in , in some implementations, more than one device included in the active pairing set can be used to simultaneously present the video of other participants in the active session 253. Figure 3C In the example shown in FIG. 1 , a first user 120 joins an active session 253 with four other participants, wherein audio and video are captured and presented as in FIG. Figure 3B , after which two additional participants with respective real-time video streams 375 and 377 have joined active session 253. System 100 continues to use second device 132 to render audio for active session 253, render real-time video streams 353, 355, 357, and 359, and capture real-time video and audio of first user 120, wherein seventh software program instance 340 receives inbound media 366 and sends outbound media 368, as in Figure 3B Described in .

[0066] However, instead of presenting all six video streams 353, 355, 357, 359, 375, and 377 together on the second device 132, the system 100 is configured to cause multiple devices included in the active pairing set (and, in some examples, determined to be capable of video presentation based on the capabilities and / or configuration information obtained for the devices) to each present a portion of the video stream for the active session 253. This allows the size of the presentation to be larger than that achieved with a single device. In some implementations, the system 100 is configured to select a primary video presentation device and, based on information about the video stream and the devices included in the active pairing set, select and use one or more additional video presentation devices to present corresponding portions of the video stream. In some examples, use of a device as an additional video presentation device is based on activity type and / or activity level; for example, if the first device 130 has not been actively used recently, the system 100 is more likely to select and use the first device 130 as an additional video presentation device. In some examples, the use of additional rendering devices can be based on the number of video streams to be rendered; for example, system 100 can be configured to limit second device 100 to rendering four video streams unless there are more than six video streams (in which case a different rendering scheme can be used).

[0067] exist Figure 3C , based on the number of video streams and the fact that the first device 130 and the second device 132 are in the active pairing set, the system 100 is presenting real-time video streams 375 and 377 on the first device 130. As a result, the fifth software program instance 310 is configured to receive and present incoming session media 386 (labeled "Media Input") for the active session 253. The fifth software program instance 310 is configured to present the real-time video included in the incoming session media 386 in various areas of the fourth UI 372 on the first display 210. In this example, the fourth UI 372 includes a fifth area 374 that presents the video stream 375 and a sixth area 376 that presents the video stream 377. In this example, the second device 132 is still used to capture the video and audio of the first user 120 and send the outgoing session media 368, and present the audio for the active session 253, as in Figure 3B In some implementations, the system 100 is configured to present audio to a participant on the same device that is used to present the video stream to the participant.

[0068] Additionally, in some implementations, fifth software program instance 310 is configured to also present second session UI controls 380 on first display 210 that are similar to and presented simultaneously with first UI controls 360. In this example, second session UI controls include UI controls 381, 382, ​​383, and 384 that have the same functionality as corresponding UI controls 361, 362, 363, and 364 shown on second display 230. Note that actuation of UI controls 381, 382, ​​and 383 results in transmission of corresponding messages by first device 130 (e.g., to control service 250 and / or second device 132), which causes system 100 to cause seventh software program instance 340 to switch video transmission, switch audio transmission, and adjust volume for audio presentation, respectively. Both UI controls 384 on first device 130 and UI controls 364 on second device 132 can be used to end participation in active session 253. Thus, first user 120 can easily submit commands for active session 253 regardless of whether his attention is directed at first device 130 or second device 132 at that time. In some examples, fifth software program instance 310 can be configured to present second session UI controls 380 on first display 210 even when fourth UI 372 is not presented.

[0069] about Figure 3B and Figure 3C For example, the device used by the system 100 to present and capture audio and video for the conference call session can be dynamically changed in response to the device being added to or removed from the active pairing set. For example, if the second device 132 is turned off or loses network connection, the system 100 can be configured to use the first device 130 instead to present and capture all audio and video for the active session 253.

[0070] Figure 4 The diagram shows Figure 1 , Figure 2A and Figure 2B A second example of the system 100 shown in , which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein a projected portion of the active session 253 initiated by a remote participant is presented via the first device 130 while maintaining the second device 132 as the primary audio-visual endpoint for the active session 253. Figure 4 From Figure 3B The state shown in continues and can be Figures 3A-3C and Figure 5A-16C Combine any of the examples described in .

[0071] exist Figure 4In the example embodiment, one of the remote participants of the active session 253 has initiated a projection portion of the active session 253. The projection portion may include, for example, "screen casting" a portion of the remote participant's display, although other projection techniques may be used. The system 100 is configured to: in response to determining that the first device 130 and the second device 132 are included in the active pairing set and the notification of the projection portion, continue to use the second device 132 as the remote participant in the active session 253. Figure 3B , and at the same time, the first device 130 is caused to operate as a projection endpoint, wherein the projection content is presented on the first display 210. Various schemes involving messages exchanged between and determined by the first device 130, the second device 132, and the control service 250 can be used. For example, the control service 250 can be configured to recognize these conditions through interaction with the pairing service 150 and the teleconferencing service 180, and send a message to the first device 130 to request it to start operating as a projection endpoint of the projection portion.

[0072] In this example, fifth software program instance 310 is configured to receive and respond to (e.g., by presenting visual projection content 424 on first display 210) projection data 410 for the projection portion of active session 253. Fifth software program instance 310 (although a different software program instance may be used) is configured to present fifth UI 420 including content display area 422 in which visual projection content 424 included in projection data 410 is presented. Additionally, in some implementations, fifth software program instance 310 is configured to also present third session UI controls 430 on first display 210, which is similar to the previous presentation of fifth software program instance 310 for the projection portion of active session 253. Figure 3C 380 in the example. In this example, the third session UI controls 430 include the UI controls 381, 382, ​​and 384 described above. In some examples, the third session UI controls 430 also include a projection view UI control 440, which controls the presentation of the projection data 410 by the fifth software program instance 310. In this example, the projection view UI control 440 includes a back navigation UI control 441 and a forward navigation UI control 442, which allow the first user 120 to asynchronously move backward or forward through Figure 4 Thus, the first user 120 can easily submit commands for the active session 253, regardless of whether his attention is directed to the first device 130 or the second device 132 at the time.

[0073] Note that the system 100 is configured to: in response to the second device 132 not being in the active pairing set and being used to join the active session 253 that includes the projection portion, use the second device 132 to receive the projection data 410 and present the visual projection content 424 on the second display 230. The system 100 is configured to use the first device 130 in response to the first device 130 not being in the active pairing set in a similar independent manner. However, as in Figure 4 As shown in , the system 100 provides an improved experience to the first user 120 when both the first device 130 and the second device 132 are in the active pairing set during the active session 253 .

[0074] Figure 5A , Figure 5B and Figure 5C The diagram shows Figure 1 , Figure 2A and Figure 2B A third example of the system 100 shown in , which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein a first user 120 initiates a projection portion of an active session 253 from a first device 130 using a UI control on a second device 132. Figures 5A-5C The third example in Figure 3B The status shown continues and can be Figure 3A-4 and Figures 6A-16C Any combination of the described examples may be used. Figure 5A , the first user 120 is initiating the casting portion of the active session 253 via the second device 132. Since the second device 132 may already be the area of ​​user focus for the active session 253, it is more efficient to initiate the casting from the second device 132. In response, using the hand 122 to provide input via the touch screen, the first user 120 actuates the UI control 510 to present a casting device list 512. The system 100 is configured to identify devices included in the active pairing set with the second device 132 that is capable of casting content (e.g., based on the capabilities and / or configuration information obtained for the device), and present the identified devices as available casting devices with corresponding casting device selector UI controls 514. In this example, the first device 130 (which has been given a human-readable identifier "PC") is the only identified casting device, and therefore only one casting device selector UI control 514 (with the text "Cast PC") is included in the casting source list 512. Here, the first user 120 actuates the casting device selector UI control 514, selecting the first device 130.

[0075] Figure 5BThe diagram illustrates a projection source selection operation that can be performed by system 100 in response to determining that the selected device provides multiple projection sources (which, in some examples, can be identified by the selected device in response to actuation of device selector UI control 514). In this example, fifth software program instance 310 is configured to identify a projection source available from first device 130. However, if the selected device does not provide multiple projection sources, system 100 can skip the projection source selection operation. Figure 5B In response to determining that the selected first device 130 provides multiple projection sources, the system 100 displays a projection source selection UI 530 to the first user 120. In this example, the system 100 is configured to present the projection source selection UI 530 via the selected first device 130 (in this example, via the fifth software program instance 310 on the first display 210), but in some examples, the system 100 may alternatively display the projection source selection UI 530 via the second device 132 on which the device selector UI control 514 has been actuated. The projection source selection UI 530 includes a projection source selector UI control for each projection source available on the selected device. As shown in Figure 5B , the first device 130 is capable of providing a window-specific projection source, and therefore, the projection source selection UI 530 includes projection source selector UI controls 532, 534, and 536 for the respective windows 522, 524, and 526. Many devices are capable of providing the entire display area as a projection source, as indicated for the first device 130 by a projection source selector UI control 538 included in the projection source selection UI 530. Figure 5B In the example shown in , the first user 120 uses the pointer 502 to actuate the projection source selector UI control 532 for the window 522 .

[0076] Figure 5C Continue with initiation of the projection portion. The system 100 is configured to, in response to actuation of the projection device selector UI control 514 and / or the projection source selector UI control 532, configure the selected first device 130 to begin transmitting projection data 540 to the remote participant for presentation. In this example, the system 100 causes the fifth software program instance 310 to generate and transmit projection data 504. Additionally, in some implementations, the fifth software program instance 310 is configured to present the fourth session UI control 540 on the first display 210, as previously described for the first display 210. Figure 3C380 in the first session UI controls 360. In this example, the fourth session UI controls 540 include the UI controls 381, 382, ​​383, and 384 described above. In some examples, the fourth session UI controls 540 also include a projection source UI control 550 that controls the projection portion of the active session 253. In this example, the projection source UI control 550 includes a projection indicator 551 (indicating that the first device 130 is projecting) and a projection termination UI control 552 (which causes the system 100 to terminate the projection portion). In some implementations, the sixth program software instance 540 is configured to include a similar projection source UI control 560 in the first session UI controls 360. In this example, the projection source UI control 560 includes a projection indicator 561 (indicating that the first device 130 is projecting) and a projection termination UI control 562 (which causes the system 100 to terminate the projection portion).

[0077] Fig. 6A , Figure 6B and Figure 6C The diagram shows Figure 1 , Figure 2A and Figure 2B A fourth example of the system 100 shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein the capture of real-time media streams is selectively performed by the first device 130 and the second device 132 based on the detection of the attention of the first user 120. Figures 6A-6C The fourth example in Figure 5A The example shown continues and can be compared with Figures 3A-5C and Figure 7A-16C Combine any of the examples described in .

[0078] exist Fig. 6A In, the system 100 is configured to identify which devices among the devices included in the active pairing set and capable of real-time video capture are of interest to the first user 120 in response to the second device 132 being in the active pairing set with the first device 130, and configure the identified devices to capture and transmit a video stream of the first user 120 for the active session 253. In some implementations, the system is configured to identify the devices capable of real-time video capture based on capabilities and / or configuration information obtained for the devices (which may reflect a temporary camera disablement initiated by the user). In some implementations, the system 100 is configured to assess user attention based on, for example, detected head and / or eye direction (e.g., via images captured by camera 217 and / or camera 237), detected user proximity (e.g., via images captured by camera 217 and / or camera 237), and / or user input activity (e.g., via a keyboard, pointer, or touch screen). In Fig. 6AIn the particular example shown, both the first device 130 and the second device 132 are capable of video capture via respective cameras 217 and 237, and based on the detected gaze direction 610 towards the second device 132, the system 100 has determined that the attention of the first user 120 is directed to the second device 132. Based on this determination, the system 100 configures the second device 132 to capture and transmit an outbound real-time video stream 620 using the second camera 237, to capture and transmit an outbound real-time audio stream 630 using the second microphone 238, and to present an inbound real-time audio stream 640 using the second speaker 241. Likewise, based on this determination, the system 100 configures the first device 130 to not use the first camera 217, the first microphone 218, or the first speaker 221 for generating or presenting media for the active session 253.

[0079] exist Figure 6B In, continue Fig. 6A In the example of FIG. 6 , based on the detected gaze direction 612 toward the first device 130, the system 100 has determined that the attention of the first user 120 is newly directed toward the first device 130. Based on this determination, the system 100 configures the second device 132 to stop transmitting the outbound real-time video stream 620, and configures the first device 130 to capture and transmit the outbound real-time video stream 622 using the first camera 217 to replace the outbound real-time video stream 620. As a result, the real-time video provided to the remote participant by the active pairing set better captures the first user 120.

[0080] As in Figure 6C As shown in , the system 100 may also be configured to identify which devices among the devices included in the active pairing set and capable of real-time audio capture are of interest to the first user 120, and configure the identified devices to capture and transmit the audio stream of the first user 120 for the active session. In some implementations, the system is configured to identify the device capable of real-time audio capture based on the capability and / or configuration information obtained for the device. Figure 6C, based on the detected gaze direction 614 towards the first device 130, the system 100 has determined that the attention of the first user 120 is directed toward the first device 130. Based on this determination, the system 100 configures the second device 132 to not transmit the outbound real-time video stream 620 or the outbound audio stream 630, and configures the first device 130 to capture and transmit the outbound real-time video stream 622 and to capture and transmit the real-time audio stream 632 using the first microphone 218 to replace the outbound real-time audio stream 630. As a result, the real-time audio provided to the remote participant by the active pairing set better captures the first user 120. In some examples, based on the capabilities and / or configuration information obtained for the devices, the system 100 may identify the second device 100 as the primary audio capture device, and therefore not use the second microphone 238 (resulting in Figure 6B This can allow the advanced microphone 238 or audio processing pipeline in the second device 132 to continue to be used to maintain audio quality. Figures 6A-6C , the system 100 has identified the second device 132 as the primary audio rendering device (e.g., based on the capabilities and / or configuration information obtained for the device), and therefore configures the second device 132 to render the real-time audio stream 640. This can allow the advanced speaker 241 to continue to be used to maintain audio quality.

[0081] Fig. 7A , Figure 7B , Figure 7C and Fig.7D The diagram shows Figure 1 , Figure 2A and Figure 2B A fifth example of the system 100 shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein media presentation is selectively performed via a first device 130 and a second device 132 based on detection of the attention of a first user 120. Figures 7A-7D The fifth example in Fig. 6A The status shown continues and can be Figure 3A-6C and Figure 8A-16C Any combination of the examples described in Figures 7A-7D , the system 100 is configured to, in response to the second device 132 being in the active pairing set with the first device 130, identify which of the devices included in the active pairing set and capable of video presentation are of interest to the first user 120, and configure the identified devices to present the selected video content for the active session 253. In some implementations, the system is configured to identify the device capable of video presentation based on capability and / or configuration information obtained for the device (which may reflect that the device is in a locked state where video cannot be presented on a display).

[0082] exist Figures 7A-7C In , the system 100 is configured to identify the most recent active speaker 751 from the remote participants of the active session 253, and configure the identified device to present the real-time video stream corresponding to the most recent active speaker 751. Figures 7A-7C In the particular example shown in , the most recent active speaker 751 is user 160a corresponding to real-time video stream 353. Fig. 7A In, as in Fig. 6A , based on the detected gaze direction 710 toward the second device 132, the system 100 has determined that the attention of the first user 120 is directed toward the second device 132. Based on this determination, the system 100 configures the second device 132 to present a real-time video stream 353 corresponding to the most recently active speaker 751 in the first area 352 (e.g., Figure 3B ), and configures the other devices in the active pairing set (in this example, the first device 130) to not present the real-time video stream 353. Fig. 7A , real-time video streams 353 , 355 , 357 , and 359 are presented by the second device 132 in corresponding areas 352 , 354 , 356 , and 358 of the third UI 342 .

[0083] exist Figure 7B In , the system 100 is configured to present the real-time video stream corresponding to the most recent active speaker 751 only on one of the devices included in the active pairing set at a time. Fig. 7A In the example of FIG. 1 , based on the detected sight line direction 712 toward the first device 130, the system 100 has determined that the attention of the first user 120 is newly directed toward the first device 130, as in Figure 6B Based on this determination, the system 100 configures the second device 132 to stop presenting the real-time video stream 353 (and the system 100 may stop transmitting the real-time video stream 353 to the second device 132), and configures the fifth software program instance 310 running on the first device 130 to receive and present the real-time video stream 353 on the first display 210 in the area 722 of the sixth UI 720. As a result, the first user 120 is more engaged in the active session 253 while participating in the activity on the first device 130. In some implementations, the sixth UI 720 includes a compact fifth session UI control 724, which can be actuated to display a fifth session UI control similar to the one in the Figure 3C In some implementations, the third UI 342 is also rearranged, and new content can be included in the third UI 342 to more efficiently utilize the space previously used to present the video of the most recent active speaker 751. Figure 7BIn the particular example shown in FIG, real-time video streams 355, 357, and 359 are moved to corresponding regions 352, 354, and 356, and region 358 is used to display content 726 (in this example, showing a live transcription of active conversation 253). Note that in response to the most recently active speaker 751 changing to a different remote participant, the system 100 will modify the video streams presented by the first device 130 and the second device 132 accordingly.

[0084] Figure 7C Shows Figure 7B , wherein the system 100 is configured to present a real-time video stream corresponding to the most recent active speaker 751 on the second device 132 (e.g., based on identifying the second device 132 as the primary audiovisual device), and a device that is included in the active pairing set and capable of video presentation is determined to be in the focus of the first user 120 if it is different from the second device 132. Figure 7B As shown in , based on the detected gaze direction 714 toward the first device 130, the system 100 has determined that the attention of the first user 120 is directed toward the first device 130. The real-time video streams 353, 355, 357, and 359 continue to be presented in areas 352, 354, 356, and 358 of the third UI 342, as shown in Fig. 7A As shown in Figure 7B Based on this determination, the system 100 configures the fifth software program instance 310 executed on the first device 130 to Figure 7B The sixth UI 720 receives and presents the real-time video stream 353 on the first display 210 in the area 722. The sixth UI 720 may also include Figure 7B 724 described in the compact fifth conversation UI control 724. Note that in response to the most recent active speaker 751 changing to a different remote participant, the system 100 will modify the video stream presented by the first device 130 in the sixth UI 720 accordingly. Figure 7B and 7C An example is illustrated in which the presentation of one real-time video stream follows the attention of the first user 120, but in some examples, multiple real-time video streams may follow the attention of the first user 120; for example, in Figure 7B In one embodiment, all four real-time video streams may be presented by the first device 130 in a manner that allows the first user 130 to utilize other areas of the first display 210 .

[0085] Fig.7D continue Figure 7C Example. Fig.7DIn the specific example shown in FIG. 1 , the compact fifth conversation UI control 724 includes one or more UI controls (not shown) that allow the first user 120 to select one of multiple video content items for the active conversation 253 for presentation via the sixth UI 720 (or otherwise by a device identified as being of interest). Exemplary video content items include, but are not limited to: a live video stream associated with the most recent active speaker 751 (e.g., Figure 7C ), a live transcript of active session 253 (as shown in Figure 7B 358 in the projection portion (as shown in Figure 4 ) and a whiteboard (virtual or real). Fig.7D , the local user 120 has selected the live transcription as the selected video content 746 shown in the area 722 of the sixth UI 720. In some line-of-sight modes, the selected video content item is stored and used in a later teleconference session involving the active pairing set. In some line-of-sight modes, Figure 7C The example of can be modified to show a live transcription, a projected portion, or a whiteboard instead of the most recent active speaker 751.

[0086] Fig. 8A and Figure 8B The diagram shows Figure 1 , Figure 2A and Figure 2B A sixth example of the system 100 shown in , which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein, while participating in a teleconference session, a first user 120 selects a file for projection and initiates the projection portion of the session via a handheld third device 134 using the selected file. Fig. 8A and Figure 8B The sixth example in Figure 3B The state shown in continues and also includes a third device 134 in the active pairing set including the first device 130 and the second device 132. Fig. 8A and Figure 8B The sixth example in can be compared with Figures 3A-7D and Figure 9A-16C Combine any of the examples described in .

[0087] exist Fig. 8A, a third software program instance 135 executed by the third device 134 in the execution environment 812 is configured to present a seventh UI 812 on the display 810 of the third device 134. The system 100 is configured to identify files recently opened by the first user 120, and present some or all of the identified files in the file list 820 included in the seventh UI 812. In some implementations, the system 100 is configured to identify and include a first file maintained by the storage service 254 and recently opened by the first user 120; for example, the control service 250 can be configured to interact with the storage service 254 to identify the first file, and report the identified first file to the third software program instance 135. In some implementations, the system 100 is configured to identify and include a second file stored locally in the third device 134 for selecting a file for projection. In some implementations, system 100 is configured to identify and include a third file stored locally on a device included in the active pairing set; for example, a file stored locally on first device 130 and recently edited by first user 120 can be reported to third software program instance 135 via system 100 through first device 130. Fig. 8A , the hand 212 of the first user 120 is used to select a file 822 from a file list 820 for projection.

[0088] In some implementations, the system 100 is configured to identify devices that are included in the active pairing set with the second device 132 and are capable of projecting content (e.g., based on capabilities and / or configuration information obtained for the devices) as candidate projection devices, and to identify a projection device from the candidate devices. In some examples, where multiple candidate projection devices are identified, the system 100 is configured to select a default projection device from the candidate projection devices. The default projection device can be selected based on the capabilities and / or configuration information obtained for the candidate projection devices; for example, the system 100 can select the default projection device based on various factors, such as, but not limited to: display resolution, display size, network connection type, network bandwidth, the ability to present the selected file (e.g., having a spreadsheet installed to open a spreadsheet file), and / or an input interface (e.g., a device with a keyboard and / or mouse may be preferred over a touch screen device). Fig. 8A In the specific example shown in , the first device 130, the second device 132, and the third device 134 are candidate projection devices, wherein the first device 130 is selected as the default projection device. In some examples where multiple candidate projection devices are identified, the system 100 is configured to provide, via the third software program instance 135, a UI control that allows the first user 120 to select one of the identified candidate projection devices as the projection device (in Fig. 8A not shown).

[0089] As mentioned above, in this example, the first device 130 is selected to project the file 822. Figure 8B In response, the system 100 causes the first device 130 to initiate the projection portion of the active session 253 for the file 822. For example, the third device 134 and / or the control system 250 can send a message to the first device 130, causing the first device 130 to open the selected file in the corresponding application and transmit the corresponding projection data for presentation to the remote participant. Figure 8B In the example shown in FIG. 8 , the first display 210 initially displays the eighth UI 832, and then for the projected portion, the fifth software program instance 310 opens the selected file to display the projected content 844 from the file in the area 842 of the ninth UI 840 for the fifth software program instance 310, as shown in FIG. Figure 5C Very similar to what is described in .

[0090] Fig. 9A , Fig. 9B and Fig. 9C The diagram shows Figure 1 , Figure 2A and Figure 2B A seventh example of the system 100 shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein a presentation of a whiteboard interface for a teleconference session is automatically transferred between devices based on device states. Figures 9A-9C The seventh example in can be compared with Figures 3A-8B and Figures 10A-16C Combine any of the examples described in .

[0091] exist Fig. 9A In , the first device 130 is not included in the active pairing set with the second device 132, and the second device 132 operates as the primary audio-visual endpoint for the active session 253, much like described in the previous example. Fig. 9A , the second device 132 is presenting a tenth UI 910 for the active session 253, wherein the tenth UI 910 includes a remote participant display area 950, which includes vertically arranged portions 952, 954, and 956, in which respective real-time video streams 953, 955, and 957 are presented. Likewise, in response to enabling a whiteboard (in this example, a virtual whiteboard capable of collaborative editing) in the active session 253, the tenth UI 910 includes a whiteboard interface 912 showing first whiteboard content 914. For purposes of this example, the second device 132 has limited or no ability to edit via the whiteboard interface 912.

[0092] Interested in having an improved interface for interacting with the whiteboard, the first user 120 has picked up a fifth user computing device 930 (embodied as a tablet computing device in this example), as indicated on a fourth display 932 of the fifth device 930, which is displayed on the Fig. 9A In some examples, a device in a locked state may be included in the active pairing set, but provide different capabilities based on whether it is locked or unlocked. For example, in the locked state, the device can be used for audio presentation or notifications but not for video presentation; while in the unlocked state, the device can be used for video presentation. In some examples, a device will not be included in the active pairing set when it is locked. In some examples, a specific application configured to interact with pairing service 150 and / or control service 250 must be running on the device to be included in the active pairing set.

[0093] exist Fig. 9B , the fifth device 930 has been unlocked by the first user 120, an eighth software program instance 960 is being run in the running environment 962 on the fifth device 930, and is included in the active pairing set with the second device 132. In response to at least the fifth device 930 being unlocked, the system 100 automatically determines that the fifth device 930 currently provides a better whiteboard interface than the second device 132, and in response to the determination, supports the fifth device 960 to stop using the second device 132 for whiteboarding. Therefore, the eighth software program instance 962 presents an eleventh UI 964, in which the first user 120 can use the stylus 124 to view the first whiteboard content 914 and add second whiteboard content 966. In response to the whiteboard not being presented on the second device 132, the tenth UI 910 is rearranged to be displayed according to, for example, Figure 7B 910 to provide a larger view of the remote participants, with real-time video streams 953, 955, and 957 presented in corresponding portions 942, 944, and 946 of the tenth UI 910, and additional teleconference session content 949 (in Fig. 9B , the chat session for the active session 253 is presented in portion 948 of the tenth UI 910.

[0094] exist Fig. 9C , the first user 120 has finished adding the second whiteboard content 966 to the virtual whiteboard using the fifth device 930 and has returned the fifth device 930 to the locked state, as shown in the display 932. The system 100 is configured to: in response to the fifth device 930 being locked and / or removed from the active pairing set with the second device 132, cause the tenth UI 910 to return to the state in Fig. 9A, including a remote participant display area 950 presenting respective real-time video streams 953, 955 and 957 and now including the arrangement shown in Fig. 9A 914 and the second whiteboard content 966 added by the first device 120 using the fifth device 930. Figures 9A-9C As shown in , the first user 120 is able to quickly and dynamically add and remove the fifth device 930 for performing specific activities for the active session 253.

[0095] Fig. 10A and Fig. 10B The diagram shows Figure 1 , Figure 2A and Figure 2B An eighth example of the system 100 shown in , which performs unified UI interaction for user activities across multiple devices included in an active pairing set, wherein a first user 120 uses a second device 132 to initiate transmission of content generated in a teleconference session to other participants via a UI on the first device 130. Fig. 10A and Fig. 10B The eighth example in Fig. 9C The example shown in , and can be combined with Figure 3A-9C and Figures 11A-16C Combine any of the examples described in .

[0096] exist Fig. 10A , the first user 120 wishes to provide a copy of the content created in the active session 253 (in this example, the third whiteboard content 1014 seen in the whiteboard interface 912 of the tenth UI 910) to the participants and / or invitees of the active session 253. To this end, using the hand 122 to provide input via the touch screen 239, the first user 120 actuates the UI control 1022 to expose and then actuates the content sharing UI control 1024.

[0097] As in Fig. 10B As shown in FIG. 1 , the system 100 is configured to automatically generate a content sharing UI control 1024 in response to the actuation of the content sharing UI control 1024 and the second device 132 being included in the active pairing set with the first device 130, and ... TM) A draft email opened in a twelfth UI 1030 presented on the first display 910. In some implementations, the system 100 can be configured to similarly respond to the termination of the active session 253. The draft email includes a first attachment 1032 including the whiteboard content 1014. In some examples, the draft email can include a second attachment 1033 including a machine-generated transcription of the active session 253. In addition, as in Fig. 10B As shown in , the recipients 1034 of the email have been automatically populated with the participants of the active conversation 253, and the subject 1036 of the email has been automatically generated. In some examples, initial text can be generated for the body 1038 of the email. By processing the draft email on the first device 130, the first user 120 benefits from features such as, but not limited to: a keyboard 219 for easier text entry for the body 1038, spell checking, access to contacts, automatic suggestions of additional contacts for the email, auto-completion of contacts, and a file browser for identifying additional attachments that can be added through a "drag and drop" interaction.

[0098] Note that the system 100 is configured to respond to actuation of the UI control 1024 in "standalone mode" by generating and sending a similar email to the participants of the active session 253, which may include using the associated UI on the second device 132, in response to the second device 132 not being in the active pairing set. However, a more efficient, full-featured, and familiar experience may be provided to the first user 120 by completing the draft and sending the email via the first device 130, which allows the first device 130 to automatically enter an active aspect, wherein it can provide an improved experience for the first user 120.

[0099] Fig.11A , Fig. 11B , Fig. 11C , Fig.11D and Fig.11E The diagram shows Figure 1 , Figure 2A and Figure 2B A ninth example of the system 100 shown in , which performs unified UI interactions for user activities across multiple devices included in an active pairing set, wherein while participating in a teleconference session, a first user 120 invites additional participants via a handheld third device 134. Figures 11A-11E The ninth example in can be compared with Figure 3A-10B and Figures 12A-16C Combine any of the examples described in . Fig.11A From Fig. 8A The example shown in continues, but the first device 130 is not active and is not included in the active pairing set that includes the second device 132 and the third device 134.

[0100] exist Fig.11A , the second device 132 operates as an audio-visual endpoint for the first user 120 to participate in the active session 253, as in Figure 3B As described in Figure 3B , sixth software program instance 340 presents third UI 342 on second display 230, wherein real-time video streams 353, 355, 357, and 359 are presented in respective portions 342, 344, 346, and 348. In addition, third UI 342 includes invitation UI control 1102, which can be actuated by first user 120 to present an invitee selection UI (not shown) on second display 230 for selecting and requesting participants to join current active session 253.

[0101] However, in this example, the first user 120 would prefer to use the third device 134 to select and request participants to join the current active session 253. For example, the first user 120 may not want the invitee selection UI to block the current content of the second display 230, be more cautious in making requests to participants, or may find the UI provided by the third device 134 more efficient or familiar than the invitee selection UI. Fig. 9A At the time shown in , the third device 134 is included in the active pairing set with the second device 132. The eighth software program instance 1110 (labeled as "eighth APP") is a contact list application running in the execution environment 812 on the third device 134, and presents a thirteenth UI 1112 with a contact list on the display 810, including the contact list application program 1110 executed by the third device 134. Fig.11A The first contact 1114 selected by the first user 120 in .

[0102] exist Fig. 11B, in response to the selection of the first contact 1114, the eighth software program instance 1110 presents a fourteenth UI 1120 displaying the details of the first contact 1114 on the display 810. In addition, the fourteenth UI 1120 includes a contact action portion 1122, which includes a plurality of UI controls for initiating an action related to the first contact 1114 by the third device 134 and / or the system 100, such as but not limited to: making a call, sending a message, and / or sending an email. The third software program instance 135 configured to interact with the control service 250 is configured to interact with elements of the service 100 (such as the control service 250, the pairing service 150, and / or the second device 132). In addition, the third software program instance 135 is configured to interact with the eighth software program instance 1110; for example, as a plug-in component called by the eighth software program instance 1110 and / or a callback registered with the eighth software program instance 1110 and / or the runtime environment 812. In response to the third software program instance 135 obtaining information from the service 100 that the third device 134 is included in the active pairing set and the first device 120 is currently participating in the active session 253, the action portion 1122 of the fourteenth UI 1120 includes a contact invitation UI control 1124, which indicates to the first user 120 that a request is made to the first contact 1114 via the fourteenth UI 1120 to join the current active session 253. Fig. 11C In the example, in response to the first user 120 having actuated the contact invitation UI control 1124, the third device 134 sends a message (not shown) to the service 100, and in response, the service 100 will issue a request to the first contact 1114 to join the active session 253. For example, the control service 250 can be configured to request the conference call service 180 to issue an invitation. In addition, a confirmation 1128 is presented on the display 810.

[0103] exist Fig.11D , the first contact 1114 has received a request from the system 100 to join the active session 253 and has accepted the request (such as, but not limited to, by clicking a link in an invitation email), resulting in the first contact 1114 being added as a participant in the active session 253. In response, the third UI 342 is displayed from Fig.11A The arrangement shown with four parts 352, 354, 356 and 357 is rearranged to Fig.11D The new arrangement is shown having a portion 1142 for presenting a live video stream 1143 of a first contact 1144 and portions 1144, 1145, 1146 and 1147 for presenting real-time video streams 353, 355, 357 and 359, respectively. Fig.11E Shows Fig.11DAn alternative to , wherein the request to the first contact 1114 to join the active session 253 is performed by initiating a real-time call via the teleconferencing system 180. Fig.11D Similar to that shown in FIG, the third UI 342 is arranged as a five-panel layout. Initially, portion 1142 indicates that a call to the first contact 1114 is pending. Once the first contact 1114 accepts the call, portion 1142 presents a real-time video stream 1143 to the first contact 1114, as shown in FIG. Fig.11D As shown in .

[0104] Fig. 12A , Fig. 12B and Fig. 12C The diagram shows Figure 1 , Figure 2A and Figure 2B A tenth example of the system 100 shown in FIG. 1 is to perform unified UI interactions for user activities across multiple devices included in an active pairing set, wherein a handheld third device 134 is temporarily recruited as a media input device for a virtual whiteboard presented by a second device 132. Figures 12A-12C The tenth example in Fig.11A The example shown continues and is consistent with Figure 3A-11E and Figures 13A-16C Combine any of the examples described in .

[0105] exist Fig. 12A , first user 120 is using third device 134 to capture a photo of portion 1212 of physical object 1210 (in this example, a portion of a notebook page). In some examples, in response to system 100 determining that third device 134 is actively paired with second device 132, a user interface control (not shown) presented by second device 132 may initiate use of third device 134 in this manner. Second device 132 may present an indication that third device 134 may be used for this purpose to facilitate first user 120 to discover this functionality.

[0106] Fig. 12BA fifteenth UI 1220 is shown presented on the display 810 and allows the first user 120 to select a captured photo 1222 and, in this example, identify a cropped image 1224 from the photo 1222. Likewise, the fifteenth UI 1220 includes an image action portion 1225 that includes a plurality of UI controls for initiating actions related to the image 1224 by the third device 134 and / or the system 100, such as, but not limited to, copying the image 1224 to a clipboard and / or emailing the image 1224. Additionally, in response to the third software program instance 135 obtaining information from the service 100 that the third device 134 is included in the active pairing set and that the first user 120 is currently participating in the active session 253, and in some examples, obtaining information that the virtual whiteboard is active in the active session 253, the image action portion 1224 of the fourteenth UI 1120 includes a whiteboard upload UI control 1226 for adding the image 1224 to the virtual whiteboard. The presentation of the whiteboard upload UI control 1226 can be achieved by interacting with the third software program instance 135. Fig. 11B and Fig. 11C Very similar to what is described in .

[0107] exist Fig. 12C In the example, the second display 230 is displaying Fig. 9A , which includes first whiteboard content 914 in whiteboard interface 912. System 100 is configured to, in response to actuation of whiteboard upload UI control 1226, obtain image 1224 from third device 134 and add new whiteboard image 1230 generated based on image 1224 to the virtual whiteboard. In various implementations, control service 250 and / or second device 132 may be configured to obtain image 1224 and add whiteboard image 1230 (e.g., via interaction with teleconferencing service 180). In some implementations, such as in Fig. 12C As shown in , the first user 120 can use the hand 122 to move, rotate, resize, or otherwise adjust the whiteboard image 1230 via the whiteboard interface 912 on the second device 132 .

[0108] Therefore, if Figures 12A-12CAs shown, the first user 120 is able to quickly and dynamically add and remove the third device 134 to be able to take advantage of its unique characteristics (in this example, a small form factor that facilitates casual capture of photos). It should be understood that the system 100 can be similarly configured to use the third device 134 to capture images for other devices included in the active pairing set with the third device 134 and / or pairing-aware applications running on such devices. For example, the techniques can be performed via the third party service 258 to copy such images to a clipboard provided by the operating environment 214 of the first device 130, thereby allowing the images to be pasted into various applications.

[0109] Fig.13A , Fig. 13B and Fig. 13C The diagram shows Figure 1 , Figure 2A and Figure 2B An eleventh example of the system 100 shown in FIG. 1 is to perform a unified UI interaction for a user activity across multiple devices included in an active pairing set, wherein a file opened for viewing on a second device 132 is opened on a first device 130 by actuating a UI control on a second device 132. Figures 13A-13C The eleventh example can be compared with Figure 3A-12C and Figures 14A-16C Combine any of the examples described in .

[0110] exist Fig.13A In the example, the first device 130 and the second device 132 are included in the active pairing set. The ninth software program instance 1310 (labeled as "ninth APP") runs on the second device 132 in the operating environment 234 and is configured to display on the second display 230 similar to the one in the example. Figure 3A The sixteenth UI 1312 of the second UI 322 shown in FIG. Figure 3A-12C In contrast to the example shown in , in this particular example, the first user 120 is not participating in a conference call session. The first user 120 is interested in opening a file to be viewed on the second device 132. Using hand 122, the first user 120 first actuates the file open UI control 1314, in response to which the system 100 identifies files recently opened by the first user 120 and presents some or all of the identified files in a file list 1316, as in Fig. 8A The file list 820 described in is very similar. Fig.13A , the hand 212 of the first user 120 is used to select a file 1318 from a file list 1316 for opening by the second device 132. In another example, the file 1318 may be sent by another user as an attachment to a message and selected using a corresponding UI control included in a UI displaying the message.

[0111] exist Fig. 13B In response to selecting file 1318, second device 132 is configured to open selected file 1318 using associated tenth software program instance 1320 (in this example, tenth software program instance 1320 is capable of opening a word processing file such as file 1318). Tenth software program instance 1320 is a pairing-aware application that is configured to interact with second software program instance 133 to interact with pairing service 150 (which may occur via a client service such as third-party service 258). As shown in Fig. 13B As shown in , tenth software program instance 1320 presents seventeenth UI 1322 on second display 230 , which includes file content portion 1324 for displaying file content 1326 stored in file 1318 .

[0112] The system 100 is configured to determine whether the active pairing set including the second device 132 includes one or more supported editing devices other than the second device 132 that can be used to edit the file 1318 (e.g., using a content creation application suitable for the file 1318). The determination can be based on capability and / or configuration information obtained for the devices included in the active pairing set. In this example, the first device 130 is identified as a supported editing device. Fig. 13B As shown in , in response to identifying a supported editing device, the system 100 is configured to present a file editing UI control 1328 in the seventeenth UI 1322 to allow the first user 120 to edit the file 1318 on the supported editing device. In some implementations, if a supported editing device is not included in the active pairing set but a paired (and not currently actively paired) supported editing device is identified, the system 100 is configured to indicate (e.g., by displaying an indication in the seventeenth UI 1322) that the paired device is valid for editing the file 1318 to make the first user 120 aware of this possibility. Fig. 13B , the hand 212 of the first user 120 is used to actuate the file editing UI control 1328.

[0113] exist Fig. 13C In the example, in response to actuation of the file editing UI control 1328, the system 100 is configured to send a message to the first device 130 (identified as a supported editing device) requesting that the first device 130 open the file 1318 for editing. In response to receiving the message (e.g., by the first software program instance 131), the first device 130 is configured to use the associated eleventh software program instance 1330 (in this example, a word processing application such as Microsoft Word). TM ) opens file 1318 for editing. Fig. 13CAs shown in FIG. 1 , the eleventh software program instance 1330 presents an eighteenth UI 1332 on the first display 210, which includes a file content portion 1334 for displaying and editing the file content 1326 stored in the file 1318. Thus, as shown in FIG. Figures 13A-13C As shown in , the first user 120 is able to conveniently and efficiently open files directly on the second device 132, and be confident that the first user 120 can quickly and dynamically open files on the active paired device, thereby providing a more capable application or convenient user input device (such as the keyboard 219 of the first device 130).

[0114] Fig.14A and Fig. 14B The diagram shows Figure 1 , Figure 2A and Figure 2B A twelfth example of the system 100 shown in , which performs a unified UI interaction for user activities across multiple devices included in an active pairing set, wherein a content element included in a file being viewed on the second device 132 is opened for inspection on the first device 130 by actuating a UI control on the second device 132. Fig.14A and Fig. 14B The twelfth example in can be compared to Figure 3A-13C and Figures 15A-16C Combine any of the examples described in .

[0115] exist Fig.14A In, it has been used to open Fig. 13B 1318 in the example, wherein the tenth software program instance 1320 presents the seventeenth UI 1322, wherein the file content portion 1324 is displaying the file content 1410 stored in the file that has been opened. The displayed file content 1410 includes a content element 1412 (in this example, a chart generated by a spreadsheet program). The system 100 is configured to determine whether the active pairing set including the second device 132 includes one or more supported inspection devices other than the second device 132 that can be used to inspect the content element 1412 (e.g., using an application suitable for opening and / or editing the content element 1412), in conjunction with Fig. 13B The identification of the editing support device for the identification file 1318 is similar to that described. In this example, the first device 130 is identified as a supporting inspection device. Fig.14AAs shown in , in response to identifying a supported inspection device for content element 1412, system 100 is configured to present a content element inspection UI control 1414 in a seventeenth UI 1322 (in this example, adjacent to content element 1412) to allow first user 120 to inspect content element 1412 on a supported inspection device. In some implementations, if a supported inspection device is not included in the active pairing set but a paired (and not actively paired) supported inspection device is identified, system 100 is configured to indicate (e.g., by displaying an indication in the seventeenth UI 1322) that the paired device is effectively inspecting content element 1412 to make first user 120 aware of this possibility. Fig.14A , the hand 212 of the first user 120 is used to actuate the element inspection UI control 1414 .

[0116] exist Fig. 14B In response to actuation of content element inspection UI control 1414, system 100 is configured to send a message to first device 130 (identified as a supported inspection device) requesting first device 130 to open content element 1412. In response to receiving the message (e.g., by first software program instance 131), first device 130 is configured to use associated twelfth software program instance 1420 (in this example, a spreadsheet application such as Microsoft Excel) to open content element 1412. TM ) obtains and opens content element 1412 and / or a file associated with content element 1412. In this example, the spreadsheet file used to generate content element 1412 has been opened by twelfth software program instance 1420, which presents nineteenth UI 1422 displaying associated file content 1424 for the spreadsheet file. Fig.14A and 14B As shown in , the first user 120 can conveniently and efficiently open the file directly on the second device 132, and be confident that the first user 120 can quickly and dynamically inspect the content elements of the file on the active paired device, thereby providing a more capable application or convenient user input device (such as the keyboard 219 of the first device 130) for inspecting the content elements. Note that although in Fig. 13B and Fig.14A UI controls 1328 and 1414 are presented within UI 1322, but in some examples, such UI controls may be presented outside of a UI used to display files.

[0117] Fig.15A , Fig. 15B and Fig. 15C The diagram shows Figure 1 , Figure 2A and Figure 2BA thirteenth example of the system 100 shown in , which performs a unified UI interaction for a user activity across multiple devices included in an active pairing set, wherein a first device 130 presents a UI control to provide a third device 134 with a geographic location identified in content presented by the first device 130. Figures 15A-15C The thirteenth example in Figure 3A-14B and Figures 16A-16C Any combination of the examples described in Fig.15A and Fig. 15B , the first device 130 and the third device 134 are included in the active pairing set.

[0118] In 15A, the first user 120 is accessing a website through a web browser application instance 1510 running on the first device 130 (e.g., in the execution environment 214) via a twentieth UI 1512 presented on the first display 210. The twentieth UI 1512 is presenting a first web page 1514, which includes a map element 1516 and an associated street address element 1517 (in this example, a text element including a street address). A pairing-aware thirteenth software program instance 1520 is configured to interact with the web browser application instance 1510 and the first software program instance 131. In some implementations, the thirteenth software program instance 1520 can operate as a plug-in component for the web browser application instance 1510, and can be configured to identify the presence of an information item of a specific information type (in this example, a geographic location information type), and obtain the information item - in this example, a geographic location corresponding to a map element in a web page (e.g., by identifying a map element generated by a known map service and obtaining a geographic location based on the map element) and / or a street address element (e.g., by identifying an HTML HTML element). <address>contact address tags and parse the street address included in the tag).

[0119] The system 100 is configured to determine whether the active pairing set including the first device 130 includes one or more guidance-supporting devices (e.g., guidance-supporting map applications using satellite-based geographic services) that can be used for navigation guidance to the indicated geographic location. The determination can be based on capabilities and / or configuration information obtained for the devices included in the active pairing set. In this example, the third device 134 is identified as a guidance-supporting device. The system 100 is configured to present a location guidance UI control 1518 in response to both obtaining the geographic location from the web page and identifying the guidance-supporting device to allow the first user 120 to use the guidance-supporting device to navigate to the obtained geographic location. Fig.15A In the example shown in , the presentation of the location guide UI control 1518 includes presenting an information bar 1519 including the location guide UI control 1518 in the twentieth UI 1512. In some examples, the location guide UI control 1518 can be presented as a right-click menu item and / or a mouse hover item for the map element 1516 and / or the street address element 1517. Fig.15A , the location guide UI control 1518 is actuated using the pointer 1504, and in response, the first device 130 sends a message (which may include a geographic location) to the system 100 indicating that the location guide UI control 1518 has been actuated.

[0120] As in Fig. 15B As shown in FIG. 1 , the system 100 is configured to: in response to the actuation of the location guidance UI control 1518, send a request to the third device 134 (identified as a navigation-supporting device) to provide the third device 134 to the Fig.15A The navigation guidance message of the geographical location obtained in Fig. 15B At the time shown, the third device 134 is in a locked state. The third device 134 is configured to, in response to receiving a message while the third device is in a locked state, present a notification 1530 on the display 810 that allows the first user 120 to initiate the requested navigation guidance (once the third device 134 is unlocked). Fig. 15C , the third device 134 has been unlocked by the first user 120, and in response to the message having been received and the third device 134 being in an unlocked state, navigation guidance to the obtained geographic location is performed via a map mapping running on the third device 134 via a twenty-first UI 1540 presented on the display 810.

[0121] Fig.16A , Fig. 16B and Fig. 16C The diagram shows Figure 1 , Figure 2A and Figure 2B A fourteenth example of the system 100 shown in , which performs a unified UI interaction for a user activity across multiple devices included in an active pairing set, wherein a first device 130 initiates a user input operation on a third device 134, resulting in the user input being delivered from the third device 134 to the initiating first device 130. Figures 16A-16C The fourteenth example in can be compared with Figure 3A-15C Any combination of the examples described in Figures 16A-16C , the first device 130 , the second device 132 , and the third device 134 are included in the active pairing set.

[0122] exist Fig.16A In the example, the first user 120 accesses a website via a twenty-second UI 1612 presented on the first display 210 through a web browser application instance 1610 running on the first device 130 (e.g., in the execution environment 214). The twenty-second UI 1612 is presented on the second web page 1614, which includes a signature input element 1616. The pairing-aware fourteenth software program instance 1620 is configured to interact with the web browser application instance 1610 and the first software program instance 131. In some implementations, the thirteenth software program instance 1620 can operate as a plug-in component for the web browser application instance 1610, and can be configured to recognize the presence of the signature input element in the web page.

[0123] The system 100 is configured to determine whether an active pairing set including the first device 130 includes one or more supported signature input devices. The determination may be based on capability and / or configuration information obtained for the devices included in the active pairing set. In this example, both the second device 132 and the third device 134 are identified as supported signature input devices. The system 100 is configured to, in response to detecting a signature input element in a web page and identifying a supported signature input device, present a corresponding signature input UI control 1618 to allow the first user 120 to use a supported signature input device to input a signature. Fig.16A In the example shown in , two exemplary signature input UI controls 1618 are shown for the second device 132, including a signature input UI control 1618aa in the information bar 1619a and a signature input UI control 1618ba adjacent to the signature input element 1616, and two exemplary signature input UI controls 1618 are shown for the third device 134, including a signature input UI control 1618ab in the information bar 1619b and a signature input UI control 1618bb adjacent to the signature input element 1616. In some examples, the signature input UI controls 1618 may be presented as right-click menu items and / or mouse hover items for the signature input element 1616. Fig.15A , the pointer 1604 is used to actuate the signature input UI control 1618bb of the third device 134.

[0124] As in Fig. 16B As shown in , system 100 is configured to, in response to actuation of signature input UI control 1618ab or 1618bb for third device 134, send a message to third device 134 requesting third device 134 to obtain a signature from first user 120. Third device 134 is configured to, in response to receiving the message (e.g., via third software program instance 135), present on display 810 a twenty-third UI 1632 for a fifteenth software program instance 1630 running on third device 134. Fifteenth software program instance 1630 is configured to obtain a signature 1634 provided using hand 122. Twenty-third UI 1632 includes a signature completion UI control 1636 to complete input of signature 1634 via third device 134. In some examples, second device 132 may be applied to a similar effect in response to actuation of signature input UI control 1618aa or 1618ba for second device 132. As shown in Fig. 16C As shown in , the system 100 is configured to: in response to actuation of the signature completion UI control 1636, transmit the signature 1634 from the third device 134 to the first device 130 and input the signature 1634 into the signature input element 1634. Figures 16A-16C As shown in , the first user 120 is able to quickly and dynamically use a device with a touch screen or other desired input device with another device in the active pairing set that lacks such an input device.

[0125] Fig.17 1700 is a flowchart illustrating an implementation of an exemplary process 1700 for providing a unified user interface between actively paired user computing devices. In some examples, some or all of the process 1700 may be combined with the implementation of the exemplary process 1700. Figure 1-16C , Fig.19 and Fig. 20 The invention may be performed with any of the features discussed, but it may also be performed with any of the other features described herein. Fig.17 In the process 1700, the first operation 1710 may include determining that a first user computing device is eligible for active pairing with a second user computing device. In the second operation 1720, the process 1700 may include determining that the first user computing device is physically proximate to the second user computing device at a first time when both the first user computing device and the second user computing device are authenticated for the same user service account. In the third operation 1730, the process 1700 may include determining that the first user computing device and the second user computing device are included together in an active pairing set at the first time based on determining that the first user computing device is eligible for active pairing with the second user computing device and determining that the first user computing device is physically proximate to the second user computing device. In the fourth operation 1740, the process 1700 includes receiving an indication of actuation of a user interface control associated with a user activity and presented by the first user computing device at the first time. In the fifth operation 1750, the process 1700 includes: in response to receiving the indication and determining that the first user computing device and the second user computing device are included together in the active pairing set at the first time, causing the second user computing device to present a second user interface associated with the user activity.

[0126] Fig.18 1 is a flowchart illustrating an implementation of an exemplary process 1800 for providing a unified user interface between actively paired user computing devices. In some examples, some or all of the process 1800 may be combined with the implementation of the exemplary process 1800. Figure 1-16C , Fig.19 and Fig. 20 The invention may be performed with any of the features discussed, but it may also be performed with any of the other features described herein. Fig.18 In a first operation 1810, process 1800 may include determining that a first user computing device is eligible for active pairing with a second user computing device. In a second operation 1820, process 1800 may include determining that the first user computing device is physically proximate to the second user computing device at a first time when both the first user computing device and the second user computing device are authenticated for the same user service account. In a third operation 1830, process 1800 includes determining that the first user computing device and the second user computing device are included together in an active pairing set at the first time based on determining that the first user computing device is eligible for active pairing with the second user computing device and determining that the first user computing device is physically proximate to the second user computing device. In a fourth operation 1840, process 1800 includes receiving an indication of an actuation of a user interface control associated with a teleconference session and presented by the first user computing device at the first time. In a fifth operation 1850, process 1800 includes, in response to receiving the indication and determining that at the first time the first user computing device and the second user computing device are included together in the active pairing set, causing the real-time component of the teleconference session to be presented by the second user computing device when a first user associated with the user service account joins the teleconference session as a first participant.

[0127] Combination Figure 1-18 Detailed examples of the described systems, devices, and techniques will be presented herein to illustrate the present disclosure and its benefits. Such usage examples should not be interpreted as limitations on the logical process embodiments of the present disclosure, nor should variations of the user interface methods from those described herein be considered outside the scope of the present disclosure. In some embodiments, in Figure 1-18 The various features described in the present invention are implemented in corresponding modules, which may also be referred to as and / or include logical units, components, units and / or mechanisms. A module may constitute a software module (eg, code embodied on a machine-readable medium) or a hardware module.

[0128] In some examples, the hardware module can be implemented mechanically, electronically, or using any suitable combination thereof. For example, the hardware module may include a dedicated circuit or logic unit configured to perform a specific operation. For example, the hardware module may include a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware module may also include a programmable logic or circuit temporarily configured by software to perform a specific operation, and may include a portion of machine-readable medium data and / or instructions for such a configuration. For example, the hardware module may include software included in a programmable processor, which is configured to run a set of software instructions. It should be appreciated that the decision to implement the hardware module mechanically, in a dedicated and permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) may be driven by cost, time, support, and engineering considerations.

[0129] Therefore, the phrase "hardware module" should be understood to cover a tangible entity that can perform a specific operation and can be configured or arranged in a certain physical manner, that is, a physical structure, a permanently configured entity (e.g., hardwired) and / or a temporary configuration (e.g., programmed) that operates or performs a specific operation described in this article in a certain way. As used in this article, a "hardware-implemented module" refers to a hardware module. Considering the example in which a hardware module is temporarily configured (e.g., programmed), each hardware module in the hardware module does not need to be configured or instantiated at any one time instance. For example, in the case where a hardware module includes a programmable processor that is configured to be a special-purpose processor by software, the programmable processor can be configured as different special-purpose processors (e.g., including different hardware modules) at different times. Software can configure one or more specific processors accordingly, for example, to constitute a specific hardware module at one time instance and to constitute different hardware modules at different time instances. A hardware module implemented using one or more processors can be referred to as "processor-implemented" or "computer-implemented".

[0130] Hardware modules can provide information to other hardware modules and receive information from other hardware modules. Therefore, the described hardware modules can be considered to be communicatively coupled. In the case where multiple hardware modules exist simultaneously, communication can be achieved by signal transmission (e.g., through appropriate circuits and buses) between two or more hardware modules in the hardware modules. In an embodiment where multiple hardware modules are configured or instantiated at different times, communication between such hardware modules can be achieved, for example, by storing and retrieving information in a memory device accessible to multiple hardware modules. For example, one hardware module can perform an operation and store the output in a memory device, and then another hardware module can access the memory device to retrieve and process the stored output.

[0131] In some examples, at least some of the operations of the method may be performed by one or more processors or processor-implemented modules. In addition, the one or more processors may also be operated to support the performance of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some of the operations may be performed by multiple computers (as an example of a machine including a processor) and / or between multiple computers, where the operations may be accessed via a network (e.g., the Internet) and / or via one or more software interfaces (e.g., application program interfaces (APIs)). The execution of specific operations in the operations may be distributed among the processors, not only residing in a single machine, but also deployed across multiple machines. The processor or processor-implemented module may be located in a single geographic location (e.g., in a home or office environment, or in a server farm), or may be distributed across multiple geographic locations.

[0132] Fig.19 19 is a block diagram 1900 illustrating an exemplary software architecture 1902, portions of which may be used in conjunction with the various hardware architectures described herein that may implement any of the features described above. Fig.19 is a non-limiting example of a software architecture, and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 1902 may be implemented in a variety of environments such as Fig. 20 The system 1900 is executed on the hardware of the machine 2000, which includes a processor 2010, a memory 2030, and an input / output (I / O) component 2050, etc. A representative hardware layer 1904 is illustrated and can represent, for example, Fig. 20 The machine 2000 of the embodiment of the present invention. The representative hardware layer 1904 includes a processing unit 1906 and associated executable instructions 1908. The executable instructions 1908 represent executable instructions of the software architecture 1902, including implementations of the methods, modules, etc. described herein. The hardware layer 1904 also includes a memory / storage device 1910, which also includes executable instructions 1908 and accompanying data. The hardware layer 1904 may also include other hardware modules 1912. The instructions 1908 maintained by the processing unit 1908 may be part of the instructions 1908 maintained by the memory / storage device 1910.

[0133] Exemplary software architecture 1902 can be conceptualized as layers, each providing various functions. For example, software architecture 1902 may include layers and components, such as operating system (OS) 1914, library 1916, framework 1918, application 1920, and presentation layer 1944. In operation, application 1920 and / or other components within a layer may invoke API calls 1924 to other layers and receive corresponding results. The illustrated layers are representative in nature, and other software architectures may include additional or different layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 1918.

[0134] OS1914 can manage hardware resources and provide public services.OS1914 can include, for example, kernel 1928, service 1930 and driver 1932. Kernel 1928 can serve as an abstraction layer between hardware layer 1904 and other software layers. For example, kernel 1928 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Service 1930 can provide other public services for other software layers. Driver 1932 can be responsible for controlling the underlying hardware layer 1904 or interface with it. For example, driver 1932 can include display driver, camera driver, memory / storage device driver, peripheral device driver (e.g., via universal serial bus (USB)), network and / or wireless communication driver, audio driver, etc., depending on hardware and / or software configuration.

[0135] The library 1916 may provide a common infrastructure that may be used by the application 1920 and / or other components and / or layers. The library 1916 generally provides functions for use by other software modules to perform tasks, rather than interacting directly with the OS 1914. The library 1916 may include a system library 1934 (e.g., a C standard library) that provides functions such as memory allocation, string manipulation, and file operations. In addition, the library 1916 may include an API library 1936, such as a media library (e.g., supporting the presentation and manipulation of image, sound, and / or video data formats), a graphics library (e.g., an OpenGL library for presenting 2D and 3D graphics on a display), a database (e.g., SQLite or other relational database functions), and a web library (e.g., WebKit that may provide web browsing functions). The library 1916 may also include a variety of other libraries 1938, thereby providing many functions for the application 1920 and other software modules.

[0136] Framework 1918 (sometimes also referred to as middleware) provides a higher level common infrastructure that can be used by applications 1920 and / or other software modules. For example, framework 1918 can provide various graphical user interface (GUI) functions, advanced resource management, or advanced location services. Framework 1918 can provide a wide range of other APIs for applications 1920 and / or other software modules.

[0137] Applications 1920 include built-in applications 1940 and / or third-party applications 1942. Examples of built-in applications 1940 may include, but are not limited to, contact applications, browser applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 1942 may include any application developed by an entity other than the vendor of a particular platform. Applications 1920 may use the functionality available via OS 1914, library 1916, framework 1918, and presentation layer 1944 to create a user interface to interact with a user.

[0138] Some software architectures use virtual machines, as illustrated by virtual machine 1948. Virtual machine 1948 provides an execution environment in which applications / modules can run as if they were on a hardware machine such as Fig. 20 The virtual machine 1948 may be hosted by a host OS (e.g., OS 1914) or a hypervisor and may have a virtual machine monitor 1946 that manages the operation of the virtual machine 1948 and interoperation with the host operating system. A software architecture that may be different from the software architecture 1902 outside the virtual machine runs within the virtual machine 1948, such as an OS 1950, a library 1952, a framework 1954, an application 1956, and / or a presentation layer 1958.

[0139] Fig. 20 2000 is a block diagram illustrating components of an exemplary machine 2000 configured to read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any of the features described herein. The exemplary machine 2000 is in the form of a computer system within which instructions 2016 (e.g., in the form of software components) for causing the machine 2000 to perform any of the features described herein can be run. In this way, the instructions 2016 can be used to implement the modules or components described herein. The instructions 2016 cause the unprogrammed and / or unconfigured machine 2000 to operate as a specific machine configured to perform the described features. The machine 2000 can be configured to operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 2000 can operate in the capacity of a server machine or a client machine in a server-client network environment, or operate as a node in a peer-to-peer or distributed network environment. The machine 2000 may be embodied as, for example, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a gaming and / or entertainment system, a smartphone, a mobile device, a wearable device (e.g., a smart watch), and an Internet of Things (IoT) device. Furthermore, although only a single machine 2000 is illustrated, the term "machine" includes a collection of machines that individually or jointly execute instructions 2016.

[0140] The machine 2000 may include a processor 2010, a memory 2030, and I / O components 2050 that may be communicatively coupled via, for example, a bus 2002. The bus 2002 may include multiple buses that couple various elements of the machine 2000 via various bus technologies and protocols. In an example, the processor 2010 (including, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, or a suitable combination thereof) may include one or more processors 2012a to 2012n that may execute instructions 2016 and process data. In some examples, one or more processors 2010 may execute instructions provided or recognized by one or more other processors 2010. The term "processor" includes a multi-core processor that includes cores that may execute instructions simultaneously. Although Fig. 20 Multiple processors are shown, but the machine 2000 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors each with a single core, multiple processors each with multiple cores, or any combination thereof. In some examples, the machine 2000 may include multiple processors distributed among multiple machines.

[0141] The memory / storage 2030 may include a main memory 2032, a static memory 2034 or other memory and a storage unit 2036, which may be accessed by the processor 2010 via the bus 2002. The storage unit 2036 and the memory 2032, 2034 store instructions 2016 that embody any one or more of the functions described herein. The memory / storage 2030 may also store temporary, intermediate and / or long-term data for the processor 2010. The instructions 2016 may also reside, in whole or in part, in the memory 2032, 2034, in the storage unit 2036, in at least one processor 2010 (e.g., in a command buffer or cache), in a memory in at least one I / O component 2050, or any suitable combination thereof during their execution. Therefore, the memory 2032, 2034, the storage unit 2036, the memory in the processor 2010, and the memory in the I / O component 2050 are examples of machine-readable media.

[0142] As used herein, "machine-readable medium" refers to a device capable of temporarily or permanently storing instructions and data that cause the machine 2000 to operate in a particular manner, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical storage media, magnetic storage media and devices, cache, network-accessible or cloud storage devices, other types of storage devices, and / or any suitable combination thereof. The term "machine-readable medium" applies to a single medium or a combination of multiple media used to store instructions (e.g., instructions 2016) for execution by the machine 2000, such that when the instructions are executed by one or more processors 2010 of the machine 2000, the machine 2000 performs one or more of the features described herein. Thus, a "machine-readable medium" may refer to a "cloud-based" storage system or storage network with a single storage device, as well as a plurality of storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0143] I / O components 2050 may include a variety of hardware components suitable for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 2050 included in a particular machine will depend on the type and / or function of the machine. For example, a mobile device (such as a mobile phone) may include a touch input device, while a headless server or IoT device may not include such a touch input device. Fig. 20 The specific examples of I / O components illustrated in the figure are by no means limiting, and other types of components may be included in the machine 2000. The grouping of I / O components 2050 is merely to simplify the discussion, and the grouping is by no means limiting. In various examples, the I / O components 2050 may include a user output component 2052 and a user input component 2054. The user output component 2052 may include, for example, a display component (e.g., a liquid crystal display (LCD) or a projector) for displaying information, an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor or a force feedback device), and / or other signal generators. The user input component 2054 may include, for example, an alphanumeric input component (e.g., a keyboard or a touch screen), a pointing component (e.g., a mouse device, a touch pad, or other pointing instruments), and / or a tactile input component (e.g., a physical button or touch screen that provides a position and / or a touch force or a touch gesture) configured to receive various user inputs, such as user commands and / or selections.

[0144] In some examples, I / O component 2050 may include biometric component 2056, motion component 2058, environment component 2060, and / or position component 2062, as well as a wide range of other physical sensor components. Biometric component 2056 may include, for example, components that detect body expressions (e.g., facial expressions, vocal expressions, hand or body gestures, or eye tracking), measure biosignals (e.g., heart rate or brain waves), and identify people (e.g., via voice, retina, fingerprint, and / or face-based recognition). Motion component 2058 may include, for example, acceleration sensors (e.g., accelerometers) and rotation sensors (e.g., gyroscopes). Environment component 2060 may include, for example, lighting sensors, temperature sensors, humidity sensors, pressure sensors (e.g., barometers), acoustic sensors (e.g., microphones for detecting ambient noise), physical proximity sensors (e.g., infrared sensing of nearby objects), and / or other components that may provide indications, measurements, or signals corresponding to the surrounding physical environment. Position component 2062 may include, for example, a position sensor (e.g., a global positioning system (GPS) receiver), an altitude sensor (e.g., a barometric pressure sensor from which altitude can be derived), and / or an orientation sensor (e.g., a magnetometer).

[0145] The I / O components 2050 may include a communication component 2064 that implements a variety of technologies operable to couple the machine 2000 to a network 2070 and / or a device 2080 via respective communication couplings 2072 and 2082. The communication component 2064 may include one or more network interface components or other suitable devices that interface with the network 2070. The communication component 2064 may include, for example, components suitable for providing wired communication, wireless communication, cellular communication, near field communication (NFC), Bluetooth communication, Wi-Fi, and / or communication via other modalities. The device 2080 may include other machines or various peripheral devices (e.g., coupled via USB).

[0146] In some examples, communication component 2064 can detect an identifier or include a component suitable for detecting an identifier. For example, communication component 2064 can include a radio frequency identification (RFID) tag reader, an NFC detector, an optical sensor (e.g., a one-dimensional or multi-dimensional barcode or other optical code), and / or an acoustic detector (e.g., a microphone that recognizes an audio signal of a tag). In some examples, location information can be determined based on information from communication component 2062, such as, but not limited to, a geographic location via an Internet Protocol (IP) address, a location via Wi-Fi, cellular, NFC, Bluetooth, or other wireless station identification and / or signal triangulation.

[0147] Although various embodiments have been described, this description is intended to be exemplary rather than restrictive, and it should be understood that more embodiments and implementations within the scope of the embodiments are possible. Although many possible feature combinations are illustrated in the drawings and discussed in this detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature of any embodiment can be used in combination with any other feature or element in any other embodiment or replace any other feature or element. Therefore, it should be understood that any feature of the features shown and / or discussed in this disclosure can be implemented together in any suitable combination. Therefore, except according to the attached claims and their equivalents, the embodiments are not limited. In addition, various modifications and changes can be made within the scope of the attached claims.

[0148] Although the foregoing has described what is considered to be the best mode and / or other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and these teachings may be applied in many applications, only some of which are described herein. The appended claims are intended to claim any and all applications, modifications, and changes that fall within the true scope of the present teachings.

[0149] Unless otherwise indicated, all measurements, values, ratings, positions, dimensions, sizes and other specifications set forth in this specification (including the claims that follow) are approximate and not exact and are intended to have a reasonable range consistent with the functions to which they are related and with customary practices in the art to which they are related.

[0150] The scope of protection is limited only by the following claims. The scope is intended and should be interpreted to be consistent with the ordinary meaning of the language used in the claims when interpreted in light of this specification and subsequent prosecution history and to include all structural and functional equivalents. Nonetheless, no claim is intended to encompass subject matter that fails to meet the requirements of the Patent Act, nor should it be interpreted in such a manner. Any unexpected admission of such subject matter is hereby disclaimed.

[0151] Except as stated above, nothing stated or described is intended or should be construed as conferring any component, step, feature, object, benefit, advantage or equivalent to the public, whether or not mentioned in the claims.

[0152] It should be understood that the terms and expressions used in this article have the ordinary meanings given to the corresponding investigations and research fields corresponding to these terms and expressions, unless a specific meaning is set forth in this article in addition. Relational terms such as "first" and "second" can be used only to distinguish one entity or action from another, without requiring or implying any actual such relationship or order between such entities or actions. The term "includes", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including the list of elements not only includes those elements, but may include other elements that are not clearly listed or elements inherent to such processes, methods, articles or devices. In the absence of further restrictions, the element starting with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0153] The Abstract of the present disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above detailed description, it can be seen that various features are grouped together in various examples in order to simplify the disclosure. This disclosure method should not be interpreted as reflecting an intention that the claims require more features than those expressly listed in each claim. On the contrary, as reflected in the following claims, the subject matter of the present invention does not lie in all the features of a single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim standing on its own as a separately claimed subject matter.< / address>

Claims

1. A system for providing a unified user interface between actively paired user computing devices, comprising one or more processors and a machine-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to: determining that the first user computing device is eligible for active pairing with the second user computing device; determining that the first user computing device is in physical proximity to the second user computing device at a first time when both the first user computing device and the second user computing device are authenticated for the same user service account; determining that the first user computing device and the second user computing device are included together in a first active pairing set at the first time based on determining that the first user computing device is eligible for active pairing with the second user computing device and determining that the first user computing device is in physical proximity to the second user computing device; receiving, at the first time, a first indication of an actuation of a first user interface control associated with the teleconference session and presented by the first user computing device; as well as In response to receiving the first indication and determining that at the first time the first user computing device and the second user computing device are included together in the first active pairing set: causing the first user computing device to display a first user interface associated with the session associated with the user service account; as well as causing the second user computing device to simultaneously display a second user interface, the second user interface being different from the first user interface and also being related to the session associated with the user service account; receiving, via a communications network, a first indication from the first user computing device, the first indication indicating that the user has performed a first interaction with the first user interface to activate a first function of the session; as well as In response to receiving the first indication, the second user computing device is caused to execute the first function of the session activated by the first interaction of the user on the first user computing device, and the second user interface is updated based on the execution of the first function on the second user computing device, while the first user computing device is simultaneously executing a second function that is not responsive to the first interaction, and simultaneously displaying the first user interface that is different from the updated second user interface for executing the second function.

2. The system according to claim 1, wherein: The instructions further cause the one or more processors to: receiving a second indication that a projection component of the teleconference session is active; and In response to receiving the second indication and determining that the first user computing device and the second user computing device are included together in the first active pairing set, the first user computing device is caused to present the projected component of the teleconference session while the real-time component of the teleconference session is being presented by the second user computing device.

3. The system according to claim 1, wherein: The instructions further cause the one or more processors to: determining that the first user's attention is directed toward the first user computing device during the teleconference session; in response to determining that the first user's attention is directed toward the first user computing device, causing the second user computing device to cease presenting a second real-time component of the teleconference session; as well as In response to determining that the first user's attention is directed to the first user computing device, causing the first user computing device to begin presenting the second real-time component of the teleconference session.

4. The system according to claim 1, wherein: The instructions further cause the one or more processors to: determining that the first user's attention has changed from the second user computing device to the first user computing device during the teleconference session; in response to determining that the first user's attention has changed from the second user computing device to the first user computing device, causing the second user computing device to cease transmitting the first user's real-time media stream captured by the second user computing device; as well as In response to determining that the first user's attention has changed from the second user computing device to the first user computing device, causing the first user computing device to begin transmitting the real-time media stream of the first user captured by the first user computing device.

5. The system according to claim 1, wherein: The instructions further cause the one or more processors to: determining that a third user computing device is included in the first active pairing set during the teleconference session; responsive to determining that the third user computing device is included in the first active pairing set, causing the third user computing device to present a second user interface element for providing a media item from the third user computing device to the teleconference session; receiving a first media item submitted via the second user interface element; as well as In response to receiving the first media item, the first media item is caused to be presented by the first user computing device or the second user computing device.

6. The system according to claim 1, wherein: The instructions further cause the one or more processors to: In response to receiving the first indication, causing the first user to join the telephone conference session as a first participant, wherein: the first user did not join the telephone conference session at the first time, and The real-time component of the teleconference session is a real-time media stream to a second participant of the teleconference session.

7. The system according to claim 1, wherein: The instructions further cause the one or more processors to: determining that the first user computing device is a projection-supported device; responsive to determining that the first user computing device is a cast-enabled device and determining that the first user computing device and the second user computing device are together included in the first active pairing set, causing the second user computing device to present a second user interface control; receiving a second indication of actuation of the second user interface control; as well as In response to the second indication, the first user computing device is caused to begin projecting electronic content via a projection component of the teleconference session.

8. A method for providing a unified user interface between actively paired user computing devices, the method comprising: determining that the first user computing device is eligible for active pairing with the second user computing device; determining that the first user computing device is in physical proximity to the second user computing device at a first time when both the first user computing device and the second user computing device are authenticated for a user service account of a first user; determining that the first user computing device and the second user computing device are included together in a first active pairing set at the first time based on determining that the first user computing device is eligible for active pairing with the second user computing device and determining that the first user computing device is in physical proximity to the second user computing device; receiving, at the first time, a first indication of an actuation of a first user interface control associated with the teleconference session and presented by the first user computing device; as well as In response to receiving the first indication and determining that at the first time the first user computing device and the second user computing device are included together in the first active pairing set: causing the first user computing device to display a first user interface associated with the session associated with the user service account; as well as causing the second user computing device to simultaneously display a second user interface, the second user interface being different from the first user interface and also being related to the session associated with the user service account; receiving, via a communications network, a first indication from the first user computing device, the first indication indicating that the user has performed a first interaction with the first user interface to activate a first function of the session; as well as In response to receiving the first indication, the second user computing device is caused to execute the first function of the session activated by the first interaction of the user on the first user computing device, and the second user interface is updated based on the execution of the first function on the second user computing device, while the first user computing device is simultaneously executing a second function that is not responsive to the first interaction, and simultaneously displaying the first user interface that is different from the updated second user interface for executing the second function.

9. The method according to claim 8, further comprising: In response to receiving the first indication, causing the first user to join the telephone conference session as a first participant, wherein: the first user did not join the telephone conference session at the first time, and The real-time component of the teleconference session is a real-time media stream to a second participant of the teleconference session.

10. The method according to claim 8, further comprising: In response to determining that the first user computing device and the second user computing device are included together in the first active pairing set: causing the first user computing device to present a first user interface control for controlling the teleconference session while the teleconference session is active; and The first user computing device is caused to present a second user interface control for controlling the teleconference session while the teleconference session is active.

11. The method according to claim 8, further comprising: receiving a second indication that a projection component of the teleconference session is active; as well as In response to receiving the second indication and determining that the first user computing device and the second user computing device are included together in the first active pairing set, the first user computing device is caused to present the projected component of the teleconference session while the real-time component of the teleconference session is being presented by the second user computing device.

12. The method according to claim 8, further comprising: determining that the first user computing device is a projection-supported device; responsive to determining that the first user computing device is a cast-enabled device and determining that the first user computing device and the second user computing device are together included in the first active pairing set, causing the second user computing device to present a second user interface control; receiving a second indication of actuation of the second user interface control; as well as In response to the second indication, the first user computing device is caused to begin projecting electronic content via a projection component of the teleconference session.

13. The method according to claim 8, further comprising: determining that a third user computing device is included in the first active pairing set during the teleconference session; In response to determining that the third user computing device is included in the first active pairing set, causing the third user computing device to present a second user interface element for issuing an invitation to the first contact to join the teleconference session; as well as In response to actuation of the second user interface element, an invitation is issued to the first contact to join the teleconference session.

14. The method according to claim 8, further comprising: determining that the first user computing device and a third user computing device are included together in the first active pairing set during the teleconference session; In response to determining that the first user computing device and a third user computing device are included together in the first active pairing set, causing the third user computing device to present a second user interface for identifying electronic content for projection; receiving an identification of first electronic content identified via the second user interface; as well as In response to receiving the identification of the first electronic content, causing the first user computing device to begin projecting the first electronic content via a projection component of the teleconference session.

15. The method according to claim 8, further comprising: determining that the first user computing device and the second user computing device are included together in the first active pairing set at a second time during the teleconference session; responsive to determining that at the second time the first user computing device and the second user computing device are included together in the first active pairing set, causing the second user computing device to present a second user interface control for generating an email including first electronic content generated in the teleconference session; receiving a second indication of actuation of the second user interface control; as well as In response to receiving the second indication, causing the first user computing device to present a draft email message including the first electronic content for transmission by an email application executing on the first user computing device.

Images