Delay compensation for synchronized shared video content within web conferencing sessions

By processing and cacheing video content on the server side and using delay compensation technology, the problem of synchronous playback of video content in the online conferencing system is solved, and efficient resource use and synchronous playback effects are achieved.

CN115053518BActive Publication Date: 2024-12-03MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202180012997.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-02-07
Filing Date
2021-01-20
Publication Date
2024-12-03
Estimated Expiration
2041-01-20

AI Technical Summary

Technical Problem

Modern network conferencing systems cannot share video content at full resolution and frame rate, and conventional screen sharing technology has inconsistent playback position of video content on different client devices due to network delay and upload speed limitations, resulting in inconsistent playback of video content on different client devices, making it impossible to achieve synchronous playback.

Method used

By processing and cacheing video content on the server side and using delay compensation technology to adapt to network delays of different client devices, ensuring that video content is played simultaneously on all devices and reducing the use of resources.

Benefits of technology

It realizes synchronous playback of video content on multiple client devices, improves network efficiency, reduces resource consumption, and ensures the full resolution and frame rate of video.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115053518B_ABST
    Figure CN115053518B_ABST
Patent Text Reader

Abstract

A system for performing delay compensation techniques to facilitate the synchronous sharing of video content during a communication session. The system enables video content shared during the communication session to be played synchronously across many different client devices used to participate in the communication session, despite different network delays associated with different client devices. The disclosed technology provides improvements over existing screen sharing technologies, which include continuously uploading shared media content streams from a sharer's device and distributing these uploaded streams to other devices. For example, the technology described herein enables a system to facilitate the sharing of video content during a communication session without initially rendering the video content at the sharer's device and simultaneously uploading the rendered video content from the sharer's device. In addition, the technology described herein includes delay compensation to adapt to changes in network delays experienced across many different client devices.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Modern network conference system promotes the virtual conference that is carried out online between people far away from each other.Usually, network conference system allows the user that is positioned at two or more remote locations to communicate interactively with each other via live, two-way video stream, audio stream or both simultaneously.Some network conference systems (for example, CISCO WEBEX provided by CISCO SYSTEMS company of San Jose, CA, GOTOMEETING provided by CITRIX SYSTEMS company of Santa Clara, CA, ZOOM provided by ZOOM VIDEO COMMUNICATIONS of Sna Jose, CA, GOOGLE HANGOUTS provided by ALPHABET company of Mountain View, CA, and TEAMS MEETING provided by MICROSOFT CORPORATION of Redmond, WA) also promote screen sharing.In this way, the participant of the communication session based on the Internet can share the video (and any other media content) being played locally on the display screen of client device.

[0002] Unfortunately, conventional screen sharing techniques utilized by modern web conferencing systems are not suitable for sharing video content at full resolution and frame rate. For example, these techniques typically require that the video be fully processed and rendered at the sharer's device before being transmitted to the web conferencing server in an upload stream. The uploaded video content is then transmitted from the web conferencing server to other devices in a download stream. Therefore, in these techniques, the video content is not available for download until it has been rendered on the sharer's device and uploaded to the web conferencing server.

[0003] A major disadvantage of these conventional screen sharing technologies (e.g., they rely on real-time uploading and downloading of video content when it is rendered at the device of the sharer) is that the web conferencing system cannot facilitate caching of any upcoming parts of the video content. Another major disadvantage is that network limitations often require that the resolution and frame rate of the video content be reduced before the video content is added to the upload stream. As a specific example, a video rendered at 60 frames per second on the device of the sharer may be reduced to only 5 frames per second before being uploaded to the web conferencing server and then distributed to other client devices. One reason for reducing frame rate and / or resolution is that the upload speed is usually much lower than the download speed on a typical network. In addition, even in the case of reduced resolution and frame rate, network delays often hinder the sharing of video content across various participant computing devices for synchronous rendering. For example, because the time spent by the video content of the stream being transmitted to different devices is different, the shared video content may be unintentionally played at different playhead positions of different client devices.

[0004] With regard to these and other considerations, the following disclosure is made. Summary of the invention

[0005] The technology disclosed in this article enables a system to perform delay compensation technology to facilitate the synchronous sharing of video content during a communication session. In short, the system enables the video content shared during a communication session to be played synchronously across many different client devices for participating in the communication session without being subjected to changes in network delays associated with many different client devices. The disclosed technology provides an improvement to existing screen sharing technology, which includes continuously uploading shared media content streams from the sharer's device and distributing these uploaded streams to other devices. For example, the technology described in this article enables a system to facilitate the sharing of video content during a communication session without initially rendering the video content at the sharer's device and uploading the rendered video content from the sharer's device at the same time. In addition, the technology described in this article includes delay compensation to adapt to changes in network delays experienced across many different client devices.

[0006] Therefore, the technology described in this article reduces or even completely eliminates any differences in the playback head position of the shared video content played across various different client devices for participating in the communication session. In addition, as described below, various embodiments of the technology described in this article enable individual client devices to cache shared video content independently, thereby enabling the shared video content to be played synchronously across various client devices with the expected full video resolution and frame rate (e.g., by eliminating the reliance on the real-time upload stream of the video content from the sharer's device). At least for the aforementioned reasons, the technology described in this article provides a technical benefit of improving the network efficiency of computing devices sharing media content during a network-based communication session (e.g., a virtual meeting). For example, such technology can improve network efficiency by eliminating the reliance on video content that is simultaneously rendered and uploaded from the sharer's device before the video content is distributed to the devices of other participants. Therefore, by deploying the technology described in this article, the use of various computing resources such as network resources, memory resources, processing resources, and power resources (e.g., "batteries") can be significantly reduced during a communication session.

[0007] In an exemplary implementation, the system facilitates a communication session (e.g., a virtual meeting or a web-based meeting) by transmitting a two-way user stream between multiple client devices. For example, individual client devices in the client devices can continuously capture (e.g., the user's) audio and video "AV" data via various input devices (such as cameras and microphones). The captured A / V data can be continuously streamed to (one or more) server devices. Upon receiving the AV data stream, (one or more) server devices transmit the stream of A / V data received from each other client device to each individual client device. In this way, each individual client device enables the use of other client devices to render the live video of the participant.

[0008] In addition to facilitating the communication session, the system also obtains media data that defines the media content that will be played simultaneously across client devices during the communication session. For example, a user can upload a video to a (one or more) server device during and / or before the communication session. Then, at some point during the communication session, while the client device is rendering live video or a simple audio stream captured at the client devices of other participants, the video hosted by the (one or more) server device can be streamed to and displayed at the client device.

[0009] During a communication session, the system receives user play instructions generated based on user input received at a particular client device to initiate playback of a video across all client devices. For example, a user at a particular client device can select a video and even adjust the playhead position of the video to a desired playhead position. The user can then select a "play" or "share" command associated with the video to cause all other client devices to begin playback of the video at the desired playhead position.

[0010] The system also determines delay values ​​for individual client devices in the client devices. Specifically, the delay data defines individual delay values ​​associated with transmitting information to individual client devices. For example, a delay value of 54 milliseconds for a particular client device indicates that the information sent to the particular client device will take 54 milliseconds to be transmitted across one or more networks and ultimately arrive at the particular client device. Determining the delay value for the client device may include periodically pinging the individual client device and measuring the amount of time it takes for the ping to return (e.g., be transmitted back to) the sending device from the receiving device. For example, assuming that each segment of these individual round-trip times is equal in time, the delay value associated with any individual client device can be assumed to be half of the measured round-trip time.

[0011] Finally, the system can respond to the user play instruction by transmitting to the client device a delayed play instruction including a customized delay value for the individual client device. For example, an individual delayed play instruction can include a delay delay value that specifies the amount of time each receiving client device is to wait (i.e., after receiving the instruction) before initiating playback of the video. In this way, all devices can be made to wait the appropriate amount of time so that even the device experiencing the highest network delay will receive its instruction. All devices can then play the video synchronously or simultaneously.

[0012] Features and technical benefits other than those explicitly described above will become apparent by reading the following detailed description and reviewing the associated drawings. This summary is provided to introduce a selection of concepts in a simplified form, which concepts will be further described in the detailed description below. This summary is not intended to identify the key or essential features of the claimed subject matter, nor is it intended to be used to assist in determining the scope of the claimed subject matter. For example, the term "technology" may refer to (one or more) systems, (one or more) methods, computer-readable instructions, (one or more) modules, algorithms, hardware logic units, and / or (one or more) operations as allowed by the context described above and throughout this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The detailed description is described with reference to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears. The same reference numbers in different figures indicate similar or identical items. References to individual items in multiple items can use reference numbers with letters in the alphabetical sequence to refer to each individual item. Generic references to items can use specific reference numbers without the alphabetical sequence.

[0014] Figure 1A An exemplary scenario is illustrated in which a system performs delay compensation techniques to facilitate synchronized sharing of video content across multiple client devices during a communication session.

[0015] Figure 1B The system shows a process after the system has dispatched multiple delayed playback instructions to multiple client devices so that the video content starts to be played synchronously across the multiple client devices. Figure 1A , although the network delay is different at different client devices in the client devices.

[0016] Figure 1C The diagram shows that after each delayed play instruction in the delayed play instruction is received at the corresponding client device Figure 1A to Figure 1B An exemplary scenario of .

[0017] Figure 1D 1 illustrates a video playback image after the system has dispatched multiple pause instructions to cause playback of the video content to pause at exactly the same playhead position across all client devices. Figures 1A to 1C , although the network delay is different at different client devices in the client devices.

[0018] Figure 2 An alternative scenario is illustrated in which the system facilitates synchronous sharing of video content during a communication session.

[0019] Figure 3A An exemplary scenario is illustrated in which the system determines a delay value associated with an individual client device used to participate in a communication session.

[0020] Figure 3B Shown with Figure 3A An exemplary scenario in which a delay value associated with an individual client device is determined at a later time than the system.

[0021] Figure 3C Shown with Figure 3B An exemplary scenario in which a delay value associated with an individual client device is determined at a later time than the system.

[0022] Figure 3D Shown with Figure 3CAn exemplary scenario in which a delay value associated with an individual client device is determined at a later time than the system.

[0023] Figure 4A An exemplary graphical user interface is shown that may be simultaneously displayed on two different client devices operating in asynchronous playback mode according to the techniques described herein.

[0024] Figure 4B An exemplary graphical user interface is shown that may be simultaneously displayed on two different client devices operating in synchronized playback mode according to the techniques described herein.

[0025] Figure 4C An exemplary graphical user interface is shown that may be simultaneously displayed on two different client devices after a master control state is transferred from a first client device to a second client device.

[0026] Figure 4D An exemplary graphical user interface is shown that may be simultaneously displayed on two different client devices, one of which is operating in an autonomous queuing state.

[0027] Figure 5 is a flow chart of an exemplary process for performing delay compensation techniques to facilitate synchronized sharing of video content across multiple client devices during a communication session.

[0028] Figure 6 is a diagram illustrating an exemplary environment 600 in which a system can implement the techniques disclosed herein.

[0029] Figure 7 A diagram showing example components of an example device (also referred to herein as a "computing device") configured to generate data for some of the user interfaces disclosed herein is illustrated. DETAILED DESCRIPTION

[0030] Figure 1AAn exemplary scenario is illustrated in which the system 100 performs delay compensation techniques to facilitate synchronized sharing of video content during a communication session 128. In short, the system 100 enables video content shared during a communication session 128 to be played synchronously (e.g., at substantially the same playhead position 122) across many different client devices 114 used to participate in the communication session 128. The techniques disclosed herein with respect to the system 100 provide improvements over existing screen sharing techniques, which require that video content be initially processed and rendered at an individual client device, while being uploaded to a web conference server for distribution to one or more other client devices (at which the video content will ultimately be rendered with a certain delay caused by network delays). For example, as described above, existing screen sharing techniques attempt to mitigate the effects of network delays by reducing the resolution and / or frame rate of the rendered video content before uploading the rendered video content to the web conference server. In this manner, the bit rate of the data stream uploaded for sharing the video content is reduced, which in part reduces the delay between the playhead positions of the video content across the devices. Unfortunately, even if the resolution and / or frame rate of the video content is reduced before being uploaded, the requirement that the video content is initially rendered at the sharer's device and then uploaded from the sharer's device before being distributed to other client devices cannot be eliminated. Since uploading video content from the sharer's device inherently requires some amount of time, existing screen sharing technologies cannot completely eliminate the delay of the playhead position caused by the delay caused by the network. On the contrary, the technology described in this article enables the system 100 to facilitate the sharing of video content during the communication session 128 without initially rendering the video content at the sharer's device and uploading the rendered video content from the sharer's device at the same time. In addition, the technology described in this article includes delay compensation to adapt to the changes in network delays experienced across many different client devices 114.

[0031] Thus, the techniques described herein reduce or even completely eliminate any differences in the playhead position 122 of the shared video content played across the various client devices 114 used to participate in the communication session 128. In addition, as described below, various embodiments of the techniques described herein enable individual client devices 114 to independently maintain a certain pre-fetched portion of the shared video content (e.g., by pre-downloading the entirety of the shared video content or by maintaining a cache of a certain upcoming portion of the shared video content), thereby enabling the shared video content to be played synchronously across the various client devices 114 at the expected full video resolution and frame rate (e.g., by eliminating the reliance on real-time upload video content streaming of the video content from the sharer). For at least the foregoing reasons, the techniques described herein provide technical benefits of improving network efficiency for computing devices to share media content during a network-based communication session (e.g., a virtual meeting). For example, such techniques can increase network efficiency by eliminating the reliance on video content that is simultaneously rendered and uploaded from the sharer's device before the video content is distributed to the devices of other participants. Thus, by deploying the techniques described herein, the use of various computing resources, such as network resources, memory resources, processing resources, and power resources (eg, "battery"), can be significantly reduced during a communication session.

[0032] As in Figure 1AAs illustrated in , the system 100 may include one or more server devices 102 and one or more client devices 114 that perform various operations described herein and / or store various types of data described herein. With respect to the client devices 114, although illustrated in the form of a laptop computing device, the client devices 114 may be in any suitable form, such as, for example, a desktop computer, a smart phone, a smart watch, a smart television, etc. In the illustrated embodiment, the server device(s) 102 may store (or otherwise access) conference data 104 that defines various aspects of a "network-based" communication session. For example, but not limited to, the conference data 104 may include a list of invitees, a scheduled time at which the "network-based" communication session will occur, and / or a link that enables a first client device 114(1) to an Nth client device 114(N) to join the "network-based" communication session (e.g., as represented by a communication session 128 spanning the first client device 114(1) to the Nth client device 114(N)) using a corresponding communication module 116. As illustrated, the conference data 104 may also include media data 106 hosted on the server device 102. The media data 106 may be video data (e.g., MPEG-4 part 14 data, or any other suitable format) uploaded to the server device 102 from one or more of the client devices 114 before and / or during the communication session 128. For example, the organizer of the communication session 128 may utilize the communication module 116 (e.g., MICROSOFT OUTLOOK, etc.) to generate a calendar object associated with the communication session 128. The calendar object may enable the communication session 128 to be added to the electronic calendars of various invitees while also providing access to the media data 106. As illustrated, the media data 106 may be hosted on the server device 102. In some embodiments, the media data 106 includes video content (e.g., MPEG-4 part 14 data) to be shared during the communication session 128. Additionally or alternatively, the media data 106 may include a link or pointer to video content hosted externally to the server device 102. For example, media data 106 may include a link to an online video hosted by a third-party video hosting service (eg, VIMEO, YOUTUBE, etc.).

[0033] In the illustrated embodiment, server device(s) 102 also include latency data 108 that defines latency values ​​associated with individual ones of client devices 114. As used herein, the term "latency value" refers to the amount of time it takes for information to be sent from a particular computing device (e.g., server device 102) to an individual one of client devices 114 over one or more networks. For example, the amount of time it takes to send information from one or more of server devices 102 to a first client device 114(1) over one or more networks may be appropriately referred to as the latency value for first client device 114(1). For purposes of non-limiting illustration, in FIG. Figure 1A The delay data 108 shown in FIG. 1 defines a first delay value of 12 milliseconds (ms) associated with a first client device 114(1), a second delay value of 21 milliseconds (ms) associated with a second client device 114(2), and an Nth delay value of 55 milliseconds (ms) associated with an Nth client device 114(N). Thus, where the delay values ​​are measured based on the time it takes for data to be transmitted from one or more server devices 102 to an individual client device 114, information sent from a server device 102 will take 12 milliseconds to reach the first client device 114(1), 21 milliseconds to reach the second client device 114(2), and so on.

[0034] In some embodiments, the server device(s) 102 are configured to deploy a delay monitor 110 to generate delay data 108 associated with individual client devices in the client devices 114. For example, during the communication session 128, the delay monitor 110 may periodically transmit information to the individual client devices 114 and may measure the amount of time it takes to receive a reply from the individual client device 114. It will be appreciated that this measured amount of time represents the round-trip time it takes for data to be transmitted from the server device 102 to a particular client device 114 and then returned from the particular client device 114 to the server device 102. Therefore, assuming that each segment of this round-trip time is equal in time, the delay value associated with any individual client device 114 can be assumed to be half of the measured round-trip time. As described in more detail below, in some embodiments, the delay monitor 110 may deploy a machine learning engine 112 to analyze various aspects of the delay data 108 (e.g., rising and / or falling trends in delay for individual client devices and / or groups of client devices) to refine the delay value associated with the individual client device 114.

[0035] exist Figures 1A to 1D About the first time T 1 To the seventh time T 7Various aspects of an exemplary scenario in which the system 100 performs delay compensation techniques to facilitate synchronized sharing of video content during a communication session 128 are described. For purposes of this discussion, these times represent the following time series, where: 1 <T 2 <T 3 <T 4 <T 5 <T 6 <T 7 (where < means "before..."). The sequence of events described as occurring at individual times within the time sequence is for illustrative purposes only. Individual ones of these events may be omitted and / or reordered relative to other ones of these events without departing from the scope of the described embodiments and the appended claims.

[0036] In the first time T 1 , the system 100 is transmitting multiple bidirectional user streams 127 between the server device 102 and the individual client devices 114 for facilitating a communication session 128. Specifically, the system 100 is shown transmitting a first bidirectional user stream 127(1) between the server device 102 and the first client device 114(1), transmitting a second bidirectional user stream 127(2) between the server device 102 and the second client device 114(2), and transmitting an Nth bidirectional user stream 127(N) between the server device 102 and the Nth client device 114(N). As used herein, the term "stream" may refer to a substantially continuous flow of data (e.g., audio and video material) between different computing devices over one or more networks. For example, a continuous flow of video content from the server device 102 to the individual client devices 114 may be properly referred to as a stream (also referred to as a "data stream"). As used herein, the term "bidirectional user stream" refers to a combination of: an upload stream that includes data captured by one or more input devices (e.g., microphone, video camera, etc.) of a particular client device 114; and a download stream that includes data captured by (one or more) input devices of at least one other client device 114.

[0037] For purposes of this discussion, it is assumed that each bidirectional user stream 127 illustrated in association with each individual client device 114 includes an upload stream having at least one of substantially real-time audio or substantially real-time video captured at the individual client device 114. Furthermore, it is assumed that each bidirectional user stream 127 illustrated in association with each individual client device 114 includes a download stream having at least one of substantially real-time audio or substantially real-time video captured at one or more of the other client devices 114 participating in the communication session 128. For example, a first bidirectional user stream 127(1) may include an upload stream of real-time data captured by a microphone and camera of the first client device 114(1), and a download stream of real-time data captured by a microphone and camera at each of the second client device 114(2) and the Nth client device 114(N). It can thus be appreciated that by transmitting a bidirectional user stream 127, the system 100 can facilitate a communication session 128 as a type of "virtual" or "web-based" gathering or conference during which individual participants can be presented with the audio and / or video of other participants while also having their own audio and / or video presented to those other participants. Figures 1A to 1D For the purpose of the present discussion, it is assumed that the communication of the bidirectional user stream 127 starts at a first time T 1 Lasts until the seventh time T 7 .

[0038] At the second time T 2 , user play instructions 130 are generated by the first client device 114(1) and transmitted therefrom to the server device 102. The user play instructions 130 may be generated based on user input received at the first client 114(1), associated with the media data 106 (identified within the conference data 104), so that video content is played synchronously at the first client device 114(1) to the Nth client device 114(N). As illustrated, the user play instructions 130 may include an indication of a first playhead position 112(1) at which to begin playing the media data 106. For example, the media data 106 may include (or point to) a sequence of frames that, when played (e.g., at a predefined frame rate), form the video content, and the user play instructions 130 may define a particular frame within the sequence at which the video content is to begin playing synchronously across the individual client devices 114. Additionally or alternatively, the user play instructions 130 may define a particular time at which the video content begins playing synchronously across the individual client devices 114.

[0039] Out of Figures 1A to 1DFor purposes of the exemplary scenarios illustrated and described in , it is assumed that the server device 102, upon receiving a user play instruction 130, begins transmitting individual media streams 132 to individual ones of the client devices 114. Specifically, the server device 102 is shown as responding to the user play instruction 130 by initiating transmission of a first media stream 132(1) to the first client device 114(1), a second media stream 132(2) to the second client device 114(2), and an Nth media stream 114(N) to the Nth client device 114(N). As used herein, the term "media stream" may refer to a flow of media data 106 (e.g., audio / video content in MP4 format) from the server device 102 to the individual ones of the client devices 114. Thus, it can be appreciated that the media streams 132 may be unidirectional in nature (e.g., as opposed to being bidirectional in nature due to having both an upload portion and a download portion).

[0040] As described above, the amount of time it takes for individual media streams 132 to reach corresponding client devices 114 varies due to delays or latencies inherently caused by transmitting data over one or more networks. As indicated by delay data 108, for example, information transmitted from server device 102 (e.g., over the Internet) will take 12 milliseconds to reach first client device 114(1), 21 milliseconds to reach second client device 114(2), and 55 milliseconds to reach Nth client device 114(N). Thus, assuming that the transmission of media streams 132 is initiated simultaneously, it can be expected that first media stream 132(1) will reach first client device 114(1) only 12 milliseconds later and before either of second media stream 132(2) or Nth media stream 132(N) reaches second client device 114(1) or Nth client device 114(N), respectively.

[0041] Based on these network delays associated with various client devices 114, at a third time T 3 , the first media stream 132(1) is shown as having arrived at the first client device 114(1), and at the third time T 3, the second media stream 132(2) has not yet arrived at the second client device 114(2) and the Nth media stream 132(N) has not yet arrived at the Nth client device 114(N). As illustrated, upon receiving some initial portion of the first media stream 132(1), the first client device 114(1) is shown as having set the media state 120 of the video content to "pause" within the first communication module 116(1). Additionally or alternatively, upon receiving the initial portion of the first media stream 132(1), the first client device 114(1) may establish a first buffer 124(1) of video content (e.g., continuously received via the first media stream 132(1)).

[0042] Now turn to Figure 1B , showing the fourth time T 4 , at the fourth time T 4 , the system 100 has dispatched a plurality of delayed play instructions 134 to various client devices 114. Specifically, the server device 102 has dispatched a first delayed play instruction 134(1) to the first client device 114(1), a second delayed play instruction 134(2) to the second client device 114(2), and an Nth delayed play instruction 134(N) to the Nth client device 114(N). Individual delayed play instructions in the delayed play instructions 134(1) may include an indication of a first play head position 122(1) at which to begin playing the video content.

[0043] In some embodiments, an individual delayed play instruction in the delayed play instructions 134(1) defines a delay delay 126 that indicates a time at which a receiving client device 114 is to begin playing video content received in a media stream 132. As a specific but non-limiting example, the delay delay 126 defined within an individual delayed play instruction 134 may indicate an amount of time that a receiving client device 114 is to wait (e.g., after receiving the delayed play instruction 134) before playing the video content. In this manner, an individual client device 114 that is experiencing less network latency than other client devices 114 may be instructed to wait an appropriate amount of time before playing the video content in order to allow all client devices 114 sufficient time to receive the corresponding media stream 132. Thus, rather than each individual client device 114 immediately beginning to play the video content upon receiving the video content (which would likely result in asynchronous playback of the video content), the individual client device 114 may be caused to wait until all other client devices have also received the video content.

[0044] To illustrate this, assume that each of the first delayed play instruction 134(1) through the Nth delayed play instruction 134(N) are dispatched simultaneously from the server device 102. Based on the delay data 108 in the illustrated scenario, further assume that the first delayed play instruction 134(1) will be received 12 milliseconds after being dispatched, the second delayed play instruction 134(2) will be received 21 milliseconds after being dispatched, and the Nth delayed play instruction 134(N) will be received 55 milliseconds after being dispatched. It should be appreciated that based on these assumptions, the first delayed play instruction 134(1) will be received by the first client device 114(1) 43 milliseconds before the Nth delayed play instruction 134(N) is received by the Nth client device 114(N), and the second delayed play instruction 134(2) will be received by the second client device 114(2) 34 milliseconds before the Nth delayed play instruction 134(N) is received by the Nth client device 114(N). In these exemplary but non-limiting cases, a suitable first delay 126(1) may be 43 milliseconds, a suitable second delay 126(2) may be 34 milliseconds, and a suitable Nth delay 126(N) may be 0 milliseconds. Here, the first client device 114(1) may delay playing the video content for 43 milliseconds after receiving the first delayed play instruction 134(1), the second client device 114(2) may delay playing the video content for 34 milliseconds after receiving the second delayed play instruction 134(2), and the Nth client device 114(N) may begin playing the video content immediately upon receiving the Nth delayed play instruction 134(N). Thus, it can be appreciated that, in some examples, each delayed play instruction 134 transmitted to a respective client device 114 may define a delay amount based on the difference between the delay initially experienced by the respective client device and the longest delay experienced by any of the client devices 114. For example, here, the Nth client device 114 (N) is experiencing the longest delay (eg, 55 ms), and the delay delay determined for each device is based on the difference between the current delay being experienced by that device 114 and the longest delay of 55 ms.

[0045] As shown in the figure, at the fourth time T 4 , each of the first client device 114 (1) and the second client device 114 (2) has received both the corresponding media stream 132 and the delayed playback instruction 134. Therefore, at the fourth time T 4 , upon receiving the initial portion of the second media stream 132(2), the second client device 114(2) has already established a second buffer 124(2) of video content. 4 , the first buffer 124(1) of the video content is larger than Figure 1A The third time T described 3 is relatively fuller (e.g., contains relatively more bytes of data). However, at the fourth time T 4 , the Nth client device 114(N) has not yet received any of the Nth media stream 132(N) or the Nth delayed play instruction 134(N). Figure 1B As shown in the figure, at the fourth time T 4 , the Nth client device 114 (N) has not yet begun caching the video content to be shared during the conference session 128. Figure 1B As shown in the figure, at time T 4 , each of the first client device 114(1) and the second client device 114(2) is shown as having set the media state 120 of the video content to “pause” within the corresponding communication module 116 (e.g., to allow the Nth client device 114(N) to receive the Nth media stream 132(N) for a period of time. In contrast, at the fourth time T 4 , the media state at the Nth client device 114 (N) remains “empty” because the Nth client device 114 (N) has not yet received the corresponding media stream 132 and / or delayed playback instruction 132.

[0046] Now turn to Figure 1C , showing the fifth time T 5 , at the fifth time T 5 , the Nth client device 114(N) has now received both the Nth media stream 132(N) and the Nth play instruction 134(N). Continuing with the example where the Nth delay time 126(N) is 0 milliseconds, upon receiving the Nth delay play instruction 134(N), the Nth client device 114(N) begins playing the video content (from the Nth media stream) without delay. For purposes of this example, it is assumed that the delay value indicated in the delay data 108 and used to determine the delay time 126 is accurate. Therefore, if the fifth time T 5 represents the time when the Nth delayed play instruction 134 (N) is received, then the fifth time T 5 This also corresponds to a 43 millisecond delay when the first delayed play instruction 134(1) is received from the first client device 114(1) and a 34 millisecond delay when the second delayed play instruction 134(2) is received from the second client device 114(2).

[0047] By complying with the delay delay 126 indicated within its respective delayed play instruction 134, each of the first through Nth client devices 114(1) through 114(N) begins playing the video content at the exact same moment at the appropriate playback position 122, despite experiencing different network delays at different client devices 114. To illustratively convey this point, each of the first through Nth media states 120(1) through 120(N) is shown as being at exactly the fifth time T 5 has been switched to "play". Thus, compared to conventional screen sharing techniques as described above, the playback position 122 of the playing video content is constant across all client devices 114 (rather than being out of sync due to variations in network latency). Furthermore, compared to these conventional screen sharing techniques, the techniques described herein do not require the shared video content to be initially rendered and uploaded from the sharer's client device before being dispatched to other client devices. Thus, the amount of network resources utilized during the conference session 128 while sharing video content is significantly reduced.

[0048] Now turn to Figure 1D , illustrates an exemplary scenario in which a pause instruction 136 is generated at an individual client device 114 so that playback of video content shared during a communication session 128 is paused at the exact same playhead position 122 across all client devices 114 participating in the communication session 128. In the illustrated scenario, based on user input received at the Nth client device 114(N) when the video content is at the second playhead position 122(2), the playback is initially paused at a sixth time T 6 A pause instruction 136 is generated. For purposes of this discussion, assume that the second playhead position 122(2) (e.g., the position at which the participant using the Nth client device 114(N) paused the video content) corresponds to "frame 100" of the video content. Thus, at the sixth time T 6 , the Nth media state 120(N) switches from “playing” to “paused” exactly at the second playhead position 122(2). In addition, at the sixth time T 6 , the Nth client device 114 (N) transmits a pause instruction 136 to the server device 102. It will be appreciated that due to T 6 At a sixth time T corresponding to the time at which the pause instruction 136 is sent from the Nth client device 114(N) but before the pause instruction 136 is relayed to either the first client device 114(1) or the Nth client device 114(N), 6, the video content will still be played at the first client device and the second client device. In some embodiments, the pause instructions 136 include data defining a specific playhead position (e.g., in terms of playhead time and / or frame) at which the video content will be collectively paused across the client devices 114. For example, as illustrated, the pause instructions 136 define the video content at the sixth time T 6 The second playhead position 122(2) is being played.

[0049] Based on the above discussion regarding networking delays, it will be appreciated that a certain amount of time will occur between when the Nth client device 114(N) sends the pause instruction 136 and when the other client devices 114 receive the pause instruction 136. Moreover, because the networking delay experienced by the first device 114(1) is different than the networking delay experienced by the second device 114(2), it will be further appreciated that even if an instance of the pause instruction 136 is transmitted to each of these devices (e.g., from the server device 102) at exactly the same time, it is still possible that the instances of the pause instruction 136 are received at the first client device and the second client device at different times. For example, as illustrated, at the seventh time T 7 , an instance of the pause instruction 136 has been received at the first client device 114(1), while another instance of the pause instruction 136 has not been received at the second client device 114(2). 7 , the media state 120 of the video content has been switched to “pause” at the first client device 114 (1). In addition, at the seventh time T 7 , the playhead position at the first client device 114(1) has been updated from the third playhead position 122(3) (e.g., frame 125 of the video content) to the second playhead position 122(2), at which the video content is paused at the Nth client device 114(N).

[0050] However, as shown, at the seventh time T 7, because the second client device 114(2) has not yet received an instance of the pause instruction 136, the media state 120 of the video content at the second client device 114(2) remains as playing. Therefore, it will be appreciated that the video content may continue to play even after passing the third playhead position 122(3) at the second client device 114(2), at least until the pause instruction 136 is received. Then, upon receiving an instance of the pause instruction 136, the video content will be paused at the second client device 114(2) and the playhead position will be updated to the second playhead position 122(2). Therefore, despite network delays, once all client devices 114 receive the instruction to pause the video content and the indication of the particular playhead position at which to pause the video content, the video content will be synchronously paused at exactly the same playhead position across all client devices 114.

[0051] The currently disclosed technology is applicable to various systems and schemes for sharing media content during a communication session while compensating for differences in network delays across multiple client devices so that the shared media content is played synchronously across multiple client devices. Various aspects of the disclosed technology are mainly described in the context of shared video content hosted on a server device 102, and the server device 102 is configured to facilitate a communication session 128 during which the video content is shared. Although the currently disclosed technology is not necessarily limited to this context, it is best to obtain an understanding of various aspects of the currently disclosed technology by discussing examples in this specific context. However, the currently disclosed technology can also be deployed in a scenario where the shared video content is not hosted on the same server device 102 that facilitates the communication session 128 (for example, by generating and / or transmitting a two-way user stream as described above). For example, the currently disclosed technology is also applicable to scenarios where video content (or any other type of media content to be shared during a communication session) is hosted by a third-party media hosting service provider, such as, for example, YOUTUBE, VIMEO, and TWITCH. In such an implementation, it should be appreciated that bidirectional user streams 127 may be transmitted by server device 102 between multiple client devices 114 as described above, while media streams 132 are received by individual client devices 114 from other servers associated with a third-party media hosting service provider.

[0052] In some embodiments, system 100 can enable playback modes associated with playback of media content to be switched between synchronous playback modes and asynchronous playback modes across client devices 114. For example, because media content is received from media stream 132 at individual client devices 114 independently of bidirectional user stream 132, it can be appreciated that the precise playback position of the media content can be independently controlled at each of the individual client devices 114 without affecting the ability to hear and / or see the video and / or audio of other participants received in the bidirectional user stream. For example, a first user participating in a communication session at a first client device 114(1) can decide to fast forward playback of the media content, while a second user participating in the communication session at a second client device 114(1) can decide to watch the media content at normal playback speed. In such a scenario, it can be appreciated that a first playback head position 122(1) of media content at a first client device 114(1) can become continuously further ahead of a second playback head position 122(2) of media content at a second client device 114(2) (i.e., as long as the playback speed at the first client device 114(1) is greater than that at the second client device 114(2)). As used herein, the term "asynchronous playback mode" may be used to refer to a playback mode that may be enabled at an individual client device so that an individual playback position of media content can be controlled at each individual client device independently of the playback position of the media content at other individual client devices. As used herein, the term "synchronized playback mode" may be used to refer to a playback mode that may be enabled at an individual client device to synchronize the playback position of media content across multiple client devices.

[0053] As an exemplary but non-limiting scenario for switching between playback modes, during a communication session 128 and while sharing media content, the system 100 may receive a first playback mode command associated with playback of the media content, wherein the first playback mode command instructs the system 100 to switch the playback mode to an asynchronous playback mode across all client devices. Then, in response to the first playback mode command, the system 100 may transmit multiple playback mode commands to multiple client devices 114, thereby enabling individual playback head positions for playback of the media content to be independently adjusted at each of the individual client devices. For example, as described above, when the first device 114(1) is in asynchronous playback mode, a user at the first client device 114(1) may adjust the playback head position at the first client device without affecting the playback head position at the second client device 114(2).

[0054] Then, at a certain point in time when the individual client device 114 is operating in the asynchronous playback mode, the system 100 may receive a second playback mode command associated with the playback of the media content, wherein the second playback mode command instructs the system 100 to switch the playback mode to a synchronized playback mode across all client devices. Then, in response to the second playback mode command, the system 100 may transmit a plurality of playback mode commands to the plurality of client devices 114, thereby causing each of the individual client devices 114 to update the playhead position to a particular playhead position specified within the received playback mode command. For example, the first device 114(1) may be configured to generate a synchronized playback mode command to simultaneously synchronize the playhead position at all client devices to the current playhead position of the media content at the first client device 114(1).

[0055] Aspects of the disclosed technology are mainly described in the context of play instructions generated based on user input, which are received at individual client devices in multiple different client devices for participating in a communication session. For example, a user can click a "play button" on his client device to initiate playback of media content across multiple different client devices. Play instructions generated based on such user input can be referred to as user play instructions in this article. However, such context is only for illustrative purposes, and many other scenarios are contemplated and within the scope of this disclosure. In another exemplary context, playback of media content across multiple client devices can be initiated or otherwise controlled by one or more server devices that perform the technology described in this article. For example, a network-based broadcast service such as an Internet-based television broadcast service (e.g., DirectTV Now) can activate a live streaming event with built-in delay control and / or compensation as described in this article in some cases. Therefore, references to "play instructions" in this specification and / or the attached claims can be used as a general term covering "(one or more) user play instructions" and other types of play instructions, and the play instructions are generated to initiate otherwise controlling the playback of media content. Similarly, reference to a "pause instruction" may be used as a general term covering both "user(s) pause instruction" and other types of pause instructions.

[0056] Figure 2 An alternative scenario is illustrated in which system 200 facilitates synchronous sharing of video content 206 during a communication session 128. As illustrated, system 200 includes one or more server devices 102 that facilitate communication sessions 128 by transmitting multiple bidirectional user streams 127 between multiple client devices 114. Figure 2 The bidirectional user flow 127 shown in FIG. 1 may be similar to that of FIG. Figures 1A to 1DFor example, in Figure 2 The first bidirectional user stream 127(1) shown in the figure can correspond to a combination of an upload stream of audio / video data captured by an input device physically installed in the first client device 114(1) and a download stream including audio / video data captured by (one or more) input devices physically installed in both the second client device and the Nth client device.

[0057] In the illustrated scenario, the conference data 104 defining aspects of the communication session 128 includes a link 202 pointing to (e.g., providing an address for) video content 206 hosted by a video hosting service 204. In some embodiments, the video hosting service 204 may be a third-party service that facilitates publicly hosted video content (e.g., hosting videos that anyone can access via the Internet - without restrictions). Additionally or alternatively, the video hosting service 204 may be a service that facilitates privately hosted video content (e.g., such that only invitees to the communication session 128 can access the video content). Individual ones of the client devices 114 may utilize the link 202 to access the video content 206 from the video hosting service 204. For example, as in Figure 2 As shown in FIG. 1 , each of the first client device 114 ( 1 ) to the Nth client device 114 (N) establishes communication with the video hosting service 204 using the link 202 , and ultimately obtains the corresponding media stream 132 , and obtains the video content 206 through the media stream 132 .

[0058] In some embodiments, such as in Figure 2 In the embodiment shown in , the communication module 116 can cache the video content 206 (which will eventually be shared during the communication session 128) before generating the user play instruction 130 in association with the video content 206. To illustrate this point, with respect to the first time T 1 and the second time T 2 describe Figure 2 The scenario, in which T 1 <T 2 As shown, at the first time T 1 , each of the client devices 114 is communicating a respective bidirectional user stream 127 with the server device 102 to facilitate a communication session 128. As further shown, at a first time T 1 , each of the client devices 114 is receiving a corresponding media stream 132 from the video hosting service 204 to independently maintain a cache 124 of video content 206 .

[0059] Then, at the second time T 2, generating a user play instruction 130 at the first client device 114(1) and transmitting it to the server device 102. Upon receiving the user play instruction 130, the server device 102 may dispatch a delayed play instruction 134 to the individual client devices 114 in the client devices 114, as described with respect to Figure 1B and Figure 1C For example, one or more server devices 102 may determine a custom delay time delay value 126 for each individual client device 114. Then, one or more server devices 102 may dispatch a delayed play instruction 134 to each client device 114, which includes the determined custom delay time delay value 126 for the client device 114. As described above, upon receiving the delayed play instruction 134, each of the client devices 114 may begin playing the video content 206 after waiting for the delay time indicated by the custom delay time delay value 126 for the client device.

[0060] Thus, in some embodiments, the delayed play instructions 134 may be dispatched to multiple client devices 114 simultaneously and may be configured to instruct the client devices 114 to defer playing the video content 206 for an amount of time that is determined based on the networking delays experienced at the individual client devices 114. Figures 1A to 1D Such an embodiment is described in the exemplary scenario of . Alternatively, the server device 102 can stagger the time at which the delayed play instructions 134 are transmitted to each specific client device 114 based on the networking delay experienced by the specific client device 114. For example, if the first client device is experiencing a networking delay of 10 milliseconds less than that of the second client device, the first delayed play instruction can be dispatched to the first client device 10 milliseconds after the second delayed play instruction is dispatched to the second client device. Then, each of the first client device and the second client device will receive its respective delayed play instruction at substantially the same time. In such an embodiment, individual client devices can begin playing the video content without delay when receiving their corresponding delayed play instructions.

[0061] Figure 3AAn exemplary scenario is illustrated in which the system 300 determines delay values ​​associated with individual client devices 114 for participating in a communication session 128. In the illustrated implementation, delay values ​​are determined on a rolling basis to accommodate changes in network delays for individual client devices 114 throughout the communication session 128. For example, the delay value for each individual client device 114 may be updated according to some predefined time interval (e.g., two seconds). Based on the predefined time interval, the system 300 may periodically "ping" the individual client device 114 to determine the current network delay being experienced at the individual client device 114. Aspects of the exemplary scenario in which the system 300 determines delay values ​​during a communication session 128 are described in detail below. FIG. 3A to FIG. 3D About the first time T 1 To the fourth time T 4 is described, where T 1 <T 2 <T 3 <T 4 .

[0062] In the first time T 1 , one or more server devices 102 facilitate a communication session 128 by transmitting a plurality of bidirectional user streams 127 between a plurality of client devices 114. In this manner, the system 300 enables participants in the communication session 128 to converse via each other's live audio / video streams (e.g., the communication session 128 may be a network-based video conference). As described above, the system 300 may utilize the delay data 108 to enable the media content shared during the communication session 128 to be played synchronously across all client devices 114. It should be appreciated that having accurate and up-to-date delay values ​​for each individual client device 114 may be useful for these purposes.

[0063] With respect to obtaining accurate and up-to-date latency values ​​on a per-device basis, the system 300 may deploy a latency monitor 110 during a communication session 128 to generate latency data 108 associated with individual client devices in the client devices 114. For example, during the communication session 128, the latency monitor 110 may periodically send pings 302 to the individual client devices 114. As used herein, the term "ping" may refer to an instance of information transmitted from a source (e.g., a server device 102) to a computing device and at least partially transmitted back from the computing device to the source. The latency monitor 110 may calculate the network latency associated with the individual client devices 114 based on the round-trip time taken for the individual pings to reach the client device 114 and then return to one or more server devices 102. For example, assuming that each segment of these individual round-trip times is equal in time, the latency value associated with any individual client device 114 may be assumed to be half of the measured round-trip time.

[0064] For the purpose of this discussion, assume that slightly earlier than the first time T 1 , the first ping 302(1) to the Nth ping 302(N) are respectively assigned to the first client device 114(1) to the Nth client device 114(N). Then, as in Figure 3A As shown in FIG. 1 , a first ping 302 ( 1 ) is sent by a first client device 114 ( 1 ) at a first time T 1 1 is transmitted back to the server device 102 and received by the server device 102. Based on the round trip time of this first ping 302(1), the latency monitor 110 determines the current network latency affecting the first client device 114(1). For the purposes of this discussion, assume that the round trip time for the first ping 302(1) is 24 milliseconds. Further assume that the latency monitor 110 is programmed to assume that the amount of time it takes for the ping 302 to travel from the server device 102 to the individual client device 114 through the one or more networks is equal to the amount of time it takes for the ping 302 to return from the individual client device 114 to the server device 102 through the one or more networks. In other words, the latency monitor 110 may assume that each segment of the round trip time is equal in time. Thus, as in Figure 3A As illustrated in , the delay monitor 110 updates the delay data 108 to define a delay value of 12 milliseconds for the first client device 114 ( 1 ).

[0065] In the first time T 1 , neither the second ping 302(2) nor the Nth ping 302(N) completes its respective round trip back to the server device 102. For this reason, as in Figure 3A As shown in the figure, at the first time T1 , the delay test on the second client device 114 (2) and the Nth client device 114 (N) is still in progress.

[0066] Now turn to Figure 3B , an exemplary scenario in which the system 300 determines a delay value associated with an individual client device 114 at a second time T 2 It is shown that the second time T 2 The second ping 302(2) corresponds to completing a round trip back to the server device 102. Based on the round trip time of the second ping 302(2), the latency monitor 110 determines the current network latency affecting the second client device 114(2). For the purposes of this discussion, assume that the round trip time of the second ping 302(2) is 42 milliseconds. Thus, as in Figure 3B As shown in the figure, in response to the second time T 2 Upon returning the second ping 302(2), the latency monitor 110 updates the latency data 108 to define a latency value of 21 milliseconds for the second client device 114(2).

[0067] At the second time T 2 , the Nth ping 302 (N) has not yet completed its round trip back to the server device 102. For this reason, as in Figure 3B As shown in the figure, at the second time T 2 , latency testing is still ongoing with respect to the Nth client device 114 (N).

[0068] Now turn to Figure 3C , an exemplary scenario in which the system 300 determines a delay value associated with an individual client device 114 at a third time T 3 It is shown that the third time T 3 corresponds to the Nth ping 302(N) completing a round trip back to the server device 102. Based on the round trip time of the Nth ping 302(N), the latency monitor 110 determines the current network latency affecting the Nth client device 114(N). For the purposes of this discussion, assume that the round trip time for the Nth ping 302(N) is 110 milliseconds. Thus, as in Figure 3C As shown in FIG. , in response to the third time T 3 Upon return of the Nth ping 302 (N), the latency monitor 110 updates the latency data 108 to define a latency value of 55 milliseconds for the Nth client device 114 (N).

[0069] In some implementations, the system 300 can be configured to generate the delay data 108 on a rolling basis such that the delay values ​​for individual client devices 114 are continuously updated during the communication session 128. For example, the system 300 can be configured to periodically ping individual client devices 114 and update the delay values ​​associated with the individual client devices 114 when these periodically transmitted pings 302 are transmitted back to the server device 102. For example, as in Figure 3D As shown in FIG. , at the fourth time T 4 , after the initial set of pings 302 have been sent and transmitted back to the server device 102, the second set of pings 304 are dispatched to the first through Nth client devices 114. As illustrated, at the fourth time T 4 , no individual 304 has yet completed its respective round trip back to the server device 102, and therefore, testing is still ongoing with respect to this second set of pings 304. Later, as individual pings in these pings 304 complete their respective round trips, the delay data 108 may be continually updated.

[0070] Now turn to Figure 4A , illustrates two exemplary graphical user interfaces that may be simultaneously displayed on a first client device 114(1) and a second client device 114(2) when those devices are operating in asynchronous playback mode. As illustrated, the first client device 114(1) is receiving a first media stream 132(1) from a server device 102 and the second client device 114(2) is receiving a second media stream 132(2) from the server device 102. Here, the first client device 114(1) and the second client device 114(2) render media content 402, such as an MP4 video, using the first media stream 132(1) and the second media stream 132(2), respectively. As further illustrated, the first client device 114(1) is receiving a first bidirectional user stream 127(1) from the server device 102 and the second client device 114(2) is receiving a second bidirectional user stream 127(2) from the server device 102. Here, the first client device 114(1) and the second client device 114(2) are respectively utilizing the first bidirectional user stream 127(1) and the second bidirectional user stream 127(2) to render live video 404 of one or more other participants of the communication session 128 (e.g., audio and / or video content captured via input devices at the client devices 114).

[0071] As in Figure 4A, media content received independently of the bidirectional user stream 132 enables the first client device 132(1) and the second client device 114(1) to adjust the playback position of the media content independently of each other while still rendering the live video and / or audio received during the communication session 128. For example, a first user participating in the communication session at the first client device 114(1) has adjusted the playback position to 5 minutes and 34 seconds, while a second user participating in the communication session at the second client device 114(2) has adjusted the playback position to 15 minutes and 44 seconds. As used herein, the term "asynchronous playback mode" may be used to refer to a playback mode enabled at individual client devices so that the individual playback position of media content can be controlled at each individual client device independently of the playhead position of the media content at other individual client devices.

[0072] Now turn to Figure 4B , illustrates two exemplary graphical user interfaces that may be displayed simultaneously on a first client device 114(1) and a second client device 114(2) after those devices have switched from an asynchronous playback mode to a synchronous playback mode. For example, during a communication session 128 and when a client device 114 is in a state such as with respect to Figure 4A When operating in the described asynchronous playback mode, a user operating a first client device 114(1) may provide user input to the first client device 114(1) to cause all (or a subset thereof) of the other client devices 114(1) to switch from the asynchronous playback mode to the synchronized playback mode. As a result of the user input, the system 100 may transmit individual playback mode commands to the individual client devices, thereby causing each of the individual client devices 114 to update the playhead position to a particular playhead position specified in the received playback mode command. For example, the first device 114(1) may be configured to generate a synchronized playback mode command that simultaneously synchronizes the playhead position at all client devices to the current playhead position of the media content at the first client device 114(1). In a specifically illustrated but non-limiting example, in response to the user input received at the first client device 114(1), the playback position at the second client device 114(2) is aligned to (e.g., synchronized to) the current playback position (i.e., 5 minutes 34 seconds) at the first client device 114(1). As used herein, the term "synchronized playback mode" may be used to refer to a playback mode enabled at an individual client device to synchronize the playback position of media content across multiple client devices.

[0073] In some implementations, user rights or permissions for controlling various aspects of playback of media content across all client devices 114 may be limited to a particular client device 114 having master control status. Figure 4B , a first client device 114(1) is displaying an indication that the device currently has a "master control state" for playback of media content, while a second client device 114(2) is displaying an indication that the device currently has a "slave control state". In some embodiments, a particular client device having a master control state with respect to the media content is allowed to Figure 4B The synchronized playback mode shown in Figure 4A . Thus, a participant in a communication session that currently has user permission to grant master control status can allow or restrict other participants in the communication session to independently control the playback position of media content at their respective client devices. For example, a participant with master control status can switch the playback mode to asynchronous playback mode so that all participants can independently control their own media playback. Then, when he or she deems it appropriate, the participant with master control status can switch the playback mode back to synchronized playback mode so that the playback position across all client devices is synchronized with the specific playhead position currently being played at the client device of a specific "master user" participant.

[0074] Furthermore, in some embodiments, the systems and techniques described herein can facilitate transferring master control states between individual client devices in a client device. Figure 4B , a first client device 114(1) is displaying a user control or button with a status of "pass-off control" and a second client device 114(2) is displaying a user control or button with a status of "request control." Here, by selecting the pass-off control button (and then selecting one or more specific participants), the participant currently having the primary control status is able to pass that status to some other participant as desired. In this regard, Figure 4C An exemplary graphical user interface is shown that may be simultaneously displayed on a first client device to a second client device after a master control state is transferred from the first client device to the second client device.

[0075] Figure 4DAn exemplary graphical user interface is shown that can be displayed simultaneously on two different client devices, one of which is operating in an autonomous queue state. Specifically, in the illustrated scenario, a first client device 114(1) is operating in a master control state and has selected a synchronized playback mode. Thus, as described above, in addition to the first device 114(1), client devices operating in a slave control state can cause any playhead position currently being rendered at the first device 114(1) to be rendered. It can be appreciated that in many cases, it may be desirable to queue (e.g., bookmark) various playhead positions so that a user of the first device 114(1) (or any device operating in a master control state) can quickly adjust the playhead position without manually scrolling (e.g., dragging) a playhead position indicator. However, there may also be situations where a participant may wish to jump to a playhead position for which no queue or bookmark exists.

[0076] In such a case, a second device 114(2) that is not operating in a master control state can be given an autonomous queue state to enable a user of the second device 114(2) to adjust his or her playhead position independently of other devices - even if some other devices are operating in a slave control state and synchronized playback mode. The user of the second device 114(2) can then move the playhead position to some desired point and send that current position to the queue at the first device 114(1). For example, as illustrated, the second device 114(2) renders the playhead position at 11:50, even if the first device 114(1) is causing other devices operating in a slave control state to render some other earlier playhead position. As further illustrated, because the user of the second device 114(2) adds the playhead position of 11:50 to the queue of any device operating in the master control state, a bookmark indicator (e.g., which is illustrated as a star in the non-limiting scenario) becomes visible at the playhead position of 11:50 along with a message "Chris has queued the playhead position at 11:50." When the user of the first device 114(1) selects the bookmark indicator, other devices participating in the conference session can immediately synchronize to that position. In this way, one or more participants in the communication session can work in the background on the second device 114(2) to manually scroll through the video content and instantly generate a bookmark that can be acted upon by the user of the first device 114(1).

[0077] Now turn to Figure 5, a flow chart of an exemplary process 500 for performing delay compensation techniques to facilitate synchronized sharing of video content across multiple client devices during a communication session is shown. It should be understood by those of ordinary skill in the art that the operations of the methods disclosed herein need not be presented in any particular order, and that it is possible and contemplated to perform some or all of the operations in (one or more) alternative orders. For ease of description and illustration, the operations have been presented in the order in which they are presented. Operations may be added, omitted, performed together, and / or performed simultaneously without departing from the scope of the appended claims.

[0078] It should also be understood that the illustrated method can be terminated at any time and need not be executed in its entirety. Some or all of the operations of the method and / or substantially equivalent operations can be performed by running computer-readable instructions contained on a computer storage medium, as defined herein. The term "computer-readable instructions" and its variants as used in the specification and claims are used broadly herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, and the like. Computer-readable instructions can be implemented on a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, and the like.

[0079] It should therefore be appreciated that the logical operations described herein are implemented as: (1) a sequence of computer-implemented actions or program modules running on a computing system such as those described herein; and / or (2) interconnected machine logic circuits or circuit modules within a computing system. The implementation is a matter of choice depending on the performance and other requirements of the computing system. Thus, the logical operations may be implemented in software, firmware, dedicated digital logic, and any combination thereof.

[0080] At operation 502 of routine 500, the system transmits multiple bidirectional user streams 127 between multiple client devices 114 to facilitate a communication session 128. For example, each of the multiple client devices 114 can capture (e.g., a user's) audio and video "A / V" data via various input devices (such as a camera and microphone). The captured A / V data can be continuously streamed to one or more server devices 102. As the A / V data streams are received from the client devices, the server device 102 can transmit the A / V data streams received from each other client device 114 to each individual client device 114. In this manner, each individual client device 114 is enabled to render live video 404 of the participants using the other client devices 114, thereby facilitating the communication session 128 as a type of virtual conference or web-based meeting. For example, as in Figure 4A and Figure 4BAs illustrated in , each of the first client device 114 ( 1 ) and the second client device 114 ( 2 ) renders a stream of live video 404 of the other participants of the communication session 128 .

[0081] At operation 504, the system obtains media data defining media content to be played simultaneously across client devices during the communication session 128. For example, at some point during the communication session 128, media content such as a video hosted by the server device 102 may be streamed to and displayed at the client device 114 while the client device 114 is rendering live video 404 or simply an audio stream captured at the client devices 114 of other participants.

[0082] At operation 506, the system determines delay data that defines customized delay values ​​for individual client devices in the client devices. Specifically, the delay data defines individual delay values ​​associated with transmitting information to individual client devices. For example, a delay value of 54 milliseconds for a particular client device indicates that information sent to the particular client device will require 54 milliseconds to be transmitted across one or more networks and ultimately reach the particular client device.

[0083] In some embodiments, determining the latency value for an individual one of the client devices includes periodically pinging the individual client device and measuring the amount of time it takes for the ping to be returned from the receiving device to (e.g., backhauled to) the sending device. FIG. 3A to FIG. 3B As described, the system can determine accurate and up-to-date latency values ​​based on each device by deploying a latency monitor 110 that calculates network latency associated with individual client devices based on the round-trip time of an individual ping to the client device and then back to one or more server devices. For example, assuming that each of these individual round-trip times is equal in time, the latency value associated with any individual client device 114 can be assumed to be half of the measured round-trip time.

[0084] In some embodiments, the delay monitor 110 can repeatedly calculate and recalculate the delay value for the individual client device based on the sequence of delay values ​​stored in the last-in-first-out (LIFO) database. For example, the delay monitor 110 can be configured to periodically transmit pings to the individual client device according to some predefined intervals (such as, for example, 1 second, 2 seconds, or any other suitable time interval). Then, as the individual client device transmits the ping back to the delay monitor 110, the most recently calculated delay value can be added to the LIFO database, and the oldest value can be deleted. In this way, the LIFO database can be used to maintain samples of delay values, which are statistically relevant due to having a suitable size (e.g., 5 values, 10 values, etc.), and are temporally relevant due to being recently calculated, that is, not stale. In an embodiment in which the LIFO database is maintained using the most recent set of delay values ​​for the individual client device, the current delay value used to determine the appropriate delay delay as described above can be a dynamic rolling average of those delay values ​​maintained in the LIFO database.

[0085] In some embodiments, the latency monitor 110 deploys a machine learning engine 112 to perform pattern recognition techniques to determine a current latency value that illustrates a trend within the latency values ​​determined for any particular client device. For example, where a recent set of latency values ​​for an individual client device reveals a trend of increasing or decreasing latency, the machine learning engine 112 may adjust the dynamic rolling average up or down, respectively, to account for the identified trend. As another example, the machine learning engine 112 may perform a browser sniffing operation for an individual client device to identify a specific web browser being used to render media content. Then, to account for variations in latency regarding how quickly different types of web browsers are able to render media content, the machine learning engine 112 may adjust the dynamic rolling average of the current latency value.

[0086] At operation 508, the system receives a user play instruction generated based on the user input received at a specific client device 114, for initiating playback of the media content across multiple client devices 114. For example, a user at a specific client device can select the media data 106 stored in association with the conference data 104 to cause the media content to begin playing at the specific client device. Then, before providing user input to cause all other client devices to begin playback of the media content at the desired play head position, the user can adjust the play head position of the media content at the specific client device to the desired play head position. In some embodiments, the user play instruction can include an indication of the desired play head position at which the media content will begin synchronous playback across all client devices 114.

[0087] At operation 510, in response to the user play instruction, the system transmits a delayed play instruction to the client device based on the delay data. As described above, the delayed play instruction is transmitted to the client device in a manner that can specify an intentional delay delay on a subset of the client devices so that playback of the media data begins synchronously across multiple client devices. For example, an individual delayed play instruction may include a delay delay value that specifies the amount of time that an individual client device is to wait after receiving the corresponding delayed play instruction before initiating playback of the media content obtained via the media stream.

[0088] It should be appreciated that the subject matter described above can be implemented as a computer-controlled device, a computer process, a computing system, or as an article such as a computer-readable storage medium. The operations of the exemplary method are shown in individual boxes and summarized with reference to those boxes. The method is illustrated as a logical flow of boxes, each of which can represent one or more operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable media, which, when executed by one or more processors, enable the one or more processors to perform the described operations.

[0089] Typically, computer executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be run in any order, combined in any order, subdivided into multiple sub-operations, and / or run in parallel to implement the described processes. The described processes can be performed by resources associated with one or more devices, such as one or more internal or external CPUs or GPUs, and / or one or more hardware logic units, such as a field programmable gate array ("FPGA"), a digital signal processor ("DSP"), or other types of accelerators.

[0090] All methods and processes described above can be embodied in software code modules run by one or more general-purpose computers or processors, and are particularly fully automated. The code modules can be stored in any type of computer-readable storage medium or other computer storage device, such as those described below. Some or all methods can alternatively be embodied in special-purpose computer hardware, such as those described below.

[0091] Any routine description, element or block in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to potentially represent a module, segment or code portion that includes one or more executable instructions for implementing a specific logical function or element in the routine. Alternative implementations are included within the scope of the examples described herein, wherein elements or functions may be deleted, or run in an order other than that shown or discussed, including running substantially synchronously or in reverse order, depending on the functions involved, as will be understood by those skilled in the art.

[0092] Figure 6 604 .

[0093] As illustrated, communication session 604 may be implemented between a plurality of client computing devices 606(1)-606(N) (where N is a number having a value of two or greater) associated with or as part of system 602. Client computing devices 606(1)-606(N) enable users (also referred to as individuals) to participate in communication session 604.

[0094] In this example, communication session 604 is hosted by system 602 over one or more networks 608. That is, system 602 can provide services that enable users of client computing devices 606(1)-606(N) to participate in communication session 604 (e.g., via live viewing and / or recorded viewing). Thus, "participants" of communication session 604 can include users and / or client computing devices (e.g., multiple users can participate in a communication session in a room via use of a single client computing device), each of which can communicate with other participants. Alternatively, communication session 604 can be hosted by one of client computing devices 606(1)-606(N) utilizing peer-to-peer technology. System 602 can also host chat conversations and other team collaboration functionality (e.g., as part of an application suite).

[0095] In some implementations, such chat conversations and other team collaboration functions are considered to be external communication sessions distinct from the communication session 604. The computing system 602 that collects participant data in the communication session 604 can be linked to such an external communication session. Thus, the system can receive information that enables connection to such an external communication session, such as date, time, session details, etc. In one example, a chat conversation can be conducted in accordance with the communication session 604. In addition, the system 602 can host the communication session 604, which includes at least a plurality of participants co-located at a meeting location (such as a conference room or auditorium) or located at different locations.

[0096] In the example described herein, client computing devices 606(1) to 606(N) participating in a communication session 604 are configured to receive and render communication data for display on a user interface of a display screen. The communication data can include a collection of various instances or streams of live content and / or recorded content. The collection of various instances or streams of live content and / or recorded content can be provided by one or more cameras such as a video camera. For example, individual streams of live or recorded content can include user data associated with a video feed provided by a video camera (e.g., audio and visual data capturing the appearance and voice of a user participating in the communication session). In some implementations, the video feed can include such audio and visual data, one or more still images, and / or one or more avatars. The one or more still images can also include one or more avatars.

[0097] Another example of an individual stream of live or recorded content can include user data including an avatar of a user participating in a communication session and audio data capturing the user's voice. Yet another example of an individual stream of live or recorded content can include user stream data including a file displayed on a display screen and audio data capturing the user's voice. Thus, various streams of live or recorded content within the communication data enable remote conferencing between a group of people and sharing content within the group. In some implementations, various streams of live or recorded content within the communication data can originate from multiple co-located video cameras in a space (such as a room) to live record or stream a presentation of one or more individuals presenting the content and one or more individuals consuming the presented content.

[0098] Participants or attendees can view the content of the communication session 604 live as the activity occurs, or alternatively, view it via a recording at a later time after the activity occurs. In the example described herein, the client computing devices 606 (1) to 606 (N) participating in the communication session 604 are configured to receive and render communication data for display on a user interface of a display screen. The communication data can include a collection of various instances or streams of live and / or recorded content. For example, an individual stream of content can include user stream data associated with a video feed (e.g., audio and visual data capturing the appearance and voice of a user participating in the communication session). Another example of an individual stream of content can include user stream data, which includes an avatar of a user participating in a conference session and audio data capturing the user's voice. Yet another example of an individual stream of content can include user stream data, which includes content items displayed on a display screen and / or audio data capturing the user's voice. Therefore, the various streams of content within the communication data enable a conference or broadcast presentation to be facilitated between a group of people dispersed across remote locations.

[0099] A participant or attendee of a communication session is a person within range of a camera or other image and / or audio capture device, such that the person's movements and / or sounds generated while viewing and / or listening to content shared via the communication session can be captured (e.g., recorded). For example, a participant may be sitting in a crowd, viewing shared content at a broadcast location where a stage presentation is taking place. Alternatively, a participant may be sitting in an office conference room, viewing shared content of a communication session with other colleagues via a display screen. Further, a participant may be sitting or standing in front of a personal device (e.g., a tablet computer, a smart phone, a computer, etc.), viewing shared content of a communication session individually in his or her office or home.

[0100] Figure 6 System 602 includes device(s) 610. Device(s) 610 and / or other components of system 602 can include distributed computing resources that communicate with each other and / or with client computing devices 606(1) to 606(N) via one or more networks 608. In some examples, system 602 can be a stand-alone system tasked with managing aspects of one or more communication sessions, such as communication session 604. As an example, system 602 can be managed by an entity such as SLACK, WEBEX, GOTOMEETING, GOOGLE HANGOUTS, etc.

[0101] The network(s) 608 may include, for example, a public network such as the Internet, a private network such as an institutional and / or personal intranet, or some combination of private and public networks. The network(s) 608 may also include any type of wired and / or wireless network, including, but not limited to, a local area network ("LAN"), a wide area network ("WAN"), a satellite network, a cable network, a Wi-Fi network, a WiMax network, a mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. The network(s) 608 may utilize communication protocols, including packet-based and / or datagram-based protocols, such as the Internet Protocol ("IP"), the Transmission Control Protocol ("TCP"), the User Datagram Protocol ("UDP"), or other types of protocols. In addition, the network(s) 608 may also include a plurality of devices that facilitate network communications and / or form the hardware foundation for the network, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, backbone devices, etc.

[0102] In some examples, network(s) 608 may also include devices that enable connection to wireless networks, such as wireless access points (“WAPs”). Examples support connection via WAPs that send and receive data over various electromagnetic frequencies (e.g., radio frequencies), including WAPs that support the Institute of Electrical and Electronics Engineers (“IEEE”) 802.7 standards (e.g., 802.7g, 802.7n, 802.7ac, etc.) and other standards.

[0103] In various examples, (one or more) devices 610 may include one or more computing devices that operate in a cluster or other grouping configuration to share resources, balance loads, improve performance, provide failover support or redundancy, or for other purposes. For example, (one or more) devices 610 may belong to various device categories, such as conventional server-type devices, desktop computer-type devices, and / or mobile devices. Therefore, although illustrated as a single type of device or server-type device, (one or more) devices 610 may include multiple device types and are not limited to a specific type of device. (One or more) devices 610 may represent, but are not limited to: a server computer, a desktop computer, a network server computer, a personal computer, a mobile computer, a laptop computer, a tablet computer, or any other type of computing device.

[0104] The client computing device (e.g., one of client computing devices 606(1)-606(N)), each of which is also referred to herein as a "data processing system") can belong to a variety of categories of devices, which can be the same as or different from device(s) 610, such as conventional client-type devices, desktop-type devices, mobile-type devices, dedicated-type devices, embedded devices, and / or wearable devices. Thus, the client computing device can include, but is not limited to, a desktop computer, a game console and / or gaming device, a tablet computer, a personal data assistant ("PDA"), a mobile phone / tablet hybrid, a laptop computer, a telecommunications device, a computer navigation-type client computing device (such as a satellite-based navigation system, including a Global Positioning System ("GPS") device), a wearable device, a virtual reality ("VR") device, an augmented reality ("AR") device, an implanted computing device, an automotive computer, a network-enabled television, a thin client, a terminal, an Internet of Things ("IoT") device, a workstation, a media player, a personal video recorder ("PVR"), a set-top box, a camera, an integrated component (e.g., a peripheral device) contained in a computing device, an appliance, or any other type of computing device. Furthermore, the client computing device may include a combination of the examples of client computing devices listed earlier, such as, for example, a desktop computer-type device or a mobile-type device combined with a wearable device, etc.

[0105] The various classes and device types of client computing devices 606(1) through 606(N) can represent any type of computing device having one or more data processing units 692 operably connected to a computer-readable medium 694, such as via a bus 616, which in some cases can include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any number of local, peripheral, and / or independent buses.

[0106] The executable instructions stored on the computer-readable medium 694 may include, for example, an operating system 619 , a communication module 620 , a profile module 622 , and other modules, programs, or applications that can be loaded and executed by the data processing unit 692 .

[0107] The client computing devices 606(1)-606(N) may also include one or more interfaces 624 to enable communication between the client computing devices 606(1)-606(N) and other networked devices (e.g., device 610) via the network(s) 608. Such network interface(s) 624 may include one or more network interface controllers (NICs) or other types of transceiver devices to send and receive communications and / or data via the network. In addition, the client computing devices(s) 606(1)-606(N) may include input / output ("I / O") interfaces (devices) 626 that support communication with input / output devices such as user input devices including peripheral input devices (e.g., game controllers, keyboards, mice, pens, voice input devices such as microphones, video cameras for obtaining and providing video feeds and / or still images, touch input devices, gesture input devices, etc.) and / or output devices including peripheral output devices (e.g., displays, printers, audio speakers, tactile output devices, etc.). Figure 6 The client computing device 606 ( 1 ) is illustrated as being connected in some manner to a display device (eg, display screen 629 (N)) that is capable of displaying a UI according to the techniques described herein.

[0108] exist Figure 6 In the exemplary environment 600 of FIG. 6 , client computing devices 606(1) to 606(N) can use their respective client modules 620 to connect to each other and / or to other external devices (one or more) to participate in a communication session 604 or to contribute activities to the collaborative environment. For example, a first user can use client computing device 606(1) to communicate with a second user of another client computing device 606(2). When executing client module 620, users can share data, which can cause client computing device 606(1) to connect to system 602 and / or other client computing devices 606(2) to 606(N) via network(s) 608.

[0109] The client computing device(s) 606(1)-606(N) may use their respective profile modules 622 to generate participant profiles (in Figure 6 ), and provide the participant profile to other client computing devices and / or to the device(s) 610 of the system 602. The participant profile may include one or more of the following: an identity of a user or a group of users (e.g., name, unique identifier (“ID”), etc.), user data (such as personal data), machine data (such as location (e.g., IP address, room in a building, etc.), and technical capabilities, etc. The participant profile may be used to register the participant for a communication session.

[0110] As in Figure 6 , device(s) 610 of system 602 include a server module 630 and an output module 632. In this example, server module 630 is configured to receive media streams 634(1) to 634(N) from individual client computing devices, such as client computing devices 606(1) to 606(N). As described above, the media streams can include video feeds (e.g., audio and video data associated with a user), audio data to be output with a presentation of an avatar of the user (e.g., an audio-only experience in which the user's video data is not transmitted), text data (e.g., a text message), file data, and / or screen sharing data (e.g., a document, slideshow, image, video, etc. displayed on a display screen), etc. Thus, server module 630 is configured to receive a collection of various media streams 634(1) to 634(N) (the collection is referred to herein as "user A / V data 634") during a live view of communication session 604. In some scenarios, not all client computing devices participating in communication session 604 provide media streams. For example, the client computing device may be merely a consuming or “listening” device, such that it merely receives content associated with the communication session 604 , but does not provide any content to the communication session 604 .

[0111] In various examples, server module 630 can select aspects of stream 634 to be shared with individual client computing devices of participating client computing devices 606(1)-606(N). Thus, server module 630 can be configured to generate session data 636 based on stream 634 and / or communicate session data 636 to output module 632. Output module 632 can then transmit communication data 639 to client computing devices (e.g., client computing devices 606(1)-606(3) participating in a real-time view of the communication session). Communication data 639 can include video, audio, and / or other content data provided by output module 632 based on content 650 associated with output module 632 and based on received session data 636.

[0112] As shown, output module 632 transmits communication data 639(1) to client computing device 606(1), and transmits communication data 639(2) to client computing device 606(2), and transmits communication data 639(3) to client computing device 606(3), etc. The communication data 639 transmitted to the client computing devices can be the same or different (e.g., the positioning of the content flow within the user interface can be different from one device to the next).

[0113] In various implementations, the device(s) 610 and / or the communication module 620 can include a GUI presentation module 640. The GUI presentation module 640 can be configured to analyze the communication data 639 for delivery to one or more of the client computing devices 606. Specifically, the UI presentation module 640 at the device(s) 610 and / or the client computing device 606 can analyze the communication data 639 to determine an appropriate manner for displaying the video, image, and / or content on the display screen 629 of the associated client computing device 606. In some implementations, the GUI presentation module 640 can provide the video, image, and / or content to a presentation GUI 646 rendered on the display screen 629 of the associated client computing device 606. The GUI presentation module 640 can cause the GUI presentation module 640 to render on the display screen 629. The presentation GUI 646 can include the video, image, and / or content analyzed by the GUI presentation module 640.

[0114] In some implementations, the presentation GUI 640 may include multiple sections or grids that may render or include video, images, and / or content for display on the display screen 629. For example, a first section of the presentation GUI 646 may include a video feed of a presenter or individual, and a second section of the presentation GUI 640 may include a video feed of an individual consuming the conference information provided by the presenter or individual. The GUI presentation module 640 may populate the first and second sections of the presentation GUI 646 in a manner that appropriately mimics an ambient experience that the presenter and individual may be sharing.

[0115] In some implementations, the GUI presentation module 640 can magnify or provide a zoomed view of the individual represented by the video feed to highlight the individual's reactions to the presenter, such as facial features. In some implementations, the presentation GUI 640 can include video feeds of multiple participants associated with a meeting, such as a general communication session. In other implementations, the presentation GUI 640 can be associated with a channel such as a chat channel, an enterprise team channel, etc. Thus, the presentation GUI 640 can be associated with an external communication session that is different from a general communication session.

[0116] Figure 7A diagram illustrating exemplary components of an exemplary device 700 (also referred to herein as a "computing device") is illustrated, and the device 700 is configured to generate data for some user interfaces disclosed herein. The device 700 can generate data including one or more parts, which can render or include video, images, virtual objects, and / or content for display on a display screen. The device 700 can identify one of the (one or more) devices described herein. Additionally or alternatively, the device 700 can represent one of the client computing devices 1106.

[0117] As illustrated, device 700 includes one or more data processing units 702, computer readable media 704, and communication interface(s) 706. The components of device 700 are operably connected, for example, via bus 709, which may include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any variety of local, peripheral, and / or independent buses.

[0118] As used herein, data processing unit(s) (such as data processing unit(s) 702 and / or data processing unit(s) 1192) may represent, for example, a CPU-type data processing unit, a GPU-type data processing unit, a field programmable gate array ("FPGA"), another type of DSP, or other hardware logic components that may be driven by a CPU in some cases. For example, but not limited to, illustrative types of hardware logic components that may be utilized include application specific integrated circuits ("ASICs"), application specific standard products ("ASSPs"), systems on chips ("SOCs"), complex programmable logic devices ("CPLDs"), and the like.

[0119] As used herein, computer-readable media (such as computer-readable media 704 and computer-readable media 1194) can store instructions that can be executed by (one or more) data processing units. The computer-readable medium can also store instructions that can be executed by an external data processing unit (e.g., by an external CPU, an external GPU) and / or by an external accelerator (e.g., an FPGA-type accelerator, a DSP-type accelerator, or any other internal or external accelerator). In various examples, at least one CPU, GPU, and / or accelerator is incorporated into the computing device, while in some examples, one or more of the CPU, GPU, and / or accelerator is external to the computing device.

[0120] Computer readable media (which may also be referred to herein as computer readable media) may include computer storage media and / or communication media. Computer storage media may include one or more of the following: volatile memory, non-volatile memory and / or other persistent and / or auxiliary computer storage media, removable and non-removable computer storage media, implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Thus, computer storage media includes media in tangible and / or physical form contained within and / or as part of a device or in hardware components external to the device, including, but not limited to: random access memory (“RAM”), static random access memory (“SRAM”), dynamic random access memory (“DRAM”), phase change memory (“PCM”), read-only memory (“ROM”), erasable programmable read-only memory (“EPROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory, compact disk read-only memory (“CD-ROM”), digital versatile disk (“DVD”), optical cards or other optical storage media, magnetic cassettes, magnetic tape, magnetic disk storage, magnetic cards or other magnetic storage devices or media, solid-state memory devices, storage arrays, network attached storage, storage area networks, hosted computer storage devices, or any other storage memory, storage devices, and / or storage media that can be used to store and maintain information for access by a computing device.

[0121] In contrast to computer storage media, communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism. As defined herein, computer storage media does not include communication media. That is, computer storage media does not include communication media that itself consists only of a modulated data signal, carrier wave, or propagation signal.

[0122] The communication interface(s) 706 may represent, for example, a network interface controller ("NIC") or other type of transceiver device to send and receive communications over a network. In addition, the communication interface(s) 706 may include one or more video cameras and / or audio devices 722 to enable generation of video feeds and / or still images, etc.

[0123] In the illustrated example, computer-readable medium 704 includes data storage 708. In some examples, data storage 708 includes a data storage device, such as a database, a data warehouse, or other types of structured or unstructured data storage devices. In some examples, data storage 708 includes a corpus and / or a relational database having one or more tables, indexes, stored procedures, etc. to enable data access, including, for example, one or more Hypertext Markup Language ("HTML") tables, Resource Description Framework ("RDF") tables, Web Ontology Language ("OWL") tables, and / or Extensible Markup Language ("XML") tables.

[0124] The data store 708 may store data for the operation of processes, applications, components, and / or modules stored in the computer-readable medium 704 and / or executed by the (one or more) data processing units 702 and / or (one or more) accelerators. For example, in some examples, the data store 708 may store session data 710 (e.g., session data 1136 as shown in FIG. 11), profile data 712 (e.g., associated with participant profiles), and / or other data. The session data 710 may include the total number of participants (e.g., users and / or client computing devices) in the communication session, activities occurring in the communication session, a list of invitees for the communication session, and / or other data related to when and how the communication session is conducted or hosted. The data store 708 may also include content data 714, such as content including video, audio, or other content for rendering and display on one or more display screens.

[0125] Alternatively, some or all of the above data can be stored on a separate memory 716 on one or more data processing units 702, such as a CPU-type processor, a GPU-type processor, an FPGA-type accelerator, a DSP-type accelerator, and / or a memory on other accelerators. In this example, the computer-readable medium 704 also includes an operating system 718 and an application programming interface (API) 710 (one or more) configured to expose the functions and data of the device 700 to other devices. In addition, the computer-readable medium 704 includes one or more modules, such as a server module 730, an output module 732, and a GUI rendering module 740, but the number of modules illustrated is only an example, and the number can become higher or lower. That is, the functions described in conjunction with the illustrated modules in this article can be performed by a smaller number of modules or a larger number of modules on one device or across multiple devices.

[0126] It should also be appreciated that many variations and modifications may be made to the examples described above, and the elements therein should be understood to be in other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure and protected by the following claims.

[0127] Example clauses

[0128] Exemplary clause A: A computer-implemented method comprising: obtaining media data defining media content to be played simultaneously at multiple client devices operating in association with a communication session; determining delay data defining a delay value associated with transmitting a signal to each of the multiple client devices participating in the communication session, wherein the delay value associated with each respective client device is determined based on evaluating connection conditions to each client device during the communication session; receiving a play instruction to initiate playback of the media content at the multiple client devices; and in response to the play instruction, causing the playback of the media content to begin across the multiple client devices with different built-in delays controlled by a delay play instruction provided to each of the client devices, wherein: each delay play instruction corresponds to a respective client device, each delay play instruction defines a delay delay amount determined based on a difference between the delay data corresponding to the respective client device and the delay data corresponding to a client device corresponding to a maximum delay value, the delay delay amount being configured to cause the respective client device to postpone initiating playback of the media data by a defined delay delay amount; and at least two delay play instructions define different delay delay amounts.

[0129] Exemplary Clause B: The computer-implemented method of Exemplary Clause A, wherein a first delay for a first client device to postpone initiating playback of the media data is determined at least in part based on a portion of the delay data associated with a second client device.

[0130] Exemplary Clause C: The computer-implemented method described in any one of Exemplary Clauses A to B, further comprising: receiving a first playback mode command associated with the playback of the media content; in response to the first playback mode command, enabling individual playback head positions for the playback of the media content to be independently adjusted at the individual client devices; receiving a second playback mode command associated with the playback of the media content, wherein the second playback mode command indicates a specific playback head position; and in response to the second playback mode command, enabling the playback of the media content to be updated to the specific playback head position across the multiple client devices.

[0131] Exemplary Clause D: A computer-implemented method as described in any of Exemplary Clauses A to C, wherein determining the delay data comprises: transmitting a plurality of pings to the plurality of client devices during the communication session; and calculating a plurality of delay values ​​based at least in part on a plurality of round-trip times associated with the plurality of pings.

[0132] Exemplary Clause E: The computer-implemented method of any of Exemplary Clauses A to D, wherein the play instruction is a user play instruction generated based on user input received at a particular client device of the plurality of client devices.

[0133] Exemplary Clause F: A computer-implemented method as described in any of Exemplary Clauses A to E, wherein each delayed playback instruction defines both: a playback head position indicated by the user playback instruction generated based on the user input received at the particular client device; and an individual delay delay determined for a corresponding individual one of the individual client devices based on a portion of the delay data corresponding to at least one other client device.

[0134] Exemplary Clause G: A computer-implemented method as described in any of Exemplary Clauses A to F, wherein at least some of the individual delayed playback instructions define individual delay delays that cause at least some of the individual client devices to postpone initiating the playback of the media data until each of the multiple client devices receives a corresponding delayed playback instruction.

[0135] Exemplary Clause H: A computer-implemented method as described in any of Exemplary Clauses A to G, wherein individual delayed playback instructions cause at least some of the individual client devices to begin caching the media content received from one or more media streams independently of one or more bidirectional user streams.

[0136] Exemplary Clause I: The computer-implemented method described in any one of Exemplary Clauses A to H, further comprising: in response to determining that the playback of the media content has been paused at a specific playback head position at a first client device, transmitting a pause instruction to a second client device for the purpose of: causing the playback of the media content to be paused at the second client device, and causing the current playback head position of the media content at the second client device to be updated to the specific playback head position.

[0137] Exemplary Clause J: A system comprising: at least one processor; and at least one memory in communication with the at least one processor, the at least one memory having computer-readable instructions stored thereon, the computer-readable instructions, when executed by the at least one processor, causing the at least one processor to: transmit multiple bidirectional user streams between multiple client devices to facilitate a communication session; receive user playback instructions from a particular client device during the communication session to initiate playback of hosted media content at the multiple client devices via multiple media streams different from the multiple bidirectional user streams; determine delay data, the delay data defining multiple delay values ​​associated with transmitting information to individual client devices among the multiple client devices during the communication session; determine individual delay delays for the individual client devices based on portions of the delay data corresponding to other client devices of the multiple client devices; and transmit multiple delayed playback instructions so that the playback of the hosted media content begins across the multiple client devices, wherein the individual delayed playback instructions include corresponding individual delay delays in the individual delay delays so that at least some of the individual client devices postpone initiating the playback of the media content after receiving the individual delayed playback instructions.

[0138] Exemplary Clause K: The system of Exemplary Clause J, wherein the individual delayed play instructions are a play head position indicated by the user play instructions.

[0139] Exemplary clause L: A system as described in any of Exemplary clauses J to K, wherein the computer-readable instructions further cause the at least one processor to: in response to receiving an asynchronous playback mode command from a master client device, enable individual playhead positions for the playback of the media content to be independently adjusted at the individual client devices.

[0140] Exemplary clause M: A system as described in any of Exemplary clauses J to L, wherein the computer-readable instructions further cause the at least one processor to: in response to receiving a synchronized playback mode command from a master client device, cause the playback of the media content to be updated across each of the multiple client devices to a specific playhead position associated with the master client device.

[0141] Exemplary Clause N: A system as described in any of Exemplary Clauses J to M, wherein determining the delay data comprises: transmitting a plurality of pings to the plurality of client devices during the communication session; and calculating the plurality of delay values ​​based on round-trip times associated with individual pings in the plurality of pings.

[0142] Exemplary clause O: A system as described in any of Exemplary clauses J to N, wherein the computer-readable instructions further cause the at least one processor to: receive a transfer control request from a first client device corresponding to a master control state, the transfer control request identifying a second client device corresponding to a slave control state; and in response to the transfer control request, update the licensing data to grant the master control state to the second client device and grant the slave control state to the first client device.

[0143] Exemplary Clause P: A system as described in any of Exemplary Clauses J to O, wherein the individual delayed playback instructions cause at least some of the individual client devices to begin caching the hosted media content from one or more media streams independently of being received by the bidirectional user stream.

[0144] A system as described in any one of exemplary clauses Q and exemplary clauses J to P, wherein the computer-readable instructions further cause the at least one processor to: receive a pause instruction from the specific client device indicating that the playback of the media content has been paused at a specific playhead position at the specific client device; and in response to the pause instruction, cause the playhead position to be updated to the specific playhead position across the multiple client devices and paused at the specific playhead position.

[0145] Exemplary clause R: A system comprising: a unit for identifying media content to be played at multiple client devices operating in association with a communication session; a unit for determining delay data, the delay data defining multiple delay values ​​associated with transmitting information to the multiple client devices during the communication session; a unit for receiving play instructions, the play instructions being generated to initiate playing of the media content at the multiple client devices; and a unit for transmitting multiple delayed play instructions so that the playback of the media content begins synchronously across the multiple client devices, wherein at least some of the individual client devices are caused to delay initiating the playback of the media content after receiving at least one corresponding delayed play instruction.

[0146] Exemplary Clause S: ​​The system described in any one of Exemplary Clauses J to L further includes: a unit for receiving a transfer control request from a first client device corresponding to a master control state, wherein the transfer control request identifies a second client device corresponding to a slave control state; and a unit for updating licensing data to provide the master control state to the second client device and to provide the slave control state to the first client device.

[0147] Exemplary clause T: The system of any of Exemplary clauses J to S, comprising: means for switching a playback mode associated with the playback of the media data between an asynchronous playback mode and a synchronous playback mode.

[0148] in conclusion

[0149] Finally, although various configurations have been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the attached representations is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed subject matter.

Claims

1. A computer-implemented method, include: transmitting user stream data between a plurality of client devices operable to facilitate a communication session, wherein instances of the user stream data provide real-time participation of users in the communication session; receiving, from a particular client device among the plurality of client devices, a sharing instruction for sharing media data, the media data defining media content to be played back across the plurality of client devices during the real-time participation in the communication session; obtaining, via the communication session and based on the sharing instructions, the media data defining the media content to be played back across the plurality of client devices during the real-time participation in the communication session, the media data being distinct from the user stream data; determining delay data defining a delay value associated with transmitting a signal to each of the plurality of client devices operative to facilitate the communication session, wherein the delay value associated with each respective client device is determined based on evaluating a connection condition to each client device during the real-time participation in the communication session; receiving a play instruction from the particular client device to initiate playback of the media content across the plurality of client devices; in response to receiving the play instruction from the particular client device, causing playback of the media content to commence across the plurality of client devices based on the delay value associated with each respective client device with a different built-in delay time delay controlled by a delayed play instruction provided to each of the plurality of client devices, wherein the playback of the media content across the plurality of client devices is effectuated during the real-time participation in the communication session; enabling asynchronous consumption of the media content by the plurality of client devices, during the asynchronous consumption, allowing independent adjustment of an individual playhead position of the playback of the media content at each of the plurality of client devices during the real-time participation in the communication of the user streaming data between the plurality of client devices operative to facilitate the communication session; receiving a synchronized playback mode command from the particular client device, the synchronized playback mode command requiring synchronized consumption of the media content based on a current playhead position of the particular client device; and In response to receiving the synchronized playback mode command from the specific client device, the multiple client devices are switched from asynchronous consumption of the media content to synchronous consumption of the media content by updating the playback of the media content across the multiple client devices to the current playhead position of the specific client device.

2. The computer-implemented method of claim 1, in, A first delay for which the first client device defers initiating the playback of the media content is determined based at least in part on the delay value associated with the second client device.

3. The computer-implemented method of claim 1, in, The determining the delay data comprises: transmitting a plurality of pings to the plurality of client devices during the communication session; and A plurality of delay values ​​are calculated based at least in part on a plurality of round trip times associated with the plurality of pings.

4. The computer-implemented method of claim 1, in, At least some of the individual delayed play instructions define individual delayed delays, which cause at least some of the multiple client devices to postpone initiating the playback of the media data until each of the multiple client devices receives a corresponding delayed play instruction.

5. A system, include: at least one processor; as well as at least one memory in communication with the at least one processor, the at least one memory having computer readable instructions stored thereon, the computer readable instructions, when executed by the at least one processor, causing the at least one processor to: transmitting a plurality of user streams between a plurality of client devices to facilitate a communication session, wherein the individual user streams provide real-time participation of the users in the communication session; receiving a sharing instruction from a first client device of the plurality of client devices to share media content across the plurality of client devices during the real-time participation in the communication session; assigning a master control state for the media content to the first client device based on the sharing instruction, wherein the master control state enables control of a playback position of the media content for each of the plurality of client devices; receiving, during the real-time participation in the communication session and based on the master control state, from the first client device a user play instruction to initiate playback of the media content across the plurality of client devices via a plurality of media streams distinct from the plurality of user streams; determining delay data defining a plurality of delay values ​​associated with transmitting information to individual client devices of the plurality of client devices during the real-time participation in the communication session; determining an individual latency delay for the individual client device based on portions of the latency data corresponding to other client devices in the plurality of client devices; transmitting a plurality of delayed play instructions to cause the playback of the media content to commence across the plurality of client devices, wherein individual delayed play instructions include corresponding individual delay delays in the individual delay delays to cause at least some of the individual client devices to defer initiating the playback of the media content after receiving the individual delayed play instructions; receiving, from the first client device, a transfer control request identifying a second client device; and In response to receiving the transfer control request, the master control state for the media content is removed from the first client device and the master control state for the media content is granted to the second client device identified via the transfer control request.

6. The system according to claim 5, in, The individual delayed play instructions include an initial play head position indicated by the user play instructions.

7. The system according to claim 5, in, Determining the delay data includes: transmitting a plurality of pings to the plurality of client devices during the communication session; and The plurality of delay values ​​are calculated based on round trip times associated with individual pings of the plurality of pings.

8. The system according to claim 5, in, The computer readable instructions further cause the at least one processor to: receiving a pause instruction from the first client device, the pause instruction indicating a particular playhead position at which the playback of the media content has been paused at the first client device; as well as In response to receiving the pause instruction, the current playhead position is updated to the specific playhead position across the plurality of client devices and paused at the specific playhead position across the plurality of client devices.

9. A system, include: means for transmitting user stream data between a plurality of client devices operable to facilitate a communication session, wherein instances of the user stream data provide real-time participation of a user in the communication session; for receiving, from a particular client device among the plurality of client devices, sharing instructions for sharing media data, the media data defining media content to be played back across the plurality of client devices during the real-time participation in the communication session; means for obtaining, via the communication session and based on the sharing instructions, the media data defining the media content to be played back across the plurality of client devices during the real-time participation in the communication session, the media data being distinct from the user stream data; means for determining delay data defining a plurality of delay values ​​associated with transmitting a signal to each of the plurality of client devices operative to facilitate the communication session, wherein the delay value associated with each respective client device is determined based on evaluating a connection condition to each client device during the real-time participation in the communication session; means for receiving a play instruction from the particular client device, the play instruction being for initiating playback of the media content across the plurality of client devices; means for causing, in response to receiving the play instruction from the particular client device, the playback of the media content to commence across the plurality of client devices with different built-in delays controlled by a delayed play instruction provided to each of the plurality of client devices based on the delay value associated with each respective client device, wherein the playback of the media content across the plurality of client devices is effectuated during the real-time participation in the communication session; means for enabling asynchronous consumption of the media content by the plurality of client devices, during which, during the real-time participation in the communication of the user streaming data between the plurality of client devices operative to facilitate the communication session, allowing independent adjustment of an individual playhead position of the playback of the media content at each of the plurality of client devices; means for receiving a synchronized playback mode command from the particular client device, the synchronized playback mode command requesting synchronized consumption of the media content based on a current playhead position of the particular client device; and A unit for switching the multiple client devices from asynchronous consumption of the media content to synchronous consumption of the media content by updating the playback of the media content across the multiple client devices to the current play head position of the specific client device in response to receiving the synchronous playback mode command from the specific client device.

10. The system according to claim 9, further comprising: include: means for receiving a transfer control request from a first client device corresponding to a master control state, wherein the transfer control request identifies a second client device corresponding to a slave control state; and Means for updating licensing data to provide the master control state to the second client device and the slave control state to the first client device.