Simulating local experiences by live streaming a shareable perspective of a live event

The platform allows for immersive and interactive sharing of live events by enabling multiple perspectives and augmented reality, addressing the limitations of existing technologies in providing a shared experience among remotely located individuals.

JP7594642B2Active Publication Date: 2024-12-04STATS LLC
View PDF 11 Cites 0 Cited by

Patent Information

Application Number
JP2023144058
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-02-06
Filing Date
2023-09-06
Publication Date
2024-12-04
Estimated Expiration
2039-02-06

AI Technical Summary

Technical Problem

Existing technologies limit the shared experience of live events to a single predefined perspective, preventing multiple remotely located individuals from interacting and experiencing the event in an immersive manner as if they were attending together.

Method used

A platform that enables participants to share a simulated experience of a live event through multiple perspectives, incorporating augmented reality content, and allows for virtual interaction between participants using a server computer system and client devices, including head-mounted displays, to dynamically update and synchronize live streams.

Benefits of technology

Enables immersive and interactive sharing of live events among remotely located individuals, enhancing the experience with augmented reality and ensuring synchronized, seamless transitions between perspectives.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007594642000003
    Figure 0007594642000003
  • Figure 0007594642000004
    Figure 0007594642000004
  • Figure 0007594642000005
    Figure 0007594642000005
Patent Text Reader

Abstract

To provide a system that enables a shared experience of a live event between a user located remotely from a live event and a participant located at the event, and enhances the shared experience with an augmented reality content.SOLUTION: In a system 700, a platform server 702 performs capturing by a plurality of source devices 704 located at a live event, displays a streaming viewpoint on a client device 706 located remotely from the live event, and dynamically update a subscription list in response to live streaming stop. The remotely located client device can select a viewpoint by a subscription list, and an access to the deleted source device 704 is disabled due to the streaming stop.SELECTED DRAWING: Figure 7
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The disclosed teachings relate to a platform that enables sharing the experience of attending a live event and enables interaction between participants. In particular, the disclosed teachings enable a shared experience of a live event between a user located remotely from the live event and an attendee located at the event, and can enhance the shared experience with augmented reality content. [Background technology]

[0002] Attending a live event, such as a sports game, a seminar, a concert, or a demonstration with friends and family, is an important social activity for many people. Existing technologies allow individuals located remotely from the live event and from each other to experience a limited shared experience of the live event. For example, an individual located remotely from the live event can watch it on a television. The remotely located individual experiences a live broadcast of the event rendered on the user's television. In another example, a spectator located at the live event can share her experience with a remotely located individual by using a mobile phone to capture images and audio of the live event as an AV signal that is streamed to a personal device capable of rendering the AV signal. Thus, the remotely located individual can share the spectator's experience of the live event. In other words, this individual is a spectator of the spectator's experience. That is, the "sharing" of the experience is only one-way.

[0003] Furthermore, sharing live content acquired and communicated by mobile phones does not allow a remotely located individual to have an immersive experience as if the individual were present at the live event. Instead, the individual can only share the experience of the spectators because the individual has no control over the experience that the spectators are providing. Similarly, remotely located individuals can only share the experience of one or more spectators, but the spectators cannot share the experience of remotely located individuals, and the experience of the same live event cannot be shared among multiple remotely located users and / or spectators. Thus, existing technologies do not allow multiple remotely located individuals and multiple local spectators to share a live event in an immersive manner as if they were participating in the live event together. Summary of the Invention

[0004] The disclosed embodiments include at least one method that includes communicatively coupling a first client device to a platform that delivers a live stream over a network to client devices, including the first client device, the live stream providing a live broadcast of the live event from a perspective captured by a source device located at the live event, including a second client device. The method includes transmitting a selection of the first perspective from the first client device. sign and communicating the first live stream to the first client device, the first live stream being captured by the first source device and, when rendered by the first client device, causing the first perspective to be displayed on the first client device.

[0005] The disclosed embodiment also includes at least one system including a server computer including a processor and a memory. The memory includes instructions that, when executed by the processor, interconnect client devices to the server computer over a network. At least some of the client devices are located at the live event and at least some of the client devices are located remotely from the live event. Further, each client device located at the live event is a source of a live stream that, when rendered by the device, displays a perspective of the live event. The server computer system can host a subscription service that dynamically updates a subscription list to include any of the client devices that are live streaming their perspective of the live event and removes client devices that have stopped live streaming their perspective of the live event, and allows selective access by any client device to any client device on the subscription list and disables access to client devices that have been removed from the subscription list.

[0006] The disclosed embodiment also includes at least one device. An example of the device is a head mounted display (HMD) device that includes a display device, an orientation sensor, a processor, and a memory. The memory includes instructions that, when executed by the processor, cause the HMD device to render a live stream to display a perspective of the live event on the display device. The live stream originates at a source device located at the live event, and the source device may include a stationary device at a fixed location of the live event and a handheld mobile device. The HMD device further causes the HMD device to determine a change in orientation of the HMD device based on the output of the orientation sensor, and dynamically track the change in orientation of the HMD device to seamlessly transition from displaying a perspective of the live stream on the stationary device to displaying a different perspective of another live stream on the handheld mobile device.

[0007] This Summary is provided to introduce a selection of concepts in a simplified form that are described in more detail below in the Detailed Description. This Summary is not intended to identify key features or essential features of the disclosed subject matter, nor is it intended to be used to limit the scope of the disclosed subject matter. Other aspects of the disclosed subject matter will become apparent from the accompanying figures and detailed description. [Brief description of the drawings]

[0008] [Figure 1] 1 illustrates a view of a live event displayed on a user interface by a client device according to some embodiments of the present disclosure. [Diagram 2] 1 illustrates a live video chat window displayed from the perspective of a live event, according to some embodiments of the present disclosure. [Diagram 3] 1 illustrates an alternative view of a live event by a client device in accordance with some embodiments of the present disclosure. [Figure 4] 1 illustrates augmented reality (AR) content integrated into a viewpoint, according to some embodiments of the present disclosure. [Diagram 5] A figure showing another example of AR content integrated into a viewpoint, according to some embodiments of the present disclosure. [Figure 6] A diagram showing different perspectives of a live event that is displayed simultaneously on a client device and includes AR content, according to some embodiments of the present disclosure. [Figure 7] FIG. 1 is a block diagram of a system implementing a platform for sharing multiple perspectives of a live event according to some embodiments of the present disclosure. [Figure 8] A diagram illustrating the relationship between users and sources of a live stream of a live event according to some embodiments of the present disclosure. [Figure 9]1 illustrates a clock synchronization process between a client and a server for synchronizing live streams of a live event according to some embodiments of the present disclosure. [Figure 10] FIG. 2 illustrates a live stream composed of packets grouped into scenes, according to some embodiments of the present disclosure. [Figure 11] 1 is a flowchart illustrating a process for subscribing to live stream content according to some embodiments of the present disclosure. [Figure 12] FIG. 1 is a block diagram of a computing device operable to implement aspects of the disclosed technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The following embodiments represent the necessary information to enable those skilled in the art to practice the embodiments and show the best mode for practicing the embodiments. Upon reading the following description in light of the accompanying figures, those skilled in the art will understand the concepts of the present disclosure and recognize applications of these concepts not specifically addressed herein. It is understood that these concepts and applications are within the scope of the present disclosure and the appended claims.

[0010] (Definitions of some terms) The terminology used herein is for the purpose of describing the embodiments only and is not intended to limit the scope of the disclosure. Where the context permits, words using the singular or plural form may also include the plural or singular form respectively.

[0011] As used herein, unless otherwise indicated, the terms "processing," "computing," "calculating," "determining," "displaying," "generating," and the like refer to the actions and processes of an electronic computing device that manipulates and transforms data represented as physical (electronic) quantities in the computer's memory or registers into other data that is similarly represented as physical quantities in the computer's memory, registers, or other such storage medium, transmission or display device.

[0012] As used herein, the terms "connected," "coupled," or variations thereof refer to any direct or indirect connection or coupling between two or more elements. The coupling or connection between the elements can be physical, logical, or a combination thereof.

[0013] As used herein, the term "live" or variations thereof can refer to something occurring in real time or near real time. For example, a live event can refer to an event that is currently in progress. A live broadcast or transmission can refer to various types of media that are broadcast without a human-perceivable delay. A live stream can refer to a form of online streaming of media that is simultaneously captured and broadcast in real time or near real time and that can be viewed by an audience. That is, a live stream refers to real-time or near-real-time streaming. Thus, live streaming can provide low latency delivery of content with human-perceivable interactivity.

[0014] As used herein, a "viewpoint" can refer to a view of a live event from a vantage point at a given location in the live event and provided by a particular source device. For example, a viewpoint can be the viewpoint of a camera located in the northeast corner of the field.

[0015] As used herein, "augmented reality" (AR), in the context of some implementations, can refer to virtual objects (e.g., generated virtually or with virtual controls) that are overlaid on a viewpoint on a real-world scene.

[0016] (overview) Attending live events such as sports games, talks, concerts, demonstrations, etc. with friends and family is an important social activity for many people. The expansion of network connectivity increasingly enables groups of people to participate in live events despite being physically distant from each other (e.g. even in different countries and continents). However, today, sharing the experience of a live event is mostly restricted to a single predefined viewpoint of the producer, which is broadcast live to all viewers. The perspective can be changed, but this change is controlled by the producer, who can then choose the appropriate viewpoint to suit the needs of the audience. sign The video is then broadcast to the viewer and then adopted by all viewers. Thus, much of what makes a shared experience memorable, such as the direct and indirect interactions and side conversations with other audience members, is lost.

[0017] The disclosed embodiments and associated algorithms address these shortcomings by participating in a live event and enabling a shared experience. The disclosed embodiments include a platform and associated systems, devices, and methods that enable participants to share a simulated experience of a real-world live event. The shared experience can include a simulation of local interactions at the live event. This allows for interactions between participants who are participating in the live event or are remotely located from it. For example, a remotely located participant can adopt the perspective of a local attendee. In some embodiments, the simulated environment incorporates live augmented reality (AR) to enhance the simulated experience. The AR features can be determined based on the viewer's preferences.

[0018] This disclosure includes sufficient details to ensure that a person skilled in the art can understand the embodiments described herein. The disclosed embodiments may be implemented without specific details or with equivalent configurations. In other examples, well-known structures and devices are represented as block diagrams to avoid unnecessarily obscuring the embodiments of the present invention. Thus, this description does not focus on the details of computing devices, sensors, and networks, including cameras, located around a venue or attached to participants of a live event, such as players in a sporting event or actors in a theater performance, or the details for generating AR content associated with an event. These aspects would be understood by those skilled in the art. Thus, these aspects are omitted here for the sake of brevity. Instead, this disclosure focuses on protocols for simulating interactions and the timely synchronized delivery of multiple streams associated with various viewpoints and sources of content.

[0019] The disclosed embodiments include a system that implements a platform for providing a simulated experience of a live event that can be shared between individuals located remotely from the live event and local attendees of the live event (collectively referred to as "participants"). The platform allows participants to virtually interact with the live event as if they were at the live event. In some embodiments, the platform captures several live streams from source devices including cameras and microphones strategically distributed at fixed locations in the live event venue, and mobile devices operated by participants attending the live event and / or individuals performing as part of the live event.

[0020] FIG. 1 shows a perspective 100 of a live event displayed by a client device ("first client device") on a user interface (UI) managed by a platform to simulate local interaction sharing the experience of the live event with other participants. In the illustrated embodiment, the live event 102 is a sports game in a stadium. The UI renders the live stream to display the perspective 100 of a source device located at the live event. The perspective 100 reflects the location of the source devices 106 at the live event and their relative positions to each other. sign Contains graphic icons that

[0021] Source devices 106 broadcast different perspectives of the same live event with their respective live streams. Examples of source devices 106 include mobile devices 106-1 through 106-5 operated by participants located at the live event, and stationary devices 106-6 and 106-7. Stationary devices 106-6 and 106-7 record the location and orientation of cameras (and microphones) that are located at fixed positions in the live event. sign The cameras are represented by camera icons that display the stadium's stadium view. In the illustrated example, the cameras are located in the stadium ceiling and near the stadium seating. Examples of mobile devices 106-1 through 106-5 include client devices such as handheld mobile devices (e.g., smartphones, tablet computers) operated by attendees or devices attached to individuals playing at the live event (e.g., devices attached to a player's helmet / clothing).

[0022] Render viewpoint 100, by user icon signThe first client device being viewed is operated by a user 108. The first client device can be located locally or remotely from the live event. Thus, if the first client device is located remotely from the live event 102, the first client device simulates a local experience of the live event. If the first client device is located at a first location of the live event 102 and the viewpoint 100 is from a second location of the live event, the viewpoint 100 simulates being at the second location.

[0023] The platform can host the live stream of the source device 106 as a selectable viewpoint of the live event 102 available to the client devices. To simulate the viewpoint 100, a first client device is communicatively coupled to the platform, which delivers the live stream collected over a network from the source device 106. The live stream provides a live broadcast of the live event 102 from the viewpoint captured by the source device 106. In operation, the platform enables the first client device to view a selection of the viewpoint 100. sign and in response, communicates a live stream of viewpoint 100 to the first client device. Thus, the live stream, as captured by the source device and rendered by the first client device, causes viewpoint 100 to be displayed on the first client device.

[0024] The platform can host a virtual interaction between a first client device (located at the live event 102 or remotely located therefrom) and a second client device, a source device. The source device, a client device, publishes a live stream and allows users to interact live with the audience via text, audio, emojis, etc., or indirectly via other streams. For example, a publisher broadcasting a view of a live event and other participants not necessarily at the event can comment on the publisher's actions or those of the broadcast (e.g., in response to the publisher's votes).

[0025] 2 illustrates a live video chat window 202 displayed on a viewpoint 200 of a live event 102 according to some embodiments of the present disclosure. Specifically, viewpoint 200 is the same as viewpoint 100 with respect to live event 102, but incorporates live video chat window 202. This live video chat window 202 is an example of an interaction with a simulated user 108 by another participant of live event 102. Specifically, user 108 interacts with a user of source device 106-1 via live video chat window 202 displayed on the viewpoint of source device 106-1 (i.e., viewpoint 100) to generate viewpoint 200.

[0026] In another example, a third client device can be communicatively coupled to the platform over a network. The first client device and the third client device can be located remotely from the live event and from each other. In this manner, the platform can host a virtual interaction that simulates a local interaction between the first client device and the third client device even though they are located remotely from the live event 102. In some implementations, the first client device and another client device can share respective perspectives of the live event 102. Thus, when the first client device is located at the location of the live event 102 and is a source device, the perspective provided by the first client device to another client device is different from that from the perspective 100.

[0027] In another example, a group of users located remotely from each other and / or from the live event can simultaneously watch the live event and dynamically adopt (or abandon) each other's perspectives while participating in the live event. For example, FIG. 3 illustrates the adoption of a perspective 300 of the live event 102 by a first client device according to some embodiments of the present disclosure. The perspective 300 is adopted from the source device 106-2. Thus, the first client device of the user 108 is rendering the live stream from the source device 106-2 to adopt the perspective 300 from the perspective of the participant operating the source device 106-2.

[0028] Thus, the virtual interaction closely simulates the local interaction of the live event 102. The immediacy of the interaction (e.g., feedback, comments) and the synchronization between the live streams ensure frictionless interaction and a high level of engagement by the participants. Quantitatively speaking, the human perception limit for a one-way audio delay of less than 100 ms is considered indistinguishable from perfectly synchronized video, while an audio delay of about 150 ms is considered the minimum acceptable level for verbal communication. The embodiments disclosed herein implement these considerations to enable synchronized consumption of multi-source or single-source digital data among a group of users.

[0029] (Augmented reality perspective) The platform allows participants to experience a perspective of a live event in which augmented reality (AR) content is incorporated to generate an "AR perspective." Any person, including a third party, can provide this AR content to enhance the shared experience. For example, the platform may support, from a first client device, a selection of AR content. sign This can be received. sign In response, the platform can augment the live stream with AR content, by the first client device overlaying the AR content onto a perspective of the live stream when rendered by the first client device. Thus, an unmodified perspective of the live stream can be enhanced with the AR content. The platform can also automatically augment the live stream with AR content that, when rendered by the first client device, displays the AR perspective.

[0030] The AR content can be live AR content or pre-recorded AR content associated with the live event. Examples of this AR content include live commentary, player statistics, content curated from other participants, advertisements, information about the live event or venue, location-based information, or a combination thereof. For example, participants can subscribe to AR content delivered by the platform as AR content created by a third party and embedded in the live stream. In this way, the live event experienced by the participants can be enriched with AR content.

[0031] 4 illustrates a perspective 400 of a rendered live stream incorporating an example of AR content that is displayed to enhance a simulated experience of a live event, according to some embodiments of the present disclosure. Specifically, the perspective 400 is the same as the perspective 100 with respect to the live event 102, but incorporates AR content 402. In the illustrated example, the user 108 is presented with the AR content 402, rendered as a virtual seller offering pizza for purchase at the perspective 400. The virtual seller is not present in the live event 102 and is therefore not included in the original unmodified live stream providing the perspective 100. Nevertheless, the seller appears in the perspective 400 as if an interaction with the virtual seller were actually occurring at the live event 102. Another participant adopting the same perspective 100 may see the same or different AR content, depending on the source of the AR content, the participant's preferences, etc.

[0032] In another example, AR content may be automatically or selectively rendered to provide information about an athlete of interest, a team of interest, a tournament, or field conditions. For example, a displayed viewpoint could be augmented with information about whether the field is wet or whether the wind across the field is calm. FIG. 5 illustrates another example of AR content incorporated into a viewpoint, according to some embodiments of the present disclosure. As shown, the AR viewpoint 500 includes AR content 502 that is displayed to enhance the viewpoint 100. The AR content includes information about an athlete associated with the source device 106-5 and is overlaid on the viewpoint 100 of the live event 102 to render the AR viewpoint 500.

[0033] In some implementations, the platform can enable a client device to simultaneously view multiple perspectives containing the same AR content. For example, FIG. 6 shows different perspectives 600-1 and 600-2 of the same live event 102 displayed simultaneously on the UI of a first client device. Each of the perspectives 600-1 and 600-2 contains the same AR content 602 from a different perspective. Thus, the platform can combine multiple perspectives containing the same or different AR content provided, for example, by a third party.

[0034] (System Environment) 7 is a block diagram of a system implementing a platform for sharing multiple perspectives of a live event according to some embodiments of the disclosure. The system 700 includes one or more server computers 702, dedicated source devices 704 (e.g., cameras), and client devices 706, interconnected via one or more networks 708, such as the Internet. As shown, the source devices 704 communicate live streams unidirectionally toward the server 702. In contrast, the client devices 706 can communicate bidirectionally with each other via the server 702. That is, at least some of the client devices 706 can publish and consume live streams.

[0035] In operation, at least some of the client devices 706 are located at the live event and some are located remotely from the live event. Each client device 706 located at the live event is a source of a live stream that, when rendered by the client device, displays a perspective of the live event. The server 702 can dynamically update the subscription list to include the source device 704 and any client devices 706 that are currently live streaming their perspective of the live event and to remove devices that have stopped live streaming their perspective of the live event. The server 702 can host a subscription service that allows selective access by any client device to any client device on the subscription list and disables access to any client device that has been removed from the subscription list.

[0036] A client device located remotely from the live event provides a virtual presence of the live event, while a client device located at the live event has both a physical and virtual presence of the live event. When present only virtually, participants could have one or more devices available to participate in the shared experience. For example, a virtual participant could experience the live event on a smartphone while on the go, while at home using a tablet computer. Participants could seamlessly transition from one device to another to maintain a continuous experience of the live event.

[0037] As participants use different devices, the platform can seamlessly update the subscribed participants' live streams and adjust the rendered environment by replacing or augmenting the participants' views. In some implementations, the platform can dynamically update other subscription lists, such as, for example, a list of client devices that were previously sources of live streams of a live event. In some implementations, the platform can enable a search operation in response to a query, for example, to search for client devices that are currently live streaming a perspective of the live event.

[0038] Network 708 may include any combination of private, public, wired, or wireless portions. Any control information or data communicated over network 708 may be encrypted or unencrypted at or along different portions of network 708. Each component of system 700 may include a combination of hardware and / or software to process data, perform functions, communicate over network 708, etc. For example, any component of system 700 may include a processor, memory or storage, a network transceiver, a display, an operating system, and application software (e.g., for providing a UI), etc. Other hardware or software components of system 700 known to those skilled in the art are not shown or described herein for the sake of brevity.

[0039] The source device 704 represents any electronic device that generates a live stream of a live event. That is, the source device 704 is the source of the live stream from different viewpoints of the live event. The source device 704 and at least some of the client devices 706 that can operate as source devices capture images and / or audio of the live event as media that is streamed to a server 702 that synchronizes the live stream that can be shared among the client devices 706. Thus, consider a live event (such as a sports game) in which multiple participants of the platform participate. These participants can interact with each other in the same way, regardless of whether they are local attendees or virtual participants.

[0040] Each source of live streams communicated to the server 702 provides a viewpoint that can be active or inactive. For example, an inactive viewpoint can correspond to a communication channel that is not currently available for live streaming. In contrast, all active viewpoints have one or more live streams that are immediately available to the client device 706. For example, a first client device providing a first live stream can adopt the perspective of a second client device by accessing a corresponding second live stream. A request from a participant to adopt a particular viewpoint can be fulfilled by any source associated with that viewpoint.

[0041] The viewpoints available to the client devices 706 include, for example, media captured by the cameras and microphones of the client devices 706. By sharing viewpoints, at least some of the client devices 706 participate in an active and unique perspective from a viewpoint of the live event that can be shared with other client devices. In addition to viewpoints provided by other client devices and cameras or microphones located at venue locations, another source of content for the live stream includes devices used by actors of the live event, such as players at a sporting event (see, for example, FIGS. 1-3). For example, at a venue for a live event, source devices can be positioned in various locations and orientations to capture media for live streaming by the platform. The platform can collect various live streams over the network 708 from available cameras at each client device or fixed locations at the venue.

[0042] The client devices 706 are shown as handheld mobile devices used by participants to interact with the system 700. Although only three client devices 706-1 to 706-3 are shown in FIG. 7, the disclosed embodiment is not so limited and may include any number of handheld mobile devices. Examples of client devices include feature phones, smart phones (such as APPLE IPHONE, SAMUSUNG GALAXY, NOKIA LUMINA, etc.), tablet computers (such as APPLE IPAD, SAMUSUNG NOTE, AMAZON FIRE, MICROSOFT SURFACE, etc.), and other devices that can access the server 702 via the network 708.

[0043] As described above, the platform can be hosted by a server 702. The server 702 can include any number of server computers that facilitate interaction between client devices 706 to obtain a live stream of the captured live event from a source device 704 or other client devices. In some implementations, the server 702 manages the platform as a service accessible by the client devices 706. For example, a mobile app managed by the server 702 can provide a UI that is displayed on the client devices 706. The UI can be used to access the live stream of the live event.

[0044] Thus, the platform facilitates sharing of a live event between participants by enabling virtual interactions between the client devices 706 participants. For example, the client devices 706 can communicate with each other via the platform by sharing content from a particular viewpoint of the live event. The client devices can also adopt the viewpoint of another participant. Thus, the client devices 706 can share information and perspectives that enable a virtual shared experience of the same live event even though the participants are not physically located together. Additionally, the shared experience can be augmented with third-party AR content to create an augmented reality environment for an enriched experience as described above.

[0045] The platform can define audience groups, which are sets of users who subscribe to the same extended shared experience of a given live event, such as a group watching or participating in a play of a game at a given time. The composition of audience groups can change dynamically as users join or leave a shared space. In some implementations, the platform can restrict access to certain live streaming content or third-party content. For example, the platform can define audience groups with limited access to private live streams and public live streams with unlimited access. In some implementations, the platform can set access levels for each live stream. A private level can allow limited access to only some client devices. A public level can allow unlimited access to any client device. Access to a viewpoint can be restricted to a list of individuals, such as close friends in some third-party social networks where the publisher provides a public interface.

[0046] Client devices 706 can subscribe to live streams of viewpoints that are active and available as a service of the platform. In some implementations, the subscription service can include a list of client devices that are currently live streaming a particular live event, or a list of participants that live stream the event on a regular basis. Thus, audience members of a live event can easily search for other audience members of the live event and subscribe to their viewpoints. In some implementations, audience members or their devices can subscribe to other client devices that attend the live event and allow them to subscribe to their viewpoints. For example, the platform can provide a list of audience members that frequently attend concerts. Prospective attendees can request a subscription to any live streams that these audience members offer. Thus, as audience members offer live streaming of the event, subscribed participants can be modified to attend the live event.

[0047] Thus, a participant virtually present at a live event, but located remotely from the event, is associated with at least one viewpoint of a source device corresponding to either another client device, which may be mobile, or a source device at a fixed location at the live event, and therefore necessarily adopts that viewpoint. Any local participant at a live event may also adopt the perspective of at least one viewpoint provided by another spectator or source device at a fixed location at the live event. Furthermore, participants physically or virtually present at a live event may interact in real time or near real time with other participants physically or virtually present at the live event, and enhance their own experience by adopting or changing to other viewpoints.

[0048] (Dialogue, perspective change, and synchronization) As described above, the disclosed embodiments enable virtual interactions between participants of a live streamed event, as if the participants were together at the live event and interacting naturally. To achieve this, the participants' virtual interactions (e.g., images and audio) must be synchronized. Thus, at least some live streams need to be synchronized with respect to a particular point in time of the live broadcast of the live event. In some embodiments, the platform can define a group of synchronized live streams that includes only synchronized live streams and excludes non-synchronized live streams (e.g., restricting access by a client device to only the group of synchronized live streams).

[0049] For example, consider a scenario in which two participants subscribe to a single viewpoint. The two participants may seek to discuss current activity occurring at a live event. The two participants may receive a live stream of the same live activity at different times. For example, the time it takes for the live stream to reach each client device from a common source may vary depending on the distance between the source device and the client device or on factors that cause latency. For example, a source device including a spectator-operated camera at a sporting event may provide a live transmission to a subscribed audience. The first participant may be a local spectator and the second participant may be remote from the live event. Assuming the same communication means, whose speed is limited by the speed of light, the arrival time to the second participant will be longer than the arrival time to the first participant. If different communication means with different latencies are used to communicate the respective live streams, the difference in arrival times will be further exacerbated. Thus, when multiple participants in different locations try to talk about a live activity, the shared experience is compromised if the participants receive the live stream at different times (i.e., because the live streams are not synchronized).

[0050] Similarly, for a participant to seamlessly change / adopt a different perspective, the live stream of the current perspective and the live stream of the new perspective need to be synchronized. For example, consider a participant who is currently observing live activity from a first perspective on one side of a venue and wants to adopt a different perspective or angle on the other side of the venue. If these perspectives are at different distances from the source and / or participant, the participant may experience a discontinuity in the live event when there is a change from the current perspective to the new perspective. For example, depending on whether the perspective is closer to the source or participant or both, and depending on the medium through which the live stream is communicated, the participant may miss brief activities or see brief activities twice. In order for the participant to experience both perspectives as simultaneous perspectives or angles of the same live event, the live stream associated with the perspective closer to the live event could be buffered and delayed for the period of time required for the live stream to reach the farther perspective. The delay should be set to the difference in distance between the two perspectives.

[0051] FIG. 8 illustrates the relationship between participants and sources of a live stream of a live event according to some embodiments of the present disclosure. A group of participants are subscribers to a live stream of a viewpoint of the live event. A participant can subscribe to multiple viewpoints simultaneously, and the platform can adapt to combine all of those views. A group of cameras and other source devices can provide live streams from various viewpoints of the same live event. However, only a subset of participants overlaps with the subset of sources that can be synchronized to enable natural interaction between participants for any viewpoint of these sources. This subset includes the viewer group and the available viewpoints that can be treated as a single synchronized group. All sources and corresponding live streams in a synchronized group have their clocks synchronized to ensure a coordinated experience that enables natural interaction, as described below. For a source to create a new viewpoint or add it to an existing one, the source must run a clock synchronization algorithm to derive the offset between the group clock and its own local clock.

[0052] (Group Clock Synchronization) FIG. 9 illustrates clock synchronization between a client device (C) and a time server (S) for rendering live streams from different sources according to some embodiments of the present disclosure. Specifically, the process of live streaming a live event may include a clock synchronization phase for multiple viewpoints to ensure a harmonious experience during interactions between participants and changes of viewpoint. During the clock synchronization phase, the client connects to a "time server" and requests its server time. After receiving the request, the server prepares a response that is appended to the server time (T). This time can be set based on an external source such as a UTC time source. After receiving the response that includes the time response, the client sets the local time to T+RTTe / 2. RTTe refers to the round trip time (RTT) of the communication during the epoch (e). The RTT is estimated as the minimum RTT of a series of exchanges between the client and the server during the previous epoch, which sets the length of the epoch to account for changes in network conditions. For example, this method, which is based at least in part on Christian's time synchronization algorithm, assumes that the RTT is divided evenly between the request and response communications.

[0053] At each new epoch, the viewpoint source broadcasts its previous RTTe to all other sources in a given synchronized group. To maintain synchronization, all viewpoint sources in a given synchronization group must maintain a relatively similar RTTe. If a source's new RTTe increases beyond a maximum value, the source must leave the synchronized group, but may remain a subscriber to the group. This maximum value may correspond to the minimum time that a human would be able to perceive a difference between live streams of the same live event. If the present embodiment does not require synchronization with an external source, depending on the application, other algorithms could be used instead with minimal modifications.

[0054] (Integration or change of perspective) 10 illustrates a live stream composed of packets grouped into scenes, according to some embodiments of the present disclosure. As shown, the live stream 1000 includes packets 1002 grouped into scenes 1004 of a live event. A client device can adopt a different perspective by rendering another live stream of another source device. The rendering of the other live stream is delayed until the start of the next scene of the live stream 1000. Thus, the client device can render the other live stream only at defined times for one or more scenes.

[0055] Specifically, the start of each scene is a discrete point in the live stream where the view can change (e.g., change viewpoint, add AR content). The length of the scene in terms of number of packets can be dynamically configured, for example, per participant. In some implementations, the length of the scene corresponds to the time it takes to subscribe to the live stream. When a participant selects a viewpoint or changes viewpoint, the client-side component initiates the process of subscribing to the live stream of the desired viewpoint without terminating the current viewpoint. If the subscription process is successful, the client's view changes to the new viewpoint at the start of the next scene after the change is initiated. This ensures a seamless experience to the viewpoint.

[0056] In some implementations, each scene is implemented as a data chunk (DataChunk) defined by a sequence of frames (Frame) associated with a consumption timestamp (ConsumptionTS). Each frame includes a frame ID (FramelD), a data timestamp (DataTS), and data (Data). The source of the live stream informs subscribed participants of the data chunks along with when to render the data chunks to the participants' viewers. signA server-side component that uses the URI can send metadata containing a usage timestamp, as shown in the following formula:

number

[0057] When a client device receives a data chunk, it can convert the server's usage timestamp to the client device's usage timestamp. If the client's usage timestamp is a time in the future compared to the current time of the client's clock, the client will wait to render (use) the associated data chunk until the local time reaches a time equal to or greater than the usage timestamp. If a data chunk arrives at the client device at a time different from or earlier than the usage timestamp, the client device can adjust the live stream to maintain synchronization or discard the data chunk.

[0058] The disclosed embodiments include a process for subscribing to content, such as AR content or a live stream. For example, FIG. 11 is a flow chart illustrating a process for subscribing to live stream content according to some embodiments of the present disclosure. In step 1102, content is selected by a participant. For example, the platform can cause a client device to display an option for the participant to select AR content.

[0059] In optional step 1104, the live stream is adapted based on participant preferences or contextual information. For example, a user can set a live commentary preference that the platform (or a third party) can use to automatically select and incorporate live AR content depending on the situation (e.g., when an athlete is in view). Thus, when available for the live event, live commentary can be automatically incorporated into the live stream rendered by the client device. In another example, the platform can determine participant preferences based on participant demographic information or profile information. For example, the platform can store a participant profile that includes preferences regarding live commentary. In another example, the platform includes a machine learning algorithm that automatically detects preferences based on interactions between participants or with the platform and makes appropriate recommendations for incorporating content into the live stream. In some implementations, the client device can automatically subscribe to a live stream and automatically unsubscribe from a first live stream. Thus, the participant's profile can be updated to reflect changes in subscriptions and content viewing.

[0060] In optional step 1104, content to be rendered on the client device can be selected based on contextual and / or temporal information of the live event. For example, a client-side component of the platform can receive content associated with the live event via a multicast channel. The multicast channel can be provisioned based on contextual information about the live event, participants' interests, participants' locations, participants' viewing devices, time of day, etc. In one example, a third-party provider provisions AR content to the platform, delivering content to participants of a particular age group and providing food and drink advertisements to adult viewers during halftime of a live football game, etc. Thus, the AR content can be tailored differently for participants with different tastes and demographics.

[0061] In step 1106, the platform can cause the selected content to be rendered on the client device in addition to the live stream at a point in time (e.g., a scene). When content is associated with a group of live stream sources that have different perspectives of the live event, clock synchronization is necessary to ensure that the content is included in the desired time interval of the live stream. In some implementations, a "wall clock" type of synchronization can be sufficient for some applications.

[0062] For example, an ARStream of ARContent embedded in one or more live streams may include content (e.g., images, text) as well as context. sign For example, the time for displaying AR content is signThe time window may include a specific point in time or window defined by a start time (ARTStart) and an end time (ARTEnd). In some implementations, the time window is defined by a number of scenes and specifies the period of time during which the AR content is set to be displayed to the viewer of the client device. sign The context in which the AR content (ARContent) is displayed sign An (ARContext) can contain one or more directives, such as three-dimensional (3D) coordinates (ARCoord) that the platform's client-side components can map to the viewer, or specific instructions, such as "floating", "bound to a billboard", etc. This is expressed in the following formula:

number

[0063] In this way, scenes of a live stream can be used as discrete points of time to change viewpoints or update the view with content. For example, scenes of a live event can provide natural points of interaction that allow changes to be made simultaneously, instead of instantly on demand, which can result in a high quality experience. Participants can request content or other parties can push content for the participants' viewers. Delivered content can include live streams with different viewpoints and AR content that is added to the live stream by a client-side component of the platform and rendered to the viewer at the start of the next scene after a subscription to the live stream is associated with the content.

[0064] The disclosed embodiments also include at least one device. An example of the device is a head mounted display (HMD) device, including a display, an orientation sensor, a processor, a memory, etc. The HMD device can render a live stream to cause the device to display a perspective of the live event. The live stream originates from a source device located at the live event. The HMD device can determine a change in orientation of the HMD device based on the output of the orientation sensor, and dynamically track the change in orientation of the HMD device to seamlessly transition from displaying a perspective of the live stream of one source device to displaying a different perspective of another live stream of another source device.

[0065] In this example, the provided content can be delivered to a virtual participant via an HMD device or in a virtual reality environment. For example, the platform can adapt the participant's perspective based on the participant's body position and gaze point (e.g., leveraging other devices around the participant). For example, when the participant turns around, the HMD display can seamlessly change the live stream source to a matching viewpoint. In some implementations, the client-side component of the platform will subscribe to all necessary viewpoints upon initialization so that it can provide an immersive environment. In some implementations, the participant can also manually change the viewpoint.

[0066] 12 is a block diagram of a computing device 1200 capable of implementing at least some aspects of the disclosed technology. Examples of computing devices include a platform server, a client device, a source device, or any combination or component thereof. The computing device 1200 can be a general-purpose computer or can be specially designed to perform the functions of the system. For example, the computing device 1200 can be a SOC, an SBC system, a desktop or laptop computer, a kiosk, a mainframe, a mesh of computer systems, a handheld mobile device, or a combination thereof.

[0067] Computing device 1200 may be a standalone device or part of a distributed system consisting of multiple networks, locations, machines, or combinations thereof. In some implementations, computing device 1200 operates as a server computer or a client device in a client-server network environment, or as a peer machine in a peer-to-peer system. In some implementations, computing device 1200 may execute one or more processes in real time, near real time, offline, batch processing, or combinations thereof.

[0068] As shown, computing device 1200 includes a bus 1202 operable to transfer data between hardware components. These components include a control 1204 (i.e., processing system), a network interface 1206, an input / output (I / O) system 1208, and a clock system 1210. Computing device 1200 may also include other components not shown or discussed in detail for brevity. Those skilled in the art will recognize any hardware and software included herein that is not shown in FIG. 12. Control 1204 includes one or more processors 1212 (e.g., CPU, ASIC, FPGA) and memory 1214 (which may include software 1216). Memory 1214 may be local, remote, or distributed, and may include, for example, volatile and / or non-volatile memory.

[0069] A software program (e.g., software 1216), when referred to as being "implemented in a computer-readable storage medium," includes computer-readable instructions stored in a memory (e.g., memory 1214). A processor (e.g., processor 1212) is "configured to execute the software program" when at least one value associated with the software program is stored in a register readable by the processor. In some embodiments, the routines executed to implement the disclosed embodiments may be implemented as part of operating system (OS) software (e.g., Microsoft Windows, Linux) or a particular software application, component, program, object, module, or sequence of instructions referred to as a "computer program."

[0070] Thus, a computer program typically comprises one or more instructions stored at various times in various memory devices of a computer (e.g., computing device 1200) and read and executed by at least one processor (e.g., processor 1212) that cause the computer to perform operations to perform functions including various aspects of the disclosed embodiments. In some embodiments, a carrier is provided that contains the aforementioned computer program product. The carrier is one of an electronic signal, an optical signal, a radio signal, or a non-transitory computer-readable storage medium (e.g., memory 1214).

[0071] Network interface 1206 may include a modem or other interface (not shown) for coupling computing device 1200 to another device over a network 1218. I / O system 1208 may operate to control various I / O devices, including peripheral devices, such as a display system 1220 (e.g., a monitor or touch-sensitive display) and one or more input devices 1222 (e.g., a keyboard and / or pointing device). Other I / O devices 1224 may include, for example, disk drives, printers, scanners, etc. Finally, clock system 1210 controls timers for use in accordance with the disclosed embodiments.

[0072] An operation of a memory device (e.g., memory 1214), such as a change of state from a binary 1 to a binary 0 (or vice versa), can comprise a perceptible physical transformation. The transformation can comprise a physical transformation of an article or thing to a different state. For example, the change of state can include the accumulation and storage of charge or the release of stored charge. Similarly, the change of state can comprise a physical change or transformation in magnetic orientation, or a physical change or transformation in molecular structure, such as from a crystalline structure to an amorphous structure, or vice versa.

[0073] Aspects of the disclosed embodiments can be described in terms of algorithms and symbolic representations of operations on data bits stored in a memory. These algorithmic descriptions and symbolic representations usually include a sequence of operations leading to a desired result. The operations require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Conventionally and conveniently, these signals are referred to as bits, values, elements, symbols, characters, terms, numbers, or the like. These and similar terms are merely associated with physical quantities and are convenient labels applied to these quantities.

[0074] Although the embodiments have been described in the context of a fully functional computer, those skilled in the art will appreciate that various embodiments are capable of being distributed as program products in a variety of forms, and that the present disclosure applies equally regardless of the particular type of machine or computer-readable medium used to actually effect the distribution.

[0075] While the present disclosure has been described with respect to several embodiments, those skilled in the art will recognize that the present disclosure is not limited to the embodiments described herein, but can be practiced with modification and alteration within the spirit and scope of the present invention. Those skilled in the art will also recognize improvements to the embodiments of the present disclosure. All such improvements are considered to be within the scope of the concepts disclosed herein. Thus, the description should be considered illustrative rather than limiting.

[0076] Although specific embodiments of the invention have been described herein for purposes of illustration, it will be appreciated from the above that various modifications can be made without departing from the scope of the invention. Accordingly, the invention is not limited except as by the appended claims.

[0077] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application Serial No. 6 / 627,126, filed February 6, 2018, and entitled "Method for Providing a Real-Time, Augmented and Shared Experience of Participating in a Live Event," which is incorporated herein by reference in its entirety.

Claims

1. A processor; Memory A server computer system comprising: The memory comprises: Instructions that, when executed by the processor, cause the server computer system to: Instructions for interconnecting a plurality of client devices over a network, comprising: a local group of the plurality of client devices located at a live event venue and a remote group located remotely from the live event venue; and each client device of the local group is a source of a live stream that, when rendered by another client device, may display a vantage point of the live event venue; instruction; instructions for dynamically updating a subscription list of a subscription service, the instructions comprising: including an indication of any client devices in the local group that are actively live streaming the live event; and removing indications of any client devices in the local group that have completed live streaming of the live event; instruction; enabling selective access to any live stream of any client device in the updated subscription list; Command; and Disabling access to any live streams for any client devices that are removed from the updated subscription list. instruction, Including, Server computer system.

2. further causing the removed indication to be added to a separate subscription list that includes indications of client devices that were previous sources of a live stream of the live event. The system of claim 1 .

3. further comprising receiving a query for a particular vantage point of the live event; and in response to the query, causing the subscription list to be searched for any client devices that satisfy the query; The system of claim 1 . [0020] Further comprising: enabling a first client device of the local group to display a first vantage point captured by a second client device of the local group; the second client device is selected from the updated subscription list. The system of claim 1 . [0021] Further comprising: enabling a first client device of the local group to live stream a first vantage point while displaying a second vantage point captured by a second client device of the local group; the second client device is selected from the updated subscription list. The system of claim 1 . [0010] Further comprising: enabling a first client device of the remote group to display a first vantage point of the live event captured by a second client device of the local group; the second client device is selected from the updated subscription list. The system of claim 1 .

7. Further comprising: enabling two client devices to simultaneously render a common live stream; The two client devices belong to the local group, or one of the two client devices belongs to the remote group; a source of the common live stream is selected from the updated subscription list; The system of claim 1 .

8. The method of claim 7, wherein the updated subscription list indicates a frequency with which the client device has ever live streamed a particular type of live event. The system of claim 1 .

9. Further, receiving an indication of a selection of a particular vantage point; and Depending on the indication of the selection: searching the updated subscription list for client devices having the particular vantage point; and providing a live stream of said particular vantage point; The system of claim 1 .

10. The method of claim 1, wherein the local group of client devices includes handheld mobile devices and stationary devices located at fixed locations. The system of claim 1 .

11. Further, hosting a virtual interaction between client devices of the local group and the remote group; The virtual interaction simulates a local interaction at the live event. The system of claim 1 .

12. Further, hosting a virtual conversation between two client devices of the local group; The virtual interaction simulates a local interaction at the live event. The system of claim 1 .

13. Further, enabling selective switching between displays of different vantage points of the live event venue based on different client devices of the local group. The system of claim 1 .

14. Further comprising receiving an indication from a particular client device to change subscriptions between different live streams of different client devices; causing the particular client device to unsubscribe from a first live stream of the different client stream; and subscribe the particular client device to a second live stream of the different client device; The system of claim 1 .

15. Further comprising: detecting activation of a first live stream from a first client device of the local group; enabling a second client device to subscribe to the first live stream; and updating a profile of the second client device to reflect the subscribing to the first live stream. The system of claim 1 .

16. Further, the live stream is augmented with AR content that, when rendered by a particular client device, displays a particular vantage point along with the AR content. The system of claim 1 .

17. Further, the method further comprises synchronizing the multiple live streams with respect to time. the subscription list includes client devices that have synchronized live streams and excludes client devices that have unsynchronized live streams; The system of claim 1 .

18. Further, receiving an indication of a selection of augmented reality (AR) content from the particular client device; and in response to the indication of the selection of the augmented reality (AR), superimposing a particular vantage point rendered at the particular client device onto the augmented reality (AR) content. The system of claim 1 .

19. Instructions that, when executed by at least one data processor of a system, cause the system to: Instructions for interconnecting a plurality of client devices over a network, comprising: a local group of the plurality of client devices located at a live event venue and a remote group located remotely from the live event venue; and each client device of the local group is a source of a live stream that, when rendered by another client device, may display a vantage point of the live event venue; instruction; instructions for dynamically updating a subscription list of a subscription service, the instructions comprising: including an indication of any client devices in the local group that are actively live streaming the live event; and removing indications of any client devices in the local group that have completed live streaming of the live event; instruction; Allowing selective access to any live stream of any client device on the updated subscription list. Command; and Disabling access to any live streams for any client devices that are removed from the updated subscription list. instruction, A non-transitory computer-readable storage medium having recorded thereon. [0023] Further comprising: enabling a first client device of the local group or the remote group to display a first vantage point of the live event captured by a second client device of the local group; the second client device is selected from the updated subscription list.

20. The non-transitory computer-readable storage medium of claim 19.

Citation Information

Patent Citations

  • Live concert / event video system and method

    JP2008520163A

  • Requesting content in a two-way network

    JP2008522490A

  • Video management system and method for identifying shooting side terminal shooting arbitrary area

    JP2016032229A

  • Camera selecting method and video distribution system

    JP2016213808A

  • Method and apparatus for distributing and / or playing content

    JP2017527230A