System and method for enabling live broadcast sessions in a virtual environment - Patents.com

Abstract

To provide users with feelings of realism and presence and an improved interaction and user experience during the broadcasting of live events.SOLUTION: A method for providing live broadcasting sessions in virtual environments includes creating a virtual environment based on virtual environment creation instructions; connecting the virtual environment to an event that is broadcast through the virtual environment; setting up a live broadcasting session from the virtual environment comprising the event broadcast; sending an invitation to at least one invitee client device, where the invitation comprises 3D (3-dimensional) coordinates of the virtual environment specified; opening up a communication channel enabling access to the at least one invitee to the virtual environment, through a user graphical representation that is positioned at the specified 3D coordinates of the virtual environment; and streaming the live broadcasting session to the at least one invitee client device.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates generally to computer systems, and more particularly to a system, method, and computer-readable medium for enabling live broadcast sessions in a virtual environment. [Background technology]

[0002] Delivery of media content can be accomplished through traditional means such as broadcast television via cable, satellite, and antenna transmissions, but such broadcast experiences typically do not allow for interaction with other remote viewers of the event that could improve the user experience.

[0003] Some social media applications also function as streaming platforms that allow the viewing of live events, such as YouTube® Live, Facebook® Live, and Twitch®, which allow users to comment on the event in real time and check various information such as the number of users watching the event live, live comments or reactions to the event, etc. However, such platforms tend to limit the user experience to a "flat" or 2D (two-dimensional) user interface, and interactions with users or hosts tend to be performed by means of text input (e.g., chat or comments), which creates a feeling of alienation. Live events such as conferences may also be conducted through distance conferencing platforms such as Skype™ and Zoom™, but the low level of realism, lack of user presence, lack of shared space, and lack of quality of possible interactions contribute to a feeling of loneliness or boredom for many users when comparing real-life experiences with these solutions.

[0004] There is a demand for technological solutions that provide users with realism, immersion, and improved interactivity and user experience during broadcasts of live events, such as sports, e-sports, live performances, conferences, and the like, that can enhance user engagement with the events. Summary of the Invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] According to one aspect of the present disclosure, a system for providing a live broadcast session in a virtual environment includes at least one server computer having at least one processor and a memory storing instructions implementing a virtual environment streaming application configured, when instructions are executed by the at least one processor, to cause the at least one server computer to: create a virtual environment based on virtual environment creation instructions from a host client device connected to the at least one server computer via a network; connect the virtual environment to an event to be broadcast through the virtual environment; set up a live broadcast session from the virtual environment including the event broadcast; send an invitation to at least one invitee client device, the invitation including 3D (three dimensional) coordinates of the virtual environment designated for the at least one invitee; upon the at least one invitee accepting the invitation, open a communication channel enabling the at least one invitee access to the virtual environment, such that the invitee client device accesses the virtual environment through a user graphical representation located at the designated 3D coordinates of the virtual environment; and stream the live broadcast session to the at least one invitee client device.

[0007] In some embodiments, to open a communication channel, the virtual environment streaming application selects between a selective forwarding unit (SFU) architecture for forwarding individual live streams from a sending client device to a receiving client device and a multipoint control unit (MCU) architecture for forwarding a combined live stream of the live streams from the sending client device to the receiving client device, the selection being based on a threshold number of live streams from the sending client device.

[0008] In some embodiments, the virtual environment streaming application allows guests to interact with each other by exchanging real-time audio and video.

[0009] In some embodiments, the virtual environment streaming application allows for selection of a limited number of guest user graphical representations that will be visible to at least one host of the live broadcast session, and at least one of the selected guests is allowed to interact with at least one host of the live broadcast session. In further embodiments, the virtual environment streaming platform allows said interactions with the at least one host to become part of a public video stream that is shared with other client devices. In further embodiments, the virtual environment streaming application further displays symbolic graphical representations of guests not included in the selection of the limited number of guests, the graphical representations being provided in the form of user graphical representations, a numerical representation indicating the number of users, or a combination thereof. In further embodiments, the virtual environment streaming application further allows for displaying graphical representations of the guests' reactions.

[0010] In some embodiments, the virtual environment streaming application allows an audience member to explore the virtual environment by moving within the virtual environment, and further, presentation of such movement by at least some of the audience members is disabled for the view of at least one host.

[0011] In some embodiments, the user graphical representation includes a user 3D virtual cutout with background removed, or a user real-time 3D virtual cutout with background removed, or a video with background removed, or a video without background removed.

[0012] In some embodiments, the field of view of the users, including the host and guests, is calculated from specified 3D coordinates of the virtual environment in which the corresponding user graphical representation is located.

[0013] In another aspect of the present disclosure, a method for providing a live broadcast session in a virtual environment is provided. The method may be executed by at least one processor executing instructions in a memory of at least one server computer, the memory storing a virtual environment streaming application configured to at least execute. In an embodiment, the method includes the steps of: creating a virtual environment based on a virtual environment creation instruction from a host client device connected to the at least one server computer via a network; connecting the virtual environment to an event to be broadcast through the virtual environment; setting up a live broadcast session to be broadcast from the virtual environment including the event broadcast; sending an invitation to at least one invitee client device, the invitation including 3D coordinates of the virtual environment designated for the at least one invitee; opening a communication channel enabling the at least one invitee access to the virtual environment upon the at least one invitee accepting the invitation, the invitee client device accessing the virtual environment through a user graphical representation located at the designated 3D coordinates of the virtual environment; and streaming the live broadcast session to the at least one invitee client device.

[0014] In some embodiments, the method further includes selecting between a selective forwarding unit (SFU) architecture for forwarding individual live streams from the sending client device to the receiving client device and a multipoint control unit (MCU) architecture for forwarding a combined live stream of the live streams from the sending client device to the receiving client device, the selecting being based on a threshold number of live streams from the sending client device.

[0015] In some embodiments, the method further comprises enabling the guests to interact with each other by exchanging real-time audio and video.

[0016] In some embodiments, the method further comprises selecting a limited number of guest user graphical representations to be visible to at least one host of the live broadcast session, where at least one of the selected guests is enabled to interact with at least one host of the live broadcast session. In a further embodiment, the method further comprises enabling said interaction with at least one host to be part of a public video stream shared with other client devices. In a further embodiment, the method further comprises displaying a symbolic graphical representation of the guests not included in the selection of the limited number of guests, where the graphical representation is provided in the form of a user graphical representation, a numerical representation indicating the number of users, or a combination thereof. In a further embodiment, the method further comprises displaying a graphical representation of the guest's reaction.

[0017] In some embodiments, creating the virtual environment includes retrieving user configuration data and customizing the virtual environment using the user configuration data.

[0018] In some embodiments, the method further includes retrieving and displaying the media from one or more third party databases.

[0019] In another aspect of the disclosure, a non-transitory computer-readable medium including instructions stored in a memory that, when executed by at least one processor, causes at least one server computer to perform steps including: creating a virtual environment based on virtual environment creation instructions from a host client device connected to the at least one server computer via a network; connecting the virtual environment to an event to be broadcast through the virtual environment; setting up a live broadcast session to be broadcast from the virtual environment including the event broadcast; sending an invitation to at least one invitee client device, the invitation including 3D coordinates of the virtual environment designated for the at least one invitee; opening a communication channel enabling the at least one invitee access to the virtual environment upon the at least one invitee accepting the invitation, the invitee client device accessing the virtual environment through a user graphical representation positioned at the designated 3D coordinates of the virtual environment; and streaming the live broadcast session to the at least one invitee client device, the invitee's view (e.g., as presented to a user of the invitee client device) being calculated from the designated 3D coordinates of the virtual environment.

[0020] The above summary does not include an exhaustive list of all aspects of the present disclosure. The present disclosure is contemplated to include all systems and methods that may be practiced from all suitable combinations of the various aspects outlined above and disclosed in the following detailed description and particularly pointed out in the claims filed with this application. Such combinations have advantages not specifically recited in the above summary. Other features and advantages will become apparent from the accompanying drawings and from the detailed description that follows below.

[0021] Certain features, aspects, and advantages of the present disclosure will become better understood with regard to the following description and the accompanying drawings. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic diagram of a system for enabling a live broadcast session in a virtual environment, according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram of server architecture options that a server may utilize depending on the number of user live streams, according to an embodiment. [Diagram 3] FIG. 1 is a schematic diagram of a host utilizing profile data to generate a virtual environment, according to an embodiment. [Figure 4] 1 is a schematic diagram of an embodiment of third party media published within a virtual environment. [Diagram 5] FIG. 1 is a block diagram of a method for enabling a live broadcast session in a virtual environment, according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] In the following description, reference is made to the drawings, which show by way of example various embodiments, which will be described below with reference to certain examples, it being understood that the embodiments may include changes in design and construction without departing from the scope of the claimed subject matter.

[0024] The present disclosure addresses at least some of the deficiencies described in the background through a system, method, and computer readable medium configured to provide a live broadcast session of an event in a virtual environment. The system, method, and computer readable medium of the present disclosure enable broadcast of an event, such as a real or digital live event or a pre-recorded event, in a virtual environment, such as a 3D virtual environment. The 3D virtual environment may connect to the event to create a live broadcast session that may be broadcast to multiple users through corresponding client devices. Invited users may access the virtual environment through a corresponding invitation having 3D coordinates such that, if the invitation is accepted, the users may be placed in the 3D coordinates of the virtual environment through the corresponding user graphical representation. The host of the event may allow a certain number of limited users (e.g., friends, special guests, or other people involved in the event) to be visible to the host through the user graphical representation, while the rest of the audience may be displayed in the form of a symbolic representation. Interactions with the audience may become part of the live broadcast session such that other users (e.g., all users or some subset of users) may watch the interaction. While current systems allow for viewing of an event and some types of interaction (e.g., comments, chat, or live video broadcast), providing users with the ability to access a live broadcast session of an event in a 3D virtual environment, as provided by the present disclosure, can enhance the user's sense of presence and the quality of experience and interaction between users and hosts of the live broadcast session by enabling a shared 3D space in which such users may communicate and interact.

[0025] FIG. 1 is a schematic diagram of a system 100 for enabling a live broadcast session in a virtual environment, according to an embodiment.

[0026] The system 100 comprises at least one server computer 102 comprising at least one processor 104 and a memory 106 storing instructions implementing a virtual environment streaming application 108. The virtual environment streaming application 108, when instructions are executed by the at least one processor 104, may be configured to create a virtual environment 110 based on virtual environment creation instructions from a host client device 122 connected to the at least one server computer 102 via a network 114. The virtual environment streaming application 108 may then connect the virtual environment 110 to an event 116 to be broadcast through the virtual environment 110. Once such a connection is established, the virtual environment streaming application 108 sets up a live broadcast session from the virtual environment 110, including a live event broadcast 118 of the event. The event may thus be instantiated and displayed by the virtual environment streaming application 108 on a projection surface within the virtual environment 110, such as a virtual client device screen (e.g., a TV, tablet, or mobile device screen), a projection screen, a projection wall, a panel, a billboard, a ceiling, etc.

[0027] The virtual environment streaming application 108 then sends an invitation to at least one invitee client device 120. Such an invitation may be sent based on a request received from an inviting client device, such as the host client device 122, or alternatively, a client device of another inviting user who may be so entitled. The invitation may be sent through any suitable channel, such as email, SMS, social media applications, messaging applications, and the like, and may include user information that may help identify the invitee, such as name, location, email address, phone number, etc. In some embodiments, the invitation includes 3D coordinates of the virtual environment 110 designated for the at least one invitee 124. Once the at least one invitee 124 accepts the invitation, the virtual environment streaming application 108 opens a communication channel enabling the at least one invitee 124 to access the virtual environment 110, and the invitee client devices 120 access the virtual environment 110 through a user graphical representation 126 positioned at the specified 3D coordinates of the virtual environment and thus presenting a representation of the user (e.g., an avatar or a real-time video broadcast) within the virtual environment 110. Finally, the virtual environment streaming application 108 streams the live broadcast session to the at least one invitee client devices 120.

[0028] In this disclosure, a "broadcast session" refers to a teleconferencing session hosted through the virtual environment streaming application 108 and including a broadcast of an event from within the virtual environment. In some embodiments, the broadcast is performed by instantiating and projecting the event onto at least one surface of the virtual environment. A broadcast session may be attended by multiple users who may watch and experience a broadcast of the event, interacting with each other and possibly with at least one host of the broadcast session to view a presentation of the event. Once an event is instantiated and projected onto at least one surface of the virtual environment during a broadcast session, all users within the virtual environment may be able to view the broadcast event if the user's field of view at a given moment includes the projection surface.

[0029] In some embodiments, at least one host may select the projection surface, may decide to turn the projection surface on or off, or switch the projection surface to a different projection surface. In some embodiments, the projection surface containing the projected event may be further enlarged or reduced, or moved to another area of ​​the virtual environment. For example, if the projection surface is a TV screen, the TV screen may be enlarged or moved to another area that is more easily viewed by the invited guests. The virtual environment broadcasting application 108 may be further configured to adjust the resolution of the projection surface containing the projected event accordingly.

[0030] In this disclosure, an event may be a live or pre-recorded real-world or digital event that can be broadcast through a broadcast session, such as, for example, e-sports, real-world sports matches, performances, fashion shows, popularity contests, movie releases, video game releases, theater plays, circus performances, news reports, and the like. The event may be introduced, narrated, commented on, and / or discussed by one or more hosts and / or moderators from within the virtual environment 110 and may enable interaction with at least some members of the audience based on their status. Such interaction may be enabled through user graphical representations of the hosts and audience members that may be generated and inserted into the virtual environment.

[0031] In some embodiments, the field of view of the guests and other users (e.g., the host 128 and other members of the audience) is calculated from the specified 3D coordinates of the virtual environment 110 where the guest user graphical representation 126 is located. Thus, each user graphical representation 126 may have a personalized viewing field set from their respective specific locations to display the virtual scenes and live events to the corresponding user from the user's position in the virtual environment 110. In some embodiments, the viewing field is updated as the user manually navigates through the virtual environment 110 via a graphical user interface. In further embodiments, the viewing field is established and updated automatically by using a virtual camera, which is automatically updated by tracking and analyzing the user's eye and head tilt data, or head rotation data, or a combination thereof.

[0032] Thus, in some embodiments, virtual environment 110 is a 3D virtual environment that allows users to view the event from within a provided virtual space shared by the users and accessed through the users' corresponding user graphical representations 126 located at specified 3D coordinates in virtual environment 110. The presentation of the users within the shared 3D virtual environment 110, along with the enabled interaction and communication between users (e.g., between invitees 124, host 128, and combinations thereof), increases the users' sense of realism, facilitates engagement with and / or attendance at the live broadcast session and corresponding live event, and provides a realistic and entertaining user experience.

[0033] In some embodiments, the virtual environment streaming application 108 allows the invitees 124 to interact with one another by exchanging real-time audio and video. Such interaction and communication is provided within the shared space of the virtual environment 110 to facilitate interaction between users. In further embodiments, other interactions between users may also be enabled, such as chatting, screen sharing, recording, voting, media file sharing, sending emoticons, virtual hugs, raising hands, shaking hands, walking, adding content, moving objects, booing, cheering, shouting, gesturing, projection, laser pointing, game playing, and purchasing, among others.

[0034] In some embodiments, the virtual environment streaming application 108 allows for the selection of a limited number of invited guest user graphical representations 126 that are visible to the at least one host 128 of the live broadcast session, and at least one of the selected limited invited guests 130 is allowed to interact with the at least one host 128 of the live broadcast session through the corresponding user graphical representation 126. The limited invited guests may be selected by the at least one host 128 or by another user with such qualifications, and may represent, for example, the host's close friends, VIPs, or people with some type of special relevance to the live broadcast session. These limited invited guests may be visible to the at least one host 128 through the user graphical representation 126. The at least one host 128 may also be visible to the at least one limited guest 130 within the virtual environment 110 by the host user graphical representation 132.

[0035] In a further embodiment, the virtual environment streaming application 108 allows interactions with at least one host 128 to become part of a public video stream shared with other client devices of the remaining audience 134. In this disclosure, the term "remaining audience" refers to any user or invitee who is not part of the limited guests 130. For example, if a member of the limited audience asks a question, shares a screen, or submits some information, such data and / or interaction may be displayed to the remaining audience 134 as part of the live broadcast from the live broadcast session. In a further embodiment, the virtual environment streaming application 108 further displays a symbolic user graphical representation 136 of the invitee from the remaining audience 134, the graphical representation being provided in the form of a user graphical representation, a numerical representation indicating the number of users, or a combination thereof. For example, if 1000 people are attending the live broadcast session, there may be 5 people who are part of the limited guests 130 and 995 people who are included as the remaining audience 134. Each of the qualified guests 130 may be represented by a corresponding user graphical representation 126, while the remaining audience 134 may be represented by a symbolic graphical representation 136, which may be in the form of a symbolic crowd including symbolic faces, whole bodies, silhouettes, shadows, masks, cartoons, and the like, that may represent the remaining audience 134.

[0036] In some embodiments, the audience members, including the limited guests 130 and the remaining audience 134, may be assigned a fixed location based on the 3D coordinates included in the invitation or may be allowed to move and explore the virtual environment from their original 3D coordinates. In further embodiments, the virtual environment broadcasting application may be further configured to allow the limited guests 130 to freely explore the virtual environment during the broadcast session without disturbing the at least one host 128 and / or to allow the limited guests 130 to disable their image to the at least one host so that such movement is not shown in the video broadcast to other client devices.

[0037] In some embodiments, the events 116 that may be broadcast during a broadcast session refer to any real or digital event occurring in real time. The real events may be broadcast in real time through at least one camera 138. However, in other embodiments, the virtual environment streaming application 108 also allows for hosting a live broadcast session of a pre-recorded event. For example, a previously recorded football match may be transferred to at least one server 102 for replay in the virtual environment 110 during the broadcast session. In further embodiments, the virtual environment streaming application 108 further allows for displaying graphical representations of guest reactions, such as from the exclusive guests 130 or the remaining audience 134. For example, such representations may be shown as emojis, including thumbs up or down, hearts, smiley faces, happy faces, sad faces, angry faces, and the like, that represent the guest reactions.

[0038] In some embodiments, the virtual environment streaming application 108 temporarily allows members from the remaining audience 110 to interact with the at least one host 128. A user from the remaining audience 134 may raise their hand through the user interface, which sends a signal to the at least one host client device 122 prompting the at least one host 128 to allow the user to temporarily interact with the at least one host 128 through a corresponding user graphical representation 136. Such interaction may be included in a live broadcast session that may be viewed by the limited guests 130 and other members from the remaining audience 134.

[0039] In some embodiments, the user graphical representation 126 includes a user 3D virtual cutout with the background removed, or a user real-time 3D virtual cutout with the background removed, or a video with the background removed, or a video without the background removed.

[0040] In the present disclosure, the user 3D virtual cutout may include a virtual replica of the user constructed from a 2D photograph uploaded by the user or from a third party source. In an embodiment, the user 3D virtual cutout is created through a 3D virtual reconstruction process through machine vision techniques that use a 2D photograph uploaded by the user or from a third party source as input data, which generates a 3D mesh or 3D point cloud of the user with the background removed. In one embodiment, the user 3D virtual cutout may have a static facial expression. In another embodiment, the user 3D virtual cutout may have a facial expression that is updated through a camera feed. In yet another embodiment, the user 3D virtual cutout may have a facial expression that can be changed through a button in a user graphical interface, such as a button that allows the user 3D virtual cutout to smile, frown, become serious, and the like. In a further embodiment, the user 3D virtual cutout displays facial expressions using a combination of the aforementioned techniques. After generating the user 3D virtual cutout, the status and / or facial expression of the user 3D virtual cutout may be constantly updated, for example, by processing a camera feed from the user.

[0041] The user real-time 3D virtual cutout may include a virtual replica of the user after removing the user background based on a real-time 2D or 3D live video stream data feed obtained from a camera. In an embodiment, the user real-time 3D virtual cutout is created through a 3D virtual reconstruction process through machine vision techniques using the user live data feed as input data by generating a 3D mesh or 3D point cloud of the user with the background removed. For example, the user real-time 3D virtual cutout may be generated from 2D video from a camera 146, which may be processed to create a holographic 3D mesh or 3D point cloud. In another example, the user real-time 3D virtual cutout may be generated from 3D video from a depth camera (e.g., LIDAR or any depth camera), which may be processed to create a holographic 3D mesh or 3D point cloud. Thus, the user real-time 3D virtual cutout graphically represents the user in three dimensions and in real time.

[0042] A background-removed video may include a video streamed to a client device where a background removal process has been implemented such that only the user is visible and thus may be displayed at the receiving client device using a polygonal structure. A non-background-removed video may include a video streamed to a client device where the video faithfully represents the camera capture and thus the user and the user's background are visible and thus may be displayed at the receiving client device using a polygonal structure. The polygonal structure may be a rectangular structure or a more complex 3D structure that is used as a virtual frame to support the video.

[0043] In some embodiments, the user graphical representation 126 is created by a process that includes receiving a live data feed captured by at least one camera 146 connected to the sending client device by at least one server computer 102, sending the captured data to a receiving client device via the server 102 or through a P2P architecture, performing background removal of the user of the sending client device to extract only images representing the user sending user, and inserting the user graphical representation 126 into the virtual environment 110 by the receiving client device.

[0044] In some embodiments, data used as input data contained in the live data feed and / or 2D photos uploaded by users or from third party sources includes, among others, 2D or 3D image data, 3D shapes, video data, media data, audio data, text data, tactile data, temporal data, 3D entities, 3D dynamic objects, text data, temporal data, metadata, priority data, security data, location data, lighting data, depth data, and infrared data.

[0045] In some embodiments, one or more hosts 132 may be moderators of a live broadcast session and may introduce, narrate, comment, and / or discuss a live or pre-recorded event while enabling interaction and communication therein and with select guests 130 and at least a portion of the remaining audience 134.

[0046] In some embodiments, the virtual environment 110 refers to a virtual scene, world, or universe in which users, including the host, limited guests 130, and the remaining audience 134, explore, subscribe to, and interact with published media content items within the virtual environment 110, communicate with each other, and the like. The virtual environment 110 has assets that may be modeled across multiple software platforms or engines. The assets may be defined in a scene description, which may be in the form of a scene graph with properties and values, or may be based on a model. A model is a description of a three-dimensional object, defined with data including shape, texture, physics, lighting, materials, etc.

[0047] The virtual environment 110 thus refers to a virtual structure (e.g., a virtual model) that may be designed through any suitable 3D modeling technique through computer assisted drawing (CAD) methods. In some embodiments, the virtual environment 110 refers to a virtual structure scanned from a real structure (e.g., a physical room) through any suitable scanning tool with an image scanning pipeline that is input through various photo, video, depth measurement, and / or simultaneous location and mapping (SLAM) scans to generate the virtual environment 110. For example, radar imaging, such as synthetic aperture radar, real aperture radar, Light Detection and Ranging (LIDAR), Inverse Aperture Radar, monopulse radar, and other types of imaging techniques, may be used to map and model real-world structures and transform them into the virtual environment 110. In other embodiments, the virtual environment 110 is a virtual structure modeled after a real structure (e.g., a real-world room, building, or facility).

[0048] The assets included in the virtual environment 110 may include, among others, any of 2D or 3D image data, 3D shapes, video data, media data, audio data, text data, tactile data, time data, 3D entities, 3D dynamic objects, text data, time data, priority data, security data, location data, lighting data, depth data, infrared data, and corresponding metadata.

[0049] In some embodiments, client device 120 may be one or more of a mobile device, a personal computer, a laptop, a game console, a media center, smart contact lenses, and a head mounted display, among others. Camera 146 / 138 may be one or more of a 2D or 3D camera, a 360 degree camera, a webcam, an RGBD camera, a CCTV camera, a professional camera, a cell phone camera, a depth camera (e.g., LIDAR), or a light field camera, among others.

[0050] In some embodiments, the virtual environment 110 utilizes a hybrid system architecture including a client-server side and a peer-to-peer (P2P) side. In an embodiment, the client-server side comprises a web or application server. The client-server side may be further configured to include a secure communication protocol, microservices, a database management system, a database, and / or a distributed message and resource delivery platform. The server side components may be provided with a client device that communicates to the server over a network. The client-server side defines the interaction between one or more clients and a server over a network, including any processing performed by the client side, the server side, or the receiving client side. In an embodiment, one or more of the corresponding clients and servers perform the required image and media processing by a combination of various rule-based task assignments. In an embodiment, the web or application server is configured to receive the client request using the secure communication protocol and process the client request by requesting the microservices or data corresponding to the request from the database using the database management system. The microservices are distributed using a distributed message and resource delivery platform using a publish-subscribe model.

[0051] The P2P side comprises a P2P communication protocol enabling real-time communication between client devices in the virtual environment 110, and a rendering engine configured to enable the client devices to perform real-time 3D rendering (e.g., user graphical representations) of live session elements included in the virtual environment in the rendering engine. An example of a suitable P2P communication protocol may be the Web Real-Time Communication (WebRTC) communication protocol, which is a collection of standards, protocols, and JavaScript APIs that combine to enable P2P audio, video, and data sharing between peer client devices. An example of a suitable rendering engine may be a 3D engine based on WebGL, which is a JavaScript API for rendering 2D and 3D graphics in any compatible web browser without the use of plug-ins and accelerated use of physics and image processing and effects by one or more processors (e.g., one or more graphic processing units (GPUs)) of the client devices. In an embodiment, the P2P side further includes a computer vision library configured to enable the client devices to perform real-time computer vision tasks in the virtual environment. An example of a suitable computer vision library may be OpenCV, which is a library of programming functions primarily configured for real-time computer vision tasks. Using such a hybrid model of communication can enable rapid P2P communication between users, reducing latency issues while providing web services and resources to each session, and enabling multiple interactions between users and with content in the virtual environment.

[0052] In some embodiments, the client devices 122 and the at least one cloud server computer 102 connect through a wired or wireless network 114. In some embodiments, the network may include millimeter-wave (mmW) or a combination of mmW and sub-6 GHz communication systems, such as fifth generation wireless systems communication (5G). In other embodiments, the system may connect through wireless local area networking (Wi-Fi). In other embodiments, the system may communicate through fourth generation wireless systems communication (4G), may be supported by a 4G communication system, or may include other wired or wireless communication systems.

[0053] In some embodiments, at least one server computer 102 is a media server computer using a multipoint control unit (MCU) topology, or a selective forwarding unit (SFU) topology, or a Traversal Using Relay NAT (TURN) topology, or a spatially analyzed media server topology (SAMS).

[0054] In some embodiments, at least the server computer 102 may be an intermediate server intended to be used to facilitate and / or optimize the exchange of data between client devices. In such embodiments, at least one cloud server may manage, analyze, process, and optimize the incoming image and multimedia streams and the forwarding of the outbound streams as a router topology (e.g., but not limited to, SFU, SAMS, multimedia server router, or image and media processing (e.g., decoding, combining, improving, mixing, enhancing, expanding, computing, manipulating, encoding)) and a forwarding server topology (e.g., but not limited to, MCU, cloud media mixer, cloud 3D renderer, and the like), or other server topology.

[0055] In embodiments in which the intermediate server is a SAMS, such media server manages, analyzes, and processes the incoming data of each issuing client device (e.g., including but not limited to metadata, priority data, data classes, spatial structure data, three-dimensional position, orientation, or motion information, images, media, scalable video codec-based video) and, in such analysis, optimizes the transfer of outbound data streams to each client device by modifying, upscaling, or downscaling media temporally (varying frame rates), spatially (e.g., different image sizes), and for level of detail, quality (e.g., different compression or encoding-based quality), and color (e.g., color resolution and range) based on the spatial three-dimensional orientation, distance, and priority relationship of a particular receiving client device user to such incoming data to achieve optimal bandwidth and computing resource utilization for one or more user client devices.

[0056] FIG. 2 is a schematic diagram of a sample server architecture option 200 that may be utilized by at least one server computer 202 of the present disclosure depending on the number of user live streams, according to an embodiment.

[0057] Upon opening a communication channel to enable interaction within the virtual environment, the server computer 202 may select between the SFU architecture and the MCU architecture to present the live streams from the sending client devices, as disclosed with reference to the system 100 of FIG. 1. Such live streams may be presented in the form of user graphical representations of the users, which may be generated from videos of these users captured by the sending client devices. In an exemplary scenario below a certain threshold of live streams from client devices at 204, the virtual environment streaming application uses the SFU architecture 206 to forward individual live streams from a minimum of one central server computer 202 to the receiving client devices, and to receive streams captured by the sending client devices from all other participants via the same at least one central server computer 202. The SFU architecture 206 utilizes programmed logic stored in the memory of the at least one server computer 202 to determine which media streams to forward to other users of the receiving client devices. Thus, because the SFU architecture 206 enables at least one central server computer 202 to receive and forward all individual media streams from all users of sending client devices to receiving client devices, such an architecture may be suitable for use in embodiments of the present disclosure with live streams below a certain threshold, such as, for example, below 5, 20, or some other threshold number of live streams from users of sending client devices.

[0058] When a live stream threshold of 208 is exceeded, the virtual environment streaming application uses the MCU architecture to forward the combined live stream to the receiving client device when opening a communication channel to enable interaction within the virtual environment. The MCU architecture assumes that each user sends their stream from the sending client device to at least one server computer 202, which decodes the stream, rescales the stream, assembles a new stream from all received streams, encodes the new stream, and sends a single stream to all other participants of the receiving client device. Thus, the MCU combines all streams received from the sending client device before sending the combined media stream to all participants of the receiving client device, which may be more efficient and appropriate for live streams above a certain threshold, such as, for example, above 5, 20, or some other threshold number of live streams from users of the sending client device.

[0059] FIG. 3 is a schematic diagram of a virtual environment generation process 300 by utilizing profile data, according to an embodiment.

[0060] In Fig. 3, the host 302 uses a graphical user interface of a virtual environment streaming application 304 through a client device to create a virtual environment 306. The virtual environment streaming application 304 may be a downloadable application or may be available online from a browser without the need to download any software program. In response to the virtual environment streaming application 304 receiving a user command to create the virtual environment 306, the virtual environment streaming application 302 retrieves user setting data from a database, such as a social media database 310, over a network 308 and uses the user setting data to generate the virtual environment 306 having a style appropriate for the user. The virtual environment streaming application 304 extracts style-related data from the retrieved user setting data, such as color, shape, decoration-related information, and the like, that can be used to generate the virtual environment 306.

[0061] In other embodiments, the profile of the host 302 is manually entered by the host 302 through a corresponding host client device. Manually entering a profile may be a feature enabled by the virtual environment streaming application 304. Such a profile may request style-related information that can be used by the virtual environment streaming application 304 to thus create the virtual environment 306. For example, the profile may request adding favorite colors, architecture, locations, backgrounds, surface area, etc. The profile may further request data such as requesting the number of users to calculate the capacity and size of the virtual environment. The virtual environment streaming application 304 may create the virtual environment using predefined templates based on the retrieved profile data, or may use more complex algorithms through machine learning and artificial intelligence to generate a new unique virtual environment from the extracted data specifically created for the user.

[0062] Once the virtual environment 306 is created, the virtual environment 306 may then retrieve the live event broadcast 312 through the virtual environment streaming application 304 over the network 308, and the live event broadcast 312 is then streamed through the virtual environment 306 by instantiating and projecting the live event onto at least one surface of the virtual environment.

[0063] FIG. 4 is a schematic diagram of a third-party media publishing process 400 within a virtual environment, according to an embodiment.

[0064] The third party media publishing process 400 is implemented by the virtual environment streaming application 402 by retrieving media content from a corresponding third party database 404 over a network 406 in response to a request from a third party, such as a company representing one or more brands. Such a request may need to be approved by one or more hosts 408 of the virtual environment. The sponsored media may then be published in the created virtual environment 410, which retrieves the live event broadcast 414 through the virtual environment streaming application 402 over a network 412. The sponsored media may be static or dynamic media data that can be instantiated and projected onto at least one surface of the virtual environment to promote the corresponding brand to attendees. For example, the sponsored media may be published on one or more billboards, panels, walls, pillars, tables, and the like. Thus, the virtual environment 410 may serve as a publishing channel for companies to promote themselves during the live event broadcast session.

[0065] 5 illustrates a method 500 for enabling a live broadcast session in a virtual environment, according to an embodiment. The method 500 for providing a live broadcast session in a virtual environment may be performed by a system, such as the system 100 of FIG. 1, using at least one processor implementing instructions in a memory of at least one server computer, the instructions including a virtual environment streaming application.

[0066] Method 500 may begin, at step 502, by creating a virtual environment based on virtual environment creation instructions from a host client device connected to at least one server computer via a network. Method 500 may then continue, at step 504, by connecting the virtual environment to an event that is to be broadcast through the virtual environment. Such an event may be a real or digital live event broadcast in real time through at least one camera. However, in other embodiments, the virtual environment streaming application may also enable hosting live broadcast sessions of pre-recorded events.

[0067] At step 506, the method may continue by setting up a live broadcast session broadcast from the virtual environment including the event broadcast. At step 508, the method 500 may continue by sending an invitation to at least one invitee client device, the invitation including 3D coordinates of the virtual environment designated for the at least one invitee. Such an invitation may be sent from a request received from an inviting client device, such as the host client device, or alternatively, a client device of another invited user. The invitation may be sent in response to the virtual environment streaming application receiving an invitation generation from the at least one host. Furthermore, the invitation may be sent through any suitable channel, such as email, SMS, social media applications, messaging applications, and the like, and may include user information that may help identify the invitee, such as name, location, email address, phone number, etc.

[0068] Once the at least one invitee accepts the invitation, the method may continue in step 510 by opening a communication channel enabling the at least one invitee access to the virtual environment. In some embodiments, the invitee client device accesses the virtual environment through a user graphical representation positioned at a specified 3D coordinate of the virtual environment. Finally, in step 512, the method may conclude by streaming the live broadcast session to the at least one invitee client device.

[0069] While particular embodiments have been described and illustrated in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of the broad invention and are not limiting thereto, and that the invention is not limited to the specific construction and arrangements shown and described, since various other modifications may occur to those skilled in the art. Accordingly, the description should be considered as illustrative and not restrictive. [Explanation of symbols]

[0070] 100 Systems 102 Server computer, server, cloud server computer 104 processors 106 Memory 108 Virtual environment streaming application, virtual environment broadcasting application 110 Virtual Environment 114 Network 116 Events, Live Events 118 Live Event Broadcast 120 Invitee Client Device, Client Device 122 Host Client Device, Client Device 124 Invited Guest 126 User Graphical Representation of Guests, User Graphical Representation 128 Host 130 Exclusive guests, exclusive guests 132 Host User Graphical Representation 134 Remaining Audience 136 Symbolic User Graphical Representation 138 Camera 146 Cameras 200 Sample Server Architecture Options 202 Server computer, server 204 Live stream from client device below threshold 206 SFU Architecture Exceeding the 208 Live Stream Threshold 210 MCU Architecture 300 Virtual environment generation process 302 Host 304 Virtual Environment Streaming Application 306 Virtual Environment 308 Network 310 Social Media Database 312 Live Event Broadcast 400 Third Party Media Publishing Process 402 Virtual Environment Streaming Application 404 Third Party Databases 406 Network 408 Host 412 Network 410 Virtual Environment 414 Live Event Broadcasting 500 ways

Claims

1. 1. A system for providing a live broadcast session in a virtual environment, comprising: at least one server computer; At least one processor; When executed by the at least one processor, the instructions creating a virtual environment based on virtual environment creation instructions from a host client device connected to the at least one server computer via a network; connecting said virtual environment to an event broadcast through said virtual environment; setting up a live broadcast session from said virtual environment, including a broadcast of said event; sending an invitation to at least one invitee client device, the invitation including three-dimensional coordinates of the virtual environment designated for the at least one invitee; Upon the at least one invitee accepting the invitation, opening a communication channel enabling the at least one invitee to access the virtual environment, wherein the invitee client device accesses the virtual environment through a user graphical representation located at the specified three-dimensional coordinates of the virtual environment; and Streaming the live broadcast session to the at least one invitee client device. a memory storing the instructions for implementing a virtual environment streaming application configured to cause the at least one server computer to: At least one server computer comprising: Equipped with the virtual environment streaming application selecting between a selective forwarding unit (SFU) architecture for forwarding individual live streams from a sending client device to a receiving client device and a multipoint control unit (MCU) architecture for forwarding a combined live stream of the live streams from the sending client device to the receiving client device, the selection being based on a threshold number of live streams from the sending client device; system.

2. The system of claim 1 , wherein the virtual environment streaming application enables guests to interact with each other by exchanging real-time audio and video.

3. 2. The system of claim 1, wherein the virtual environment streaming application enables selection of a limited number of guest user graphical representations that are visible to at least one host of the live broadcast session, and at least one selected guest is enabled to interact with the at least one host of the live broadcast session.

4. The system of claim 3, wherein the virtual environment streaming application enables the interaction with the at least one host to become part of a public video stream shared with other client devices.

5. The system of claim 3 , wherein the virtual environment streaming application further displays symbolic graphical representations of guests not included in the selection of the limited number of guests.

6. The system of claim 3 , wherein the virtual environment streaming application further enables displaying a graphical representation of guest reactions.

7. The system of claim 3 , wherein the virtual environment streaming application enables an audience to explore the virtual environment by moving within the virtual environment.

8. The system of claim 7, wherein presentation of movement by at least one member of the audience is disabled for the view of the at least one host.

9. The system of claim 1 , wherein a user's field of view, including a host and guests, is calculated from the specified three-dimensional coordinates of the virtual environment in which the corresponding user graphical representation is located.

10. 1. A method performed by at least one processor executing instructions in a memory of at least one server computer, comprising: creating a virtual environment based on virtual environment creation instructions from a host client device connected to the at least one server computer via a network; connecting the virtual environment to an event that is broadcast through the virtual environment; setting up a live broadcast session broadcast from said virtual environment including a broadcast of said event; sending an invitation to at least one invitee client device, the invitation including three-dimensional coordinates of the virtual environment designated for the at least one invitee; if the at least one invitee accepts the invitation, opening a communication channel allowing the at least one invitee access to the virtual environment, the invitee client device accessing the virtual environment through a user graphical representation located at the specified three-dimensional coordinates of the virtual environment; streaming the live broadcast session to the at least one invitee client device; selecting between a selective forwarding unit (SFU) architecture for forwarding individual live streams from a sending client device to a receiving client device and a multipoint control unit (MCU) architecture for forwarding a combined live stream of the live streams from the sending client device to the receiving client device, the selecting being based on a threshold number of live streams from the sending client device; A method comprising:

11. 11. The method of claim 10, further comprising the step of enabling the guests to interact with each other by exchanging real-time audio and video.

12. 11. The method of claim 10, further comprising the step of selecting a limited number of invitee user graphical representations to be visible to at least one host of the live broadcast session, wherein at least one selected invitee is enabled to interact with the at least one host of the live broadcast session.

13. The method of claim 12 , further comprising the step of enabling the interaction with the at least one host to become part of a public video stream shared with other client devices.

14. The method of claim 12 , further comprising the step of displaying a symbolic graphical representation of guests not included in said selection of said limited number of guests.

15. The method of claim 12 further comprising the step of displaying a graphical representation of the guest's reactions.

16. The step of creating a virtual environment comprises: Retrieving user configuration data; and customizing the virtual environment using the user configuration data; and The method of claim 10, comprising:

17. The method of claim 10 , wherein the guest's field of view is calculated from the specified three-dimensional coordinates of the virtual environment.

18. When the instructions are executed by at least one processor, creating a virtual environment based on virtual environment creation instructions from a host client device connected to at least one server computer via a network; connecting the virtual environment to an event that is broadcast through the virtual environment; setting up a live broadcast session broadcast from said virtual environment including a broadcast of said event; sending an invitation to at least one invitee client device, the invitation including three-dimensional coordinates of the virtual environment designated for the at least one invitee; Upon the at least one invitee accepting the invitation, opening a communication channel enabling the at least one invitee access to the virtual environment, wherein the invitee client device accesses the virtual environment through a user graphical representation located at the specified three-dimensional coordinates of the virtual environment; streaming the live broadcast session to the at least one invitee client device, wherein a field of view of the invitee is calculated from the specified three-dimensional coordinates of the virtual environment; selecting between a selective forwarding unit (SFU) architecture for forwarding individual live streams from a sending client device to a receiving client device and a multipoint control unit (MCU) architecture for forwarding a combined live stream of the live streams from the sending client device to the receiving client device, the selection being based on a threshold number of live streams from the sending client device; 23. A non-transitory computer-readable medium comprising instructions stored in a memory that causes the at least one server computer to perform steps including:

Images