Method and apparatus for providing performance content

Abstract

A method of operation of a rendering server according to one embodiment of the present disclosure comprises: identifying at least one object from image information; generating a rendering image for the at least one object; transmitting the rendering image so that it is displayed on a user terminal; extracting at least one event information corresponding to a specific time interval from the image information; and providing the at least one event information to a host server so as to be synchronized according to the specific time interval.

Description

Technology Field

[0001] The present disclosure relates to a technology for providing performance content, and more specifically, to a content provision technology for providing virtual performance content including user-to-user interaction to a number of remote user terminals at a certain level of quality. Background Technology

[0002] Recently, coinciding with the COVID-19 crisis, various performance planning companies are introducing online digital performances that combine ICT such as VR and AR to the public, and are actively attempting new online revenue models.

[0003] Online digital VR performances featuring large numbers of participants in the form of avatars within virtual spaces are being showcased, and market demand for digital humans capable of expressing realistic and natural emotions is increasing.

[0004] Currently, expressing such photorealistic and natural digital humans on mobile devices is difficult due to issues such as a lack of computing power. Furthermore, even in the case of VR video streaming provided as 360-degree video, there are limitations on the freedom of viewpoint, and since it is video-based, it is difficult to expect real-time interaction between users participating in a performance.

[0005] In the future, for virtual online performances in the form of a metaverse, there is a growing need for services that allow large audiences to participate in the performance, albeit indirectly, alongside performers with photorealistic meta-human quality provided by tools such as Unreal Engine.

[0006] However, it is technically difficult to guarantee a certain level of quality for such high-quality virtual performance content on multiple client mobile devices accessed remotely. The problem to be solved

[0007] The purpose of the present disclosure is to provide a method for visualizing high-quality performances in real time and enabling interaction among audiences on multiple remote user terminals.

[0008] More specifically, the present disclosure aims to provide a dual object-specific streaming-multiplayer host server structure that supports a multiplayer host server for individually rendering high-quality performer avatar images using high-performance server resources, encoding them into a video format for streaming, compositing and synthesizing the results with scenes on client-side mobile devices, and supporting free navigation of the audience and interaction among the audience even in a streaming environment. means of solving the problem

[0009] A method of operation of a rendering server according to one embodiment of the present disclosure comprises: identifying at least one object from image information; generating a rendering image for the at least one object; transmitting the rendering image so that it is displayed on a user terminal; extracting at least one event information corresponding to a specific time interval from the image information; and providing the at least one event information to a host server so as to be synchronized according to the specific time interval.

[0010] In addition, the above image information may be extracted from performance content and be visually identifiable information.

[0011] In addition, the above at least one object may be at least one performer participating in the performance content.

[0012] In addition, the above at least one event information may be provided to the host server via a Remote Procedure Call (RPC).

[0013] Additionally, the method includes the steps of: acquiring depth information corresponding to the rendering image; generating encoded data packed according to a predetermined format based on the rendering image and the depth information; and providing the encoded data to a media server.

[0014] In addition, the method further includes the step of obtaining location information of the user terminal, and is characterized in that at least one event information is synchronized according to the location information.

[0015] Additionally, the at least one event information includes transform data for at least one of background information and audience information extracted from the image information, the background information includes information for at least one of special effects, animations, and objects of the image information, and the audience information includes information for at least one of location, gestures, and interactions of the user of the user terminal.

[0016] An apparatus of a rendering server according to one embodiment of the present disclosure comprises: a transmitting and receiving unit; and at least one control unit operably connected to the transmitting and receiving unit, wherein the at least one control unit may be configured to identify at least one object from image information, generate a rendering image for the at least one object, transmit the rendering image so that it is displayed on a user terminal, extract at least one event information corresponding to a specific time interval from the image information, and provide the at least one event information to a host server so as to be synchronized according to the specific time interval.

[0017] In addition, the above image information may be extracted from performance content and be visually identifiable information.

[0018] In addition, the above at least one object may be at least one performer participating in the performance content.

[0019] In addition, the above at least one event information may be provided to the host server via a Remote Procedure Call (RPC).

[0020] Additionally, the at least one control unit may be further configured to acquire depth information corresponding to the rendering image, generate encoding data packed according to a predetermined format based on the rendering image and the depth information, and provide the encoding data to a media server.

[0021] In addition, the at least one control unit is further configured to acquire location information of the user terminal, and the at least one event information can be synchronized according to the location information.

[0022] Additionally, the at least one event information includes transform data for at least one of background information and audience information extracted from the image information, the background information includes information for at least one of special effects, animations, and objects of the image information, and the audience information includes information for at least one of location, gestures, and interactions of the user of the user terminal.

[0023] A system for providing an online performance service according to one embodiment of the present disclosure comprises a media server; a host server; and a user terminal. The media server is configured to identify at least one object from image information, generate a rendering image for the at least one object, transmit the rendering image so that it is displayed on the user terminal, extract at least one event information corresponding to a specific time interval from the image information, and provide the at least one event information to the host server so that it is synchronized according to the specific time interval. The host server is configured to acquire the at least one event information from the media server, generate event data by synchronizing the at least one event information according to the specific time interval, and provide the event data to the user terminal. The user terminal may be configured to acquire the rendering image from the media server, acquire the event data from the host server, and display the rendering image and the event data according to the specific time interval. Effects of the invention

[0024] The present disclosure has the advantage of providing high-quality virtual reality content with a certain level of quality even on HMDs of different performance levels and low-performance client mobile terminals other than PCs, and supporting interaction between audiences even in a streaming environment, by performing rendering on the server side for performance elements and performances requiring high performance and high quality, and transmitting rendered scenes in object units to client-side mobile devices for synthesis and synchronization. Brief explanation of the drawing

[0025] FIG. 1 is a block diagram illustrating the hardware configuration of a performance content provision system according to one embodiment of the present disclosure. FIG. 2 illustrates the detailed configuration of each device constituting a performance content system according to one embodiment of the present disclosure. FIG. 3 illustrates an example of implementation in which a performance content provision system according to one embodiment of the present disclosure performs rendering processing for each performer. FIG. 4 illustrates an implementation example regarding the operation of encoding a performance content providing system according to one embodiment of the present disclosure, a viewpoint texture and a depth texture for each performer. FIG. 5 illustrates an implementation example for a performance content providing system according to one embodiment of the present disclosure to perform decoding of viewpoint textures and depth textures and mesh deformation processing for each performer. FIG. 6 illustrates an example in which audience gesture information is implemented at a user terminal in a performance content providing system according to one embodiment of the present disclosure. FIG. 7 illustrates an example of a performance content provision system according to one embodiment of the present disclosure unfolding performance content over time. FIG. 8 is a flowchart relating to the operation of a performance content provision system according to one embodiment of the present disclosure. Specific details for implementing the invention

[0026] Phrases such as "in some embodiments" or "in one embodiment" appearing in various places in this specification do not necessarily refer to the same embodiment.

[0027] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a specific function. Additionally, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as algorithms executed on one or more processors. Furthermore, the present disclosure may employ prior art for electronic configuration, signal processing, and / or data processing, etc. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

[0028] Furthermore, the connecting lines or connecting members between the components depicted in the drawings are merely illustrative of functional connections and / or physical or circuit connections. In the actual device, connections between components may be represented by various alternative or added functional connections, physical connections, or circuit connections.

[0030] FIG. 1 is a block diagram illustrating the hardware configuration of a performance content provision system according to one embodiment of the present disclosure.

[0031] Specifically, a content delivery system may be a system used to create or provide performance content, and a performance may refer to an artistic act provided to an audience by a performer using their knowledge, skills, or abilities.

[0032] Referring to FIG. 1, a performance content providing system according to one embodiment of the present disclosure may be composed of four main elements. More specifically, the performance content providing system may include a performance rendering server, a media server, a host server, and a user terminal.

[0033] The Performance Rendering Server can utilize high-performance server resources to render on a per-performer basis, encode the rendering into a video format, and transmit it to the Media Server.

[0034] A performer may be, for example, an actor or a musician, and specifically, may refer to at least one of the performers who delivers lines, performs actions, or plays music in a specific scene.

[0035] Rendering at the performer unit level may include an operation to identify at least one performer by distinguishing it from other background elements. Accordingly, the performance rendering server may identify or extract an area corresponding to at least one performer from the performance content, and then perform an individual rendering operation on the identified or extracted area.

[0036] A media server can receive encoded data from a performance rendering server and provide it to a user terminal. Here, the media server may be configured to transmit and receive signals or data from the performance rendering server as a configuration independent of the performance rendering server, or it may be a device included in the performance rendering server.

[0037] The media server can provide encoded data in a manner that provides streaming services to user terminals.

[0038] The user terminal can unpack the results streamed from the media server and visualize them in the form of deformed sprites.

[0039] The user terminal may be a mobile device, including a mobile phone or a head-mount display (HDM), used by an audience member watching the performance. The audience member may acquire performance content through their user terminal. Specifically, the user terminal may acquire and display performance content generated by extraction and reprocessing centered on a specific performer from a media server.

[0040] The host server can acquire transform data such as events, performers, and audience, as well as audience gesture information, from the performance rendering server to synchronize or manage user terminals. The host server may also be referred to as a multi-play host server.

[0042] FIG. 2 illustrates the detailed configuration of each device constituting a performance content system according to one embodiment of the present disclosure.

[0043] Referring to Fig. 2, the Performance Rendering Server can configure a scene using assets (stage, performer, object, etc.). That is, the Performance Rendering Server needs to configure the scene so that the performance content provision system can perform individual server rendering for each performance element, such as the performance stage, performer, and audience space, in advance.

[0044] Here, settings such as rendering quality, resolution, and FPS for the performance stage, group objects, and performers constituting the scene can be performed at this stage. The performance rendering server can process the configured scene into natural movements at the performer level using high-performance server resources and perform rendering individually for various viewpoints. At this time, a process of packing the individual rendering results and depth information into a target video format can be performed. Additionally, source compression suitable for network transmission bandwidth can be performed to pack into a format suitable for transmission and to ensure fast transmission even in various network environments.

[0045] The performance rendering server can transmit compressed video encoding data to the media server. At this time, the compressed encoding data can be transmitted via a wired or wireless network.

[0046] The media server can provide a real-time video service capable of generating live output for video broadcast and streaming transmission from compressed encoded data received from a performance rendering server, and can convert the format and package of real-time video content into other formats and packages.

[0047] The reason for converting the content is to provide a format and package that can be processed by various playback devices, such as mobile devices. Decoding is performed on the converted and received streaming video, and an unpacking process can be performed frame by frame. Through the unpacking process, textures to be applied to individual performers are separated, and the separated textures are textured onto a performer planar mesh similar to a sprite, after which the planar mesh can be deformed using the depth information obtained from the unpacking.

[0048] In parallel, the performance rendering server can connect with the multiplayer host server via RPC (Remote Procedure Call) to share performance event information (performance effects such as fireworks). RPC is an inter-process communication method that enables the execution of remote functions or procedures in a different address space without the need for separate coding for remote control. The multiplayer host server delivers events to the participating mobile clients connected to it, ensuring they are triggered at the same time.

[0049] Here, event information may include information regarding surrounding effects, etc., associated with the performance. For example, event information can be understood to include all recognizable situational information occurring in connection with the performance, such as information about the audience, including audience movements, gestures, responses, and cheers, or special effects occurring during the performance.

[0050] The multi-play host server can also serve as a relay for communication regarding the synchronization of audience avatars on mobile clients. Additionally, transform information resulting from positional changes caused by performer animations can be updated on each mobile client via the multi-play host server. Data such as audience navigation, transform changes, and gesture information can also be relayed and synchronized between mobile client terminals through the multi-play host server.

[0051] Finally, a process can be performed to provide the most natural appearance of the performer even during continuous movement by subtly shifting the LookAt of the performer displayed on the mobile device according to the movement of the audience within the mobile client, and switching the texture state to that viewpoint when approaching a predefined viewpoint.

[0053] FIG. 3 illustrates an example of implementation in which a performance content provision system according to one embodiment of the present disclosure performs rendering processing for each performer.

[0054] Referring to Fig. 3, three camera views (Left, Center, Right) and an additional view (zoom) can be set to look at three performers (left, main, and right actors), and at the start of the performance, the animated appearance of the performers can be rendered from the pre-set camera views and saved as a texture. This process is performed at runtime and is called Render Textures; this may be the reason why server-level resources are required to perform multiple views simultaneously.

[0055] Render textures are a special type of texture that is created and updated at runtime. By utilizing render textures, you can set an area on the canvas and render the scene as seen in the camera view to that area, which has the advantage of using the material's render texture just like a normal texture.

[0056] In addition, depth textures may be generated during this process. A packing process can be performed to align each of these generated textures into a single video frame according to the target video format. An encoding process is then performed on these texture video frames using a video format such as H.264, and the encoded result can be transmitted to a media server via a protocol such as WebRTC.

[0057] WebRTC stands for Web Real-Time Communication, which enables real-time communication on the web and apps (Android, iOS) using cameras, microphones, etc., without the need for software. A WebRTC media server can be described as a server that mediates and distributes WebRTC-based media streams. In particular, platforms such as InstaLive, YouTube Live, and Twitch use RTMP for real-time streaming. However, WebRTC has lower latency than RTMP and enables streaming communication similar to real-time with almost no delay, making it a suitable protocol for the audience participation-enabled performance streaming environment of the present invention.

[0058] The mobile client can perform the process of decoding the video to convert it into a collection of frame-unit textures (Decoding) and splitting it into individual performers (Unpacking). The mobile client can load viewpoint textures for each performer and apply them to the performer's planar mesh. Subsequently, to express the natural appearance of the performers, a process of deforming the planar mesh using depth texture information can be performed. Since the depth information transmitted from the server can be considered as brightness values ​​corresponding to the depth information of the performer for each viewpoint, this process of deforming the vertices within each planar mesh corresponding to these values ​​in the direction of the viewpoint can be described as a form of mesh warping.

[0059] At the same time, performer transform data can be transmitted to the Performance Rendering Server via the Multiplayer Host Server (which performs the role of synchronizing transform data, etc.). Using the performer transform data and transform data based on the audience position, the orientation of the performer mesh best suited for the position is updated (usually set to face the audience; if a transition to a specific viewpoint position (e.g., Left->Center) is required, a texture transition (Left->Center) is also performed).

[0061] FIG. 4 illustrates an implementation example regarding the operation of encoding a performance content providing system according to one embodiment of the present disclosure, a viewpoint texture and a depth texture for each performer.

[0062] Referring to Fig. 4, a total of 12 color render textures and 9 depth textures per viewpoint can be generated according to the three camera viewpoints (Left, Center, Right) and an additional viewpoint (zoom) looking at the three performers (left, main, right actor). At this time, physically based rendering is applied, and data values ​​reflecting the characteristics of the object can be applied to the reflectance BRDF based on the position of the observation point and the light source. Since the 9 depth textures can be considered as single brightness values, they can be assigned to the R, G, and B channels respectively and packed for efficiency to prevent the video format from becoming too large.

[0064] FIG. 5 illustrates an implementation example for a performance content providing system according to one embodiment of the present disclosure to perform decoding of viewpoint textures and depth textures and mesh deformation processing for each performer.

[0065] The performance content delivery system decodes video data received in streaming formats such as H.264 on a frame-by-frame basis. For color render textures, it uses a chroma key shader to make areas outside the performer within the frame transparent, and manages the data separately for each performer. The system separates the performance content into color render textures and depth textures. It enables the color render textures to be individually applied as viewpoint textures for each performer, and for depth textures, it performs the process of generating individual depth textures by unpacking the information assigned to the R, G, and B channels. Based on the audience's relative position to the performer (Left, Center, or Right within the Player Moving Zone), the performance content delivery system applies the viewpoint color textures for each performer to a planar mesh and can modify it using individual depth textures. The reason for performing individual depth textures for each performer is that the shape of the performer mesh viewed from a single point in time is not in a full 3D form; therefore, the system uses the individual depth texture corresponding to that point in time to modify the performer mesh to be as similar as possible to the performer mesh on the server. Through this, the system can perform LookAt correction of the performer mesh based on the audience's subtle positional movements to minimize the possibility that unrestored mesh parts that should not be visible to the audience.

[0067] FIG. 6 illustrates an example in which audience gesture information is implemented at a user terminal in a performance content providing system according to one embodiment of the present disclosure.

[0068] In order to express opinions among the audience and the level of response to the performance, the performance content delivery system needs to relay and synchronize the audience's gestures. The audience can select a gesture they wish to express through their user terminal via the UI exemplified in Fig. 6 or select a dance suitable for the performance, thereby allowing the audience avatar to perform the corresponding action and synchronize it with other audience members' terminals through a multi-player host server.

[0070] FIG. 7 illustrates an example of a performance content provision system according to one embodiment of the present disclosure unfolding performance content over time.

[0071] Each audience member participating in the multi-host server can wait in a specific waiting room referred to as the Lobby, and the system can provide a content service in which various effects (bubble map, change effect, whale effect, etc.), including intros and scene changes, are synchronized and unfolded.

[0073] FIG. 8 is a flowchart relating to the operation of a performance content provision system according to one embodiment of the present disclosure.

[0074] In the following, operations described as being performed by the system may also be understood as operations performed by each device constituting the system, such as a media server, a user terminal, etc.

[0075] In step (S110), the system identifies at least one object from the image information.

[0076] More specifically, the system can process and analyze image information.

[0077] In addition, the system can capture image information, convert the image into a digital format, perform preprocessing and feature extraction, and identify objects contained in the image.

[0078] In addition, the system can use various techniques such as machine learning, deep learning, image segmentation, and feature matching to identify objects in an image. The system can also store object information identified by an object identification module.

[0079] A system according to one embodiment of the present disclosure may further perform functions such as identifying multiple objects in an image, identifying objects under various image types and various lighting conditions, learning and improving over time, providing real-time object identification, and working with various types of image sources such as video or live camera feeds and various formats such as text or voice.

[0080] Here, the object may include performers participating in the performance. More specifically, the performers may include actors and guests who constitute the performance content. In addition to performers, backgrounds, props, etc., may be included in the object of the present disclosure.

[0081] The system can identify areas of interest for performance creators or audiences by distinguishing and identifying objects from other components.

[0082] In step (S120), the system generates a rendering image for at least one object.

[0083] The system can acquire or identify data describing an object, such as shape, color, and texture, and generate a 3D representation of the object based on said data. Subsequently, the system can apply a rendering algorithm to the 3D representation to generate a final image that realistically represents the object. The rendered image can be generated as information that can be output to a display for viewing.

[0084] In step (S130), the system transmits the rendered image so that it is displayed on the user terminal.

[0085] Here, the rendered image can be transmitted to a user terminal, such as a computer, smartphone, or tablet, and displayed on the screen. This operation may include transmitting the rendered image to the user terminal over a network, such as the Internet. This operation can be used in various applications, such as remote visualization, remote collaboration, or remote rendering. Through this, the user can access the rendered image via a network connection.

[0086] Rendered images transmitted to a user terminal can be encoded into a predetermined format for streaming. In this case, the encoding operation can be performed individually on different devices constituting the system.

[0087] In step (S140), the system extracts at least one event information corresponding to a specific time interval from the image information.

[0088] Here, event information refers to information related to the performance that is not the performance content itself. More specifically, event information may include audience information or background information. For example, it may be information regarding special effects used in the performance (e.g., fireworks) or audience response. Event information can be understood to encompass all immersive experiences obtained by audience members who participate offline at the performance site.

[0089] Event information can be extracted from image information. However, in addition to this, the system can extract or obtain event information based on sound information or a predetermined database.

[0090] Event information may correspond to a specific time interval. More specifically, event information may correspond to at least one of the time intervals of the performance content itself or other event information. For example, if a fireworks event occurs at the time a performer appears, the fireworks event and the time of the performer's appearance may correspond. Additionally, if the audience cheers during the fireworks event, the fireworks event and the time of the audience's cheering may correspond.

[0091] In step (S150), the system provides at least one event information to the host server so as to be synchronized according to the specific time interval.

[0092] The system can provide synchronized event information to a user terminal so that a user utilizing the performance content can realistically experience both the event information and the performance content. Specifically, the system provides extracted event information to a host server for synchronization, and the host server can synchronize the event information and the performance content for a specific time interval and then transmit it to the user terminal.

[0093] The host server performs real-time time synchronization of event information, enabling users to experience generated events immediately.

[0095] A performance content provision system according to one embodiment of the present disclosure provides the effect of enabling a user to experience not only high-quality content but also time-synchronized event information in a smooth environment by rendering high-definition performance content through a separate media server and providing it to a user terminal, and performing time synchronization of event information through a separate host server and providing it to a user terminal.

[0097] The embodiments of the present invention described above are not implemented only through devices and methods, but may also be implemented through a program that realizes a function corresponding to the configuration of the embodiments of the present invention or a recording medium on which such program is recorded.

[0098] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.

[0099] The above descriptions are specific embodiments for carrying out the present disclosure. The present disclosure will include not only the embodiments described above, but also embodiments that are simply modified or can be easily modified. Furthermore, the present disclosure will include technologies that can be easily modified and implemented using the embodiments. Accordingly, the scope of the present disclosure should not be limited to the embodiments described above, but should be defined by the claims set forth below as well as equivalents to the claims of this invention.

Claims

Claim 1 A method of operation of a rendering server comprising: identifying at least one object from image information; generating a rendering image for the at least one object; transmitting the rendering image so that it is displayed on a user terminal; extracting at least one event information corresponding to a specific time interval from the image information; and providing the at least one event information to a host server so as to be synchronized according to the specific time interval, wherein the image information is extracted from performance content, the at least one object is at least one performer participating in the performance content, the at least one event information includes transform data for at least one of background information and audience information extracted from the image information, the background information includes information for at least one of special effects, animations, and objects of the image information, the audience information includes information for at least one of location, gestures, and interactions of a user of the user terminal, and the host server transmits the transform data to the user terminal so as to synchronize the event information and the performance content with respect to the specific time interval. Claim 2 delete Claim 3 delete Claim 4 A method according to claim 1, wherein at least one event information is provided to the host server via a Remote Procedure Call (RPC). Claim 5 A method according to claim 1, comprising: a step of obtaining depth information corresponding to the rendering image; a step of generating encoded data packed according to a predetermined format based on the rendering image and the depth information; and a step of providing the encoded data to a media server. Claim 6 A method according to claim 1, further comprising the step of obtaining location information of the user terminal, wherein at least one event information is synchronized according to the location information. Claim 7 delete Claim 8 A device for a rendering server comprises: a transmitting and receiving unit; and at least one control unit operably connected to the transmitting and receiving unit, wherein the at least one control unit is configured to identify at least one object from image information, generate a rendering image for the at least one object, transmit the rendering image so that it is displayed on a user terminal, extract at least one event information corresponding to a specific time interval from the image information, and provide the at least one event information to a host server so as to be synchronized according to the specific time interval, wherein the image information is extracted from performance content, and the at least one object is at least one performer participating in the performance content, and the at least one event information includes transform data for at least one of background information and audience information extracted from the image information, wherein the background information includes information for at least one of special effects, animations, and objects of the image information, and the audience information includes information for at least one of location, gestures, and interactions of a user of the user terminal, and wherein the host server transmits the transform data to the user terminal so as to synchronize the event information and the performance content with respect to the specific time interval. Claim 9 delete Claim 10 delete Claim 11 In claim 8, the device wherein at least one event information is provided to the host server via a Remote Procedure Call (RPC). Claim 12 In claim 8, the device is further configured such that at least one control unit acquires depth information corresponding to the rendering image, generates encoding data packed according to a predetermined format based on the rendering image and the depth information, and provides the encoding data to a media server. Claim 13 A device according to claim 8, wherein the at least one control unit is further configured to acquire location information of the user terminal, and the at least one event information is synchronized according to the location information. Claim 14 delete Claim 15 In a system providing online performance services, a media server; a host server; The media server is configured to identify at least one object from image information, generate a rendering image for the at least one object, transmit the rendering image so that it is displayed on the user terminal, extract at least one event information corresponding to a specific time interval from the image information, and provide the at least one event information to a host server so that it is synchronized according to the specific time interval; the host server is configured to obtain the at least one event information from the media server, generate event data by synchronizing the at least one event information according to the specific time interval, and provide the event data to the user terminal; the user terminal is configured to obtain the rendering image from the media server, obtain the event data from the host server, and display the rendering image and the event data according to the specific time interval; the image information is extracted from performance content, and the at least one object is at least one performer participating in the performance content; the at least one event information includes transform data for at least one of background information and audience information extracted from the image information; the background information includes information for at least one of special effects, animations, and objects of the image information; and the audience information is the A system comprising information on at least one of the location, gesture, and interaction of a user of a user terminal, wherein the host server transmits the transform data to the user terminal to synchronize the event information and the performance content with respect to the specific time interval.

Images