Server, method, and computer program

Abstract

SOLUTION: In a live streaming system, a server includes a circuitry that is configured to perform: generating an emulator in response to a request from a first user terminal of a first user; launching an application via the emulator; receiving streaming data and interaction data via the application; rendering the streaming data with the interaction data; recording the rendered streaming data and interaction data as a clip; and storing the clip for access from the first user terminal of the first user.EFFECT: The clips may be generated in a more efficient and accurate manner, and a more immersive experience on watching the clips may be provided. Moreover, the review and share of clips may be more flexible. Therefore, the user experience may be improved.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present disclosure relates to information and communication technologies, and more particularly to a server, a method, and a computer program in live streaming.

Background Art

[0002] Some applications and platforms provide live streaming services that allow live streamers and viewers to interact. A live streamer may perform a performance to cheer for viewers, or a viewer may send a gift to support the live streamer.

[0003] Viewers may clip the highlight parts for later review. Also, with clips, users can capture and share the most unique moments from a broadcast. Non-Patent Document 1 discloses a method for creating, editing, and sharing clips from a broadcast.

[0004] However, in order to provide a more immersive experience when viewing clips, further improvement of current archive services is needed. Therefore, a more efficient and accurate archive service is required.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

[0006] A server according to an embodiment of the present disclosure is a server including a circuit, wherein the circuit is configured to perform functions of generating an emulator in response to a request from a first user terminal of a first user, starting an application via the emulator, receiving streaming data and interaction data via the application, rendering the streaming data with the interaction data, recording the rendered streaming data and interaction data as a clip, and saving the clip so that it can be accessed from the first user terminal of the first user.

[0007] A method according to another embodiment of the present disclosure includes steps of generating an emulator in response to a request from a first user terminal of a first user, starting an application via the emulator, receiving streaming data and interaction data via the application, rendering the streaming data with the interaction data, recording the rendered streaming data and interaction data as a clip, and saving the clip so that it can be accessed from the first user terminal of the first user.

[0008] A computer program according to another embodiment of the present disclosure causes a server to realize functions of generating an emulator in response to a request from a first user terminal of a first user, starting an application via the emulator, receiving streaming data and interaction data via the application, rendering the streaming data with the interaction data, recording the rendered streaming data and interaction data as a clip, and saving the clip so that it can be accessed from the first user terminal of the first user.

[0009] According to the present disclosure, there is a possibility that the clip is generated more efficiently and accurately, and a more immersive experience is provided when viewing the clip. Furthermore, there is a possibility that the clip can be reviewed and shared more flexibly. Therefore, the user experience may be improved.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0011] Hereinafter, the same or similar components, members, procedures, or signals shown in each drawing are denoted by the same reference numerals in all the drawings, and redundant descriptions are appropriately omitted. Also, some members that are not important in the description of each drawing are omitted.

[0012] The live streaming system 1 based on some embodiments of the present disclosure provides an enhanced function for smoothly communicating and interacting between users. More specifically, it is something that makes viewers and live streamers enjoy themselves in a technical way.

[0013] FIG. 1 shows a schematic diagram showing the configuration of a live streaming system 1 according to some embodiments of the present disclosure. The live streaming system 1 provides a live streaming service for a live streamer (also called a liver, streamer, or distributor) LV and viewers (also called an audience) AU (AU1, AU2...) to communicate with each other in real time. As shown in FIG. 1, the live streaming system 1 can include a server 10, a user terminal 20, and user terminals 30 (30a, 30b...). The user terminal 20 may be a live streamer, and the user terminal 30 may be a viewer. In some embodiments, the live streamer and the viewer may be called users. The server 10 can include one or more information processing devices connected via a network NW. The user terminals 20 and 30 may be, for example, mobile terminals such as smartphones, tablets, notebook PCs, recorders, portable game machines, wearable terminals, or stationary computers such as desktop PCs. The server 10, the user terminal 20, and the user terminal 30 may be communicably connected by any type of wired or wireless network NW.

[0014] An app provider (not shown) that provides the live streaming system 1, the live streamer LV, the viewer AU, and the server 10 is involved. The live streamer LV can be a person who records content such as their own songs, talks, performances, game streaming, etc. on their own user terminal 20, uploads it to the server 10, and distributes the content in real time. In some embodiments, the live streamer LV can communicate with the viewer AU via the live streaming.

[0015] The app provider can provide a platform for the content to be live-streamed on the server 10. In some embodiments, the app provider may be a media or manager that manages the real-time communication between the live streamer LV and the viewer AU. The viewer AU can access the platform through the user terminal 30 and select and view the content they want to watch. The viewer AU can perform operations for interacting with the live streamer, such as commenting on or cheering for the live streamer, through the user terminal 30. The live streamer providing the content can respond to the comments and cheers. The response of the live streamer can be sent to the viewer AU by means of video and / or audio, etc. Therefore, the mutual communication between the live streamer and the viewer can be achieved.

[0016] As used herein, "live streaming" can refer to data transmission that enables a viewer AU to substantially play and view content recorded by a live streamer LV through a user terminal 30 via a user terminal 20. In some embodiments, "live streaming" may refer to streaming realized by the above-described data transmission. The live streaming can be realized by known technologies such as HTTP live streaming, CMAF (Common Media Application Format), WebRTC (Web Real-Time Communications), RTMP (Real-Time Messaging Protocol), and MPEG DASH. The live streaming may further include an embodiment in which the viewer AU can play or view the content with a specific delay while the live streamer is recording the content. The degree of the delay is preferably small enough that at least the live streamer LV and the viewer AU can communicate with each other. However, live streaming is different from so-called on-demand distribution. More specifically, the on-demand distribution may refer to storing all data of the recorded content in a server and providing the data from the server to the user at a random timing in response to a user request.

[0017] As used herein, "streaming data" can refer to data including image data and audio data. More specifically, the image data (which may also be referred to as video data) may be generated by the image capture functions of the user terminals 20 and 30. The audio data (which may also be referred to as audio data) may be generated by the audio input functions of the user terminals 20 and 30. The streaming data may be played back on the user terminals 20 and 30 so that the user can view content related to the user. In some embodiments, between the generation of the streaming data on the user terminal of the live streamer and the playback on the user terminal of the viewer, processing such as compression, decompression, encoding, decoding, transcoding, etc., to change the data format and size specifications is assumed. Before and after such processing, the content (video and audio) is not substantially changed. Therefore, in the current embodiments of the present disclosure, it is explained that the streaming data before processing and the streaming data after processing are the same. That is, when the streaming data generated by the user terminal of the live streamer is played back on the user terminal of the viewer via the server 10, the streaming data generated by the user terminal of the live streamer, the streaming data that has passed through the server 10, and the streaming data received and played back by the user terminal of the viewer are all the same streaming data.

[0018] As shown in FIG. 1, the live streamer LV provides live streaming. The user terminal 20 of the live streamer generates streaming data by recording the video and / or audio of the streamer, and transmits it to the server 10 via the network NW. At the same time, the user terminal 20 can display the video VD on the display of the user terminal 20 and check the streaming content of the live streamer LV.

[0019] Viewers AU1 and AU2 of user terminals 30a and 30b that request the platform to provide the live streaming of the live streamer can receive streaming data corresponding to the live streaming via the network NW, play the received streaming data, display videos VD1 and VD2 on the display, and output audio from speakers and the like. The videos VD1 and VD2 respectively displayed on the user terminals 30a and 30b are substantially the same as the videos recorded by the user terminal of the live streamer LV, and the audio output from the user terminals 30a and 30b is substantially the same as the audio recorded by the user terminal of the live streamer LV.

[0020] The recording at the user terminal 20 of the live streamer may be simultaneous with the playback of the streaming data at the user terminals 30a and 30b of the viewers AU1 and AU2. When viewer AU1 inputs a comment regarding the content of live streamer LV into user terminal 30a, server 10 displays the comment in real time on the user terminal 20 of the live streamer and also on the user terminals 30a and 30b of viewers AU1 and AU2 respectively. When the live streamer LV responds to the comment, the response is output as text, image, video, or audio from the user terminals 30a and 30b of viewers AU1 and AU2, enabling communication between the live streamer LV and viewers AU1 and AU2. Therefore, the live streaming system can realize two-way communication live streaming.

[0021] FIG. 2 is a block diagram showing the functions and configuration of the user terminal 20 shown in FIG. 1 according to an embodiment of the present disclosure. The user terminal 30 has the same functions and configuration as the user terminal 20. The blocks depicted in the block diagrams in this specification represent devices such as a computer's CPU, hardware such as mechanical parts, and software such as a computer program that represents functional blocks implemented by the cooperation of these elements. Therefore, those skilled in the art will understand that the functional blocks can be implemented in various ways by a combination of hardware and software.

[0022] The live streamer LV and the viewer AU can download and install the live streaming application (live stream app) of the present disclosure from a download site to the user terminals 20 and 30 via the network NW. Alternatively, the live stream app may be pre-installed on the user terminals 20 and 30. By executing live streaming by the user terminals 20 and 30, the user terminals 20 and 30 can communicate with the server 10 via the network NW and realize a plurality of functions. The functions realized by executing the live stream app by the user terminals 20 and 30 (more specifically, a processor such as a CPU) will be described below as the functions of the user terminals 20 and 30. The functions are basically the functions that the live stream app causes the user terminals 20 and 30 to realize. In some embodiments, these functions may be transmitted from the server 10 to the web browsers of the user terminals 20 and 30 via the network NW and realized by being executed by a computer program of the web browser. The computer program may be written in a programming language such as HTML (Hyper Text Markup Language).

[0023] The user terminal 20 includes a streaming unit 100 and a viewing unit 200. In some embodiments, the streaming unit 100 is configured to record the user's audio and / or video data and generate streaming data to be transmitted to the server 10. The viewing unit 200 is configured to receive and play the streaming data from the server 10. In some embodiments, the user can operate the streaming unit 100 during a broadcast or operate the viewing unit 200 when viewing the stream. In some embodiments, the user terminal that operates the streaming unit 100 can be referred to as a live streamer or the user terminal that generates the streaming data. The user terminal that operates the viewing unit 200 can be referred to as a viewer or the user terminal that plays the streaming data.

[0024] The streaming unit 100 can include a video control unit 102, an audio control unit 104, a distribution unit 106, and a UI control unit 108. The video control unit 102 may be connected to a camera (not shown), and the video is controlled by the camera. The video control unit 102 can obtain the video data from the camera. The audio control unit 104 may be connected to a microphone (not shown), and the audio is controlled by the microphone. The audio control unit 104 can obtain the audio data from the microphone.

[0025] When the distribution unit 106 receives streaming data including video data from the video control unit 102 and audio data from the audio control unit 104, it transmits the data to the server 10 via the network NW. In some embodiments, the distribution unit 106 transmits the streaming data in real time. That is, the generation of the streaming data from the video control unit 102 and the audio control unit 104 and the distribution by the distribution unit 106 are executed simultaneously.

[0026] The UI control unit 108 controls the UI of the live streamer. The UI control unit 108 is connected to a display (not shown) and is configured to generate the streaming data for the partner to whom the distribution unit 106 transmits and plays the streaming data for display on the display. The UI control unit 108 is configured to display an object to be operated or an object to receive an instruction on the display and to receive a tap input from the live streamer.

[0027] The viewing unit 200 may include a UI control unit 202, a rendering unit 204, and an input transmission unit 206. The viewing unit 200 is configured to receive streaming data from the server 10 via the network NW. The UI control unit 202 controls the UI of the viewer. The UI control unit 202 is connected to a display (not shown) and / or a speaker (not shown) and is configured to display video on the display and output audio from the speaker by playing the streaming data. In some embodiments, displaying video on the display and outputting audio from the speaker may be referred to as "playing the streaming data". The UI control unit 202 is connected to an input unit such as a touch panel, a keyboard, or a display and can obtain an input from the user.

[0028] The rendering unit 204 may be configured to render the streaming data from the server 10 and the frame image. The frame image may include an input from the user, a comment input by the viewer, and a user interface object for receiving data received from the server 10. The input transmission unit 206 is configured to receive the user input from the UI control unit 202 and transmit it to the server 10 via the network NW.

[0029] In some embodiments, the user input may be, for example, the selection of a live stream, the input of a comment, the sending of a gift, the following or unfollowing of a user, the voting in an event, a game, etc., by clicking on an object on the screen of the user terminal. For example, when the user terminal of the viewer clicks on a gift object on the screen to send a gift to the live streamer, the input transmission unit 206 may generate gift information and transmit it to the server 10 via the Internet NW.

[0030] FIG. 3 is a block diagram of a server 10 according to some embodiments of the present disclosure. The server 10 may include a streaming information unit 302, a relay unit 304, a recording unit 306, a processing unit 308, a context unit 310, a clip unit 312, a stream DB 320, a user DB 322, an archive DB 324, a context DB 326, and a clip DB 328.

[0031] When the streaming information unit 302 receives a live streaming request from the user terminal 20 of the live streamer via the network NW. Upon receiving the request, the streaming information unit 302 registers the information of the live streaming in the stream DB 320. In some embodiments, the information of the live streaming may be the stream ID of the live streaming and / or the live streamer ID of the live streamer corresponding to the live streaming.

[0032] When the streaming information unit 302 receives a request for providing the information of the live streaming from the viewing unit 200 of the user terminal 30 from the viewer via the network NW, the streaming information unit 302 refers to the stream DB 320 and generates a list of available live streamings.

[0033] Thereafter, the streaming information unit 302 transmits the list to the user terminal 30 via the network NW. The UI control unit 202 of the user terminal 30 generates a live streaming selection screen based on the list and displays the list on the display of the user terminal 30.

[0034] When the input transmission unit 206 of the user terminal 30 receives the selection of a live streaming by the viewer on the live streaming selection screen, it generates a delivery request including the stream ID of the selected live streaming and transmits it to the server 10 via the network. The streaming information unit 302 can start providing the live streaming specified by the stream ID in the delivery request to the user terminal 30. The streaming information unit 302 can update the stream DB 320 and add the viewer ID of the viewer of the user terminal 30 to the live streamer ID of the stream ID.

[0035] In the live streaming started by the streaming information unit 302, the relay unit 304 can relay the transmission of the live streaming from the user terminal 20 of the live streamer to the user terminal 30 of the viewer. During the playback of the streaming data, the relay unit 304 can receive a signal indicating a user input from the viewer from the input transmission unit 206. The signal indicating the user input may be an object designation signal indicating the designation of an object displayed on the display of the user terminal 30. The object designation signal may include the viewer ID of the viewer, the live streamer ID of the live streamer who distributes the live streaming being viewed by the viewer, and the object ID designated by the object. When the object is a gift or the like, the object ID may be a gift ID or the like. Similarly, during the playback of the streaming data, the relay unit 304 can receive a signal indicating a user input of the live streamer, such as the object designation signal, from the streaming unit 100 of the user terminal 20.

[0036] The recording unit 306 may be configured to record the live streaming. In some embodiments, the recording unit 306 may automatically or manually record the live streaming based on the settings of the user terminal 20 of the streamer. For example, before starting the live streaming, the streamer may turn on the automatic archiving function to automatically record and archive the live streaming. The recording unit 306 may start recording the live streaming when the streamer starts the distribution and stop recording when the live streaming ends. Also, in some embodiments, the streamer or the viewer may manually clip the live streaming during the live streaming.

[0037] In some embodiments, the recording unit 306 may record the live streaming for a maximum period of the archival content, such as 8 hours. For example, the recording unit 306 may record the live streaming for a period such as up to 8 hours. If the duration of the live streaming is less than 8 hours, the recording unit 306 may record and archive the live streaming. However, if the live streaming is longer than 8 hours, the recording unit 306 may record the live streaming for 8 hours and further start another recording of the next 8 hours or less.

[0038] In some embodiments, the archival content of the live streaming may be stored in a storage such as Google Cloud, and the data of the archival content may be registered in the archival DB 322 for reference and further processing. In some embodiments, any possible transmission protocol such as HTTP Live Streaming (HLS) may be applied between the server and the user terminal. In some embodiments, the recording unit 306 may receive the live streaming from a streaming source and record the live streaming as archival content.

[0039] Different transmission protocols may be used during the communication of the live streaming. Here, HTTP Live Streaming (HLS) will be taken as an example for explanation. HTTP Live Streaming (HLS) is an HTTP-based streaming media network transmission protocol proposed by Apple. HLS is composed of an M3U8 segment index file and a transport stream (TS) segment.

[0040] The M3U8 file may also be referred to as an index file of ts segments and may be used to store the download addresses of ts segments on the server. The user terminal can sequentially read out the ts segments based on the M3U8 file. The ts segment may also be called a video clip obtained by splitting the entire video file or video stream. Each ts segment may include a plurality of frames of video. One ts segment may be about 1 to 2 seconds, for example. Each ts segment may include one or more GOP (Group Of Picture) structures including a plurality of frames of video. One GOP may include different frame types such as I frames, P frames, and B frames. The SEI message can be inserted into and saved in the I frame.

[0041] HLS splits the entire audio and video stream into small HTTP-based files for download, and only a part of the file may be downloaded each time. When playing the media stream, the viewer may choose to download the same resource from many different alternative sources at different rates so that the streaming media session adapts to different data rates.

[0042] Different streaming sources may provide different formats of streaming data such as FLV or M3U8. The recording unit 306 may receive the streaming data from the streaming source and convert and process the format of the streaming data into any type of format. The recording unit 306 may record and store the archive content as streaming data in any possible format. Here, an M3U8 playlist using ts segments will be taken as an example for explanation. The recording unit 306 may record the streaming data of the live streaming as ts segments for the processing unit 308 to check and store in the storage.

[0043] The processing unit 308 may be configured to process the archived content. In some embodiments, the processing unit 308 may check the archived content and store it as a transport stream (TS) segment. The processing unit 308 may further generate an M3U8 playback list for the TS segment. In some embodiments, the processing unit 308 may hold the archived content in storage and the database for a specific period, such as 7 days, 14 days, etc. Thereby, the memory is used efficiently.

[0044] The context unit 310 may be configured to handle the context of interaction information. Here, the context may refer to a rendering context. The context may include interaction information from the viewer. For example, the viewer may send a message or a gift to the streamer, and the context may include information such as messages, gift messages, animations, gift animations, etc. The context unit 310 may receive interaction information from the viewer and store the interaction information in the context DB 324. In some embodiments, the context unit 310 may include a plurality of working units for processing contexts from different users.

[0045] In some embodiments, the context unit 310 may receive interaction information from viewers via a backend server. In some embodiments, the context unit 310 may receive interaction information from a streamer terminal via a backend server. For example, the streamer may send a message to the viewer. In some embodiments, the context unit 310 may directly receive interaction information from the backend server. For example, the backend server may send a message notifying that a VIP user is online and participating in the live streaming, and the context may include interaction information of the VIP online notification. In some embodiments, the backend server may send interaction information including an identifier such as UTC time information. In some embodiments, the context unit 310 may receive any type of interaction information such as text, image, animation, notification, etc. during the live streaming.

[0046] The clip unit 312 may be configured to process clip content. In some embodiments, the clip unit 312 may obtain streaming data and interaction data and send them to the ECM for rendering and recording. Individually, the interaction data may be any possible interactive information in the live streaming room, such as messages, comments, gifts, follows, games, VIP online notifications, entrance animations, etc.

[0047] In some embodiments, the streaming data and the interaction data may be archive data and context data from the archive DB 324 and the context DB 326. In some embodiments, the streaming data and the interaction data may be, for example, live streaming data or live interaction data from a streaming server, a backend server, etc. That is, the streaming data and the interaction data may be archive data and context data from the archive DB 324 and the context DB 326, or from real-time live streaming.

[0048] In some embodiments, the clip unit 312 may store the clip content in a storage and a database. Also, in some embodiments, the clip unit 312 may generate a link to the clip content. In some embodiments, the clip content may be stored permanently or deleted after a certain period such as 7 days, 14 days, etc. In some embodiments, the setting of the clip content may be flexibly determined according to actual needs.

[0049] FIG. 4 is a table showing an exemplary data structure of the stream DB 320 of FIG. 3. The stream DB 320 holds information regarding a currently ongoing live stream. The stream DB 320 stores a stream ID for identifying a live stream on the live streaming platform provided by the live streaming system 1, a live streamer ID for identifying the live streamer providing the live stream, and a viewer ID for identifying the viewer of the live stream, in association with each other.

[0050] FIG. 5 is a table showing an exemplary data structure of the user DB 322 in FIG. 3. The user DB 322 holds information about users. The user DB 322 stores, in association with each other, a user ID for identifying a user, a point for specifying points accumulated by the user, a level for identifying the level of the user, and a status for identifying the status of the user. The point is an electronic value that circulates within the live streaming platform. The level may be an indicator of the amount of activity or engagement of the user in the live streaming platform. The status may be the identity or membership status of the user in the live streaming platform.

[0051] FIG. 6 shows a table showing an exemplary data structure of the archive DB 322 in FIG. 3. The archive DB 322 holds information about archive content. The archive DB 322 stores, in association with each other, an archive ID for identifying the archive content, a timestamp for identifying the time information of the archive content, a duration for identifying the duration of the archive content, and an archive URL for identifying the location of the archive content.

[0052] FIG. 7 shows an exemplary data structure of the context DB 324 in FIG. 3. The context DB 324 holds information about context data. The context DB 324 stores, in association with each other, a context ID for identifying the context data, a type for identifying the type of the context data, and a timestamp for identifying the time information of the context. In some embodiments, any type of interaction information from a live streamer terminal, a viewer terminal, or a backend server, such as a message, a comment, a gift, a follow, a game, a VIP online notification, etc., may be queried from the context unit 310.

[0053] FIG. 8 is a table showing an exemplary data structure of the clip DB328 of FIG. 3. The clip DB328 holds information regarding clip content. The clip DB328 stores, in association with each other, a clip ID that identifies the clip content, a duration that identifies the duration of the clip content, and a clip URL that specifies the location of the clip content. In some embodiments, the clip DB328 may further include information regarding archive data and context data, such as, for example, an archive ID, a context ID, and the like.

[0054] FIG. 9 is an exemplary functional structure of the live streaming system 1 according to some embodiments of the present disclosure. As shown in FIG. 9, the streamer may turn on the archive function in the application. In some embodiments, the streamer may switch an icon of the archive function to indicate that it desires to archive the live streaming. The backend may notify the archive control manager to register an archive task for the streamer. In some embodiments, the backend may further notify the streamer that the setup of the archive function has been successful, such as by messaging.

[0055] The streamer may further start live streaming by pushing streaming data to the streaming server. In some embodiments, the streamer may start live streaming by means of a mobile terminal such as a smartphone. In some embodiments, the streamer may start the live streaming by software on a computer such as OBS (Open Broadcaster Software). The archive service may be applied to any type of streaming method from the streamer. The recording unit 306 may extract the live streaming for recording. In some embodiments, the recording unit 306 may insert an identifier into the frame of the archived video while recording the live streaming. For example, the identifier may be inserted into the SEI (Supplemental Enhancement Information) of the frame of the archived video respectively. Therefore, the frame of the archived video may include identifier information.

[0056] In some embodiments, the duration of the ts segment may be 1 second, and the ts segment may include one GOP structure. One GOP structure may include one I frame and several P frames. The I frame may be a main frame, and the P frame may be an auxiliary frame. For example, the I frame may include the main frame information of the original frame of the live streaming, and the P frame may include some auxiliary information such as the difference between the current I frame and the previous I frame. According to this embodiment, the transmission amount can be reduced.

[0057] In some embodiments, one TS segment may include one GOP having one I-frame, which may be used to insert an SEI message having an identifier. That is, one TS segment may include one identifier such as UTC time information. In some embodiments, one TS segment may include more GOPs having more I-frames by recording or connection, so one TS segment may include multiple UTC time information and the like.

[0058] SEI can be referred to as text data inserted into an audio and video bitstream to convey additional information. SEI is a standard NAL (Network Abstraction Layer) in H.264 video coding and can include various types of data describing various characteristics of the video. Also, the SEI message can include any user-defined data. Further, the SEI message can indicate how the video is recommended to be post-processed or displayed without affecting the core decoding process.

[0059] In some embodiments, the identifier may be included in the SEI message. In some embodiments, the identifier may include a UUID (Universal Unique Identifier) dedicated to archival video. The identifier may be stored in the frame of the archival video. In some embodiments, the identifier may be a unique series of numbers, letters, text, symbols, combinations thereof, and the like. In some embodiments, the identifier may distinguish the frames of the video or the TS segments of the video from each other. That is, the identifier of the frame of the video or each TS segment of the video may be unique and different from others.

[0060] In some embodiments, the identifier may be the time information of the frame of the archived video. In some embodiments, the time information may be relative time. For example, the time information may indicate the time length of the frame based on the start of the archived video, or the time length of the frame based on the previous frame of the archived video, etc.

[0061] In some embodiments, the time information may be the absolute time information of the live streaming. For example, while recording the live streaming as an archived video, the recording unit 306 may save the absolute time of each frame. In some embodiments, the absolute time may be a standard time such as Greenwich Mean Time (GMT), Coordinated Universal Time (UTC), etc. For example, when a streamer starts a live streaming from 2022-08-05T06:00:00Z to 2022-08-05T07:00:00Z, the recording unit 306 may write a UTC timestamp to the SEI message of the frame of the archived video and save the SEI message to the corresponding frame of the archived video.

[0062] In some embodiments, the identifier may be a tag tagged by the recording unit 306 during recording. For example, the recording unit 306 may add a tag to the frame of the archived video. The backend server may further send a context having the corresponding tag to the context unit 310. Therefore, when an archived viewer queries a context having the tag, the context unit 310 may send the context having the corresponding tag to the archived viewer. In some embodiments, the identifier may be a unique code that identifies the frames of the archived content from each other and connects each frame to the corresponding context. In some embodiments, the code may be generated by the server 10 or another third-party server, etc.

[0063] In some embodiments, AI tagging technology may be applied to insert tags into each frame of the archived video. Here, AI tagging may refer to the process of tagging media files with metadata using artificial intelligence. For example, the recording unit 306 may automatically add tags with specific functions, and the backend server may generate a context with corresponding tags. The AI tagging may detect audio, video, text, images, animations, etc. from streamers or viewers, and generate tags to be inserted into the corresponding frames of the archived video. For example, when a streamer is talking about a car, an AI tag may be inserted during the conversation. The backend server may further generate information such as "the conversation about the car is in progress" or an advertisement video of the car, insert the corresponding tags, and send them to the context unit 310. Thereby, the archived video can better understand the topic that the streamer was talking about. Also, if a viewer fails to hear the key points of the streamer's speech, the viewer may access the archived video with more information displayed on the screen.

[0064] In some embodiments, the AI tagging may be applied based on the location information or time information of the streamer or viewer. For example, when a streamer is broadcasting from a café, the recording unit 306 may insert a tag indicating the location of the café. The archived viewer may receive the information of the café while checking the archived content. Further, the recording unit 306 may insert tags periodically, such as every hour, and generate a context of a message with corresponding tags to indicate the time the archived viewer has watched or to notify the archived viewer to take a break.

[0065] In some embodiments, the viewer may pull the live streaming from the streaming server to watch the live streaming. The viewer may communicate (interact) with the streamer through comments, gifts, etc., and the interaction information may be sent to the streamer or other viewers via the backend server. In some embodiments, the backend server may further send the interaction information to the context unit 310. The context unit 310 may include one or more working units for processing the interaction information. For example, the working unit may write the interaction information into the context DB 324.

[0066] In some embodiments, when the backend server sends the interaction information to the context unit 310, the interaction information may also include an identifier. More specifically, the identifier may be the time information for the backend server to receive the interaction information. In some embodiments, the time information may be relative time or absolute time such as GMT time, UTC time, etc. For example, the viewer may send a message to the streamer via the backend server, and the backend server may save the time information as an identifier in the interaction information. In some embodiments, the context unit 310 may further receive the interaction information having an identifier and write it into the context DB 324.

[0067] In some embodiments, another viewer's user terminal (which may also be referred to as an archive viewer) may request archive content from the archive service. The processing unit 308 may generate an M3U8 playback list for the ts segment. The M3U8 playback list may include a list of ts segments and identifier information for each frame or ts segment. The identifier information may include, for example, UTC information for each frame or ts segment respectively.

[0068] When the archive viewer plays the archive content, the user terminal of the archive viewer may analyze the SEI message in the frame of the archive video and obtain the identifier information including the UTC information. In some embodiments, the user terminal of the archive viewer may further query the context unit 310 about the corresponding context based on the UTC information. In some embodiments, the user terminal of the archive viewer may query the context based on the time point that the archive viewer wants to play. For example, the archive viewer may play the archive video from the beginning of the archive video or from a specific time point of the archive video. The user terminal of the archive viewer may analyze the SEI message in the frame of the archive content that the archive viewer wants to play and obtain the identifier information. For example, if the archive viewer plays from the beginning of the archive video and the UTC information is 2022-08-05T06:00:00Z, the user terminal of the archive viewer may query the context having the time stamp of the UTC information.

[0069] In some embodiments, the rendering unit 204 may render archive content having a context of interaction information based on the identifier information. For example, the rendering unit 204 may render a frame of the archive video in the context of interaction information based on UTC information. Thus, the archive viewer may view the archive video containing all the interaction information. In some embodiments, the interaction information may be information in the live streaming room excluding video and audio streaming data. For example, the interaction information may be a message that the viewer has clicked the "like" button, or a message that the streamer has changed the title of the streaming room. The interaction information may be an animation in which the viewer obtains the title, an animation in which a dragon flies around in the streaming room, or the like.

[0070] According to this embodiment, the context unit 310 may receive and transmit interaction information in any form, and may simply transmit the interaction information to the viewer's terminal based on the UTC information. Thus, interaction information in any form can be displayed and synchronized with the archive content, and the user experience can be improved.

[0071] In some embodiments, another viewer user terminal (which may also be referred to as a clip viewer) may request a clip from the live streaming while viewing the live streamer in the live streaming room. For example, the viewer may clip a highlight moment in the live streaming room and share the highlight with other social media, etc. In some embodiments, the viewer may click a button such as the clip object 612 in FIG. 11 to send a request for clipping to the server 10. In some embodiments, the clip may be automatically generated based on parameters in the live streaming room, such as the popularity of the live streaming, the number of viewers, comments, etc.

[0072] In some embodiments, the viewer may determine the length, start time, end time, etc. of the clip. The viewer may also determine, for example, the resolution, thumbnail, video format, etc. When the setting of the clip from the viewer is completed, the information of the request may be sent to the backend server. The backend server may generate a task queue in the task queue system for the request. In some embodiments, the task queue system may be an internal system or a third-party system such as the Redis Queue System. In such a system, tasks are queued and processed asynchronously by worker processes or servers.

[0073] In some embodiments, the queue of requests may be further sent to a load balancer server. The load balancer server may distribute requests from multiple sources to multiple instances or worker processes within the queue, which may help with workload distribution, improving system availability, improving system scalability, and traffic control. In some embodiments, the load balancer server may further distribute the queue of requests to an Emulator Control Manager (ECM) for recording the clips. In some embodiments, the load balancer may be an internal load balancer of server 10 or a service provided by a third-party service provider.

[0074] FIG. 10 is a schematic diagram showing a functional configuration example of a live streaming system 1 according to some embodiments of the present disclosure. As shown in FIG. 10, the ECM may include one or more virtual machines VM. The virtual machine (VM) is intended to simulate another computing environment. The function of the virtual machine is to provide an abstraction layer that enables multiple independent virtual computing environments. This brings flexibility, manageability, and enhanced security to the computing environment.

[0075] The load balancer server may distribute the queue of the request to one or more virtual machines. The virtual machine may generate one or more emulators EM in response to the request. The "emulator" may refer to software or a hardware tool that can simulate a hardware platform or a device and enable it to be executed in a different environment. That is, the emulator may enable software designed for one system to be executed on another system by simulating the target hardware or software environment. In some embodiments, the emulator EM may be an Android Package in the Android operating system, or other packages in other platforms. In the Android environment, developers often use Android emulators to test and debug applications, and these emulators may need to load and execute application packages with the file extension.apk.

[0076] In some embodiments, one or more applications APP may be launched or simulated by the emulator EM. That is, the application APP from the APP provider may be launched or simulated by the emulator EM. In some embodiments, the emulator EM may simulate a user terminal such as a smartphone or a computer. After the launch or simulation of the application by the emulator EM is completed, the emulator EM may receive archive data or context data through the application. Further, the emulator EM may render the archive data together with the context data from, for example, the archive DB324 and the context DB326 through the application.

[0077] In some embodiments, the emulator EM may record the rendered archive data and the context data as a clip. For example, information such as the length, start time, and end time of the clip may be used to determine the portion to be rendered or recorded. Once the rendering and recording of the clip are completed, the clip may be further stored in storage. In some embodiments, the storage may be an internal area within the server 10 or a service provided by a third-party service provider such as Google Cloud Storage. In some embodiments, when the recording of the clip is completed, an API may be sent to the queue system to delete the queue of the request from the task queue system.

[0078] In some embodiments, a link to the stored clip may be generated and sent to the clip viewer. The clip viewer may further view the clip or share the clip on other social media sites. In some embodiments, the link may be a website, a URL, a thumbnail, etc. The clip viewer may view the clip or re-edit the clip via the user terminal.

[0079] For the archive viewer, when the client terminal requests archive content, the streaming data and interaction data are reconstructed. According to some embodiments, this may have the advantage that any interaction data, such as expired or old gifts and comments at the time of playing the archive (for example, gifts by collaboration with external copyright holders, where the period during which such gifts can be used on the live streaming service platform is defined), can be easily deleted from the archive content. In other services, a method is implemented in which the entire view of the live streaming is recorded (like a screenshot) during the live time and the recorded data is used for the archive. In this case, gifts / comments cannot be deleted from the archive later.

[0080] For the clip viewer, the above embodiments may be for generating clip content in response to a user's request while maintaining the above advantages regarding the archive content. Thus, the emulator is used for generating clip content but not for generating archive content. More specifically, the archive content is not generated by the emulator. However, in some embodiments, the archive content may also be generated by the emulator or the like. In some embodiments, the design of the archiving and clipping functions may be determined flexibly.

[0081] Figures 11 to 14 are exemplary screen images of a live streaming room screen 600 displayed on the display of the user terminal 20 of the live streamer or the user terminal 30 of the viewer. When the viewer selects and enters the live streaming room, the live streaming room screen 600 of the live streamer may be displayed on the display. The live streaming room screen 600 may include a live streamer information object 602, a live streamer image 604, a message zone 606, a message input box 608, a gift object 610, a clip object 612, and the like.

[0082] The viewer may communicate with the live streamer. For example, the viewer may tap the gift object 610, and a gift list (not shown) may be displayed on the screen 600. The gift list may include a plurality of gift objects, and the viewer may select a gift object to send to the live streamer. As shown in FIG. 11, when the viewer clicks the gift object, the effect E and information IN of the gift object may be displayed on the screen 600.

[0083] In some embodiments, the viewer may clip the live stream by clicking the clip object 612. For example, the viewer may clip the portion of the live stream while the effect E is displayed on the screen 600, and may also capture the expression of the live streamer when receiving the gift. According to some embodiments, the viewer may save or share highlights within the live stream for later viewing or to promote the performance from the live streamer.

[0084] When the clip object 612 is clicked, as shown in FIG. 12, a clipping tool 614 may be displayed on the screen 600. In some embodiments, the clipping tool 614 may include a portion of the live streaming, such as the streaming data, archive data, and / or content data. The portion P of the live streaming may be determined flexibly. For example, 90 seconds of the live streaming from 90 seconds before or 90 seconds after the time when the viewer clicks the clip object 612 may be provided as the portion P. Also, the portion P may be from 45 seconds before to 45 seconds after the time when the viewer clicks the clip object 612. In some embodiments, the duration of the portion P may be determined flexibly.

[0085] In addition, in some embodiments, an editing tool 616 may be provided to determine information about the clip, such as the length, duration, start time, end time, etc. of the clip, for the clip viewer. In some embodiments, the editing tool 616 may also include, for example, duration, title, viewer information, live streamer information, etc. In some embodiments, the viewer may adjust, for example, resolution, quality, size, etc. In some embodiments, the functions of the editing tool 616 may be determined flexibly according to actual needs.

[0086] In some embodiments, after the information about the clip is determined, the viewer may further click a clip generation button 618 to generate the clip. When the clip generation button 618 is clicked, a request for the clip may be sent to the server 10, and the clip may be generated. The generation of the clip may be realized according to the processes shown in FIGS. 9 and 10.

[0087] When the generation of the clip is completed, information about the clip may be displayed on the screen 600 so that the clip viewer can view it. For example, a thumbnail TH of the clip may be displayed, and as shown in FIG. 13, a share button S and a download button D may also be displayed on the screen 600 so that the viewer can share, download, etc. In some embodiments, a link such as a URL may also be provided. In some embodiments, the layout and UI of the clipping function may be determined flexibly.

[0088] In some embodiments, the live streaming clip CP may be saved to the user's profile page PR as shown in FIG. 14. In some embodiments, the profile page PR may include a tab T for the user to click to obtain access to the clip CP from the user. The user may obtain access to the clip CP by clicking on the tab T on the profile page PR or the like. In some embodiments, the layout and UI may be determined flexibly according to actual needs.

[0089] FIG. 15 is an exemplary sequence diagram showing the operation of the configuration of the live streaming system 1 according to some embodiments of the present disclosure. In some embodiments, the viewer may perform an operation such as clicking on a clip object 612 to clip a part of the live stream. As shown in FIG. 15, when the clip object 612 is clicked by the viewer, a clipping request may be sent to the backend server (S302).

[0090] In some embodiments, the backend may further send the request, including the deep link, to the API server (S304). More specifically, the backend may send a request to the API server, and this request includes a deep link. This type of request may include a specific path, parameter, or identifier indicating that the API server should perform a specific action, and may direct the user to specific content or functions within the mobile application. In some embodiments, the deep link may be linked to actions such as the generation of an emulator, the launch of an application, etc. In some embodiments, the API server may be an internal API server within server 10 or a server provided by a third - party provider.

[0091] In some embodiments, the API server may further create a record job and send the queue of the request to the load - balancer server (S306). The load - balancer server may further distribute the request to one or more emulators to generate a clip (S308). In some embodiments, when steps 306 and 308 are successful (S310) and the ECM server is ready to start recording, the API server may send a "success" response indicating that the request has been received to the backend, and then the recording process may be started with the emulator EM (S322).

[0092] In some embodiments, if steps 306 and 308 are not successful, the API server may send a "failure" response along with a reason to the backend (S312). In some embodiments, the reason for the "failure" response may be a requesting time - out, server overload, etc. Here, "requesting time - out" may refer to a timeout being requested during a network request. A timeout occurs when a response or result is not received within a specified time. For example, if a "success" response is not received within 300 seconds, a "failure" response may be sent to the backend server along with the reason of "requesting time - out". In some embodiments, if the ECM server is overloaded, a "failure" response may be sent to the backend server along with the reason of "server overload".

[0093] In some embodiments, the backend server may resend the request to the API server along with a deep - link after a certain period of sleep (S314) (S316). Here, a certain period of sleep may refer to waiting for a retry time and then resending the request to the API server after the retry time. For example, the backend server may determine a retry time such as 5, 10, 30 seconds, etc. The backend server may further send the next request regarding the clip after the retry time. In some embodiments, the API server may further create a record job and send the queue of the request to a load - balancer server (S318). The load - balancer server may further distribute the request to one or more emulators to generate a clip (S320).

[0094] After the above process is completed and a "success" response is sent to the backend, the recording process in the emulator EM may be started (S322). The emulator may record the clip based on the information of the request such as length, start time, end time, etc. After the recording of the clip is completed, the clip may be stored in a storage such as Google Cloud Storage, and a link to the clip may be created (S324). The emulator EM may further notify the backend of the completion of recording and uploading, and the link, etc. In some embodiments, the link may be a URL indicating the location of the clip content in the storage.

[0095] In some embodiments, the link may be determined by the backend server while the backend sends the request to the API server in step S304. The clip content may be uploaded to the storage based on the link included in the API request from the backend. That is, the API request from the backend may include the link information, whereby after the recording is completed, the emulator EM may upload the clip content to the link. Therefore, when the recording and uploading of the clip content are completed, the backend may also receive the link information of the clip content. In some embodiments, the clip may be stored in a format such as an.mp4 file. In some embodiments, the format of the clip content may be determined flexibly.

[0096] In some embodiments, the backend server may receive a notification about the completion of the recording and uploading of the clip content (S326). For example, the emulator EM may notify the backend about the completion of the recording and uploading of the clip. Also, in some embodiments, the emulator EM may send the link, etc. to the backend server. Further, in some embodiments, the backend server may request information about the clip from the emulator EM, etc.

[0097] In some embodiments, the user may further obtain access to the clip content (S328). For example, the user may check, download, or share the clip content. Also, in some embodiments, the information of the clip content may be displayed on the user's personal page so that the user can access it. For example, the user may open the clip tab on the profile page and check the clip content.

[0098] When the user requests to obtain access to the clip content, the backend server may allow the user to access the clip content in the storage (S330). In some embodiments, when the user opens the APP, the information of the clip content may be displayed on the user's profile page, etc. In some embodiments, the user may click the download button or share button, whereby the user is guided to the clip content via the link and can download, transfer, etc. the clip content. According to this embodiment, the highlights in the live streaming can be clipped more efficiently and smoothly.

[0099] In some embodiments, after receiving archive data and context data from the archive DB324 and the context DB326 respectively, the emulator EM may render the archive data and the context data for recording. Also, in some embodiments, the emulator EM may receive streaming data and context data for recording from the streaming server and the backend server. The rendering of the archive data and the context data may be based on the time stamps of the data, etc.

[0100] In some embodiments, the clip viewer may watch the live streaming and clip a part of the live streaming as a highlight. Also, in some embodiments, the clip viewer may watch the archive video and clip a part of the archive as a highlight. In some embodiments, the live streamer and the viewer may clip a part of the live streaming as a highlight, and the highlight may also be automatically clipped according to parameters such as the popularity, number of comments, and viewer comments in the live streaming room. Also, in some embodiments, the highlight may be automatically clipped by machine learning techniques, etc.

[0101] In some embodiments, the server 10 may automatically archive the live streaming. Also, the server 10 may archive the live streaming in response to a request from the live streamer or the viewer. In some embodiments, when the live streamer turns on the auto-archive function, a portion of the live streaming may be clipped as a highlight. In some embodiments, even if the live streamer does not turn on the auto-archive function, the server 10 may clip a portion of the live streaming from the streaming data in the streaming server and the context data in the backend server. In some embodiments, the settings for the archive and clip may be determined flexibly.

[0102] Compared with rendering and recording via the user terminal, the quality of rendering and recording on the server 10 may be able to provide a higher-quality clip. More specifically, recording a clip on the user terminal side may face unexpected problems. For example, a LINE app message or update notification may suddenly pop up, which may reduce the video quality. Furthermore, rendering and recording on the server 10 may also provide adjustments to various parameters. For example, the resolution and size of the clip may be determined by the viewer or the server 10, so that the clip may be suitable for various devices such as mobile phones, tablets, and computers.

[0103] In some embodiments, the retry time is a fixed value or increases as the number of retries increases. Here, the "number of retries" may refer to the number of times the backend server retries sending the request including the deep link to the API server. In some embodiments, the retry time may increase based on the number of retries. In some embodiments, the relationship between the number of retries and the retry time may be determined flexibly according to actual needs.

[0104] According to the present disclosure, there is a possibility that the clip is generated more efficiently and accurately, and a more immersive experience is provided when viewing the clip. Furthermore, there is a possibility that the clip can be reviewed and shared more flexibly. Therefore, the user experience may be improved.

[0105] FIG. 16 is a schematic block diagram of computer hardware for executing system configuration and processing according to some embodiments of the present disclosure. The information processing apparatus 900 in FIG. 16 is configured to realize, for example, the server 10 and the user terminals 20 and 30 according to some embodiments of the present disclosure.

[0106] The information processing apparatus 900 includes a CPU 901, a read-only memory (ROM) 903, and a random access memory (RAM) 905. Further, the information processing apparatus 900 may include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input unit 915, an output unit 917, a storage unit 919, a drive 921, a connection port 925, and a communication unit 929. The information processing apparatus 900 may include an imaging device (not shown) such as a camera. The CPU 901 is an example of a hardware configuration for realizing the functions realized by the components described in this specification. The functions described in this specification may be realized by circuitry programmed to realize the described functions. The circuitry programmed to realize the functions described in this specification includes a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a general-purpose processor, an application-specific processor, an integrated circuit, ASICs (Application Specific Integrated Circuits), and / or combinations thereof. In this specification, a unit that realizes a specific function may be realized as circuitry programmed to realize the function. This unit includes, but is not limited to, a streaming unit 100, a video control unit 102, an audio control unit 104, a distribution unit 106, a UI control unit 108, a viewing unit 200, a UI control unit 202, a rendering unit 204, an input transmission unit 206, a streaming information unit 302, a relay unit 304, a recording unit 306, a processing unit 308, a context unit 310, a clip unit 312, a stream DB 320, a user DB 322, an archive DB 324, a context DB 326, and a clip DB 328.

[0107] The CPU 901 functions as an arithmetic processing unit and a control unit, and controls the overall operation or a part of the operation of the information processing apparatus 900 according to various programs recorded in the ROM 903, the RAM 905, the storage unit 919, or the removable recording medium 923. For example, the CPU 901 controls the operations of all the functional units included in the server 10 and the user terminals 20 and 30 in the above-described embodiment. The ROM 903 stores programs, operation parameters, etc. used by the CPU 901. The RAM 905 temporarily stores programs used when the CPU 901 executes and parameters that appropriately change when the programs are executed. The CPU 901, the ROM 903, and the RAM 905 are connected to each other via a host bus 907 composed of an internal bus such as a CPU bus. The host bus 907 is connected to an external bus 911 such as a peripheral component interconnect / interface (PCI) bus via the bridge 909.

[0108] The input unit 915 is a device operated by a user, such as a mouse, a keyboard, a touch panel, a button, a switch, or a lever. The input unit 915 may be a device that converts a physical quantity into an electrical signal, such as an audio sensor (such as a microphone), an acceleration sensor, an inclination sensor, an infrared sensor, a depth sensor, a temperature sensor, or a humidity sensor. The input unit 915 may be, for example, a remote control device that uses infrared rays or another type of radio wave. Alternatively, the input unit 915 may be an external connection terminal 927 such as a mobile phone corresponding to the operation of the information processing apparatus 900. The input unit 915 includes an input control circuit that generates an input signal based on information input from the user and outputs the generated input signal to the CPU 901. The user operates the input unit 915 to input various data and give an instruction for a processing operation to the information processing apparatus 900.

[0109] The output unit 917 includes a device capable of visually or auditorily notifying the user of the acquired information. The output unit 917 may be, for example, a display device such as an LCD, PDP, or OLED, an audio output device such as a speaker or headphones, a printer, or the like. The output unit 917 outputs the result obtained by the process executed by the information processing device 900 in the form of video such as text or images, sound such as audio, or the like.

[0110] The storage unit 919 is a data storage device and is an example of the storage unit of the information processing device 900. The storage unit 919 includes, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like. The storage unit 919 stores the programs executed by the CPU 901, various data, and various data acquired from the outside.

[0111] The drive 921 is a reader / writer for a removable recording medium 923 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built in or externally attached to the information processing device 900. The drive 921 reads the information recorded on the mounted removable recording medium 923 and outputs it to the RAM 905. The drive 921 writes records to the mounted removable recording medium 923.

[0112] The connection port 925 is a port used to directly connect a device to the information processing device 900. The connection port 925 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. The connection port 925 may be an RS-232C port, an optical audio terminal, an HDMI (High-Definition Multimedia Interface (registered trademark)) port, or the like. When an external connection terminal 927 is connected to the connection port 925, various data can be exchanged between the information processing device 900 and the external connection terminal 927.

[0113] The communication unit 929 is, for example, a communication interface including a communication device for connecting to a communication network NW. The communication unit 929 may be, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (registered trademark), or wireless USB (WUSB).

[0114] The communication unit 929 may be, for example, a router for optical communication, a router for ADSL (asymmetric digital subscriber line), or a modem for various communications. For example, the communication unit 929 uses a predetermined protocol such as TCP / IP to transmit and receive signals on the Internet and to transmit and receive signals with other communication devices. The communication network NW to which the communication unit 929 is connected is a network established by a wired connection or a wireless connection. The communication network NW is, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0115] The imaging device (not shown) is, for example, a device that images the real space using an imaging element such as a CCD (charge-coupled device) or a CMOS (complementary metal-oxide semiconductor), and various members such as a lens for controlling the formation of a subject image on the imaging element, and generates an imaging image. The imaging device may image a still image or a moving image.

[0116] As described above, the live streaming system 1 of the present disclosure has been described with reference to the embodiments. The above-described embodiments are merely described for the purpose of explanation. Rather, it is easily conceivable by those skilled in the art that the above-described components and processes of the embodiments can be combined in various ways and various changes can be made, and these are also included in the technical scope of the present disclosure.

[0117] The processes described in this specification, particularly those described using flowcharts or flow diagrams, can be subject to omission of some of the constituent steps, addition of steps not explicitly included in the constituent steps, and / or rearrangement of the step order. Such processes that are subject to such omission, addition, or rearrangement are also included within the scope of this disclosure, provided they do not depart from the gist of this disclosure.

[0118] In some embodiments, at least some of the functions performed by the server 10 may be performed by entities other than the server 10. For example, they may be performed by the user terminal 20 or 30. In some embodiments, at least some of the functions performed by the user terminal 20 or 30 may be performed by entities other than the user terminal 20 or 30. For example, they may be performed by the server 10. In some embodiments, the rendering of the frame image may be performed by the user terminal such as the viewer, server, live streamer, etc.

[0119] Furthermore, the systems or methods described in the above embodiments may be provided by non-transitory computer-readable storage devices such as solid-state storage devices, optical disk storage devices, magnetic disk storage devices, or computer program products. Alternatively, the program may be downloaded from a server via the Internet.

[0120] As described above, the technical content and features of this disclosure have been explained. However, those with ordinary knowledge in the technical field to which this disclosure pertains can make many more variations and modifications without departing from the teachings and disclosures of this disclosure. Therefore, the scope of this disclosure is not limited to the already disclosed embodiments, but is the scope included in the appended claims, which includes other variations and modifications that do not depart from this disclosure.

Description of Reference Numerals

[0121] 1 Live Streaming System 10 Server 20 User Terminal 100 Streaming Unit 102 Video control unit 104 Audio control unit 106 Distribution unit 108 UI control unit 200 Viewing unit 202 UI control unit 204 Rendering unit 206 Input transmission unit 208 Cache unit 210 Queue unit 212 Processing unit 250 Cache DB 252 Data queue DB 254 Cache method lookup table 30, 30a, 30b User terminal 302 Streaming information unit 304 Relay unit 306 Recording unit 308 Processing unit 310 Context unit 312 Clip unit 320 Stream DB 322 User DB 324 Archive DB 326 Context DB 328 Clip DB 600 Screen 602 Object 604 Image 606 Message zone 608 Message input box 610 Gift object 612 Clip object 614 Clip tool 616 Editing tool 618 Clip generation button 900 Information processing device 901 CPU 903 ROM 905 RAM 907 Host Bus 909 Bridge 911 External Bus 913 Interface 915 Input Unit 917 Output Unit 919 Storage Unit 921 Drive 923 Removable Recording Medium 925 Connection Port 927 External Connection Terminal 929 Communication Unit LS Live Streaming LV Streamer NW Network P Portion AU1, AU2 Audience S302 - S322 Steps VD, VD1, VD2 Video

Claims

1. A server including a circuit, wherein the circuit has a function of generating an emulator in response to a request from a first user terminal of a first user; has a function of starting an application via the emulator; has a function of receiving streaming data and interaction data via the application; has a function of rendering the streaming data with the interaction data; has a function of recording the rendered streaming data and interaction data as a clip; has a function of saving the clip so as to be accessible from the first user terminal of the first user; A server, characterized in that it is configured to execute the above.

2. Further including a function of transmitting the streaming data to a second user terminal of a second user, and a function of transmitting the interaction data to the second user terminal of the second user, wherein the streaming data and the interaction data are respectively from archive data and context data and are for rendering and displaying on the second user terminal. The server according to claim 1, characterized by this.

3. The server according to claim 1, characterized in that the clip is generated via the emulator by taking in the streaming data and the interaction data as inputs.

4. The streaming data and the interaction data are rendered based on time information inserted therein, wherein the time information is relative time or absolute time of GMT time or UTC time. The server according to claim 1, characterized by this.

5. The server according to claim 1, characterized in that the length, start time, and end time of the clip are determined by the first user terminal of the first user.

6. The server according to claim 1, characterized in that the interaction data includes information on messages, comments, gifts, follows, games, VIP online notifications, or entrance animations.

7. The streaming data is from archive data or live streaming data, and the interaction data is from context data or live interaction data. The server according to claim 1, characterized in that...

8. A function of determining a retry time in response to the failure of the generation of the emulator; A function of sending the next request regarding the generation of the emulator after the retry time; The server according to claim 1, further comprising the above, characterized in that...

9. A method, comprising: Generating an emulator in response to a request from a first user terminal of a first user; Starting an application via the emulator; Receiving streaming data and interaction data via the application; Rendering the streaming data with the interaction data; Recording the rendered streaming data and interaction data as a clip; Saving the clip so as to be accessible from the first user terminal of the first user; A method, characterized by comprising the above.

10. A computer program, causing a server to: Generate an emulator in response to a request from a first user terminal of a first user; Start an application via the emulator; Receive streaming data and interaction data via the application; Render the streaming data with the interaction data; Record the rendered streaming data and interaction data as a clip; Save the clip so as to be accessible from the first user terminal of the first user; A computer program, characterized by causing the above to be realized.

Images