Systems and methods for live streaming

Abstract

To provide a system and a method for achieving a flexible comment function desired by a viewer in a live streaming platform.SOLUTION: In a method for live streaming, a program causes one or more computers to execute a step of acquiring a comment from a viewer V1 in a live stream room, a step of displaying a selection user interface to the viewer, a step of acquiring a selection result from the viewer via the selection user interface, and a step of expressing the comment in the live stream room on the basis of the selection result.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to streaming, and in particular to live streaming. [Background technology]

[0002] As exemplified by live streaming services, real-time online interactions have become a part of everyday life. A variety of platforms and providers offer live streaming services, creating fierce competition. It is important for platforms to provide the services users desire.

[0003] Japanese Patent Application Laid-Open Publication No. 2019-144311 discloses a system that supports streamers in replying to viewers' comments. Summary of the Invention

[0004] A method according to one embodiment of the present invention is a method for live streaming executed by one or more computers, comprising steps of obtaining comments from viewers in a live stream room, displaying a selection user interface to the viewers, obtaining a selection from the viewers via the selection user interface, and showing comments in the live stream room based on the selection.

[0005] A system according to one embodiment of the present invention is a system for live streaming, which includes one or more processors, and the one or more computer processors execute machine-readable instructions to perform the following steps: acquiring comments from viewers in a live stream room; displaying a selection user interface to the viewers; acquiring selections from the viewers via the selection user interface; and expressing comments in the live stream room based on the selections.

[0006] A non-transitory computer-readable medium containing a program for live streaming according to one embodiment of the present invention causes one or more computers to perform the following steps: acquiring comments from viewers in a live stream room; displaying a selection user interface to the viewers; acquiring selections from the viewers via the selection user interface; and expressing comments in the live stream room based on the selections. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram illustrating the configuration of a live streaming system 1 according to some embodiments of the present invention. [Figure 2] 2 is a block diagram illustrating the functionality and configuration of user terminal 30 of FIG. 1 in accordance with some embodiments of the present invention. [Figure 3] 2 is a block diagram illustrating the functionality and configuration of the server of FIG. 1 in accordance with some embodiments of the present invention. [Figure 4] 4 is a table illustrating an exemplary data structure of the stream DB 310 of FIG. 3. [Figure 5] 4 is a table illustrating an exemplary data structure of the user DB 312 of FIG. 3. [Figure 6] 4 is a table illustrating an example data structure of the gift DB 314 of FIG. 3. [Figure 7] 10 is a table showing an exemplary data structure of a comment format DB 330. [Figure 8] 10 is a table showing an exemplary data structure of a comment content DB 332. [Figure 9] 1 is an exemplary flow chart according to some embodiments of the present invention. [Figure 10] 1 is an exemplary flow chart according to some embodiments of the present invention. [Figure 11] 1 is an example of a live stream room in accordance with some embodiments of the present invention. [Figure 12] 1 is an example of a live stream room in accordance with some embodiments of the present invention. [Figure 13] 1 is an example of a live stream room in accordance with some embodiments of the present invention. [Figure 14] 1 is an example of a format selection UI according to some embodiments of the present invention. [Figure 15] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, identical or similar components, parts, steps, or signals shown in each drawing will be denoted by the same reference numerals in all drawings, and redundant explanations will be omitted as appropriate. In addition, some parts that are not important in the explanation of each drawing will be omitted.

[0009] The ability to comment on streaming platforms is an important feature that allows viewers to interact with streamers, so it is desirable for platforms to offer flexible commenting features.

[0010] FIG. 1 is a schematic diagram illustrating the configuration of a live streaming system 1 according to some embodiments of the present invention. The live streaming system 1 provides a live streaming service for streaming streamers (also referred to as livestreamers, anchors, distributors, or livestreamers) LV and viewers (also referred to as audiences) AU (AU1, AU2, ...) to interact or communicate in real time. As shown in FIG. 1, the live streaming system 1 includes a server 10, a user terminal 20, and a user terminal 30 (30a, 30b, ...). In some embodiments, the streamers and viewers may be collectively referred to as users. The server 10 may include one or more information processing devices connected to a network NW. The user terminals 20 and 30 may be, for example, mobile terminal devices such as smartphones, tablets, laptops, recorders, portable game consoles, and wearable devices, or stationary devices such as desktop PCs. The server 10, the user terminals 20, and the user terminals 30 are connected to each other so as to be able to communicate with each other via various wired or wireless networks NW.

[0011] The live streaming system 1 includes a broadcaster LV, a viewer AU, and an administrator (or app provider, not shown) who manages the server 10. The broadcaster LV records content on his / her user terminal 20 and directly or indirectly uploads it to the server 10, thereby broadcasting the content in real time. Examples of the content include the broadcaster's own singing, talking, performance, gameplay, or any other content. The administrator provides a platform for live streaming the content on the server 10 and mediates or manages real-time interactions between the broadcaster LV and the viewer AU. The viewer AU accesses the platform with his / her user terminal 30, selects and watches desired content, and performs operations such as leaving comments, cheering, or sending gifts via the user terminal 30 during the live streaming of the selected content. The broadcaster LV, who is broadcasting the content, may respond to the comments, cheers, or gifts. The response is transmitted to the viewer AU via video and / or audio, establishing two-way communication.

[0012] The term "live streaming" may refer to a data transmission mode that allows content recorded by a user terminal 20 of the distributor LV to be played back and viewed on a user terminal 30 of the viewer AU in substantially real time, or may refer to live broadcasting realized by such a transmission mode. The live streaming may be realized using existing live streaming technologies such as HTTP live streaming, CMAF (Common Media Application Format), WebRTC (Web Real-Time Communications), RTMP (Real-Time Messaging Protocol), etc. Live streaming includes a transmission mode that allows the viewer AU to view content simultaneously with a predetermined delay as the content is recorded by the distributor LV. The delay may be long enough to allow interaction between the distributor LV and the viewer AU. Note that live streaming is distinguished from so-called on-demand distribution, in which all recorded data of the content is stored on a server and then provided to users from the server upon their request.

[0013] The term "video data" as used herein refers to data including image data (also referred to as video data) generated using the imaging function of the user terminal 20 or 30 and audio data generated using the audio input function of the user terminal 20 or 30. The video data is played on the user terminal 20 or 30 so that the user can view the content. In some embodiments, it is assumed that processing to change the format, size, or data specifications of the video data, such as compression, decompression, encoding, decoding, or transcoding, is performed on the video data between the generation of the video data on the distributor's user terminal and the playback of the video data on the viewer's user terminal. However, since the content (e.g., video and audio) represented by the video data remains substantially unchanged before and after such processing, the video data after such processing is referred to herein as being identical to the video data before such processing. In other words, when video data is generated on the distributor's user terminal and then played on the viewer's user terminal via the server 10, the video data generated on the distributor's user terminal, the video data passing through the server 10, and the video data received and played on the viewer's user terminal are all the same video data.

[0014] In the example shown in Fig. 1, the distributor LV provides live streaming data. The user terminal 20 of the distributor LV records video and audio of the distributor LV to generate the streaming data, and the generated data is transmitted to the server 10 via the network NW. At the same time, the user terminal 20 displays the recorded video VD of the distributor LV on the display of the user terminal 20, allowing the user to check the live streaming content currently being produced by the distributor LV.

[0015] The user terminals 30a and 30b of the viewers AU1 and AU2 who have requested to view the live streaming of the distributor LV on the platform receive video data related to the live streaming (hereinafter, may be referred to as "live streaming video data") via the network NW, play the received video data, display the videos VD1 and VD2 on the display, and output audio from speakers. The videos VD1 and VD2 displayed on the user terminals 30a and 30b are substantially the same as the video VD captured by the user terminal 20 of the distributor LV, and the audio output from the user terminals 30a and 30b is substantially the same as the audio recorded by the user terminal 20 of the distributor LV.

[0016] The recording of video and audio on the user terminal 20 of the distributor LV and the playback of video data on the user terminals 30a and 30b of the viewers AU1 and AU2 are performed substantially simultaneously. When the viewer AU1 inputs a comment about the content provided by the distributor LV to the user terminal 30a, the server 10 displays the comment in real time on the user terminal 20 of the distributor LV and also displays the comment on the user terminals 30a and 30b of the viewers AU1 and AU2, respectively. When the distributor LV reads the comment and engages in a conversation corresponding to the comment, the video and audio of the conversation are displayed on the user terminals 30a and 30b of the viewers AU1 and AU2, respectively. This interactive behavior is recognized as a conversation between the distributor LV and the viewer AU1. This allows the live streaming system 1 to realize live streaming that enables two-way communication rather than one-way communication.

[0017] FIG. 2 is a block diagram illustrating the functions and configuration of the user terminal 30 of FIG. 1 according to some embodiments of the present invention. The user terminal 20 has the same or similar functions and configuration as the user terminal 30. Each block in FIG. 2 and the subsequent block diagrams may be realized by hardware elements such as a computer CPU or a mechanical device, or by software elements such as a computer program. The functional blocks may be realized by cooperative operations between these elements. Therefore, those skilled in the art will understand that these functional blocks may be realized in various forms using a combination of hardware and software.

[0018] The distributor LV and the viewer AU may download a live streaming application program (hereinafter referred to as a live streaming application) from a download site via the network NW and install it on the user terminals 20 and 30. Alternatively, the live streaming application may be pre-installed on the user terminals 20 and 30. When the live streaming application is executed on the user terminals 20 and 30, the user terminals 20 and 30 communicate with the server 10 via the network NW and implement or execute various functions. Hereinafter, functions implemented by the user terminals 20 and 30 (processors such as CPUs) on which the live streaming application is executed are described as functions of the user terminals 20 and 30. These functions are actually realized by the live streaming application on the user terminals 20 and 30. In some embodiments, these functions may be written in a programming language such as HTML (HyperText Markup Language), transmitted from the server 10 to a web browser on the user terminals 20 and 30 via the network NW, and implemented by a computer program executed by the web browser.

[0019] The user terminal 30 includes a distribution unit 100 and a viewing unit 200. The distribution unit 100 generates video data in which the video and audio of the user (or user side) are recorded, and provides the video data to the server 10. The viewing unit 200 receives the video data from the server 10 and plays the video data. The user starts the distribution unit 100 when performing live streaming, and starts the viewing unit 200 when viewing video. The user terminal on which the distribution unit 100 is started is the distributor's terminal, i.e., the user terminal that generates the video data. The user terminal on which the viewing unit 200 is started is the viewer's terminal, i.e., the user terminal on which the video data is reproduced and played.

[0020] The distribution unit 100 includes an imaging control unit 102, an audio control unit 104, a video transmission unit 106, and a distributor-side UI control unit 108. The imaging control unit 102 is connected to a camera (not shown in FIG. 2) and controls imaging performed by the camera. The imaging control unit 102 acquires image data from the camera. The audio control unit 104 is connected to a microphone (not shown in FIG. 2) and controls audio input from the microphone. The audio control unit 104 acquires audio data from the microphone. The video transmission unit 106 transmits video data including the image data acquired by the imaging control unit 102 and the audio data acquired by the audio control unit 104 to the server 10 via the network NW. The video data is transmitted in real time by the video transmission unit 106. That is, the generation of the video data by the imaging control unit 102 and the audio control unit 104 and the transmission of the generated video data by the video transmission unit 106 are performed substantially simultaneously. The distributor-side UI control unit 108 controls the UI (user interface) of the distributor. The distributor-side UI control unit 108 may be connected to a display (not shown in FIG. 2 ) and displays a video on the display by playing the video data transmitted by the video transmission unit 106. The distributor-side UI control unit 108 may display an operation object or an instruction permission object on the display and receive input from the distributor who taps on the object.

[0021] The viewing unit 200 includes a viewer-side UI control unit 202, an overlay information generation unit 204, and an input information transmission unit 206. The viewing unit 200 receives video data related to live streaming, in which the broadcaster, the viewer who is the user of the user terminal 30, and other viewers participate, from the server 10 via the network NW. The viewer-side UI control unit 202 controls the viewer's UI. The viewer-side UI control unit 202 is connected to a display and a speaker (not shown in FIG. 2 ) and plays the received video data, displays the video on the display, and outputs the audio from the speaker. The state in which the video is output to the display and the audio is output from the speaker can be referred to as a state in which the video data is being played. The viewer-side UI control unit 202 is also connected to an input means (not shown in FIG. 2 ), such as a touch panel, keyboard, or display, and acquires user input via the input means. The overlay information generation unit 204 overlays a predetermined frame image on an image generated from video data from the server 10. The frame image includes various user interface objects (hereinafter simply referred to as "objects") for receiving input from the user, comments entered by the viewer, and / or information acquired from the server 10. The input information transmission unit 206 transmits the user input acquired by the viewer-side UI control unit 202 to the server 10 via the network NW.

[0022] 3 is a block diagram illustrating the functions and configuration of the server 10 of FIG. 1 according to some embodiments of the present invention. The server 10 includes a distribution information providing unit 302, a relay unit 304, a gift processing unit 306, a payment processing unit 308, a stream DB 310, a user DB 312, a gift DB 314, an acquisition unit 320, a processing unit 322, a comment format DB 330, a comment content DB 332, and a machine learning DB 334.

[0023] When receiving a notification or request to start live streaming from the user terminal 20 on the distributor side via the network NW, the distribution information providing unit 302 registers in the stream DB 310 a stream ID for identifying this live streaming and the distributor ID of the distributor performing the live streaming.

[0024] When the distribution information providing unit 302 receives a request for information about live streams from the viewing unit 200 of the user terminal 30 on the viewer side via the network NW, the distribution information providing unit 302 obtains or checks currently available live streams from the stream DB 310 and creates a list of available live streams. The distribution information providing unit 302 sends the created list to the requesting user terminal 30 via the network NW. The viewer-side UI control unit 202 of the requesting user terminal 30 generates a live stream selection screen based on the received list and displays it on the display of the user terminal 30.

[0025] When the input information sending unit 206 of the user terminal 30 receives the viewer's selection on the live stream selection screen, the input information sending unit 206 generates a delivery request including the stream ID of the selected live stream and sends the request to the server 10 via the network NW. The delivery information providing unit 302 starts providing the live stream specified by the stream ID included in the received delivery request to the requesting user terminal 30. The delivery information providing unit 302 updates the stream DB 310 so that the viewer ID of (or the corresponding) the stream ID includes the user ID of the viewer of the requesting user terminal 30.

[0026] The relay unit 304 relays the video data from the user terminal 20 on the broadcaster side to the user terminal 30 on the viewer side during the live streaming initiated by the broadcast information providing unit 302. The relay unit 304 receives a signal representing a user input by a viewer from the input information transmitting unit 206 during the live streaming or playback of the video data. The signal representing the user input may be an object designation signal that designates an object displayed on the display of the user terminal 30. The object designation signal may include the viewer ID of the viewer, the broadcaster ID of the broadcaster of the live stream being viewed by the viewer, and an object ID that identifies the object. When the object is a gift, the object ID is a gift ID. Similarly, the relay unit 304 receives a signal representing a user input made by a broadcaster during playback of the video data (or during live streaming) from the broadcast unit 100 of the user terminal 20. The signal may be an object designation signal.

[0027] Furthermore, the signal representing the user input may be a comment input signal containing a comment entered by the viewer into the user terminal 30 and the viewer ID of the viewer. Upon receiving the comment input signal, the relay unit 304 transmits the comment and the viewer ID contained in the signal to the user terminal 20 of the distributor and the user terminals 30 of other viewers. In these user terminals 20, 30, the viewer-side UI control unit 202 and the overlay information generation unit 204 display the received comment on a display associated with the received viewer ID.

[0028] The gift processing unit 306 increases the points of the broadcaster based on the points of the gift identified by the gift ID included in the object designation signal, and updates the user DB 312. Specifically, the gift processing unit 306 refers to the gift DB 314 to identify points to be assigned to the gift ID included in the received object designation signal. Then, the gift processing unit 306 updates the user DB 312 and adds the identified points to the points of the broadcaster ID (or corresponding to the broadcaster ID) included in the object designation signal.

[0029] In response to receiving the object designation signal, the payment processing unit 308 processes payment for the gift from the viewer. Specifically, the payment processing unit 308 refers to the gift DB 314 to determine the price points of the gift identified by the gift ID included in the object designation signal. Then, the payment processing unit 308 updates the user DB 312 and subtracts the determined price points from the points of the viewer identified by the viewer ID included in the object designation signal.

[0030] FIG. 4 is a table illustrating an exemplary data structure of the stream DB 310 in FIG. 3. The stream DB 310 holds information about currently running live streams. The stream DB 310 stores a stream ID, a broadcaster ID, and a viewer ID in association with each other. The stream ID is an ID for identifying a live stream in the live streaming platform provided by the live streaming system 1. The broadcaster ID is a user ID for identifying the broadcaster who provides the live stream. The viewer ID is a user ID for identifying a viewer of the live stream. In the live streaming platform provided by the live streaming system 1 according to some embodiments, when a user starts a live stream, the user becomes a broadcaster, and when the same user watches a live stream broadcast by another user, the user also becomes a viewer. Therefore, the distinction between a broadcaster and a viewer is not fixed, and a user ID registered as a broadcaster ID at one time may be registered as a viewer ID at another time.

[0031] FIG. 5 is a table illustrating an exemplary data structure of the user DB 312 of FIG. 3. The user DB 312 holds information about users. The user DB 312 stores user IDs and points in association with each other. The user ID identifies a user. The points correspond to points held by the corresponding user. The points are electronic value circulated within the live streaming platform. In some embodiments, when a streamer receives a gift from a viewer during a live broadcast, the streamer's points increase by a value corresponding to the gift. The points are used, for example, to determine the amount of compensation (e.g., money) that the streamer receives from an administrator of the live streaming platform. In some embodiments, when the streamer receives a gift from a viewer, an amount corresponding to the gift may be given instead of the points.

[0032] FIG. 6 is a table illustrating an exemplary data structure of the gift DB 314 of FIG. 3. The gift DB 314 stores information about gifts available to the viewer during the live streaming. Gifts are electronic data. Gifts may be purchased with points or money, or may be provided free of charge. Viewers may give gifts to the broadcaster. Giving a gift to a broadcaster is also referred to as using a gift, sending a gift, or throwing a gift. Some gifts can be used immediately upon purchase, while others can be used later by the viewer who purchased them. When a viewer gives a gift to a broadcaster, the broadcaster is awarded a number of points corresponding to the gift. When a gift is used, an effect associated with the gift may be generated. For example, an effect (such as a visual or auditory effect) corresponding to the gift may be displayed on the live streaming screen.

[0033] The gift DB 314 stores gift IDs, awarded points, and price points in association with one another. The gift ID is used to identify a gift. The awarded points are the amount of points awarded to a broadcaster when the gift is given to the broadcaster. The price points are the amount of points paid for using (purchasing) the gift. While watching a live stream, a viewer can give a desired gift to a broadcaster by paying the price points of the desired gift. The price points can be paid using an appropriate electronic payment method. For example, the viewer may pay the price points to the administrator. Alternatively, payment may be made by bank transfer or credit card. The administrator can arbitrarily set the relationship between the awarded points and the price points. For example, the awarded points may be set to equal the price points. Alternatively, the price points may be set to the points obtained by multiplying the awarded points by a predetermined coefficient such as 1.2, or the price points may be set to the points obtained by adding a predetermined fee to the awarded points.

[0034] 7 is a table showing an exemplary data structure of the comment format DB 330. The comment format DB 330 stores comment format types, points, preview files, and parameter files in association with each other.

[0035] The comment format type stores various format types for expressing a comment, with each format type having multiple options. For example, the format type "size" may have different size options for expressing a comment. In some embodiments, size A < size B < size C. For example, the format type "sound effects" may have multiple sound effects that can be played when expressing the comment. The sound effects may or may not be related to the content of the comment.

[0036] For example, the format type "volume" may have different volumes for expressing the comments. In some embodiments, expressing the comments may include reading out the content of the comments. For example, the format type "audio format" may have different voice types for expressing the comments, such as a male voice, a female voice, a robot voice, an anime voice, etc. For example, the format type "movement pattern" may have different movement patterns of the comments when expressing the comments. The different patterns may be different directions, different speeds, and / or different trajectories of the comment movement.

[0037] The points store the number of points that a user (e.g., viewer) needs to spend if they choose a specific format type to express their comment. For example, if a user wants to express their comment in size A, the user pays 50 points. For example, if a user wants their comment to be read aloud in volume B, the user pays 100 points.

[0038] The preview files store URLs of the preview files, and each preview file corresponds to a specific format type of the comment expression. In some embodiments, the preview file corresponding to "size" or "movement pattern" may have an image or video format. In some embodiments, the preview file corresponding to "sound effects," "volume," or "audio format" may have an audio format.

[0039] The parameter file stores URLs of parameter setting files, each of which corresponds to a specific format type of comment representation. For example, the parameter setting file URL_SA_para contains parameters necessary to represent (or render) the comment in size A. For example, the parameter setting file URL_MB_para contains parameters necessary to represent (or render) the comment according to movement pattern B.

[0040] 8 is a table showing an exemplary data structure of the comment content DB 332. The comment content DB 332 stores time, comment ID, comment content, user ID, selection result, total points, and format file in association with each other.

[0041] The time specifies the timing or period for receiving the comment. The comment ID is configured to specify each comment. The comment content stores the content of the comment. The user ID specifies the user who sent the comment.

[0042] The selection result stores the selection result of the format type selected by the user for the comment. For example, user U2 selects "size C + volume A + movement pattern B" for the expression of comment CM1. For example, user U1 selects "size A + sound effect B" for the expression of comment CM2. The selection result is associated with the size, sound effect, volume, audio format, and / or movement pattern of the comment expression.

[0043] The total points are the number of points that the user must pay in accordance with the selection result. The selection result is associated with the number of points that the viewer is willing to pay.

[0044] In this embodiment, the total points are the sum of the points for each selected format type. For example, to express the comment CM1, user U2 needs to pay 280 points, which is the sum of 150 points (size C), 50 points (volume A), and 80 points (movement pattern B). For example, to express the comment CM2, user U1 needs to pay 120 points, which is the sum of 50 points (size A) and 70 points (sound effect B). In other embodiments, different calculations may be applied to the total points.

[0045] The format file stores the URL of the format file. Each format file contains data, code, or instructions that indicate how a comment is to be represented in the stream room. In some embodiments, the processing unit 322 (or the distribution information providing unit 302, or the relay unit 304) references the format file of the comment to represent the comment in the stream room.

[0046] In some embodiments, the data in the comment content DB 332 may be acquired or stored by the acquisition unit 320. The total points may be calculated by the processing unit 322.

[0047] The machine learning DB 334 may store various machine learning models and AI models. For example, one or more large-scale language models (LLMs) may be included in the machine learning DB 334. Various types of TTS (Text to Speech) models may also be included.

[0048] FIG. 9 illustrates an exemplary flow according to some embodiments of the present invention.

[0049] In step S900, the acquiring unit 320 acquires comments from viewers in the live stream room, and the viewers may input the comments in a comment input area via their user terminals.

[0050] In step S902, the processing unit 322 displays a selection user interface on the user terminal of the user. The selection user interface includes various format types that the viewer can select to express the comment. The number of points corresponding to each format type may also be displayed in the selection user interface.

[0051] In step S904, the acquisition unit 320 acquires candidate selection results from the viewer through the selection user interface. For example, the viewer may click one or more format types on the selection user interface. The candidate selection results may be a combination of the format types before the combination of the format types is determined as a final selection result.

[0052] In step S906, the processing unit 322 displays the effect (or preview effect) of the selection candidate to the viewer on the user terminal before the selection candidate is determined as a selection result by the viewer. For example, the selection candidate may include a format type of "size A, volume A." The processing unit 322 refers to the comment format DB 330, accesses the corresponding preview file (by URL_SA and URL_VA), and displays the preview effect by the preview file.

[0053] In some embodiments, the preview effect for one format type is displayed immediately after the viewer selects that format type, hi some embodiments, the preview effect for a subsequently selected format type is superimposed on the preview effect for a previously selected format type.

[0054] In step S908, the acquiring unit 320 acquires a selection result (or a final selection result) from the viewer through the selection user interface. The final selection result corresponds to the latest selection result candidate. The processing unit 322 may refer to the comment format DB 330, access the corresponding parameter file, and set the corresponding format file in the comment content DB 332 according to the parameter file.

[0055] For example, as shown in Fig. 8, user U1 selects "size A, sound effect B" as the selection result for expressing comment CM2. The processing unit 322 accesses the corresponding parameter files through URL_SA_para and URL_EB_para in the comment format DB 330, and sets the format file in URL_CM2 according to the two parameter files.

[0056] In step S910, the processing unit 322 renders the comment in the live stream room according to the selection result, that is, the processing unit 322 renders the comment according to the format file set according to the parameter file determined by the selection result.

[0057] In this embodiment, the comments from the viewer are not displayed in the live stream room until the format selection is confirmed by the viewer.

[0058] FIG. 10 illustrates an exemplary flow according to some embodiments of the present invention.

[0059] In step S1000, the acquiring unit 320 acquires comments from viewers in the live stream room, and the viewers may input the comments in a comment input area via their user terminals.

[0060] In step S1002, the processing unit 322 displays the comment in the stream room in a default format.

[0061] In step S1004, the processing unit 322 displays the selected object associated with the comment displayed in the default format. The selected object may be displayed only on the user terminal of the viewer who sent the comment.

[0062] In step S1006, the obtaining unit 320 obtains a selection signal for the selected object from the viewer, that is, the viewer selects or clicks the selected object.

[0063] Steps S1008, S1010, S1012, S1014, and S1016 are the same as or similar to steps S902, S904, S906, S908, and S910, respectively.

[0064] In this embodiment, the comments are initially displayed in a default format within the live stream room and are later re-presented according to the selections determined by the viewer. This mechanism can draw the streamer's attention by re-emphasizing previously submitted comments.

[0065] 11 illustrates an example of a live stream room according to some embodiments of the present invention. The user interface is from the perspective of a viewer V1. The enhancement or emphasis of the comment occurs during (or at the time of) the comment entry.

[0066] At timing t1, viewer V1 inputs the comment in the comment input area, and selects the "Highlight Send" button to highlight the comment.

[0067] At timing t2, a format selection user interface (UI) for selecting how to highlight the comment is displayed to the viewer. The UI displays main format types, options for the selected main format type, corresponding points, a preview area, and a confirm button. In this embodiment, viewer V1 selects "Size" and then selects "Size C," causing the comment to be displayed in the preview area with the "Size C" effect (this is a selection result candidate). Viewer V1 then selects "Confirm" to confirm the selection. In some embodiments, the format selection UI is displayed by the processing unit 322 by referencing the comment format DB 330.

[0068] At timing t3, viewer V1's comment with the "size C" effect is displayed in the stream room. This display may be performed by, for example, the processing unit 322 with reference to the comment content DB 332. In some embodiments, the processing unit 322 causes the viewer to pay a number of points corresponding to the selection on the streaming platform. For example, the processing unit 322 may refer to the comment content DB 332 for a total point value for expressing the comment and deduct the number of points from the viewer's account in the user DB 312.

[0069] In some embodiments, the processing unit 322 may cooperate with the broadcast information providing unit 302 or the relay unit 304 to render the comment.

[0070] 12 illustrates an example of a live stream room according to some embodiments of the present invention. The user interface is from the perspective of a viewer V1. The enhancement or emphasis of the comment occurs after the comment is entered.

[0071] At timing t1, viewer V1 inputs the comment in the comment input area, and then selects the "Send" button to send the comment.

[0072] At timing t2, the comment from viewer V1 is displayed in a default format in the comment display area within the stream room. A "highlight" button associated with the comment is displayed. In some embodiments, the highlight button associated with the comment from viewer V1 is displayed only to viewer V1. Viewer V1 then selects the "highlight" button to highlight the comment. A format selection UI such as the UI shown in FIG. 11 is then displayed. The viewer may then select how to highlight the comment. Once the selection is determined, the comment is re-presented in the stream room according to the selected format. The previous comment in the default format may or may not be maintained in the comment display area.

[0073] In some embodiments, for comments that have already been submitted, only comments submitted within a predetermined period (from the current time) may be highlighted. For example, the processing unit 322 may refer to the comment content DB 332 to check the timing of each comment and display only the highlight button associated with the comment submitted within the predetermined period. This may encourage viewers to use the highlight feature in a timely manner and improve revenue for the streaming platform.

[0074] In some embodiments, the comment format DB 330 may be implemented on the user terminal side rather than on the server side. For example, the data of the comment format DB 330 may be transmitted from the server to the user terminal as code or configuration data. When the comment format DB 330 is implemented on the user terminal side, the preview process may be performed by a processing unit in the user terminal, thereby speeding up the process and saving communication resources between the user terminal and the server. After the user inputs a comment in the comment input area and selects the format, the user terminal may transmit the comment content along with corresponding expression parameters to the server. The server may then distribute the comment to the stream room in the selected format, so that the highlighted comment can be seen by all users in the stream room.

[0075] In some embodiments, a cache memory in the user terminal or server may be used to store past comments submitted by the viewer, and the lifespan of the stored comments may be set to a predetermined period during which the comments may be highlighted.

[0076] Figure 13 shows an example of a live stream room according to some embodiments of the present invention. The user interface is from the perspective of viewer V1.

[0077] At timing t1, viewer V1 enables the highlighting function for the comment, and the format selection UI is displayed. Viewer V1 selects "Size" and "Size C." The comment is displayed in size C in the preview area.

[0078] At timing t2, viewer V1 further selects "Movement Pattern" and "Pattern A." The effect of "Movement Pattern A" is displayed overlaid on the effect of "Size C" in the preview area. Viewer V1 then selects "Confirm" to confirm the selection.

[0079] In some embodiments, the processing unit 322 may access the comment content in the comment content DB 332 and the parameter files and preview files for multiple format types in the comment format DB 330 to display a preview of the superimposed effect.

[0080] At timing t3, the comment from viewer V1 is displayed in the stream room with the effect of "size C" and the effect of "movement pattern A." In some embodiments, the comment is also displayed in the comment display area in a default format.

[0081] Figure 14 shows an example of a format selection UI according to some embodiments of the present invention, which is displayed when the viewer selects to use the highlight feature for a comment.

[0082] For each format type, a preview (or preview) and adjustment / selection object are displayed separately. The viewer can scroll the "Size Format" scroll bar to adjust the resize and view the corresponding point number. The viewer can select and listen to different sound effects. The viewer can scroll the "Volume Format" scroll bar to adjust the volume of the message being read out loud. The viewer can select different movement patterns based on the image (or animated image or thumbnail video). Some or all of the selected effects may be superimposed or combined to determine the selection result of the format. The processing unit 322 displays the comment in the stream room according to the selection result.

[0083] In some embodiments, the format type for highlighting a comment may include pinning or fixing the comment in a location within the stream room for a period of time, meaning that the comment will not be replaced (or washed away) by a subsequent comment.

[0084] In some embodiments, the format type for expressing a comment may include a female voice or a male voice. For example, the processing unit may use the machine learning DB 334 to read the comment content in a corresponding voice. In some embodiments, the processing unit may use an AI model in the machine learning DB 334 to detect the mood of the comment content and read the comment content with a corresponding intonation. In some embodiments, if a sound effect is selected to express a comment, the processing unit may use an AI model in the machine learning DB 334 to detect the mood of the comment content and adjust the sound effect (e.g., volume, frequency, pitch, etc.) based on the mood of the comment content.

[0085] In some embodiments, the voice reading the comments may be the voice (or animated voice) of a character from a well-known game or animation. In some embodiments, the streaming platform may collaborate with or obtain permission from a well-known intellectual property (IP). The voice may then be used to generate audio / sound gifts for use in events (or campaigns) on the streaming platform.

[0086] In some embodiments, the format type for expressing a comment may include reading or displaying the comment in a different language. For example, the processing unit may use an AI model in the machine learning DB 334 to translate the comment content and read or display the translated comment in the stream room.

[0087] Some embodiments of the present invention allow viewers to express their comments in a way that makes it easier for them to draw the streamer's attention. For example, when a streamer is live streaming or playing a game, the streamer may not be able to see the comments of the viewers in the stream room. The sound effects and speech readout function may be useful for conveying the comments to the streamer.

[0088] The hardware configuration of the information processing device will be described with reference to Fig. 15. Fig. 15 is a block diagram showing an example of the hardware configuration of an information processing device based on some embodiments of the present invention. The information processing device 900 shown in the figure can realize, for example, the server 10 and / or the user terminals 20 and 30 in some embodiments.

[0089] The information processing device 900 includes a CPU 901, a ROM (read-only memory) 903, and a RAM (random access memory) 905. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 925, and a communication device 929. The information processing device 900 also includes an imaging device such as a camera (not shown). The information processing device 900 may also include a DSP (digital signal processor) or an ASIC (application-specific integrated circuit) in addition to or instead of the CPU 901.

[0090] The CPU 901 functions as an arithmetic processing device and control device, controlling all or part of the operation of the information processing device 900 in accordance with various programs stored in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 923. For example, the CPU 901 controls the overall operation of each functional unit included in the server 10 and the user terminals 20 and 30 in some embodiments. The ROM 903 stores programs and calculation parameters used by the CPU 901. The RAM 905 functions as a main storage device that stores programs used during execution by the CPU 901 and parameters that change appropriately during execution. The CPU 901, ROM 903, and RAM 905 are interconnected by a host bus 907, which may be an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 909.

[0091] The input device 915 may be a device operated by a user, such as a mouse, keyboard, touch panel, button, switch, or lever, or a device that converts a physical quantity into an electrical signal, such as a sound sensor typified by a microphone, an acceleration sensor, a tilt sensor, an infrared sensor, a depth sensor, a temperature sensor, or a humidity sensor. The input device 915 may be, for example, a remote control device that uses radio waves such as infrared, or an external connection device 927 such as a mobile phone that supports operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user or a detected physical quantity and outputs the signal to the CPU 901. The user operates the input device 915 to input various data to the information processing device 900 and instruct it to operate.

[0092] The output device 917 is a device that can visually or audibly notify the user of acquired information. The output device 917 may be, for example, a display such as an LCD, PDP, or OLED, an audio output device such as a speaker or headphones, or a printer. The output device 917 outputs the processing results of the information processing device 900 as video such as text or an image, or audio such as voice.

[0093] The storage device 919 is a data storage device and is configured as an example of a storage unit of the information processing device 900. The storage device 919 is, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs executed by the CPU 901, various data, various data acquired from the outside, and the like.

[0094] The drive 921 is a reader / writer of a removable recording medium 923 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the information processing device 900. The drive 921 reads information recorded on the attached removable recording medium 923 and outputs it to the RAM 905. Furthermore, the drive 921 writes information to the attached removable recording medium 923.

[0095] The connection port 925 is a port for directly connecting a device to the information processing device 900. The connection port 925 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. Furthermore, the connection port 925 may be an RS-232C port, an optical audio terminal, an HDMI (High-Definition Multimedia Interface) port, or the like. By connecting the external connection device 927 to the connection port 925, various data can be exchanged between the information processing device 900 and the external connection device 927.

[0096] The communication device 929 is, for example, a communication interface formed by a communication device for connecting to the network NW. The communication device 929 may be, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (trademark), or wireless USB (WUSB). Furthermore, the communication device 929 may be a router for optical communication, a router for ADSL (asymmetric digital subscriber line), or a modem for various communications. The communication device 929 transmits and receives signals, etc., over the Internet or with other communication devices using a predetermined protocol such as TCP / IP. The communication network NW connected to the communication device 929 is a wired or wireless network, such as the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication. The communication device 929 functions as a communication device.

[0097] The imaging device (not shown) is a device that captures an image of real space using an imaging element such as a CCD (charge-coupled device) or a CMOS (complementary metal-oxide semiconductor), and various components such as a lens for controlling the formation of a subject image on the imaging element, and generates a captured image. The imaging device may capture either a still image or a moving image.

[0098] The above describes the configuration and operation of the live streaming system 1 according to this embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible with respect to the combination of each component and each process, and that such modifications are also within the scope of the present invention.

[0099] The processes and procedures described in the present invention, in addition to those explicitly described, can be implemented by software, hardware, or any combination thereof. For example, the processes and procedures described herein can be implemented by implementing logic corresponding to the processes and procedures in media such as integrated circuits, volatile memory, non-volatile memory, non-transitory computer-readable media, magnetic disks, etc. Furthermore, the processes and procedures described herein can be implemented as computer programs corresponding to the processes and procedures, and can be executed by various types of computers.

[0100] Furthermore, the system or method described in the above embodiments may be integrated into a program stored in a non-transitory computer-readable medium such as a solid-state storage device, an optical disk storage device, a magnetic disk storage device, etc. Alternatively, the program may be downloaded from a server via the Internet and executed by a processor.

[0101] Although the technical contents and features of the present invention have been described above, those skilled in the art can still make many variations and modifications without departing from the teachings and disclosure of the present invention. Therefore, the scope of the present invention is not limited to the embodiments already disclosed, but is within the scope of the claims, including other variations and modifications that do not depart from the present invention. [Explanation of symbols]

[0102] 1. Communication Systems 10 Servers 20 User terminal 30, 30a, 30b User terminal LV Streamer AU1, AU2 viewers VD, VD1, VD2 video NW Network 30 User terminals 100 delivery units 102 Imaging control unit 104 Audio control unit 106 Video Transmission Unit 108 Streamer UI Control Unit 200 viewing units 202 Viewer-side UI control unit 204 Superposition Information Generation Unit 206 Input information transmission unit 302 Distribution Information Provision Unit 304 Relay unit 306 Gift Processing Unit 308 Payment Processing Unit 310 Stream DB 312 User DB 314 Gift DB 320 Acquired Units 322 Processing Unit 330 Comment Format DB 332 Comment Content DB 334 Machine Learning DB 900 Information Processing Equipment 901 CPU 903 ROM 905 RAM 907 Host Bus 909 Bridge 911 External Bus 913 Interface 915 Input Device 917 Output Device 919 Storage Device 921 Drive 923 Removable Recording Media 925 connection port 927 External Connection Device 929 Communication Equipment

Claims

1. 1. A method of live streaming performed by a server, comprising: obtaining comments from viewers in the live stream room; a step of displaying to the viewer a selection user interface for allowing the viewer to select an effect to be applied to the acquired comment, the selection user interface displaying selection result candidates and corresponding points in association with each other; obtaining a selection result candidate selected by the viewer via the selection user interface; a step of presenting the comment to which an effect corresponding to the selection result candidate has been applied to the viewer before the selection result candidate is confirmed as a selection result by the viewer; obtaining the confirmed selection from the viewer via the selection user interface; and expressing the comment in the live stream room based on the selection.

2. 2. The method of claim 1, wherein if the viewer selects and confirms multiple possible selection results, the selection is associated with the sum of multiple point values ​​corresponding to the multiple possible selection results.

3. 2. The method of claim 1, wherein the selection is associated with a volume at which the comment is expressed.

4. 2. The method of claim 1, wherein the selection is associated with a sound effect representing the comment.

5. 2. The method of claim 1, wherein the selection is associated with a movement pattern that represents the comment.

6. 2. The method of claim 1, wherein the selection is related to a size in which the comment is represented.

7. 2. The method of claim 1, wherein the selection is associated with a language in which the comment is expressed.

8. The method of claim 1 , wherein the rendering of the comment comprises reading aloud the content of the comment in the live stream room.

9. The method of claim 1 , wherein the comment is not displayed in the live stream room until the comment is expressed in the live stream room based on the selection result.

10. 2. The method of claim 1, further comprising the step of having the viewer pay a number of points corresponding to the selection on a streaming platform that provides the live stream room.

11. 1. A system for live streaming, comprising one or more processors, wherein the one or more processors execute machine-readable instructions to: obtaining comments from viewers in the live stream room; displaying said comments in a default format within said live stream room; displaying a selected object associated with the comment displayed in the default format only if the comment was acquired within a predetermined period from the current timing; and enabling the viewer to select an effect to be applied to the comment displayed in the live stream room in response to the viewer's selection of the selected object.

12. The method according to claim 1, wherein the step of presenting includes a step of enabling the content of the comment to be read aloud and / or a volume representing the comment and / or a sound effect representing the comment to be heard or heard.

13. The method of claim 1 , wherein the effect applied to the comment corresponds to a mood detected from the content of the comment.

Images