Systems and methods for live streaming

Abstract

We provide a platform suitable for live streaming. [Solution] The live streaming method includes the steps of: obtaining comments from viewers in a live stream room (S900); displaying a selection user interface to the viewers (S904); obtaining selection results from the viewers via the selection user interface (S908); and expressing the comments in the live stream room based on the selection results (S910).

Description

Technical Field

[0001] The present invention relates to streaming, and more particularly to live streaming.

Background Art

[0002] As represented by live streaming services, real-time communication on the Internet has penetrated into daily life. Various platforms and providers offer live streaming services, and the competition is fierce. For platforms, it is important to provide services that users desire.

[0003] Japanese Unexamined Patent Application Publication No. 2019-144311 discloses a system that supports a streamer's reply to a viewer's comment.

Summary of the Invention

[0004] A method according to an embodiment of the present invention is a method for live streaming executed by one or more computers, including the steps of obtaining comments from viewers in a live stream room, displaying a selection user interface to the viewers, obtaining selection results from the viewers via the selection user interface, and presenting comments in the live stream room based on the selection results.

[0005] A system according to an embodiment of the present invention is a system for live streaming, including one or more processors, and the one or more computer processors execute machine-readable instructions to obtain comments from viewers in a live stream room, display a selection user interface to the viewers, obtain selection results from the viewers via the selection user interface, and present comments in the live stream room based on the selection results.

[0006] A non-temporary computer-readable medium including a program for live streaming according to one embodiment of the present invention causes one or more computers to perform the steps of: obtaining comments from viewers in a live stream room; displaying a selection user interface to the viewers; obtaining selection results from the viewers via the selection user interface; and representing the comments in the live stream room based on the selection results. [Brief explanation of the drawing]

[0007] [Figure 1] This is a schematic diagram showing the configuration of a live streaming system 1 based on some embodiments of the present invention. [Figure 2] This block diagram shows the functions and configuration of the user terminal 30 in Figure 1, based on some embodiments of the present invention. [Figure 3] This block diagram shows the functions and configuration of the server in Figure 1, based on some embodiments of the present invention. [Figure 4] Figure 3 is a table showing an example data structure for the stream DB310. [Figure 5] Figure 3 is a table showing an example data structure for user DB312. [Figure 6] Figure 3 is a table showing an example data structure for the gift DB314. [Figure 7] This table shows an example data structure for DB330 in comment format. [Figure 8] This table shows an example data structure for DB332, based on the comments. [Figure 9] This is an exemplary flow based on some embodiments of the present invention. [Figure 10] This is an exemplary flow based on some embodiments of the present invention. [Figure 11] This is an example of a livestream room based on some embodiments of the present invention. [Figure 12] This is an example of a livestream room based on some embodiments of the present invention. [Figure 13] This is an example of a livestream room based on some embodiments of the present invention. [Figure 14] This is an example of a format selection UI based on some embodiments of the present invention. [Figure 15] This is a block diagram showing an example of the hardware configuration of an information processing device based on some embodiments of the present invention. [Modes for carrying out the invention]

[0008] In the following drawings, identical or similar components, members, procedures, or signals are denoted by the same reference numerals, and redundant explanations are omitted where appropriate. Furthermore, some components that are not essential to the description of each drawing are omitted.

[0009] The ability to comment on streaming platforms is a crucial feature that allows viewers to interact with streamers. Therefore, it is desirable for platforms to offer flexible commenting capabilities.

[0010] Figure 1 is a schematic diagram showing the configuration of a live streaming system 1 based on some embodiments of the present invention. The live streaming system 1 provides a live streaming service for real-time interaction or communication between a streaming streamer (also called a live streamer, anchor, broadcaster, or live streamer) LV and a viewer (also called an audience) AU (AU1, AU2...). As shown in Figure 1, the live streaming system 1 includes a server 10, a user terminal 20, and user terminals 30 (30a, 30b...). In some embodiments, the streamer and the viewer may be collectively referred to as users. The server 10 may include one or more information processing devices connected to a network NW. The user terminals 20 and 30 may be mobile devices such as smartphones, tablets, notebook PCs, recorders, portable game consoles, wearable devices, or stationary devices such as desktop PCs. The server 10, the user terminals 20, and the user terminals 30 are connected to each other so as to be able to communicate with each other via various wired or wireless networks NW.

[0011] The live streaming system 1 includes a broadcaster (LV), a viewer (AU), and an administrator (or application provider, not shown) who manages the server 10. The broadcaster (LV) is a person who broadcasts content in real time by recording content on their user terminal 20 and uploading it directly or indirectly to the server 10. Examples of such content may include the broadcaster's own songs, talks, performances, gameplay, or any other content. The administrator provides a platform for live streaming the content on the server 10 and mediates or manages real-time interaction between the broadcaster (LV) and the viewer (AU). The viewer (AU) accesses the platform on their user terminal 30 and selects and watches the desired content. While the selected content is live streaming, the viewer (AU) performs actions such as sending comments, support, and gifts via their user terminal 30. The broadcaster (LV) broadcasting the content may respond to these comments, support, or gifts. Such responses are sent to the viewer (AU) via video and / or audio, establishing two-way communication.

[0012] The term "live streaming" may refer to a data transmission mode that enables content recorded on the user terminal 20 of the broadcaster LV to be played back and viewed substantially in real time on the user terminal 30 of the viewer AU, or it may refer to a live broadcast realized by such a transmission mode. Such live streaming may be realized using existing live streaming technologies such as HTTP live streaming, CMAF (Common Media Application Format), WebRTC (Web Real-Time Communications), RTMP (Real-Time Messaging Protocol), etc. Live streaming includes a transmission mode that allows the viewer AU to view the content with a predetermined delay simultaneously with the recording of the content by the broadcaster LV. The length of this delay may be such that it is sufficient for communication between the broadcaster LV and the viewer AU to take place. Such live streaming is distinguished from so-called on-demand delivery, where all recorded data of the content is stored on a server and then provided to the user from the server upon user request.

[0013] The term "video data" as used herein refers to data that includes image data (also called video data) generated using the imaging function of the user terminal 20 or 30, and audio data generated using the audio input function of the user terminal 20 or 30. The video data is played back on the user terminals 20 and 30 so that the user can view the content. In some embodiments, it is assumed that between the generation of the video data on the broadcaster's user terminal and the playback of the video data on the viewer's user terminal, processing is performed on the video data to change its format, size, or data specifications, such as compression, decompression, encoding, decoding, or transcoding. However, since the content represented by the video data (e.g., video and audio) does not substantially change before and after such processing, in this specification, the video data after such processing is described as being identical to the video data before such processing. In other words, when video data is generated on the broadcaster's user terminal and then played back on the viewer's user terminal via the server 10, the video data generated on the broadcaster's user terminal, the video data passing through the server 10, and the video data received and played back on the viewer's user terminal are all the same video data.

[0014] In the example shown in Figure 1, the broadcaster LV provides live streaming data. The broadcaster LV's user terminal 20 records the broadcaster LV's video and audio to generate the streaming data, and the generated data is transmitted to the server 10 via the network NW. At the same time, the user terminal 20 displays the recorded video VD of the broadcaster LV on its display, allowing the user terminal 20 to check the live streaming content currently being broadcast by the broadcaster LV.

[0015] Each of the user terminals 30a and 30b of the viewers AU1 and AU2, which requested to watch the live streaming of the distributor LV on the platform, receives video data related to the live streaming (hereinafter may be referred to as "video data of the live streaming") via the network NW, plays back the received video data, displays the videos VD1 and VD2 on the display, and outputs audio from the speaker. The videos VD1 and VD2 displayed on the user terminals 30a and 30b are substantially the same as the video VD captured by the user terminal 20 of the distributor LV, and the audio output from the user terminals 30a and 30b is substantially the same as the audio recorded by the user terminal 20 of the distributor LV.

[0016] The recording of video and audio on the user terminal 20 of the distributor LV and the playback of the video data on the user terminals 30a and 30b of the viewers AU1 and AU2 are performed substantially simultaneously. When the viewer AU1 inputs a comment on the content provided by the distributor LV into the user terminal 30a, the server 10 displays the comment on the user terminal 20 of the distributor LV in real time, and also displays the comment on the user terminals 30a and 30b of the viewers AU1 and AU2 respectively. When the distributor LV reads the comment and develops a talk corresponding to the comment, the video and audio of the talk are displayed on the user terminals 30a and 30b of the viewers AU1 and AU2 respectively. This interactive operation is recognized as a conversation being established between the distributor LV and the viewer AU1. Thereby, in the live streaming system 1, a live streaming that enables two-way communication rather than one-way communication is realized.

[0017] Figure 2 is a block diagram showing the functions and configuration of the user terminal 30 in Figure 1, based on some embodiments of the present invention. The user terminal 20 has the same or similar functions and configuration as the user terminal 30. Each block in Figure 2 and subsequent block diagrams may be implemented by hardware elements such as a computer CPU or mechanical devices, and by software such as computer programs. Functional blocks may be implemented by the coordinated operation of these elements. Therefore, it will be understood by those skilled in the art that these functional blocks can be implemented in various forms by combinations of hardware and software.

[0018] The broadcaster (LV) and the viewer (AU) may download a live streaming application program (hereinafter referred to as the live streaming application) from a download site via the network (NW) and install it on the user terminals 20 and 30. Alternatively, the live streaming application may be pre-installed on the user terminals 20 and 30. When the live streaming application is executed on the user terminals 20 and 30, the user terminals 20 and 30 communicate with the server 10 via the network (NW) and implement or execute various functions. Hereinafter, the functions implemented by the user terminals 20 and 30 (processors such as CPUs) on which the live streaming application is executed will be described as functions of the user terminals 20 and 30. These functions are actually realized on the user terminals 20 and 30 by the live streaming application. In some embodiments, these functions may be written in a programming language such as HTML (HyperText Markup Language), sent from the server 10 via the network (NW) to the web browser of the user terminals 20 and 30, and implemented by a computer program executed by the web browser.

[0019] The user terminal 30 includes a distribution unit 100 and a viewing unit 200. The distribution unit 100 generates video data in which the video and audio of the user (or the user side) are recorded, and provides the video data to the server 10. The viewing unit 200 receives video data from the server 10 and plays the video data. When the user performs live streaming, the distribution unit 100 is activated, and when the user views a video, the viewing unit 200 is activated. The user terminal in which the distribution unit 100 is activated is the terminal of the distributor, that is, the user terminal that generates the video data. The user terminal in which the viewing unit 200 is activated is the terminal of the viewer, that is, the user terminal in which the video data is reproduced and played.

[0020] The distribution unit 100 includes an imaging control unit 102, an audio control unit 104, a video transmission unit 106, and a distributor-side UI control unit 108. The imaging control unit 102 is connected to a camera (not shown in Figure 2) and controls the imaging performed by the camera. The imaging control unit 102 acquires image data from the camera. The audio control unit 104 is connected to a microphone (not shown in Figure 2) and controls the audio input from the microphone. The audio control unit 104 acquires audio data from the microphone. The video transmission unit 106 transmits video data, including the image data acquired by the imaging control unit 102 and the audio data acquired by the audio control unit 104, to the server 10 via the network NW. The video data is transmitted in real time by the video transmission unit 106. In other words, the generation of the video data by the imaging control unit 102 and the audio control unit 104, and the transmission of the generated video data by the video transmission unit 106, are performed substantially simultaneously. The broadcaster-side UI control unit 108 controls the broadcaster's UI (user interface). The broadcaster-side UI control unit 108 may be connected to a display (not shown in Figure 2), and by playing the video data transmitted by the video transmission unit 106, it displays video on the display. The broadcaster-side UI control unit 108 may display operation objects and instruction permission objects on the display and accept input from the broadcaster by tapping these objects.

[0021] The viewing unit 200 includes a viewer-side UI control unit 202, an overlay information generation unit 204, and an input information transmission unit 206. The viewing unit 200 receives video data related to live streaming from the server 10 via the network NW, in which the broadcaster, the viewer (a user of the user terminal 30), and other viewers participate. The viewer-side UI control unit 202 controls the viewer's UI. The viewer-side UI control unit 202 is connected to a display and a speaker (not shown in Figure 2), and plays the received video data to display the video on the display and output sound from the speaker. The state in which the video is output to the display and the sound is output from the speaker can be called the state in which "video data is being played". The viewer-side UI control unit 202 is also connected to input means such as a touch panel, keyboard, and display (not shown in Figure 2), and acquires user input via these input means. The overlay information generation unit 204 overlays a predetermined frame image onto the image generated from the video data from the server 10. The frame image includes various user interface objects (hereinafter simply referred to as "objects") for receiving input from the user, comments entered by the viewer, and / or information obtained from the server 10. The input information transmission unit 206 transmits the user input obtained by the viewer-side UI control unit 202 to the server 10 via the network NW.

[0022] Figure 3 is a block diagram showing the functions and configuration of the server 10 in Figure 1, based on some embodiments of the present invention. The server 10 includes a distribution information provision unit 302, a relay unit 304, a gift processing unit 306, a payment processing unit 308, a stream DB 310, a user DB 312, a gift DB 314, an acquisition unit 320, a processing unit 322, a comment format DB 330, a comment content DB 332, and a machine learning DB 334.

[0023] When the broadcaster receives a notification or request to start a live stream via the network NW from the user terminal 20 on the broadcaster's side, the broadcast information provision unit 302 registers the stream ID for identifying the live stream and the broadcaster ID of the broadcaster performing the live stream in the stream DB 310.

[0024] When the distribution information provision unit 302 receives a request for information about live streams from the viewing unit 200 of the user terminal 30 on the viewer side via the network NW, the distribution information provision unit 302 retrieves or checks currently available live streams from the stream DB 310 and creates a list of available live streams. The distribution information provision unit 302 sends the created list to the requesting user terminal 30 via the network NW. The viewer-side UI control unit 202 of the requesting user terminal 30 generates a live stream selection screen based on the received list and displays it on the user terminal 30's display.

[0025] When the input information transmission unit 206 of the user terminal 30 receives the viewer's selection result on the live stream selection screen, the input information transmission unit 206 generates a distribution request including the stream ID of the selected live stream and sends the request to the server 10 via the network NW. The distribution information provision unit 302 begins providing the requesting user terminal 30 with the live stream specified by the stream ID included in the received distribution request. The distribution information provision unit 302 updates the stream DB 310 to include the user ID of the viewer of the requesting user terminal 30 in the viewer ID of the stream ID (or the corresponding viewer ID).

[0026] The relay unit 304 relays the video data from the broadcaster's user terminal 20 to the viewer's user terminal 30 in the live streaming initiated by the distribution information provision unit 302. During the live streaming or playback of the video data, the relay unit 304 receives a signal from the input information transmission unit 206 representing user input from the viewer. The signal representing user input may be an object designation signal that specifies an object displayed on the user terminal 30's display. The object designation signal may include the viewer's viewer ID, the broadcaster ID of the broadcaster of the live stream the viewer is watching, and an object ID that identifies the object. If the object is a gift, the object ID is a gift ID. Similarly, the relay unit 304 receives a signal from the distribution unit 100 of the user terminal 20 representing user input made by the broadcaster during the playback of the video data (or during live streaming). This signal may be an object designation signal.

[0027] Furthermore, the signal representing the user input may also be a comment input signal that includes a comment entered by the viewer into the user terminal 30 and the viewer's viewer ID. Upon receiving the comment input signal, the relay unit 304 transmits the comment and the viewer ID included in the signal to the broadcaster's user terminal 20 and to the user terminals 30 of other viewers. In these user terminals 20 and 30, the viewer-side UI control unit 202 and the overlay information generation unit 204 display the received comment on a display associated with the viewer ID that was also received.

[0028] The gift processing unit 306 updates the user database 312 by increasing the points of the sender based on the points of the gift identified by the gift ID included in the object designation signal. Specifically, the gift processing unit 306 refers to the gift database 314 to identify the points to be awarded for the gift ID included in the received object designation signal. Subsequently, the gift processing unit 306 updates the user database 312 and adds the identified points to the points of the sender ID (or the points corresponding to the sender ID) included in the object designation signal.

[0029] The payment processing unit 308 processes payment for a gift from a viewer in response to receiving the object designation signal. Specifically, the payment processing unit 308 refers to the gift DB 314 to identify the price points of the gift, which are identified by the gift ID included in the object designation signal. The payment processing unit 308 then updates the user DB 312 and subtracts the identified price points from the points of the viewer, which are identified by the viewer ID included in the object designation signal.

[0030] Figure 4 is a table showing an exemplary data structure of the Stream DB 310 in Figure 3. The Stream DB 310 holds information about the currently running live stream. The Stream DB 310 stores the Stream ID, Streamer ID, and Viewer ID in relation to each other. The Stream ID is an ID used to identify a live stream on the live streaming platform provided by the Live Streaming System 1. The Streamer ID is a user ID used to identify the streamer providing the live stream. The Viewer ID is a user ID used to identify the viewers of the live stream. In some embodiments of the Live Streaming System 1, when a user starts a live stream, that user becomes a streamer, and when the same user watches a live stream broadcast by another user, that user also becomes a viewer. Therefore, the distinction between streamer and viewer is not fixed, and a user ID registered as a streamer ID at one time may be registered as a viewer ID at another time.

[0031] Figure 5 is a table showing an exemplary data structure of the user DB 312 in Figure 3. The user DB 312 holds information about users. The user DB 312 stores the user ID and the points in relation to each other. The user ID identifies the user. The points correspond to the points held by the corresponding user. The points are electronic value that circulates within the live streaming platform. In some embodiments, when a streamer receives a gift from a viewer during a live stream, the streamer's points increase by the value corresponding to the gift. The points are used, for example, to determine the amount of compensation (such as money) that the streamer receives from the administrator of the live streaming platform. In some embodiments, when the streamer receives a gift from a viewer, the streamer may be given an amount of money corresponding to the gift instead of points.

[0032] Figure 6 is a table showing an exemplary data structure of the gift DB314 in Figure 3. The gift DB314 holds information about gifts that viewers can use during the live stream. Gifts are electronic data. Gifts may be purchased with points or money, or they may be provided free of charge. Gifts can be sent by viewers to the streamer. Sending a gift to a streamer is also called using a gift, sending a gift, or throwing a gift. Some gifts can be used immediately upon purchase, while others can be used later by the viewer at any time after purchase. When a viewer sends a gift to a streamer, the streamer is awarded a corresponding amount of points for that gift. When a gift is used, an effect associated with that gift may occur. For example, an effect (such as a visual or auditory effect) corresponding to the gift may be displayed on the live streaming screen.

[0033] The gift database 314 stores the gift ID, awarded points, and price points in an associated manner. The gift ID is used to identify the gift. The awarded points are the amount of points awarded to the broadcaster when a gift is sent to them. The price points are the amount of points paid for the use (purchase) of the gift. Viewers can send a desired gift to a broadcaster by paying the price points for the desired gift while watching a live stream. Payment of the price points can be made by any appropriate electronic payment method. For example, payment may be made by the viewer paying the price points to the administrator. Alternatively, payment may be made by bank transfer or credit card. The administrator can arbitrarily set the relationship between the awarded points and the price points. For example, it may be set so that awarded points = price points. Alternatively, the price points may be set by multiplying the awarded points by a predetermined coefficient such as 1.2, or by adding a predetermined fee point to the awarded points.

[0034] Figure 7 is a table showing an exemplary data structure for the comment format DB330. This comment format DB330 stores comment format types, points, preview files, and parameter files in an interconnected manner.

[0035] The comment format type stores a variety of format types for representing comments, and each format type has multiple options. For example, the format type "size" may have different size options for representing comments. In some embodiments, size A < size B < size C. For example, the format type "sound effect" may have multiple sound effects that can be played when representing the comment. These sound effects may or may not be related to the content of the comment.

[0036] For example, the format type "Volume" may have different volumes for representing the comment. In some embodiments, the representation of the comment may include reading the content of the comment aloud. For example, the format type "Voice Format" may have different voice types for representing the comment, such as a male voice, a female voice, a robot voice, or an anime voice. For example, the format type "Movement Pattern" may have different movement patterns of the comment when representing the comment. Different patterns may be different directions, different speeds, and / or different trajectories of the comment movement.

[0037] This point system stores the number of points a user (such as a viewer) must spend if they select a specific format type for expressing their comment. For example, if a user wants to express their comment in size A, they pay 50 points. For example, if a user wants their comment read aloud at volume B, they pay 100 points.

[0038] The preview file stores the URL of the preview file, and each preview file corresponds to a specific format type of comment expression. In some embodiments, the preview file corresponding to "size" or "movement pattern" may be in image or video format. In some embodiments, the preview file corresponding to "sound effect," "volume," or "audio format" may be in audio format.

[0039] The parameter file stores the URLs of parameter setting files, with each parameter setting file corresponding to a specific format type of comment representation. For example, the parameter setting file URL_SA_para contains the parameters necessary to represent (or render) the comment at size A. For example, the parameter setting file URL_MB_para contains the parameters necessary to represent (or render) the comment according to movement pattern B.

[0040] Figure 8 is a table showing an example data structure of the comment content DB332. The comment content DB332 stores the time, comment ID, comment content, user ID, selection result, total points, and format file in an interrelated manner.

[0041] The specified time specifies the timing or period during which comments were received. The specified comment ID is configured to identify each comment. The specified comment content stores the content of the comment. The specified user ID identifies the user who sent the comment.

[0042] The selection result stores the format type selection result chosen by the user for the comment. For example, user U2 selects "Size C + Volume A + Movement Pattern B" for the expression of comment CM1. For example, user U1 selects "Size A + Sound Effect B" for the expression of comment CM2. The selection result is associated with the size, sound effect, volume, audio format, and / or movement pattern of the comment expression.

[0043] The total points represent the number of points the user must pay based on their selection. The selection is associated with the number of points the viewer pays.

[0044] In this embodiment, the total points are the sum of the points for each selected format type. For example, user U2 would need to pay 280 points for the expression of comment CM1, which is the sum of 150 points (size C), 50 points (volume A), and 80 points (movement pattern B). For example, user U1 would need to pay 120 points for the expression of comment CM2, which is the sum of 50 points (size A) and 70 points (sound effect B). In other embodiments, a different calculation may be applied to the total points.

[0045] The format file stores the URL of the format file. Each format file contains data, code, or instructions that indicate how the comment will be represented in the stream room. In some embodiments, the processing unit 322 (or the distribution information provision unit 302, or the relay unit 304) refers to the format file of the comment in order to represent the comment in the stream room.

[0046] In some embodiments, the data in the comment content DB332 may be acquired or stored by the acquisition unit 320. The total points may be calculated by the processing unit 322.

[0047] The machine learning database 334 may store various machine learning models and AI models. For example, it may contain one or more large language models (LLMs). It may also contain various types of text-to-speech (TTS) models.

[0048] Figure 9 shows an exemplary flow based on some embodiments of the present invention.

[0049] In step S900, the acquisition unit 320 acquires comments from viewers in the live stream room. Viewers may enter their comments in the comment input area via their user terminal.

[0050] In step S902, the processing unit 322 displays a selection user interface on the user's terminal. The selection user interface includes various format types that the viewer can choose to express the comment. The point values ​​corresponding to each format type may also be displayed on the selection user interface.

[0051] In step S904, the acquisition unit 320 acquires candidate selections from the viewer via the selection user interface. For example, the viewer may click on one or more format types on the selection user interface. The candidate selections may be combinations of format types before the final selection result is determined.

[0052] In step S906, the processing unit 322 displays the effect (or preview effect) of the candidate selection result on the user terminal for the viewer before the viewer decides on the candidate selection result. For example, the candidate selection result may include a format type "Size A, Volume A". The processing unit 322 refers to the comment format DB 330, accesses the corresponding preview file (via URL_SA and URL_VA), and displays the preview effect using the preview file.

[0053] In some embodiments, the preview effect for one format type is displayed immediately after the viewer selects that format type. In some embodiments, the preview effect for a later selected format type is superimposed on the preview effect for a previously selected format type.

[0054] In step S908, the acquisition unit 320 acquires the selection result (or final selection result) from the viewer via the selection user interface. The final selection result corresponds to the latest selection result candidate. The processing unit 322 may refer to the comment format DB 330, access the corresponding parameter file, and set the corresponding format file in the comment content DB 332 according to the parameter file.

[0055] For example, as shown in Figure 8, user U1 selects "Size A, Sound Effect B" as the result of the selection for representing comment CM2. The processing unit 322 accesses the corresponding parameter files through URL_SA_para and URL_EB_para in the comment format DB 330, and sets the format file in URL_CM2 according to those two parameter files.

[0056] In step S910, the processing unit 322 expresses the comment in the live stream room according to the selection result. In other words, the processing unit 322 expresses the comment according to the format file set according to the parameter file determined by the selection result.

[0057] In this embodiment, the comment from the viewer will not be displayed in the live stream room until the viewer has made a decision on the format selection.

[0058] Figure 10 shows an exemplary flow based on some embodiments of the present invention.

[0059] In step S1000, the acquisition unit 320 acquires comments from viewers in the live stream room. Viewers may enter their comments in the comment input area via their user terminal.

[0060] In step S1002, the processing unit 322 displays the comment in the stream room in the default format.

[0061] In step S1004, the processing unit 322 displays the selection object associated with the comment, which is displayed in the default format. The selection object may be displayed only on the user terminal of the viewer who sent the comment.

[0062] In step S1006, the acquisition unit 320 acquires a selection signal from the viewer regarding the selected object. In other words, the viewer has selected or clicked the selected object.

[0063] Processes S1008, S1010, S1012, S1014, and S1016 are the same as or similar to processes S902, S904, S906, S908, and S910, respectively.

[0064] In this embodiment, the comment is initially displayed in the live stream room in a default format and then re-expressed later according to the viewer's selection. This mechanism can attract the streamer's attention by highlighting previously submitted comments.

[0065] Figure 11 shows an example of a live stream room based on some embodiments of the present invention. The user interface is from the perspective of viewer V1. The enhancement or emphasis of the comment is performed while (or at the time of) the comment is being entered.

[0066] At timing t1, viewer V1 enters the comment in the designated comment input area. Viewer V1 selects the "Highlight and Send" button to highlight the comment.

[0067] At timing t2, a format selection user interface (UI) for selecting how to highlight the comment is displayed to the viewer. The UI displays the main format type, the options for the selected main format type, corresponding points, a preview area, and a confirm button. In this embodiment, viewer V1 selects "Size" and then "Size C," at which point the comment is displayed in the preview area with the "Size C" effect (this is the selection result candidate). Viewer V1 then selects "Confirm" to confirm the selection result. In some embodiments, the format selection UI is displayed by the processing unit 322 by referring to the comment format DB 330.

[0068] At timing t3, viewer V1's comment with the "Size C" effect is displayed in the stream room. This display may be performed, for example, by the processing unit 322, by referring to the comment content DB 332. In some embodiments, the processing unit 322 causes the viewer to pay a number of points corresponding to the selection result on the streaming platform. For example, the processing unit 322 may refer to the comment content DB 332 for the total points to represent the comment and deduct that number of points from the viewer in the user DB 312.

[0069] In some embodiments, the processing unit 322 may cooperate with the distribution information provision unit 302 or the relay unit 304 to express the comment.

[0070] Figure 12 shows an example of a live stream room based on some embodiments of the present invention. The user interface is from the perspective of viewer V1. The enhancement or emphasis of the comment is performed after the comment is entered.

[0071] At timing t1, viewer V1 enters the comment in the designated comment input area. Viewer V1 then selects the "Send" button to submit the comment.

[0072] At timing t2, the comment from viewer V1 is displayed in the comment display area of ​​the stream room in the default format. A "Highlight" button associated with the comment is displayed. In some embodiments, the highlight button associated with the comment from viewer V1 is displayed only to viewer V1. Viewer V1 then selects the "Highlight" button to highlight the comment. Subsequently, a format selection UI, such as the UI shown in Figure 11, is displayed. The viewer may then choose how to highlight the comment. Once the selection is determined, the comment is re-presented in the stream room according to the selected format. The previous comment in the default format may or may not be retained in the comment display area.

[0073] In some embodiments, for comments that have already been submitted, only comments submitted within a predetermined period (from the current time) may be highlighted. For example, the processing unit 322 may refer to the comment content DB 332 to check the timing of each comment and display only the highlight button associated with the comment submitted within the predetermined period. This can encourage viewers to use the highlighting function in a timely manner and improve the revenue of the streaming platform.

[0074] In some embodiments, the comment format DB330 may be implemented on the user terminal side rather than the server side. For example, the data of the comment format DB330 may be sent from the server to the user terminal as code or config data. When the comment format DB330 is implemented on the user terminal side, the preview process may be performed by a processing unit within the user terminal, thereby speeding up the process and saving communication resources between the user terminal and the server. Once the user enters a comment into the comment input area and determines the format selection result, the user terminal may send the comment content along with the corresponding expression parameters to the server. The server may then deliver the comment in the selected format to the stream room so that the highlighted comment is visible to all users in the stream room.

[0075] In some embodiments, cache memory on the user terminal or server may be used to store past comments submitted by the viewer. The lifespan of the stored comments may be set as a predetermined period, and the comments may be highlighted within that period.

[0076] Figure 13 shows an example of a live stream room based on some embodiments of the present invention. The user interface is from the perspective of viewer V1.

[0077] At timing t1, viewer V1 enables the highlighting function for the comment, and the format selection UI is displayed. Viewer V1 selects "Size" and "Size C". The comment is then displayed in the preview area in size C.

[0078] At timing t2, viewer V1 further selects "Movement Pattern" and "Pattern A". The effect of "Movement Pattern A" is displayed overlaid on the effect of "Size C" in the preview area. After that, viewer V1 selects "Confirm" to finalize the selection.

[0079] In some embodiments, the processing unit 322 may access the comment content in the comment content DB 332 and access the parameter files and preview files of multiple format types in the comment format DB 330 to display a preview of the superimposed effect.

[0080] At timing t3, the comment from viewer V1 is displayed in the stream room with the effect of "Size C" and the effect of "Movement Pattern A". In some embodiments, the comment is also displayed in the comment display area in the default format.

[0081] Figure 14 shows an example of a format selection UI based on some embodiments of the present invention. This UI is displayed when the viewer chooses to use the highlighting function for a comment.

[0082] For each format type, a preview (or audition) and the adjustment / selection objects are displayed separately. The viewer can scroll the "Size Format" scroll bar to adjust the size and check it along with the corresponding point count. The viewer can select and listen to different sound effects. The viewer can scroll the "Volume Format" scroll bar to adjust the volume at which the message is read aloud. The viewer can select different movement patterns based on the image (or animated image or thumbnail video). Some or all of the selected effects may be superimposed or combined to determine the selection result for the format. The processing unit 322 displays the comment in the stream room according to the selection result.

[0083] In some embodiments, the formatting type for highlighting a comment may include pinning or fixing the comment in a certain position within the streamroom for a certain period of time. Pinning or fixing means that the comment will not be replaced (or streamed away) by subsequent comments.

[0084] In some embodiments, the format type for representing a comment may include a female voice or a male voice. For example, the processing unit may use the machine learning DB334 to read the comment content aloud in the corresponding voice. In some embodiments, the processing unit may use an AI model in the machine learning DB334 to detect the tone of the comment content and read the comment content aloud with the corresponding intonation. In some embodiments, if a sound effect is selected to represent the comment, the processing unit may use an AI model in the machine learning DB334 to detect the tone of the comment content and adjust the sound effect (e.g., volume, frequency, pitch, etc.) based on the tone of the comment content.

[0085] In some embodiments, the voice reading the comment may be the voice of a character (or anime voice) from a well-known game or animation. In some embodiments, the streaming platform may collaborate with or obtain permission from a prominent intellectual property (IP). The voice may then be used to generate audio / sound gifts to be used in events (or campaigns) on the streaming platform.

[0086] In some embodiments, the format type for representing comments may include reading or displaying the comments in different languages. For example, the processing unit may use an AI model in the machine learning DB334 to translate the comment content and read or display the translated comment in the stream room.

[0087] Some embodiments of the present invention enable viewers to present their comments in a way that is more likely to attract the streamer's attention. For example, when a streamer is live streaming or playing a game, the streamer may not be able to see comments from viewers in the stream room. The sound effects and text-to-speech functions may be helpful in conveying comments to the streamer.

[0088] The hardware configuration of the information processing device will be described with reference to Figure 15. Figure 15 is a block diagram showing an example of the hardware configuration of an information processing device based on some embodiments of the present invention. The information processing device 900 shown in the figure can, for example, realize the server 10 and / or the user terminals 20, 30 in some embodiments.

[0089] The information processing device 900 includes a CPU 901, a ROM (read-only memory) 903, and a RAM (random access memory) 905. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 925, and a communication device 929. Furthermore, the information processing device 900 may include an imaging device such as a camera (not shown). In addition to the CPU 901, or in place of the CPU 901, the information processing device 900 may include a DSP (digital signal processor) or an ASIC (application-specific integrated circuit).

[0090] The CPU 901 functions as an arithmetic processing unit and control unit, and controls the operation of all or part of the information processing unit 900 according to various programs stored in the ROM 903, RAM 905, storage device 919, or removable recording medium 923. For example, the CPU 901 controls the overall operation of each functional unit included in the server 10 and the user terminals 20 and 30 in some embodiments. The ROM 903 stores programs and arithmetic parameters used by the CPU 901. The RAM 905 functions as main memory, storing programs used when the CPU 901 is running and parameters that change as appropriate during execution. The CPU 901, ROM 903, and RAM 905 are interconnected by a host bus 907, which may be an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.

[0091] The input device 915 may be a user-operated device such as a mouse, keyboard, touch panel, button, switch, or lever, or a device that converts physical quantities into electrical signals, such as a sound sensor represented by a microphone, an acceleration sensor, a tilt sensor, an infrared sensor, a depth sensor, a temperature sensor, or a humidity sensor. The input device 915 may also be, for example, a remote control device that uses radio waves such as infrared rays, or an external connection device 927 such as a mobile phone that is compatible with the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on the information entered by the user or the detected physical quantity and outputs it to the CPU 901. The user inputs various data to the information processing device 900 and instructs its operation by operating the input device 915.

[0092] The output device 917 is a device that can notify the user of acquired information visually or audibly. The output device 917 may be, for example, a display such as an LCD, PDP, or OLED, an audio output device such as a speaker or headphones, or a printer. The output device 917 outputs the processing results from the information processing device 900 as text, images or other visual content, or audio such as sound.

[0093] The storage device 919 is a data storage device and is configured as an example of a storage unit of the information processing device 900. The storage device 919 is, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 919 stores programs executed by the CPU 901, various data, and various data acquired from external sources.

[0094] The drive 921 is a reader / writer for removable recording media 923 such as magnetic disks, optical disks, magneto-optical disks, and semiconductor memory, and is built into or attached to the information processing device 900. The drive 921 reads information recorded on the installed removable recording media 923 and outputs it to the RAM 905. Furthermore, the drive 921 writes data to the installed removable recording media 923.

[0095] The connection port 925 is a port for directly connecting equipment to the information processing device 900. The connection port 925 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. Furthermore, the connection port 925 may be an RS-232C port, an optical audio terminal, or an HDMI (High-Definition Multimedia Interface) port. By connecting the external connection device 927 to the connection port 925, various types of data can be exchanged between the information processing device 900 and the external connection device 927.

[0096] The communication device 929 is, for example, a communication interface formed by a communication device for connecting to the network NW. The communication device 929 may be, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (trademark), or wireless USB (WUSB). Furthermore, the communication device 929 may be a router for optical communication, a router for ADSL (asymmetric digital subscriber line), or a modem for various types of communication. The communication device 929 transmits and receives signals, etc., to and from the Internet or other communication devices using a predetermined protocol such as TCP / IP. The communication network NW connected to the communication device 929 is a network connected by wire or wireless, and may be, for example, the Internet, a home LAN, infrared communication, radio communication, satellite communication, etc. The communication device 929 realizes the function of a communication device.

[0097] The imaging device (not shown) is a device that captures images of real space and generates captured images using an image sensor such as a CCD (charge-coupled device) or CMOS (complementary metal-oxide-semiconductor) and various components such as lenses for controlling the imaging of a subject on the image sensor. The imaging device may capture still images or moving images.

[0098] The configuration and operation of the live streaming system 1 according to this embodiment have been described above. This embodiment is merely one example, and various changes can be made to the combination of each component and each process, and it will be understood by those skilled in the art that such changes are also within the scope of the present invention.

[0099] The processes and procedures described herein can be implemented by software, hardware, or any combination thereof, in addition to those explicitly described. For example, the processes and procedures described herein can be implemented by implementing the logic corresponding to those processes and procedures on a medium such as an integrated circuit, volatile memory, non-volatile memory, non-temporary computer-readable media, or magnetic disk. Furthermore, the processes and procedures described herein can be implemented as computer programs corresponding to those processes and procedures and can be executed by various types of computers.

[0100] Furthermore, the systems or methods described in the above embodiments may be integrated into a program stored on a non-temporary computer-readable medium such as a solid-state memory device, optical disk memory device, or magnetic disk memory device. Alternatively, the program may be downloaded from a server via the Internet and executed by a processor.

[0101] Although the technical content and features of the present invention have been described above, any person with ordinary skill in the art to which the present invention pertains can make many further modifications and alterations without departing from the teachings and disclosures of the present invention. Therefore, the scope of the present invention is not limited to the embodiments already disclosed, but is the scope of the claims, including other modifications and alterations that do not depart from the present invention. [Explanation of symbols]

[0102] 1. Communication System 10 servers 20 User Terminals 30, 30a, 30b User terminals LV streamer AU1, AU2 viewers VD, VD1, VD2 video NW Network 30 User terminals 100 distribution units 102 Imaging control unit 104 Audio Control Unit 106 Video transmission unit 108 Streamer-side UI control unit 200 viewing units 202 Viewer-side UI control unit 204 Superposition Information Generation Unit 206 Input Information Transmission Unit 302 Distribution Information Provision Unit 304 Relay Unit 306 Gift Processing Unit 308 Payment Processing Unit 310 Stream DB 312 User DB 314 Gift Database 320 units acquired 322 Processing Units 330 Comment-format DB 332 Comment Content Database 334 Machine Learning Database 900 Information Processing Equipment 901 CPU 903 ROM 905 RAM 907 Host Bus 909 Bridge 911 External bus 913 Interface 915 Input device 917 Output device 919 Storage devices 921 Drive 923 Removable recording media 925 connection ports 927 External connection device 929 Communication equipment

Claims

1. A method of live streaming performed by a server, The process of obtaining comments from viewers in the live stream room, The steps include: displaying a selection user interface to the viewer; The process of obtaining the selection result from the viewer via the selection user interface, Based on the selection results, the process of expressing comments within the live stream room, A method characterized by including

2. The method according to claim 1, characterized in that the selection result is associated with the number of points paid by the viewer.

3. The method according to claim 1, characterized in that the selection result is associated with the volume at which the comment is expressed.

4. The method according to claim 1, characterized in that the selection result is associated with a sound effect that expresses the comment.

5. The method according to claim 1, characterized in that the selection result is associated with a movement pattern that represents the comment.

6. The method according to claim 1, characterized in that the selection result is associated with the size of the comment being expressed.

7. The method according to claim 1, characterized in that the selection result is associated with the language used to express the comment.

8. The method according to claim 1, characterized in that the expression of the comment includes reading aloud the content of the comment within the live stream room.

9. moreover, The process involves obtaining selection result candidates from the viewer via the selection user interface, and the selection result candidates corresponding to the selection result, The steps include: expressing the effects of the candidate selection result to the viewer before the viewer decides on the candidate selection result as the selected result; The method according to claim 1, characterized by including

10. The method according to claim 1, characterized in that the comment is not displayed in the live stream room until the comment is expressed in the live stream room based on the selection result.

11. moreover, The steps include: displaying the aforementioned comments in the live stream room in the default format; The steps include: displaying the selected object associated with the comment displayed in the default format; The process of obtaining a selection signal related to the selected object from the viewer, The method according to claim 1, wherein the display of the selection user interface to the viewer is performed in response to the acquisition of the selection signal.

12. Furthermore, the method according to claim 1, characterized in that it includes the step of having the viewer pay a number of points corresponding to the selection result on the streaming platform providing the live stream room.

13. A system for live streaming, comprising one or more processors, wherein one or more processors execute machine-readable instructions. The process involves obtaining comments from viewers in a live stream room, and displaying a selection user interface to the viewers. The process of obtaining the selection result from the viewer via the selection user interface, Based on the selection results, the process of expressing comments within the live stream room, A system characterized by performing the following.

14. A non-temporary computer-readable medium containing a program for live streaming, wherein the program is on one or more computers. The process of obtaining comments from viewers in the live stream room, The steps include: displaying a selection user interface to the viewer; The process of obtaining the selection result from the viewer via the selection user interface, Based on the selection results, the process of expressing comments within the live stream room, A non-temporary computer-readable medium characterized by causing the execution of a certain action.

Images