Server, computer program, and method

Abstract

To improve convenience of a viewer by providing a summary reflecting the current state of live distribution to the viewer who participates in the live distribution from the middle.SOLUTION: The server includes an acquisition unit and a generation unit, the acquisition unit acquires first time-series data representing a content of the live-streaming recorded with progress of the live-streaming at a first time point during the progress of the live-streaming, and the generation unit generates summary information at the first time point of the live-streaming based on the acquired first time-series data. The acquisition means acquires, at a second time point during the progress of the livestreaming and after the first time point, second time-series data representing the content of the livestreaming recorded with the progress of the livestreaming, and the generation means generates summary information at the second time point of the livestreaming based on the acquired second time-series data.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a server, a computer program and a method. [Background technology]

[0002] With the development of IT technology, the way information is exchanged has also changed. During the Showa era, one-way communication, such as through newspapers and television, was the norm. In the Heisei era, the widespread use of mobile phones and personal computers and significant improvements in internet speeds led to the rise of instant, two-way communication services such as chat services. Furthermore, as storage costs decreased, on-demand video streaming services became more popular. Now, in the Reiwa era, with the increasing functionality of smartphones and further improvements in network speeds, such as those exemplified by 5G, services that enable real-time communication through video, particularly live streaming services, are rapidly gaining popularity. Live streaming services are seeing a rapid increase in users, particularly among young people, as they allow everyone to share the same fun time, even when they are far apart.

[0003] Patent Literature 1 discloses a technology that generates advice information for a broadcaster and displays it on the broadcaster's screen during live streaming of a video. This allows the broadcaster to perform live streaming while referring to the advice information displayed in real time. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-075401 [Non-patent literature]

[0005] [Non-Patent Document 1] "Demo of AI responding to the contents of an image in text", ExaWizards, Inc., URL: https: / / techblog.exawizards.com / entry / 2019 / 02 / 15 / 175416 Summary of the Invention [Problem to be solved by the invention]

[0006] According to the technology described in Patent Document 1, a broadcaster can obtain information about viewers who have newly joined a live broadcast. However, the technology described in Patent Document 1 does not provide any benefits to viewers who have newly joined or are planning to join a live broadcast.

[0007] A key feature of live streaming is that, because there is no script, the content changes constantly based on interaction and communication with viewers, potentially moving in a direction different from what was originally expected. Therefore, it is difficult for viewers who join a live stream midway through to quickly grasp the flow and content of the stream. Even if the title and thumbnail reflect the content of the live stream that the streamer envisioned before the stream began, the actual content often develops in a different direction than originally expected as the stream progresses, resulting in a discrepancy between the title and thumbnail and the actual content. In such cases, viewers who join midway through, expecting the content depicted in the title and thumbnail, may be confused by the actual content of the live stream.

[0008] The present disclosure has been made in consideration of these issues, and its purpose is to provide technology that can increase convenience for viewers who join a live broadcast midway by providing a summary that reflects the current state of the live broadcast. [Means for solving the problem]

[0009] One aspect of the present invention relates to a server including an acquisition means and a generation means, wherein the acquisition means acquires, at a first time point during the progress of a live stream, first time series data representing the content of the live stream that has been recorded as the live stream progresses, the generation means generates summary information for the first time point during the live stream based on the acquired first time series data, and the acquisition means acquires, at a second time point during the progress of the live stream that is later than the first time point, second time series data representing the content of the live stream that has been recorded as the live stream progresses, and the generation means generates summary information for the second time point during the live stream based on the acquired second time series data.

[0010] Another aspect of the present invention is a computer program that enables a terminal to perform the following functions: send a request to a server via a network; receive summary information of an ongoing live broadcast, the summary information having content that may vary depending on the timing of the request, from the server via the network; start playing a video related to the live broadcast; and display the summary information on a display in synchronization with the start of playback of the video.

[0011] In addition, any combination of the above components, or mutual substitution of the components or expressions of the present invention between devices, methods, systems, computer programs, recording media storing computer programs, etc., are also valid aspects of the present invention. [Effects of the Invention]

[0012] According to the present invention, convenience for viewers can be improved by providing a summary that reflects the current state of the live distribution to viewers who join the live distribution midway. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram illustrating a configuration of a live distribution system according to an embodiment of the present disclosure. [Figure 2]2 is a block diagram showing the functions and configuration of the user terminal of FIG. 1. [Figure 3] FIG. 2 is a block diagram showing the functions and configuration of the server in FIG. 1. [Figure 4] FIG. 4 is a data structure diagram showing an example of a stream DB in FIG. 3. [Figure 5] 4 is a data structure diagram showing an example of a user DB of FIG. 3. FIG. [Figure 6] FIG. 4 is a data structure diagram showing an example of a gift DB in FIG. 3. [Figure 7] FIG. 10 is an explanatory diagram showing the relationship between the progress of live distribution and the generated summary. [Figure 8] 10 is a flowchart showing a series of processes related to dynamic summary generation of live streaming. [Figure 9] FIG. 10 is a representative screen diagram of a live broadcast selection screen displayed on the display of the user terminal of an active user. [Figure 10] FIG. 10 is a diagram illustrating a representative screen of a live streaming room screen displayed on the display of a viewer's user terminal. [Figure 11] FIG. 10 is a representative screen diagram of a live streaming room screen with a summary area superimposed thereon, displayed on the display of a viewer's user terminal. [Figure 12] FIG. 10 is a representative screen diagram of a live streaming room screen with a summary area superimposed thereon, displayed on the display of a viewer's user terminal. [Figure 13] FIG. 10 is a representative screen diagram of a live streaming room screen with a comment candidate display area superimposed thereon, displayed on the display of a viewer's user terminal. [Figure 14] FIG. 10 is a representative screen diagram of a distribution preparation screen displayed on the display of a user terminal of a distributor. [Figure 15] FIG. 10 is a representative screen diagram of a live streaming room screen displayed on the display of a user terminal of a broadcaster. [Figure 16] 1 is a block diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment of the present invention. [Figure 17]FIG. 10 is a representative screen diagram of a live streaming room screen on which a summary area according to a modified example is superimposed and displayed on the display of a viewer's user terminal. [Figure 18] FIG. 10 is a representative screen diagram of a live streaming room screen in preview mode displayed on the display of a user terminal. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, the same or equivalent components, parts, processes, and signals shown in each drawing will be denoted by the same reference numerals, and redundant explanations will be omitted where appropriate. In addition, some of the parts that are not important for the explanation will be omitted in each drawing.

[0015] In a live streaming system according to an embodiment, if a viewer joins a live stream midway, the viewer is provided with a summary of the situation generated by a machine learning model (e.g., a language analysis model such as GPT) from the conversation that has taken place in the live stream up to that point. The summary is information that summarizes the topics and flow of the conversation in the live stream, iconic events, the streamer's actions, activities, and feelings, viewer comments, behaviors, gifting status, the atmosphere of the live stream, and the content of the live stream. This allows a viewer who joins a live stream midway through to quickly understand what has been discussed in the live stream so far and the current flow of the conversation by reading the provided summary. As a result, even viewers who join midway through the live stream can easily join in on the conversation, allowing them to enjoy the live stream more, thereby improving retention and engagement.

[0016] FIG. 1 is a schematic diagram illustrating the configuration of a live streaming system 1 according to an embodiment of the present disclosure. The live streaming system 1 provides an interactive live streaming service that allows real-time interaction between broadcasters (also referred to as live streamers) LV and viewers (also referred to as audiences) AU (AU1, AU2, ...). As shown in FIG. 1, the live streaming system 1 includes a server 10, a broadcaster-side user terminal 20, and a viewer-side user terminal 30 (30a, 30b, ...). In addition to broadcasters broadcasting live streams and viewers watching live streams, there are also users who log in to the live streaming platform but do not broadcast or watch. These users are called active users. Broadcasters, viewers, and active users may be collectively referred to as users. The server 10 may be configured with one or more information processing devices connected to a network NW. The user terminals 20 and 30 may be mobile devices such as smartphones, tablet devices, laptop PCs, recorders, portable game consoles, or wearable devices, or may be stationary devices such as desktop PCs. The server 10, the user terminal 20, and the user terminal 30 are connected to each other via various wired or wireless networks NW so that they can communicate with each other.

[0017] The live streaming system 1 involves a broadcaster LV, a viewer AU, and an administrator (not shown) who manages the server 10. The broadcaster LV records and films content such as his or her own singing, talking, performance, fortune telling, and game commentary on his or her own user terminal 20 and uploads it directly to the server 10, thereby transmitting the content in real time. The administrator provides a platform for the live streaming of content on the server 10 and also mediates or manages real-time interactions between the broadcaster LV and the viewer AU. The viewer AU accesses the platform with a user terminal 30, selects desired content, and views it. During the live streaming of this content, the viewer AU performs operations via the user terminal 30 to comment, cheer, or request a fortune telling. The broadcaster LV, who provides the content, responds to such comments, cheers, or requests, and these responses are transmitted to the viewer AU via video and / or audio, thereby establishing two-way communication.

[0018] As used herein, "live streaming" may refer to a data transmission method that enables content recorded on a user terminal 20 of a distributor LV to be played back and viewed on a user terminal 30 of a viewer AU in substantially real time, or may refer to the distribution itself achieved by such a transmission method. Live streaming may be achieved using existing live streaming technologies such as HTTP Live Streaming, Common Media Application Format, Web Real-Time Communications, Real-Time Messaging Protocol, and MPEG DASH. Live streaming includes a transmission method in which a viewer AU can view content with a predetermined delay while the distributor LV is recording the content. The delay is at least long enough to enable communication between the distributor LV and the viewer AU. However, live streaming is distinct from on-demand distribution, in which the entire recorded content is temporarily stored on a server and then provided to users from the server at any time upon request.

[0019] In this specification, "video data" refers to data including image data (also referred to as video data) generated by the imaging function of the user terminals 20 and 30 and audio data (also referred to as audio data) generated by the audio input function of the user terminals 20 and 30. The video data is played back on the user terminals 20 and 30, allowing users to view content. In this embodiment, it is assumed that processing such as compression, decompression, encoding, decoding, and transcoding is performed to change the format, size, and specifications of the data between when the video data is generated on the distributor's user terminal and when it is played back on the viewer's user terminal. Since the content (e.g., moving images and audio) represented by the video data before and after such processing remains substantially unchanged, this embodiment describes the video data after such processing as being the same as the video data before such processing. In other words, when video data is generated on the distributor's user terminal and then played back on the viewer's user terminal via the server 10, the video data generated on the distributor's user terminal, the video data passing through the server 10, and the video data received and played back on the viewer's user terminal are all the same video data.

[0020] In the example of Fig. 1, a broadcaster LV is live streaming a talk. The user terminal 20 of the broadcaster LV records the image and audio of the broadcaster LV while he or she is talking, generating video data, and transmits the video data to the server 10 via the network NW. The user terminal 20 also displays the recorded video image VD of the broadcaster LV on the display of the user terminal 20, allowing the broadcast content by the broadcaster LV to be confirmed.

[0021] User terminals 30a and 30b of viewers AU1 and AU2 who have requested the platform to watch the live broadcast of broadcaster LV each receive video data related to the live broadcast via network NW, and play the received video data to display videos VD1 and VD2 on their displays and output audio from their speakers. The videos VD1 and VD2 displayed on each user terminal 30a and 30b are substantially identical to the video VD captured by user terminal 20 of broadcaster LV, and the audio output from each user terminal 30a and 30b is substantially identical to the audio recorded by user terminal 20 of broadcaster LV.

[0022] Audio and video recording on the user terminal 20 of the broadcaster LV and playback of video data on the user terminals 30a and 30b of viewers AU1 and AU2 are performed substantially simultaneously. When one viewer AU1 inputs a comment on the content of the broadcaster LV's conversation into the user terminal 30a, the server 10 displays the comment in real time on the user terminal 20 of the broadcaster LV and also on the user terminals 30a and 30b of each viewer AU1 and AU2. After reading the comment, the broadcaster LV expands the conversation by overlaying the content. The video and audio of the conversation are output to the user terminals 30a and 30b of each viewer AU1 and AU2, thereby recognizing that a conversation between the broadcaster LV and viewer AU1 has been established. In this way, the live streaming system 1 realizes live streaming that enables two-way communication, not one-way communication.

[0023] Figure 2 is a block diagram showing the functions and configuration of user terminal 20 in Figure 1. User terminal 30 has the same functions and configuration as user terminal 20. Each block shown in Figure 2 and the following block diagrams can be realized in hardware terms by elements and mechanical devices such as a computer CPU, and in software terms by a computer program, etc., but here we depict functional blocks realized by the cooperation of these. Therefore, those skilled in the art who have read this specification will understand that these functional blocks can be realized in various ways by combining hardware and software.

[0024] The distributor LV and the viewer AU download and install a live streaming application program (hereinafter referred to as a live streaming app) according to this embodiment from a download site to the user terminals 20 and 30 via the network NW. Alternatively, the live streaming app may be pre-installed on the user terminals 20 and 30. When the user terminals 20 and 30 execute the live streaming app, the user terminals 20 and 30 communicate with the server 10 via the network NW and realize various functions. Hereinafter, functions realized by the user terminals 20 and 30 (or their processors, such as CPUs) executing the live streaming app will be described as functions of the user terminals 20 and 30. These functions are actually functions realized by the live streaming app on the user terminals 20 and 30. Note that in other embodiments, these functions may be realized by a computer program written in a programming language such as HTML (HyperText Markup Language), which is transmitted from the server 10 to a web browser on the user terminals 20 and 30 via the network NW and executed by the web browser.

[0025] The user terminal 20 comprises a distribution unit 100 that generates video data recording the user's image and voice and provides it to the server 10, a viewing unit 200 that acquires the video data from the server 10 and plays it back, and a non-distribution processing unit 400 that processes requests from active users. A user activates the distribution unit 100 when distributing, the viewing unit 200 when viewing, and the non-distribution processing unit 400 when searching for a live broadcast they want to watch, viewing the profile of a broadcaster, or viewing archives. A user terminal on which the distribution unit 100 is active is on the broadcaster's side, i.e., the user terminal that generates the video data, a user terminal on which the viewing unit 200 is active is on the viewer's side, i.e., the user terminal that plays back the video data, and a user terminal on which the non-distribution processing unit 400 is active is the user terminal of an active user.

[0026] The distribution unit 100 includes an imaging control unit 102, an audio control unit 104, a video transmission unit 106, a distribution-side UI control unit 108, and a distribution-side communication unit 110. The imaging control unit 102 is connected to a camera (not shown in FIG. 2) and controls imaging by the camera. The imaging control unit 102 acquires image data from the camera. The audio control unit 104 is connected to a microphone (not shown in FIG. 2) and controls audio input via the microphone. The audio control unit 104 acquires audio data from the microphone. The video transmission unit 106 transmits video data including the image data acquired by the imaging control unit 102 and the audio data acquired by the audio control unit 104 to the server 10 via the network NW. The video transmission unit 106 transmits the video data in real time. That is, the generation of video data by the imaging control unit 102 and the audio control unit 104 and the transmission of the generated video data by the video transmission unit 106 are performed substantially simultaneously.

[0027] The distribution-side UI control unit 108 controls the UI for the distributor. The distribution-side UI control unit 108 is connected to a display (not shown in FIG. 2) and displays a moving image on the display by playing back the moving image data to be transmitted by the moving image transmission unit 106. The distribution-side UI control unit 108 is connected to input means (not shown in FIG. 2) such as a touch panel, keyboard, or display, and acquires input from the distributor via these input means. The distribution-side UI control unit 108 superimposes a predetermined frame image on the moving image. The frame image includes various user interface objects (hereinafter simply referred to as objects) for receiving input from the distributor, comments entered by viewers, and information acquired from the server 10. The distribution-side UI control unit 108 accepts, for example, a tap input on an object by the distributor.

[0028] The distribution-side communication unit 110 controls communication with the server 10 during live distribution. The distribution-side communication unit 110 transmits the contents of input by the distributor acquired by the distribution-side UI control unit 108 to the server 10 via the network NW. The distribution-side communication unit 110 receives various types of information associated with the live distribution from the server 10 via the network NW.

[0029] The viewing unit 200 includes a viewing-side UI control unit 202 and a viewing-side communication unit 204. The viewing-side communication unit 204 controls communication with the server 10 during live distribution. The viewing-side communication unit 204 receives video data related to the live distribution in which the distributor and viewers participate from the server 10 via the network NW.

[0030] The viewer-side UI control unit 202 controls the UI for viewers. The viewer-side UI control unit 202 is connected to a display and a speaker (not shown in FIG. 2 ) and plays received video data to display video images on the display and output audio from the speaker. The output of images on the display and audio from the speaker can be collectively referred to as "video data being played." The viewer-side UI control unit 202 is connected to input devices (not shown in FIG. 2 ), such as a touch panel, keyboard, and display, and acquires input from viewers via these input devices. The viewer-side UI control unit 202 superimposes a predetermined frame image on an image of video data acquired from the server 10. The frame image includes various objects for receiving input from viewers, comments entered by the viewers, and information acquired from the server 10. The viewer-side communication unit 204 transmits the content of the viewer's input acquired by the viewer-side UI control unit 202 to the server 10 via the network NW.

[0031] The non-broadcast processing unit 400 includes a non-broadcast UI control unit 402 and a non-broadcast communication unit 404. The non-broadcast UI control unit 402 controls the UI for active users. For example, the non-broadcast UI control unit 402 generates a live broadcast selection screen that displays a list of live broadcasts that can currently be participated in and accepts the selection of a live broadcast by active users, and displays this on the display. The non-broadcast UI control unit 402 generates a profile screen for any user and displays this on the display. The non-broadcast UI control unit 402 generates a search screen that accepts search keywords for searching for broadcasters, and displays this on the display. The non-broadcast UI control unit 402 generates a search result display screen that includes search results for broadcasters, and displays this on the display. The non-broadcast UI control unit 402 plays archives generated by recording and filming past live broadcasts.

[0032] The non-broadcast communication unit 404 controls communication with the server 10 outside of live broadcasting. The non-broadcast communication unit 404 receives information for generating a live broadcast selection screen, broadcaster search results, information for generating a profile screen, and archive data from the server 10 via the network NW. The non-broadcast communication unit 404 transmits the contents of input by active users to the server 10 via the network NW.

[0033] Fig. 3 is a block diagram showing the functions and configuration of the server 10 of Fig. 1. The server 10 includes a distribution information providing unit 302, a relay unit 304, a gift processing unit 308, a payment processing unit 310, a stream DB 314, a user DB 318, a gift DB 320, a summary generating unit 322, a detail generating unit 324, a summary generation model 326, a detail generation model 328, and a candidate comment generating unit 330.

[0034] FIG. 4 is a data structure diagram illustrating an example of the stream DB 314 in FIG. 3. The stream DB 314 stores information about live streaming currently being performed. The stream DB 314 stores video data, including images and audio, of videos streamed by streamers. The stream DB 314 stores, in association with each other, a stream ID that identifies a live streaming on the live streaming platform provided by the live streaming system 1; a streamer ID that is a user ID that identifies the streamer of the live streaming; a viewer ID that is a user ID that identifies the viewer of the live streaming; a streaming time that is the time from the start of the live streaming to the present; a title of the live streaming set by the streamer before the start of the live streaming; characteristics of a summary generation model set by the streamer before the start of the live streaming; a streaming content tag that represents the content of the live streaming; image data of the live streaming up to the present; audio data of the live streaming up to the present; a history of comments posted during the live streaming; a history of gifts used during the live streaming; a current summary of the live streaming; and current detailed information about the live streaming. The summary and detailed information will be described later.

[0035] In the live streaming platform provided by the live streaming system 1 according to this embodiment, when a user performs a live stream, that user becomes a streamer, and when the same user watches a live stream broadcast by another user, that user becomes a viewer. Therefore, the distinction between streamer and viewer is not fixed, and a user ID registered as a streamer ID at one time may be registered as a viewer ID at another time.

[0036] The broadcast content tag for a live broadcast may be a tag specified by the broadcaster when the live broadcast begins, or it may be a tag obtained by a model generated by machine learning analyzing the live broadcast in real time.

[0037] The live broadcast title and broadcast content tags (if specified in advance) represent the content of the live broadcast, but are static information that does not change as the live broadcast progresses. In contrast, broadcast content tags generated by real-time analysis, image data, audio data, comment history, gift history, and viewer counts represent the content of the live broadcast and are dynamic information or time-series data that change as the live broadcast progresses. Time-series data has a structure in which data is arranged along a time axis. The time-series data is recorded in the stream DB 314 as the live broadcast progresses.

[0038] FIG. 5 is a data structure diagram showing an example of the user DB 318 in FIG. 3. The user DB 318 stores information about users. The user DB 318 includes a user ID that identifies a user, points owned by the user, rewards granted to the user, desired topic tags that indicate topics designated by the user as being of interest, and participating event IDs that identify events in which the user is participating. Events are events related to live streaming held on the live streaming platform provided by the live streaming system 1. There are various types of events, such as rankings and prize winning events.

[0039] Points are electronic value circulated within the live streaming platform. Users purchase points using credit cards or other payment methods. Rewards are electronic value defined within the live streaming platform and are an indicator used to determine the amount of money a streamer receives from the live streaming platform administrator. On the live streaming platform, when a viewer gives a gift to a streamer during or outside of a live stream, the viewer's points are consumed and the streamer's reward increases accordingly.

[0040] Fig. 6 is a data structure diagram showing an example of the gift DB 320 in Fig. 3. The gift DB 320 stores information about gifts that can be used by viewers in live streaming. A gift is electronic data that has the following characteristics. ·Can be purchased with points or money, or given for free. Something that viewers can give to the streamer. Giving a gift to a streamer is also called using a gift or throwing a gift. Some gifts are purchased and used at the same time, while others can be used at any time by the viewer after purchase. When viewers give gifts to streamers, the streamer will receive a corresponding reward. When a gift is used, an effect associated with the gift may occur. For example, an effect corresponding to the gift may appear on the live streaming room screen.

[0041] The gift DB 320 stores a gift ID that identifies a gift, a reward that is awarded to a broadcaster when the broadcaster gives the gift, and reward points that are the reward to be paid when using the gift, in association with each other. Viewers can give a desired gift to a broadcaster by paying reward points for the gift while watching a live broadcast. The reward points may be paid by an appropriate electronic payment method, such as by the viewer paying reward points to the administrator. Alternatively, payment by bank transfer or credit card may be used. The administrator can arbitrarily set the relationship between the reward and reward points. For example, the reward may be set to the reward points. Alternatively, the reward may be multiplied by a predetermined coefficient such as 1.2, and the reward points may be set to the points obtained by adding a predetermined handling fee point to the reward.

[0042] Returning to FIG. 3 , the summary generation model 326 is a pre-trained machine learning model that receives time-series data of a live stream as input and outputs text summarizing the live stream (hereinafter simply referred to as a summary). Since the output summary changes as the input time-series data changes, the output summary can be considered to be the summary at the time the corresponding time-series data was acquired. The machine learning model may be implemented using known machine learning techniques, such as BERT (Bidirectional Encoder Representations from Transformers) or GPT (Generative Pretrained Transformer). Since machine learning models for generating summaries are known, they will not be described in detail herein. As described above, the time-series data includes non-text data such as image data and audio data. If necessary, known image analysis techniques may be used to convert image data into text data representing the content (see, for example, Non-Patent Document 1). To convert audio data into text, known STT (Speech to Text) techniques may be used. In this way, non-text data may be converted into text data and then input to the summary generation model 326. In addition to time-series data, static information about the live stream may also be input to the summary generation model 326. For example, information about an event in which a live broadcaster is participating, which is registered in the user DB 318 , may be input to the summary generation model 326 .

[0043] The summary generation model 326 can be set to one of multiple different personalities. Before starting live streaming, the broadcaster specifies the personality of the model to be used when generating a summary on a broadcast preparation screen. When the broadcast information providing unit 302 receives a broadcast start instruction including the specified personality from the broadcaster's terminal via the network, it adjusts the summary generation model 326 to the specified personality. For example, if the personality "cool" is specified, the summary generation model 326 is adjusted to output text with a cool tone, and if the personality "hot" is specified, the summary generation model 326 is adjusted to output text with a hot tone. This personality setting may be achieved using known prompt engineering techniques. The summary generation model 326 may use the personality as one of its input parameters, or an instance of the summary generation model 326 may be generated and used for each ongoing live streaming or for each of multiple different personalities.

[0044] The detail generation model 328 is a pre-trained machine learning model that receives time-series data of a live stream as input and outputs text representing detailed information about the live stream (hereinafter simply referred to as detailed information). The detailed information about a live stream describes the live stream in more detail than the summary of the live stream, but is longer than the summary. Since the output detailed information changes as the input time-series data changes, the output detailed information can be said to be the detailed information as of the time when the corresponding time-series data was acquired. In addition to the time-series data, static information about the live stream may also be input to the detail generation model 328.

[0045] One of several different personalities can be set for the detailed generative model 328. Before starting live streaming, the broadcaster specifies the personality of the model to be used when generating a summary on the broadcast preparation screen. When the broadcast information providing unit 302 receives a broadcast start instruction with the specified personality from the broadcaster's terminal via the network, it adjusts the detailed generative model 328 to the specified personality.

[0046] When the distribution information providing unit 302 receives a distribution start instruction to start live distribution from the distributor's user terminal 20 via the network NW, it registers a stream ID identifying the live distribution, the distributor ID of the distributor of the live distribution, the title included in the distribution start instruction, the specified characteristics included in the distribution start instruction, and a distribution content tag included in the distribution start instruction in the stream DB 314. At the same time, the distribution information providing unit 302 begins acquiring time-series data of the live distribution, i.e., viewer information, image data, audio data, comment history, and gift history, and recording the acquired time-series data in the stream DB 314.

[0047] When the distribution information providing unit 302 receives a request for information about live distribution from the non-distribution communication unit 404 of the user terminal of an active user via the network NW, it references the stream DB 314 and generates a list of live distributions that are currently available for viewing. The distribution information providing unit 302 transmits the generated list to the user terminal that made the request via the network NW. The non-distribution UI control unit 402 of the user terminal that made the request generates a live distribution selection screen based on the received list and displays it on the display of the user terminal.

[0048] When the non-streaming UI control unit 402 of the user terminal accepts the active user's selection of a live stream on the live stream selection screen, it generates a stream request including the stream ID of the selected live stream and transmits it to the server 10 via the network NW. The stream information providing unit 302 obtains a summary and detailed information of the live stream identified by the stream ID included in the received stream request from the stream DB 314 and transmits it to the requesting user terminal via the network NW. The stream information providing unit 302 also begins providing the live stream identified by the stream ID included in the received stream request to the requesting user terminal. The stream information providing unit 302 updates the stream DB 314 so that the viewer ID of the stream ID includes the user ID of the active user of the requesting user terminal. As a result, the active user becomes a viewer of the selected live stream.

[0049] The relay unit 304 relays the transmission of video data from the broadcaster's user terminal 20 to the viewer's user terminal 30 during live streaming initiated by the streaming information providing unit 302. The relay unit 304 receives, from the viewer-side communication unit 204, a signal indicating a user input by a viewer during live streaming, i.e., playback of video data. The signal indicating user input may be an object designation signal indicating designation of an object displayed on the display of the user terminal 30. The object designation signal includes the viewer ID of the viewer, the broadcaster ID of the broadcaster performing the live streaming being viewed by the viewer, and an object ID identifying the object. If the object is a gift icon, the object ID is a gift ID. In this case, the object designation signal becomes a gift use signal indicating the viewer's use of a gift for the broadcaster. Similarly, the relay unit 304 receives, from the broadcaster-side communication unit 110 of the broadcasting unit 100 of the user terminal 20, a signal indicating a user input by a broadcaster during playback of video data, for example, an object designation signal.

[0050] The gift processing unit 308 updates the user DB 318 so as to increase the distributor's reward according to the reward for the gift identified by the gift ID included in the gift use signal. The gift processing unit 308 refers to the gift DB 320 and identifies the reward corresponding to the gift ID included in the received gift use signal. The gift processing unit 308 updates the user DB 318 so as to add the identified reward to the reward corresponding to the distributor ID included in the gift use signal.

[0051] In response to receiving the gift use signal, the payment processing unit 310 processes the payment of the gift value by the viewer. The payment processing unit 310 refers to the gift DB 320 and identifies the value points of the gift identified by the gift ID included in the gift use signal. The payment processing unit 310 updates the user DB 318 to deduct the identified value points from the points of the viewer identified by the viewer ID included in the gift use signal.

[0052] The summary generation unit 322 periodically acquires time-series data and generates summaries. The summary generation unit 322 periodically acquires time-series data of ongoing live streaming from the stream DB 314, for example, once every five minutes. The summary generation unit 322 generates a summary of the ongoing live streaming at the time of acquisition of the time-series data based on the acquired time-series data. The summary generation unit 322 inputs the acquired time-series data to the summary generation model 326 and acquires the summary output by the summary generation model 326 as the summary at the time of acquisition of the time-series data. The summary generation unit 322 updates the stream DB 314 with the acquired summary. For example, when the summary generation unit 322 acquires a new summary of a live streaming, it overwrites the summary previously stored for that live streaming in the stream DB 314 with the acquired summary.

[0053] The first time series data acquired by the summary generation unit 322 at a first time point during the progress of live streaming is different from the second time series data acquired by the summary generation unit 322 at a second time point during the progress of the live streaming, which is later than the first time point. Therefore, the summary generated based on the first time series data at the first time point may differ from the summary generated based on the second time series data at the second time point.

[0054] FIG. 7 is an explanatory diagram showing the relationship between the progress of a live stream and the generated summary. The top row of FIG. 7 shows the timeline of the progress of the live stream. In this example, the title is specified by the streamer before the start of the stream, so the title is fixed to the text "Tarot card readings!" regardless of the progress of the live stream. However, the topic of the live stream, which was initially fortune-telling, changes to casual conversation at time t3.

[0055] The summary generator 322 generates a standard summary at the start of live streaming and registers it in the stream DB 314. The standard summary (also called a default) is text that does not depend on the time-series data of the live streaming but is generated based on information (static information) entered by the streamer before the start of streaming, or fixed text that is independent of the streamer, such as "Streaming just started." Subsequently, at time t1 during the live streaming, the summary generator 322 acquires time-series data Dt1 from the stream DB 314. This time-series data Dt1 represents the contents of fortune-telling, which was the topic of the live streaming up to time t1. For example, if the streamer performed fortune-telling for three viewers by time t1, the time-series data Dt1 includes data listing the fortune-telling results for those three viewers in chronological order. The summary generator 322 inputs the acquired time-series data Dt1 into the summary generation model 326, acquires the summary output by the summary generation model 326 as summary X1 for time t1, and registers it in the stream DB 314.

[0056] After that, viewer M joins the live broadcast at time t2, which is before the next summary update time. At this time, the broadcast information providing unit 302 obtains summary X1 at time t1 from the stream DB 314 and transmits it to viewer M's user terminal via the network NW. This allows viewer M to immediately understand that, as the title indicates, fortune telling is currently being performed in the live broadcast.

[0057] Then, at time t3, the topic changes from fortune telling to casual conversation. Later, at time t4 while the live streaming is in progress, the summary generation unit 322 acquires time-series data Dt4 from the stream DB 314. This time-series data Dt4 represents the content of the live streaming up to time t4, that is, originally fortune telling, but then the user started chatting midway through. The summary generation unit 322 inputs the acquired time-series data Dt4 into the summary generation model 326, acquires the summary output by the summary generation model 326 as summary X3 for time t4, and registers it in the stream DB 314.

[0058] After that (after time t4), viewer N joins the live broadcast at time t5, which is before the next summary update time. At this time, the broadcast information providing unit 302 obtains summary X3 for time t4 from the stream DB 314 and transmits it to viewer N's ​​user terminal via the network NW. This allows viewer N to immediately understand that, unlike the title, casual conversation has recently taken place in the live broadcast.

[0059] Returning to FIG. 3 , the detail generation unit 324 generates detailed information about an ongoing live broadcast as of the time-series data acquisition time point based on the time-series data acquired by the summary generation unit 322. The detail generation unit 324 inputs the acquired time-series data to the detail generation model 328 and acquires the detailed information output by the detail generation model 328 as the detailed information as of the time-series data acquisition time point. The detail generation unit 324 updates the stream DB 314 with the acquired detailed information. For example, when the detail generation unit 324 acquires new detailed information about a certain live broadcast, it overwrites the detailed information about that live broadcast that was previously held in the stream DB 314 with the acquired detailed information.

[0060] When the candidate comment generation unit 330 receives a distribution request for a live broadcast, it acquires time-series data of the ongoing live broadcast identified by the stream ID included in the distribution request from the stream DB 314. The candidate comment generation unit 330 generates a plurality of different comment candidates based on the acquired time-series data. The candidate comment generation unit 330 transmits the generated plurality of comment candidates to the requesting user terminal via the network NW.

[0061] The candidate comment generation unit 330 inputs the acquired time-series data into a comment candidate generation model and acquires multiple different comment candidates output by the comment candidate generation model. The comment candidate generation model is a pre-trained machine learning model that inputs time-series data of a live broadcast and outputs multiple candidate comments that are suitable for viewers to post at the time the time-series data was acquired. If the input time-series data changes, the multiple comment candidates that are output also change, so the content of the multiple comment candidates may differ depending on the timing of the broadcast request.

[0062] The operation of the live distribution system 1 configured as above will now be described. 8 is a flowchart showing a series of processes involved in dynamic summary generation of live streaming. The streaming information providing unit 302 receives a streaming start instruction accompanied by streaming setting information including characteristics specified by the streamer (S202). The summary generating unit 322 adjusts the summary generation model 326 using the characteristics specified in the streaming setting information received in step S202 (S204). The streaming information providing unit 302 starts providing live streaming in response to the streaming start instruction received in step S202 (S206). The streaming information providing unit 302 starts registering time-series data of the started live streaming (S208). The summary generating unit 322 updates the stream DB 314 so that a default sentence is registered in the summary of the started live streaming (S210).

[0063] The distribution information providing unit 302 determines whether a new viewer has joined the live distribution that began in step S206 (S212). If a new viewer has joined (Y in S212), the distribution information providing unit 302 extracts a summary of the live distribution from the stream DB 314 and transmits it to the user terminal of the participating viewer when the video of the live distribution begins (S214). Then, the process returns to step S212.

[0064] If the user is not participating in step S212 (N in S212), the distribution information providing unit 302 determines whether or not a distribution end instruction has been received from the user terminal of the broadcaster (S216). If the instruction has been received (Y in S216), the distribution information providing unit 302 ends providing the live broadcast (S218). If the instruction has not been received (N in S216), the summary generating unit 322 determines whether or not a predetermined period of time, for example, 5 minutes or 15 minutes, has elapsed since the last summary update for the live broadcast started in step S206 (S220). If the predetermined period of time has not elapsed (N in S220), the process returns to step S212.

[0065] If the time has elapsed (Y in S220), the summary generation unit 322 acquires time-series data of the live distribution started in step S206 from the stream DB 314 (S222). The summary generation unit 322 inputs the acquired time-series data to the summary generation model 326 and acquires a summary output by the summary generation model 326 (S224). The summary generation unit 322 updates or replaces the summary of the live distribution started in step S206 in the stream DB 314 with the summary acquired in step S224 (S226). Then, the process returns to step S212.

[0066] FIG. 9 is a representative screen diagram of a live stream selection screen 600 displayed on the display of an active user's user terminal. The live stream selection screen 600 includes thumbnails 602 representing each live stream in a list of currently viewable live streams received from the server 10. The non-streaming UI control unit 402 generates the live stream selection screen 600 based on the list of live streams acquired from the server 10 and displays it on the display. When the non-streaming UI control unit 402 accepts a thumbnail selected by the active user on the live stream selection screen 600, it generates a stream request including the stream ID of the live stream corresponding to the selected thumbnail and transmits the stream request to the server 10 via the network NW. The stream information providing unit 302 obtains a summary and detailed information of the live stream identified by the stream ID included in the received stream request from the stream DB 314 and transmits the summary and detailed information to the requesting user terminal via the network NW. The stream information providing unit 302 also transmits multiple comment candidates generated in response to the stream request to the requesting user terminal via the network NW. The non-distribution communication unit 404 receives the summary, detailed information, and multiple comment candidates thus transmitted from the server 10 via the network NW. As described above, the summary and detailed information are information about a live broadcast that is currently viewable, i.e., currently in progress, and the content may vary depending on the timing of the broadcast request.

[0067] FIG. 10 is a representative screen diagram of a live streaming room screen 608 displayed on the display of a viewer's user terminal. When an active user taps a thumbnail 602 on the live streaming selection screen 600 of FIG. 9, the viewer-side UI control unit 202 starts displaying the live streaming room screen 608 of FIG. 10 on the display and starts playing the video related to the live streaming on the live streaming room screen 608. The live streaming room screen 608 displays video images generated on the broadcaster's user terminal 20 in real time. The live streaming room screen 608 includes a video image 610 of the broadcaster obtained by playing video data received from the server 10, a gift object 612, a comment input area 616, a comment display area 618, a viewing end button 620, and a summary display object 622. The viewer-side UI control unit 202 generates a live streaming room screen 608 by superimposing other objects, namely, a gift object 612, a comment input area 616, a comment display area 618, an end viewing button 620, and a summary display object 622, on a video image 610 obtained by playing video data.

[0068] The comment display area 618 may include comments entered by the viewer, comments entered by other viewers, and notifications from the system. The notifications from the system may include information indicating who gave which gift to the broadcaster and information indicating that a new viewer has joined the live broadcast. The viewer UI control unit 202 generates a comment display area 618 including comments from other viewers received from the server 10 and notifications from the system, and includes the generated comment display area 618 on the live broadcasting room screen 608.

[0069] The comment input area 616 accepts comments entered by viewers. The viewer-side communication unit 204 generates a comment input signal including the comment entered in the comment input area 616 and transmits the signal to the server 10 via the network NW. At the same time, the viewer-side UI control unit 202 updates the comment display area 618 to display the comment entered in the comment input area 616.

[0070] The view end button 620 is an object for receiving an instruction from the viewer to stop viewing the live broadcast. The unnecessary object 624 is an object for stopping the display of the summary display object 622. When a tap on the unnecessary object 624 is detected, the viewer-side UI control unit 202 ends the display of the summary display object 622 and the unnecessary object 624.

[0071] The viewer UI control unit 202 of the user terminal displays a summary of the live streaming received from the server 10 on the display in synchronization with the start of playback of the video related to the live streaming. In the example of Fig. 10, the viewer UI control unit 202 first displays a summary display object 622 on the display in synchronization with the start of playback of the video related to the live streaming, and when it receives designation of the summary display object 622 (for example, when it detects a tap on the summary display object 622), it generates a summary area 626 that displays the summary received from the server 10 in response to the streaming request. The viewer UI control unit 202 displays the generated summary area 626 on the live streaming room screen 608.

[0072] FIG. 11 is a representative screen diagram of the live streaming room screen 608, displayed on the display of the viewer's user terminal, with a summary area 626 superimposed. FIG. 11 corresponds to the live streaming room screen display when viewer M participates in the live streaming in FIG. 7. The summary area 626 displays a detail object 628 in conjunction with the summary. The summary area 626 includes a summary of the participated live streaming and the detail object 628. When the viewer UI control unit 202 receives a designation of the detail object 628, it generates a detail information area (not shown) that displays detailed information received from the server 10 in response to the streaming request. The viewer UI control unit 202 displays the generated detail information area on the live streaming room screen 608.

[0073] In other embodiments, instead of or in addition to providing a summary display area, the summary may be displayed in the comment display area 618 or output as audio. Alternatively, if a live streaming assistant using a machine learning model is provided on the live streaming room screen, the assistant may be caused to output the summary.

[0074] 12 is a representative screen diagram of a live streaming room screen 608 displayed on the display of a viewer's user terminal, with a summary area 630 superimposed. FIG. 12 corresponds to the live streaming room screen display when viewer N participates in the live streaming in FIG. 7. The summary area 630 displays a detail object 628 in conjunction with the summary. The summary area 630 includes a summary of the live streaming in which the viewer participated, and the detail object 628.

[0075] The viewer UI control unit 202 of the user terminal displays the multiple comment candidates received from the server 10 on the display in conjunction with the provision of the summary. In the example of FIG. 12 , the viewer UI control unit 202 stops displaying the summary area 630 when a predetermined period of time, for example, 10 seconds, has elapsed since the display of the summary area 630 began. Alternatively, the viewer UI control unit 202 stops displaying the summary area 630 when it detects a tap on an area other than the summary area 630. After stopping display of the summary area 630, the viewer UI control unit 202 generates a comment candidate display area 636 that displays the multiple comment candidates received from the server 10. The viewer UI control unit 202 displays the generated comment candidate display area 636 on the live streaming room screen 608.

[0076] FIG. 13 is a representative screen diagram of the live streaming room screen 608, displayed on the display of the viewer's user terminal, with a comment candidate display area 636 superimposed. FIG. 13 corresponds to the state after the display of the summary area 630 in FIG. 12 has finished. The comment candidate display area 636 includes a first candidate object 632 that displays a first comment candidate in text, and a second candidate object 634 that displays a second comment candidate in text. When the viewer-side communication unit 204 detects a tap on the first candidate object 632, it generates a comment input signal including the first comment candidate and transmits it to the server 10 via the network NW. At the same time, the viewer-side UI control unit 202 updates the comment display area 618 to display the first comment candidate. The same process is performed when the second candidate object 634 is tapped.

[0077] Some viewers are not good at interacting, and even when they join a live broadcast, they may find it difficult to make the first comment. In response to this, by providing a system that allows viewers to select from appropriate comment suggestions and then comment, the hurdle to making the first comment can be lowered or made easier to comment, thereby revitalizing live broadcasts.

[0078] 14 is a representative screen diagram of a distribution preparation screen 670 displayed on the display of the distributor's user terminal. The distribution preparation screen 670 is displayed before the distributor starts live distribution and accepts distribution settings from the distributor. The distribution preparation screen 670 has a video image 672 of the distributor obtained by playing video data to be transmitted by the video transmission unit 106, a distribution setting area 674 for accepting distribution settings, and a distribution start button 676.

[0079] The distribution setting area 674 has an event setting area 682 that accepts input or selection of an event in which the distributor will participate, a tag setting area 684 that accepts input or selection of a distribution content tag that represents the content of the live distribution, a personality setting area 678 that accepts input or selection of the personality of the summary generation model 326 or the detail generation model 328, and a title setting area 680 that accepts input of a title. The distributor inputs desired settings in the distribution setting area 674 and taps the distribution start button 676. When the distributor's user terminal detects a tap on the distribution start button 676, the distribution side communication unit 110 generates distribution setting information including the event, distribution content tag, personality, and title that are input in the distribution setting area 674 at that time, and transmits a distribution start instruction along with the generated distribution setting information to the server 10 via the network NW.

[0080] FIG. 15 is a representative screen diagram of a live streaming room screen 650 displayed on the display of the broadcaster's user terminal. The live streaming room screen 650 displays video generated on the broadcaster's user terminal in real time. The live streaming room screen 650 includes a video 652 of the broadcaster obtained by playing back video data to be transmitted by the video transmission unit 106, a comment display area 654, a broadcast end button 656, a summary display area 658, an update object 660, and a hidden object 662. The summary display area 658, the update object 660, and the hidden object 662 are displayed in association with one another. The broadcasting-side UI control unit 108 generates the live streaming room screen 650 by superimposing other objects, namely, the broadcast end button 656, the comment display area 654, the summary display area 658, the update object 660, and the hidden object 662, on the video 652 obtained by playing back the video data.

[0081] The distribution end button 656 is an object for receiving an instruction from the distributor to stop providing live distribution.

[0082] The summary display area 658 displays a summary of the live streaming that will be displayed on the display of the user terminal of a viewer who is about to join the live streaming. The distribution-side communication unit 110 periodically generates a summary provision request, for example, once every five minutes, and transmits it to the server 10 via the network NW. When the summary generation unit 322 receives a summary provision request from the user terminal of the distributor, it obtains a summary of the live streaming being performed by the distributor from the stream DB 314. The summary generation unit 322 transmits the obtained summary to the user terminal that made the request via the network NW. The distribution-side UI control unit 108 generates a summary display area 658 including the text of the received summary and displays it on the live streaming room screen 650.

[0083] When a tap on the update object 660 is detected, the delivery-side communication unit 110 generates an update request and transmits it to the server 10 via the network NW. When the summary generation unit 322 receives the update request from the user terminal of the distributor, it acquires time-series data of the live distribution performed by the distributor and generates a summary. The summary generation unit 322 transmits the generated summary to the user terminal that issued the request via the network NW. The delivery-side UI control unit 108 updates the display in the summary display area 658 with the received summary text.

[0084] When a tap on the hidden object 662 is detected, the delivery-side UI control unit 108 stops displaying the summary display area 658, the updated object 660, and the hidden object 662.

[0085] The summary display area 658 allows the broadcaster to check the summary that will be presented to viewers who newly join their live broadcast, and if it is inappropriate, change it using the update object 660. If the summary display is distracting, the broadcaster can hide the summary using the hide object 662. A rejection object may be displayed in association with the summary display area 658 on the live broadcast room screen 650. When a rejection object is designated, the user terminal generates a summary display rejection signal and transmits it to the server 10. For live broadcasts for which the server 10 has received a summary display rejection signal, the server 10 will not provide a summary to newly joined viewers. This allows the broadcaster to prevent unwanted summaries from being provided.

[0086] In the above-described embodiments, examples of the database (DB) are a hard disk and a semiconductor memory. Furthermore, based on the description in this specification, those skilled in the art will understand that each unit can be realized by a CPU (not shown), an installed application program module, a system program module, a semiconductor memory that temporarily stores the contents of data read from a hard disk, or the like.

[0087] According to the live streaming system 1 of this embodiment, a summary of a live stream is generated based on the time-series data of the live stream. As the live stream progresses and the time-series data is updated, the content of this summary is also updated. This summary is provided to viewers who join the live stream midway. Therefore, by viewing the summary, viewers who join midway can quickly grasp what is happening in the live stream and the flow of the conversation. As a result, viewers can smoothly enter the circle of communication in the live stream, improving their satisfaction. In addition, since this summary more accurately represents the content of the live stream compared to static information such as a title, mismatches between the live stream and viewers can be reduced or eliminated.

[0088] In the live streaming system 1 according to this embodiment, the summary generation model 326 processes time-series data to generate summaries. This enables real-time generation and updating of summaries, which is difficult to achieve manually. Unlike VOD, live streaming requires real-time summary generation, and applying a machine learning model as in this embodiment makes it possible to achieve such real-time generation.

[0089] In the live streaming system 1 according to this embodiment, the server 10 receives information for adjusting the summary generation model 326 and the detail generation model 328 from the user terminal of the live stream broadcaster when the live stream starts. The summary generation model 326 and the detail generation model 328 are adjusted according to this information. This makes it possible to generate summaries and detailed information that are more in line with the broadcaster's intentions, allowing each broadcaster to assert their own unique features.

[0090] The hardware configuration of an information processing device according to this embodiment will be described with reference to Fig. 16. Fig. 16 is a block diagram showing an example of the hardware configuration of an information processing device according to this embodiment. The illustrated information processing device 900 can realize, for example, each of the server 10 and the user terminals 20 and 30 according to this embodiment.

[0091] The information processing device 900 includes a CPU 901, a read-only memory (ROM) 902, and a random-access memory (RAM) 903. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 925, and a communication device 929. The information processing device 900 also includes an imaging device (not shown) such as a camera. The CPU 901 is an example of a hardware configuration for realizing functions realized by the components described herein. The functions described herein may be realized by circuits programmed to realize the described functions. Circuits programmed to realize the functions described herein include a central processing unit (CPU), a digital signal processor (DSP), a general-purpose processor, an application-specific processor, an integrated circuit, an application-specific integrated circuit (ASIC), and / or a combination thereof. In this specification, a unit that achieves a specific function may be realized as a circuit programmed to achieve that function.

[0092] The CPU 901 functions as an arithmetic processing unit and control unit, and controls all or part of the operations within the information processing device 900 in accordance with various programs recorded in the ROM 902, RAM 903, storage device 919, or removable recording medium 923. For example, the CPU 901 controls all of the operations of the functional units included in the server 10 and the user terminals 20 and 30 in this embodiment. The ROM 902 stores programs and calculation parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901, as well as parameters that change as appropriate during the execution. The CPU 901, ROM 902, and RAM 903 are interconnected by a host bus 907, which is an internal bus such as a CPU bus. The host bus 907 is further connected to an external bus 911, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 909.

[0093] The input device 915 may be, for example, a device operated by a user, such as a mouse, keyboard, touch panel, button, switch, or lever, or may be a device that converts physical quantities into electrical signals, such as a sound sensor such as a microphone, an acceleration sensor, a tilt sensor, an infrared sensor, a depth sensor, a temperature sensor, or a humidity sensor. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device 927 such as a mobile phone that supports operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user or a sensed physical quantity and outputs the signal to the CPU 901. The user operates the input device 915 to input various data to the information processing device 900 or to instruct processing operations.

[0094] The output device 917 is configured by a device that can notify the user of acquired information visually or audibly. The output device 917 can be, for example, a display such as an LCD, PDP, or OELD, an audio output device such as a speaker or headphones, or a printer. The output device 917 outputs the results obtained by processing by the information processing device 900 as video such as text or images, or as sound such as audio.

[0095] The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900. The storage device 919 is configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs and various data executed by the CPU 901, as well as various data acquired from the outside.

[0096] The drive 921 is a reader / writer for a removable recording medium 923 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the information processing device 900. The drive 921 reads information recorded on the attached removable recording medium 923 and outputs it to the RAM 903. The drive 921 also writes information to the attached removable recording medium 923.

[0097] The connection port 925 is a port for directly connecting a device to the information processing device 900. The connection port 925 can be, for example, a Universal Serial Bus (USB) port, an IEEE 1394 port, or a Small Computer System Interface (SCSI) port. The connection port 925 can also be an RS-232C port, an optical audio terminal, or a High-Definition Multimedia Interface (HDMI) (registered trademark) port. By connecting an external device 927 to the connection port 925, various types of data can be exchanged between the information processing device 900 and the external device 927.

[0098] The communication device 929 is, for example, a communication interface configured with a communication device for connecting to a network NW. The communication device 929 can be, for example, a communication card for a wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 929 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The communication device 929 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP / IP. The communication network NW connected to the communication device 929 is a network connected by wire or wirelessly, such as the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication. The communication device 929 also functions as a communication unit.

[0099] An imaging device (not shown) such as a camera is a device that captures real space and generates a captured image using an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and various components such as a lens for controlling the formation of a subject image on the imaging element. The imaging device may capture still images or moving images.

[0100] The above describes the configuration and operation of the live streaming system 1 according to the embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each process, and that such modifications are also within the scope of the present disclosure.

[0101] In the embodiment, a summary of a live broadcast is presented to a viewer when the viewer joins an ongoing live broadcast, but this is not limited to this. For example, a summary may be provided to a viewer when the viewer starts viewing an archive of the live broadcast, a clip (video) cut out from the archive, a video on demand (VoD) video, a profile video, or a preview of the live broadcast.

[0102] In the embodiment, the summary generation model 326 outputs summaries in text format. However, the present invention is not limited to this. The summary generation model 326 may generate summary information that includes marks, effects, audio, still images, and videos (such as explanatory videos) instead of or in addition to text. Alternatively, in the case of live streaming using an AR app or a live walk, the summary generation model 326 may acquire actual videos captured by the camera of the broadcaster's user terminal, live videos, recorded videos, or the current location using a GPS function, and generate a summary based on the acquired information. Such a summary may include, for example, text describing the broadcaster's walking route so far. Such text may include, for example, text to the effect that the broadcaster has traveled to such and such places and is now at such and such a place.

[0103] FIG. 17 is a representative screen diagram of a live streaming room screen 608 displayed on the display of a viewer's user terminal, with a summary area 690 according to a modified example superimposed. Upon receiving time-series data, the summary generation model according to this modified example outputs the positive index, negative index, frequent gifter rate, and mood as summary information. The summary area 690 displays this summary information received from the server 10 at the start of viewing the live streaming. In the example of FIG. 17, the positive index (P) is 352, the negative index (N) is 120, the frequent gifter rate is 45%, and the mood is "calm." The positive index increases in response to positive comments or comments from the streamer. The negative index increases in response to negative comments or comments from the streamer. In another modified example, the positive index and negative index may be combined and expressed as a single parameter. For example, the positive index may be expressed as 60% (the negative rate is 100-60=40%). The frequent gifter rate indicates the proportion of communication volume from high-value gifters to the total communication volume of the live streaming. For example, if the total number of comments is 1000, and the total number of comments from the highest-value gifters (i.e., the viewers with the highest, second, and third highest total gifting amounts in the live stream) is 500, then the frequent customer rate is 500 / 1000 = 0.5, or 50%. Mood indicates the mood of the live stream, and in addition to quiet, there are moods such as bright / dark / complaining / high-energy / low-energy / everyone encouraging each other.

[0104] In addition to the example summary information shown in Figure 17, summary information may also be generated that includes the number, type, and trend of gifts used in a live stream, gifts desired by the streamer and their usage status, a set list for a music stream, information on which songs have been played so far and which song is currently playing, and information on which game has been played so far and which game is currently being played, and information on which stage is being played and what stage it is, for a game stream. While the embodiments describe a case where a summary is generated using a machine learning model, this is not limiting. In particular, when the above-mentioned marks and parameters are used as summary information, summary information may be generated using a predetermined mathematical formula or lookup table. Alternatively, the server may obtain information about the live stream and its streamer obtained from an external Internet source, information about the live stream and streamer registered on a live streaming platform, or clips and archived videos of the streamer, and generate a summary based on the obtained information.

[0105] In the embodiment, the summary generation unit 322 periodically acquires time-series data and generates summaries. However, this is not limiting. A summary may be generated when a viewer participates in a live broadcast and provided to the viewer. Referring to the example of FIG. 7 , the summary generation unit acquires time-series data Dt2 from the stream DB 314 at time t2 when viewer M participates in the live broadcast, and generates a summary for time t2 based on the acquired time-series data Dt2. The distribution information providing unit provides the generated summary to viewer M's user terminal. The summary generation unit acquires time-series data Dt5 from the stream DB 314 at time t5 when viewer N participates in the live broadcast, and generates a summary for time t5 based on the acquired time-series data Dt5. The distribution information providing unit provides the generated summary to viewer N's ​​user terminal. Furthermore, the summary generation unit may generate summaries according to the viewer's attributes. In this case, the summary generation unit acquires the desired topic tags of viewers who participated in the live broadcast from the user DB 318 in addition to the time series data, and generates a summary based on the acquired time series data and desired topic tags. The desired topic tags are input to the summary generation model in addition to the time series data. The summary generation model outputs different summaries even for the same time series data if the desired topic tags are different. In addition to desired topic tags, attributes such as gender, age group, region, language used, charged amount, number of years, and ban history may be used to generate summaries.

[0106] In the embodiment, a case where a comment candidate is generated in response to a distribution request has been described, but this is not limiting, and comment candidates may be generated periodically. In the embodiment, a reason for generating a comment candidate may be generated in association with the comment candidate, and the comment candidate and the reason may be presented to the viewer in association with each other.

[0107] In the embodiment, a summary display object is first displayed when viewing of a live broadcast begins, and when the object is tapped, the summary is displayed. However, this is not limited to this, and the summary may be displayed directly on the live broadcast room screen when viewing begins.

[0108] In the embodiment, the case where the generated summary text is displayed on the live streaming room screen has been described, but this is not limiting. For example, the summary generation model 326 may display each character, word, or sentence it generates on the live streaming room screen, thereby realizing text streaming that displays the summary as it is being generated. In this case, the summary may be gradually updated based on events that occur during the live streaming after a viewer joins the room midway.

[0109] In the embodiment, a case has been described in which a personality for adjusting the summary generation model 326 and the detailed generation model 328 is received from the user terminal of the live broadcaster when starting live broadcasting, but this is not limited to this, and other adjustment information such as a data acquisition period and a prohibited word list may be received instead of / in addition to the personality, and the model may be adjusted based on this.

[0110] In the embodiment, when a thumbnail is selected on the live streaming selection screen 600, a viewer participates in the live streaming, and a summary is also displayed on the live streaming room screen 608. However, this is not limiting. For example, a summary of the live streaming may be displayed on a preview screen of the live streaming.

[0111] In this variation, when a viewer selects a live broadcast, the viewer is not immediately allowed to enter the live broadcast, but is first allowed to watch the live broadcast in preview mode. In preview mode, the viewer can watch the live broadcast without notifying the broadcaster or other viewers of their entry, and a summary of the live broadcast is provided to the viewer. The viewer can read the summary in preview mode, understand the content of the live broadcast, and enter if interested, or exit without notifying the broadcaster or other viewers.

[0112] In this variation, the user terminal waits for detection of a tap on a thumbnail displayed on the live streaming selection screen. When the tap is detected, the user terminal displays a live streaming room screen in preview mode on the display. FIG. 18 is a representative screen diagram of a live streaming room screen 708 in preview mode displayed on the display of the user terminal. The live streaming room screen 708 in preview mode displays video generated by the broadcaster's user terminal 20 in real time. The live streaming room screen 708 includes a video 710 of the broadcaster obtained by playing video data received from the server 10, a comment display area 718, a summary display area 722 that displays a summary of the live streaming received from the server 10, an entry inquiry pop-up 724 that asks viewers whether they want to enter the live streaming, and a preview frame 726. The preview frame 726 indicates that the live streaming is being viewed in preview mode. The preview frame 726 is an object added to distinguish the live streaming room screen in normal mode, as shown in FIG. 10, from the live streaming room screen in preview mode. Due to the presence of this preview frame 726, it can be said that the live streaming room screen in normal mode and the live streaming room screen in preview mode are displayed in different ways.

[0113] In preview mode, immediately after a viewer selects a thumbnail, the server is configured not to notify the broadcaster or other viewers of information about the viewer. For example, the presence or absence of a viewer in preview mode does not affect information associated with the live broadcast (such as comments or viewer lists). Alternatively, the server may manage viewers in preview mode in a different manner from other viewers. For example, viewers in preview mode may be managed using a viewer list that is different from the viewer list in which other viewers are registered, or a flag indicating that a viewer in preview mode is assigned to the viewer list. When the server is requested by the broadcaster to provide a viewer list, the server may exclude viewers with the flag set from the list it provides.

[0114] In the preview mode, the viewer's input of information is limited. Specifically, the live streaming room screen 708 in the preview mode does not have a gift object or a comment input area.

[0115] The viewer taps to select the desired option from "Yes" (i.e., enter the room) or "Leave the broadcast" displayed in the entry inquiry pop-up 724. When "Leave the broadcast," i.e., leave the room, is selected, the user terminal performs the exit process. When "Yes," i.e., enter the room, is selected, the user terminal transitions the display from the live broadcasting room screen in preview mode to the live broadcasting room screen in normal mode (equivalent to the live broadcasting room screen in FIG. 10 excluding the summary display object 622 and unnecessary objects 624).

[0116] According to this modification, by reading the summary in addition to viewing the preview, the user can more quickly and accurately understand the content of the live broadcast and can decide whether or not to subscribe.

[0117] Instead of or in addition to the text summary displayed in the summary display area 722, marks and parameters such as those shown in FIG. 17 may be presented as summary information.

[0118] In an embodiment, the server may change the summary output format automatically or as specified by the broadcaster depending on the type of live streaming. The broadcaster's designation of a personality is one example. Alternatively, the server may identify the type of live streaming (music streaming, fortune-telling streaming, chat streaming, game streaming, etc.) from the broadcaster's information registered in the user database 318, and determine the summary output format depending on the identified type. For example, if the type is music streaming, the summary is set to include the set list, the name of the song currently being played, and the number of the song.

[0119] The conversion rates from gift points to rewards in the embodiments are merely examples, and may be set appropriately by, for example, an administrator of the live distribution system.

[0120] The technical concept of the embodiments may be applied to virtual live streaming or live commerce, in which an avatar that moves in synchronization with the streamer's movements is used instead of the streamer's image. Furthermore, in the embodiments, a case has been described in which a server relays video data related to a live stream generated on the streamer's user terminal and transmits it to the viewer's user terminal, but this is not limited to this. For example, the technical concept of the embodiments may be applied when a virtual streamer is set up instead of an actual streamer. The virtual streamer is, for example, an AI virtual streamer whose appearance is an avatar, whose voice is generated by a TTS (Text-To-Speech) engine, and whose remarks are obtained from a machine learning model that uses viewer comments as input. In this case, the streamer's user terminal does not exist, and the streamer's processing is performed by the server.

[0121] In the processing procedures described in this specification, particularly in processing procedures described using flow diagrams and flowcharts, it is possible to omit some of the processes (steps) that make up the processing procedures, to add processes that are not explicitly stated as processes that make up the processing procedures, and / or to change the order of the processes, and processing procedures in which such omissions, additions, or changes in order have been made are also included within the scope of this disclosure as long as they do not deviate from the intent of this disclosure.

[0122] At least some of the functions realized by the server 10 may be realized by a device other than the server 10, for example, the user terminals 20 and 30. At least some of the functions realized by the user terminals 20 and 30 may be realized by a device other than the user terminals 20 and 30, for example, the server 10. For example, the superimposition of a predetermined frame image onto an image of video data performed on a viewer's user terminal may be performed on the server 10 or on the distributor's user terminal.

Claims

1. A server including an acquisition means and a generation means, the acquiring means acquires, at a first time point during the progress of the live streaming, first time-series data representing the content of the live streaming that is recorded as the live streaming progresses; the generation means generates summary information at the first time point of the live broadcast based on the acquired first time-series data; the acquiring means acquires second time-series data representing content of the live streaming recorded as the live streaming progresses at a second time point during the live streaming that is later than the first time point; The generating means is a server that generates summary information at the second time point of the live distribution based on the acquired second time series data.

2. The server of claim 1 further comprises a transmitting means for transmitting summary information at the first time point via a network to terminals of viewers who joined the live distribution before the second time point, and for transmitting summary information at the second time point via a network to terminals of viewers who joined the live distribution after the second time point.

3. The server according to claim 1, wherein the generation means generates summary information at the first time point by inputting the first time series data into a machine learning model, and generates summary information at the second time point by inputting the second time series data into the machine learning model.

4. The server according to claim 3 , further comprising a receiving means for receiving information for adjusting the machine learning model from a terminal of a broadcaster of the live broadcast when the live broadcast starts.

5. The server according to claim 1 , wherein the acquisition means and the generation means periodically acquire the time-series data and periodically generate the summary information.

6. The server according to claim 1 , wherein the first time point is a time point when a viewer joins the live distribution, and the second time point is a time point when another viewer joins the live distribution.

7. 7. The server according to claim 6, wherein said generating means generates summary information at said first time point according to attributes of said viewer, and generates summary information at said second time point according to attributes of said other viewer.

8. The ability to send requests to a server over a network; a function of receiving summary information of an ongoing live broadcast from the server via the network, the summary information having content that may vary depending on the timing of the request; A function to start playing the video related to the live distribution; and a computer program for causing the terminal to realize a function of displaying the summary information on a display in synchronization with the start of playback of the video.

9. The function of displaying the information is a function of displaying an object on the display in synchronization with the start of playback of the video; 9. The computer program according to claim 8, further comprising a function of displaying the summary information on the display when the designation of the object is accepted.

10. the receiving function includes a function of receiving, together with the summary information, detailed information about the live broadcast that includes more detailed content than the summary information; The function of displaying the information is a function of displaying an object on the display in conjunction with the provision of the summary information; The computer program according to claim 8 , further comprising: a function of providing the detailed information when the designation of the object is accepted.

11. a function of receiving from the server via the network a plurality of comment candidates whose contents may vary depending on the timing of the request; The computer program according to claim 8 , further comprising a function of displaying the plurality of comment candidates on the display in conjunction with the provision of the summary information.

12. the function of starting includes a function of starting playback of the video related to the live distribution in a first mode; the function of displaying includes a function of providing the summary information in synchronization with the start of playback of the video in the first mode; The computer program further comprises: causing the terminal to realize a function of starting playback of the video in a second mode in response to reception of a predetermined user input in the first mode; 9. The computer program according to claim 8, wherein disclosure of information about the user of the terminal is more restricted in the first mode than in the second mode.

13. At a first time point during the progress of the live streaming, first time series data representing the content of the live streaming is acquired, the first time series data being recorded as the live streaming progresses; generating summary information for the first time point of the live broadcast based on the acquired first time-series data; acquiring second time-series data representing content of the live streaming, the second time-series data being recorded as the live streaming progresses, at a second time-series during the live streaming that is later than the first time-series; generating summary information for the second time point of the live broadcast based on the acquired second time series data.

Images