Server, computer program and method

Abstract

To increase convenience for viewers who join a live broadcast midway by providing a summary that reflects the current state of the live broadcast. [Solution] The server is equipped with an acquisition means and a generation means, wherein the acquisition means acquires first time series data representing the content of the live streaming recorded as the live streaming progresses at a first point in time during the live streaming, and the generation means generates summary information at the first point in time of the live streaming based on the acquired first time series data, and the acquisition means acquires second time series data representing the content of the live streaming recorded as the live streaming progresses at a second point in time during the live streaming that is later than the first point in time, and the generation means generates summary information at the second point in time of the live streaming based on the acquired second time series data.

Description

[Technical field]

[0001] The present disclosure relates to a server, a computer program and a method. [Background technology]

[0002] With the development of IT technology, the way information is exchanged has also changed. In the Showa era, one-way information transmission such as newspapers and television was the norm. In the Heisei era, mobile phones and personal computers became widespread and Internet communication speeds improved significantly, leading to the rise of instant two-way communication services such as chat services, and on-demand video distribution services became more popular as storage costs decreased. Now, in the Reiwa era, with the increasing functionality of smartphones and further improvements in network speeds such as 5G, services that enable real-time communication through video, especially live streaming services, are rapidly gaining recognition. Live streaming services are seeing an increase in users, especially among young people, as they allow everyone to share the same fun time even when they are far apart.

[0003] Patent Literature 1 discloses a technology that generates advice information for a distributor during live streaming of a video and displays it on the distributor's screen. This allows the distributor to perform live streaming while referring to the advice information displayed in real time. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-075401 [Non-patent literature]

[0005] [Non-Patent Document 1] "A demo of an AI responding to the contents of an image in text", ExaWizards, Inc., URL: https: / / techblog.exawizards.com / entry / 2019 / 02 / 15 / 175416 Summary of the Invention [Problem to be solved by the invention]

[0006] According to the technology described in Patent Document 1, a distributor can obtain information about viewers who have newly joined a live distribution. However, the technology described in Patent Document 1 does not provide any benefit to viewers who have newly joined or are planning to join a live distribution.

[0007] The nature of live streaming is that, since there is no script, the content changes from moment to moment due to interaction and communication with viewers, and there is a possibility that it will move in a direction different from the initial assumption. Therefore, it is difficult for viewers who join a live streaming in the middle to grasp the flow and content of the live streaming in a short time. Even if the title and thumbnail reflect the content of the live streaming that the broadcaster envisioned before the live streaming started, in reality, as the live streaming progresses, the story progresses in a direction different from the initial assumption, and as a result, there is often a discrepancy between the title and thumbnail and the actual content. In such cases, viewers who join in the middle, expecting the content shown in the title and thumbnail, may be confused when they see the actual content of the live streaming.

[0008] The present disclosure has been made in consideration of these problems, and its purpose is to provide technology that can increase convenience for viewers who join a live broadcast midway by providing a summary that reflects the current state of the live broadcast. [Means for solving the problem]

[0009] One aspect of the present invention relates to a server including an acquisition means and a generation means, in which the acquisition means acquires first time series data representing the content of the live streaming recorded as the live streaming progresses at a first time point during the progress of the live streaming, the generation means generates summary information for the first time point of the live streaming based on the acquired first time series data, the acquisition means acquires second time series data representing the content of the live streaming recorded as the live streaming progresses at a second time point during the progress of the live streaming that is later than the first time point, and the generation means generates summary information for the second time point of the live streaming based on the acquired second time series data.

[0010] Another aspect of the present invention is a computer program that causes a terminal to perform the following functions: sending a request to a server via a network, receiving summary information of an ongoing live distribution, the summary information being different in content depending on the timing of the request from the server via the network, starting playback of a video related to the live distribution, and displaying the summary information on a display in synchronization with the start of playback of the video.

[0011] In addition, any combination of the above components, or mutual substitution of the components or expressions of the present invention between an apparatus, a method, a system, a computer program, a recording medium storing a computer program, etc., are also valid aspects of the present invention. Effect of the Invention

[0012] According to the present invention, convenience for viewers who join a live distribution midway is increased by providing a summary that reflects the current state of the live distribution. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram illustrating a configuration of a live distribution system according to an embodiment of the present disclosure. [Diagram 2]2 is a block diagram showing the functions and configuration of the user terminal of FIG. 1. [Diagram 3] 2 is a block diagram showing the functions and configuration of the server in FIG. 1. [Figure 4] 4 is a data structure diagram showing an example of a stream DB in FIG. 3. [Diagram 5] 4 is a data structure diagram showing an example of a user DB in FIG. 3. [Figure 6] 4 is a data structure diagram showing an example of a gift DB in FIG. 3. [Figure 7] FIG. 11 is an explanatory diagram showing the relationship between the progress of a live broadcast and a generated summary. [Figure 8] 13 is a flowchart showing a series of processes related to dynamic summary generation of live streaming. [Figure 9] FIG. 13 is a representative screen diagram of a live streaming selection screen displayed on the display of the user terminal of an active user. [Figure 10] FIG. 13 is a representative screen diagram of a live streaming room screen displayed on the display of a viewer's user terminal. [Figure 11] FIG. 13 is a representative screen diagram of a live streaming room screen with a summary area superimposed thereon, which is displayed on the display of a viewer's user terminal. [Figure 12] FIG. 13 is a representative screen diagram of a live streaming room screen with a summary area superimposed thereon, which is displayed on the display of a viewer's user terminal. [Figure 13] FIG. 13 is a representative screen diagram of a live streaming room screen with a comment candidate display area superimposed thereon, which is displayed on the display of a viewer's user terminal. [Figure 14] FIG. 13 is a representative screen diagram of a distribution preparation screen displayed on the display of a distributor's user terminal. [Figure 15] FIG. 13 is a representative screen diagram of a live streaming room screen displayed on the display of a user terminal of a broadcaster. [Figure 16] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to an embodiment of the present invention; [Figure 17]FIG. 13 is a representative screen diagram of a live streaming room screen with a summary area according to a modified example superimposed thereon, which is displayed on the display of a viewer's user terminal. [Figure 18] FIG. 13 is a representative screen diagram of a live streaming room screen in preview mode displayed on the display of a user terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the same or equivalent components, members, processes, and signals shown in each drawing will be given the same reference numerals, and duplicated explanations will be omitted as appropriate. Also, some of the members that are not important for the explanation will be omitted in each drawing.

[0015] In the live streaming system according to the embodiment, when a viewer joins a live streaming midway, the viewer is provided with a summary of the situation generated by a machine learning model (for example, a language analysis model such as GPT) from the conversation in the live streaming up to that point. The summary is information summarizing the conversation topics and flow of the conversation in the live streaming, symbolic events, the behavior, activities, and feelings of the broadcaster, the comments of the viewers, the behavior and gifting situation, the atmosphere of the live streaming, and the content of the live streaming. As a result, a viewer who joined the live streaming midway can quickly understand what has been discussed in the live streaming up to now and what the current conversation flow is by reading the provided summary. As a result, even a viewer who joined midway can easily join the conversation in the live streaming, so the viewer can enjoy the live streaming more, and retention and engagement are improved.

[0016] FIG. 1 is a schematic diagram showing a configuration of a live distribution system 1 according to an embodiment of the present disclosure. The live distribution system 1 provides an interactive live distribution service in which a distributor (also called a live streamer) LV and a viewer (also called an audience) AU (AU1, AU2, ...) can communicate in real time. As shown in FIG. 1, the live distribution system 1 includes a server 10, a user terminal 20 on the distributor side, and a user terminal 30 (30a, 30b, ...) on the viewer side. In addition to distributors who distribute live distribution and viewers who watch live distribution, there are also users who log in to the live distribution platform but do not distribute or watch. Such users are called active users. The distributors, viewers, and active users may be collectively called users. The server 10 may be configured by one or more information processing devices connected to a network NW. The user terminals 20 and 30 may be, for example, mobile terminals such as smartphones, tablet terminals, laptop PCs, recorders, portable game consoles, and wearable devices, or may be stationary devices such as desktop PCs. The server 10, the user terminal 20, and the user terminal 30 are connected to each other so as to be able to communicate with each other via various wired or wireless networks NW.

[0017] The live distribution system 1 involves a distributor LV, a viewer AU, and an administrator (not shown) who manages the server 10. The distributor LV is a person who transmits content in real time by recording and filming his / her own songs, talks, performances, fortune telling, game commentary, etc. on his / her own user terminal 20 and uploading the content to the server 10 as is. The administrator provides a platform for live distribution of content on the server 10, and also mediates or manages real-time exchanges between the distributor LV and the viewer AU. The viewer AU accesses the platform with the user terminal 30, selects desired content, and views it. During the live distribution of this content, the viewer AU performs operations to make comments, cheer, or request fortune telling through the user terminal 30, and the distributor LV providing the content responds to such comments, cheers, or requests, and the reactions are transmitted to the viewer AU by video and / or audio, thereby establishing two-way communication.

[0018] In this specification, "live distribution" may mean a data transmission mode that realizes a state in which content recorded on a user terminal 20 of a distributor LV is played back and made viewable on a user terminal 30 of a viewer AU in substantially real time, or may mean distribution itself realized by such a transmission mode. Live distribution may be realized using existing live distribution technologies such as HTTP Live Streaming, Common Media Application Format, Web Real-Time Communications, Real-Time Messaging Protocol, and MPEG DASH. Live distribution includes a transmission mode in which a viewer AU can view content with a predetermined delay while a distributor LV is recording the content. The amount of delay is at least as large as the amount of communication between the distributor LV and the viewer AU. However, live distribution is distinguished from so-called on-demand distribution in which the entire recorded data of the content is temporarily stored on a server, and the data is provided to a user from the server at any time thereafter in response to a request from the user.

[0019] In this specification, "moving image data" refers to data including image data (also called video data) generated by the imaging function of the user terminals 20 and 30, and audio data (also called audio data) generated by the audio input function of the user terminals 20 and 30. The moving image data is played back on the user terminals 20 and 30, allowing users to view the content. In this embodiment, it is assumed that processing such as compression, decompression, encoding, decoding, transcoding, and other processing is performed to change the format, size, and specifications of the data from when the moving image data is generated on the distributor's user terminal until it is played back on the viewer's user terminal. Since the contents (e.g., moving images and audio) represented by the moving image data before and after such processing are substantially unchanged, this embodiment will be described assuming that the moving image data after such processing is the same as the moving image data before such processing. In other words, when moving image data is generated on the distributor's user terminal and then played back on the viewer's user terminal via the server 10, the moving image data generated on the distributor's user terminal, the moving image data passing through the server 10, and the moving image data received and played on the viewer's user terminal are all the same moving image data.

[0020] In the example of Fig. 1, a broadcaster LV is live-streaming a talk. The user terminal 20 of the broadcaster LV records and records the image and sound of the broadcaster LV while he or she is talking, generating video data, and transmits the video data to the server 10 via the network NW. The user terminal 20 also displays the recorded video image VD of the broadcaster LV on the display of the user terminal 20, thereby enabling the broadcast content by the broadcaster LV to be confirmed.

[0021] The user terminals 30a, 30b of the viewers AU1, AU2 who have requested the platform to watch the live broadcast of the broadcaster LV each receive video data related to the live broadcast via the network NW, and play the received video data to display the videos VD1, VD2 on the display and output audio from the speaker. The videos VD1, VD2 displayed on each user terminal 30a, 30b are substantially identical to the video VD captured by the user terminal 20 of the broadcaster LV, and the audio output from each user terminal 30a, 30b is also substantially identical to the audio recorded by the user terminal 20 of the broadcaster LV.

[0022] The recording and recording in the user terminal 20 of the distributor LV and the playback of the video data in the user terminals 30a and 30b of the viewers AU1 and AU2 are performed substantially simultaneously. When one viewer AU1 inputs a comment on the content of the talk of the distributor LV to the user terminal 30a, the server 10 displays the comment in real time on the user terminal 20 of the distributor LV and also on the user terminals 30a and 30b of each viewer AU1 and AU2. When the distributor LV reads the comment and develops a talk that overlaps the content, the video and audio of the talk are output on the user terminals 30a and 30b of each viewer AU1 and AU2, and it is recognized that a conversation between the distributor LV and the viewer AU1 has been established. In this way, the live distribution system 1 realizes live distribution that enables two-way communication, not one-way.

[0023] Fig. 2 is a block diagram showing the functions and configuration of the user terminal 20 in Fig. 1. The user terminal 30 has the same functions and configuration as the user terminal 20. Each block shown in Fig. 2 and the following block diagrams can be realized in hardware terms by elements and mechanical devices such as a computer CPU, and in software terms by a computer program, etc., but here, functional blocks realized by the cooperation of these are depicted. Therefore, those skilled in the art who have read this specification will understand that these functional blocks can be realized in various ways by combining hardware and software.

[0024] The distributor LV and the viewer AU download and install a live distribution application program (hereinafter, referred to as a live distribution app) according to this embodiment from a download site to the user terminals 20 and 30 via the network NW. Alternatively, the live distribution app may be pre-installed in the user terminals 20 and 30. When the live distribution app is executed by the user terminals 20 and 30, the user terminals 20 and 30 communicate with the server 10 via the network NW to realize various functions. Hereinafter, functions realized by the user terminals 20 and 30 (processors such as CPUs) executing the live distribution app will be described as functions of the user terminals 20 and 30. These functions are actually functions that the live distribution app causes the user terminals 20 and 30 to realize. Note that in other embodiments, these functions may be realized by a computer program written in a programming language such as HTML (HyperText Markup Language), which is transmitted from the server 10 to a web browser of the user terminals 20 and 30 via the network NW and executed by the web browser.

[0025] The user terminal 20 includes a distribution unit 100 that generates video data recording the user's image and voice and provides it to the server 10, a viewing unit 200 that acquires the video data from the server 10 and plays it, and a non-distribution processing unit 400 that processes requests from active users. The user activates the distribution unit 100 when distributing, the viewing unit 200 when viewing, and the non-distribution processing unit 400 when searching for a live broadcast to watch, viewing a distributor's profile, or viewing an archive. A user terminal on which the distribution unit 100 is active is a user terminal on the distributor side, that is, a user terminal on the generating side of video data, a user terminal on which the viewing unit 200 is active is a user terminal on the viewer side, that is, a user terminal on the playing side of video data, and a user terminal on which the non-distribution processing unit 400 is active is a user terminal of an active user.

[0026] The delivery unit 100 includes an imaging control unit 102, an audio control unit 104, a video transmission unit 106, a delivery-side UI control unit 108, and a delivery-side communication unit 110. The imaging control unit 102 is connected to a camera (not shown in FIG. 2) and controls imaging by the camera. The imaging control unit 102 acquires image data from the camera. The audio control unit 104 is connected to a microphone (not shown in FIG. 2) and controls audio input by the microphone. The audio control unit 104 acquires audio data from the microphone. The video transmission unit 106 transmits video data including the image data acquired by the imaging control unit 102 and the audio data acquired by the audio control unit 104 to the server 10 via the network NW. The video transmission unit 106 transmits the video data in real time. That is, the generation of video data by the imaging control unit 102 and the audio control unit 104 and the transmission of the generated video data by the video transmission unit 106 are substantially performed simultaneously.

[0027] The distribution side UI control unit 108 controls the UI for the distributor. The distribution side UI control unit 108 is connected to a display not shown in FIG. 2, and displays a moving image on the display by playing back the moving image data to be transmitted by the moving image transmission unit 106. The distribution side UI control unit 108 is connected to input means such as a touch panel, a keyboard, and a display not shown in FIG. 2, and acquires input from the distributor via the input means. The distribution side UI control unit 108 superimposes a predetermined frame image on the moving image. The frame image includes various user interface objects (hereinafter simply referred to as objects) for receiving input from the distributor, comments input by the viewer, and information acquired from the server 10. The distribution side UI control unit 108 accepts, for example, a tap input on an object by the distributor.

[0028] The delivery-side communication unit 110 controls communication with the server 10 during live delivery. The delivery-side communication unit 110 transmits the contents of the input by the distributor acquired by the delivery-side UI control unit 108 to the server 10 via the network NW. The delivery-side communication unit 110 receives various types of information associated with the live delivery from the server 10 via the network NW.

[0029] The viewing unit 200 includes a viewing side UI control unit 202 and a viewing side communication unit 204. The viewing side communication unit 204 controls communication with the server 10 during live distribution. The viewing side communication unit 204 receives video data related to the live distribution in which the distributor and the viewers participate from the server 10 via the network NW.

[0030] The viewer-side UI control unit 202 controls the UI for the viewer. The viewer-side UI control unit 202 is connected to a display and a speaker (not shown in FIG. 2), and plays the received video data to display a video image on the display and output audio from the speaker. The output of an image on the display and audio from the speaker can be collectively referred to as "video data being played". The viewer-side UI control unit 202 is connected to input means (not shown in FIG. 2), such as a touch panel, a keyboard, or a display, and acquires input by the viewer through the input means. The viewer-side UI control unit 202 superimposes a predetermined frame image on the image of the video data acquired from the server 10. The frame image includes various objects for receiving input from the viewer, comments input by the viewer, and information acquired from the server 10. The viewer-side communication unit 204 transmits the contents of the input by the viewer acquired by the viewer-side UI control unit 202 to the server 10 via the network NW.

[0031] The non-broadcast processing unit 400 includes a non-broadcast UI control unit 402 and a non-broadcast communication unit 404. The non-broadcast UI control unit 402 controls a UI for active users. For example, the non-broadcast UI control unit 402 generates a live broadcast selection screen that displays a list of live broadcasts that can be currently participated in and accepts the selection of a live broadcast by an active user, and displays it on the display. The non-broadcast UI control unit 402 generates a profile screen of an arbitrary user and displays it on the display. The non-broadcast UI control unit 402 generates a search screen that accepts search keywords for searching for broadcasters, and displays it on the display. The non-broadcast UI control unit 402 generates a search result display screen including search results for broadcasters, and displays it on the display. The non-broadcast UI control unit 402 plays back an archive generated by recording and filming past live broadcasts.

[0032] The non-broadcast communication unit 404 controls communication with the server 10 outside of live broadcasting. The non-broadcast communication unit 404 receives information for generating a live broadcast selection screen, broadcaster search results, information for generating a profile screen, and archive data from the server 10 via the network NW. The non-broadcast communication unit 404 transmits the contents of input by active users to the server 10 via the network NW.

[0033] Fig. 3 is a block diagram showing the functions and configuration of the server 10 in Fig. 1. The server 10 includes a distribution information providing unit 302, a relay unit 304, a gift processing unit 308, a payment processing unit 310, a stream DB 314, a user DB 318, a gift DB 320, a summary generating unit 322, a detail generating unit 324, a summary generation model 326, a detail generation model 328, and a candidate comment generating unit 330.

[0034] FIG. 4 is a data structure diagram showing an example of the stream DB 314 in FIG. 3. The stream DB 314 holds information on live streaming currently being performed. The stream DB 314 holds video data including images and audio of a video streamed by a streamer. The stream DB 314 holds a stream ID that identifies a live streaming on a live streaming platform provided by the live streaming system 1, a streamer ID that is a user ID that identifies a streamer of the live streaming, a viewer ID that is a user ID that identifies a viewer of the live streaming, a streaming time that is the time from the start of the live streaming to the present, a title of the live streaming set by the streamer before the start of the live streaming, a characteristic of a summary generation model set by the streamer before the start of the live streaming, a streaming content tag that represents the content of the live streaming, image data of the live streaming up to the present, audio data of the live streaming up to the present, a history of comments posted in the live streaming, a history of gifts used in the live streaming, a summary of the live streaming at the present time, and detailed information of the live streaming at the present time in association with each other. The summary and detailed information will be described later.

[0035] In the live streaming platform provided by the live streaming system 1 according to the present embodiment, when a user performs live streaming, the user becomes a streamer, and when the same user watches a live stream by another user, the user becomes a viewer. Therefore, the distinction between streamer and viewer is not fixed, and a user ID registered as a streamer ID at one time may be registered as a viewer ID at another time.

[0036] The broadcast content tag for a live broadcast may be a tag specified by the broadcaster when the broadcaster starts the live broadcast, or it may be a tag obtained by a model generated by machine learning analyzing the live broadcast in real time.

[0037] The title of the live broadcast and the broadcast content tag in the case of advance specification represent the contents of the live broadcast, but are static information that does not change as the live broadcast progresses. In contrast, the broadcast content tag based on real-time analysis, image data, audio data, comment history, gift history, and viewer count represent the contents of the live broadcast and are dynamic information or time series data that change as the live broadcast progresses. Time series data has a structure in which data is arranged along a time axis. The time series data is recorded in the stream DB 314 as the live broadcast progresses.

[0038] 5 is a data structure diagram showing an example of the user DB 318 in FIG. 3. The user DB 318 holds information about users. The user DB 318 includes a user ID for identifying a user, points owned by the user, rewards given to the user, desired topic tags indicating topics designated by the user as being of interest, and participating event IDs for identifying events in which the user is participating. An event is an event related to live streaming held on a live streaming platform provided by the live streaming system 1. There are various types of events, such as rankings and prize winning events.

[0039] Points are electronic value circulated within the live streaming platform. Users purchase points by credit card or other payment methods. Rewards are electronic value defined within the live streaming platform and are an indicator for determining the amount of money that a broadcaster receives from the administrator of the live streaming platform. On the live streaming platform, when a viewer gives a gift to a broadcaster during or outside of a live streaming, the viewer's points are consumed and the broadcaster's reward increases accordingly.

[0040] Fig. 6 is a data structure diagram showing an example of the gift DB 320 in Fig. 3. The gift DB 320 holds information about gifts that can be used by viewers in live distribution. A gift is electronic data that has the following characteristics. -Can be purchased with points or money, or given free of charge. Something that viewers can give to the streamer. Giving a gift to a streamer is also called using a gift or throwing a gift. In some cases, the purchase and use of the gift occur simultaneously as a set, while in other cases the viewer can use the gift at any time after purchasing it. When a viewer gives a gift to a streamer, the streamer will receive a corresponding reward. When a gift is used, an effect associated with the gift may occur. For example, an effect corresponding to the gift may appear on the live streaming room screen.

[0041] The gift DB320 stores a gift ID that identifies a gift, a given reward that is given to a broadcaster when the gift is given to the broadcaster, and a value point that is a value to be paid when the gift is used, in association with each other. A viewer can give a gift to a broadcaster by paying the value points of a desired gift while watching a live broadcast. The payment of the value points may be made by an appropriate electronic payment means, for example, the viewer may pay the value points to the administrator. Alternatively, payment may be made by bank transfer or credit card. The relationship between the given reward and the value points can be arbitrarily set by the administrator. For example, the given reward may be set to the value points. Alternatively, the value points may be set to points obtained by multiplying the given reward by a predetermined coefficient such as 1.2, or the value points may be set to points obtained by adding a predetermined commission point to the given reward.

[0042] Returning to FIG. 3, the summary generation model 326 is a pre-trained machine learning model that receives time-series data of a live stream as input and outputs text (hereinafter simply referred to as a summary) that represents a summary of the live stream. If the input time-series data changes, the output summary also changes, so it can be said that the output summary is the summary at the time when the corresponding time-series data was acquired. The machine learning model may be realized using a known machine learning technique such as BERT (Bidirectional Encoder Representations from Transformers) or GPT (Generative Pretrained Transformer). The machine learning model that generates a summary is known, so it will not be described in detail in this specification. As described above, the time-series data includes non-text data such as image data and audio data. If necessary, a known image analysis technique may be used to convert image data into text data that represents the content of the image data (see, for example, Non-Patent Document 1). A known Speech to Text (STT) technique may be used to convert audio data into text. In this way, the non-text data may be converted into text data before being input to the summary generation model 326. In addition to the time-series data, static information of the live stream may be input to the summary generation model 326. For example, information on an event in which a live broadcaster is participating and which is registered in the user DB 318 may be input to the summary generation model 326 .

[0043] The summary generation model 326 can be set to one of a plurality of different personalities. On a distribution preparation screen before the start of live distribution, the distributor specifies the personality of the model to be used when generating a summary. When the distribution information providing unit 302 receives a distribution start instruction with the specified personality from the distributor's terminal via a network, the summary generation model 326 is adjusted to the specified personality. For example, if the personality "cool" is specified, the summary generation model 326 is adjusted to output text with a cool tone, and if the personality "hot" is specified, the summary generation model 326 is adjusted to output text with a hot tone. This personality setting may be realized using a known prompt engineering technique. The summary generation model 326 may use the personality as one of the input parameters, or an instance of the summary generation model 326 may be generated and used for each ongoing live distribution or for each of a plurality of different personalities.

[0044] The detail generation model 328 is a pre-trained machine learning model that receives time series data of a live stream as input and outputs text representing detailed information about the live stream (hereinafter simply referred to as detailed information). The detailed information about a live stream describes the content of the live stream in more detail than the summary of the live stream, but is a longer text than the summary. If the input time series data changes, the output detailed information also changes, so it can be said that the output detailed information is the detailed information at the time when the corresponding time series data was acquired. In addition to the time series data, static information about the live stream may be input to the detail generation model 328.

[0045] One of a number of different personalities can be set for the detailed generative model 328. On the distribution preparation screen before the start of live distribution, the distributor specifies the personality of the model to be used when generating a summary. When the distribution information providing unit 302 receives a distribution start instruction with the specified personality from the distributor's terminal via the network, it adjusts the detailed generative model 328 to the specified personality.

[0046] When the distribution information providing unit 302 receives a distribution start instruction to start live distribution from the distributor's user terminal 20 via the network NW, it registers a stream ID identifying the live distribution, a distributor ID of the distributor of the live distribution, a title included in the distribution start instruction, a specified character included in the distribution start instruction, and a distribution content tag included in the distribution start instruction in the stream DB 314. In addition, the distribution information providing unit 302 starts acquiring time-series data of the live distribution, i.e., viewers, image data, audio data, comment history, and gift history, and recording the acquired time-series data in the stream DB 314.

[0047] When the distribution information providing unit 302 receives a request for providing information about live distribution from the non-distribution communication unit 404 of the user terminal of an active user via the network NW, it references the stream DB 314 and generates a list of live distributions that can currently be viewed. The distribution information providing unit 302 transmits the generated list to the user terminal that made the request via the network NW. The non-distribution UI control unit 402 of the user terminal that made the request generates a live distribution selection screen based on the received list and displays it on the display of the user terminal.

[0048] When the non-delivery UI control unit 402 of the user terminal accepts the selection of live streaming by the active user on the live streaming selection screen, it generates a delivery request including the stream ID of the selected live streaming and transmits it to the server 10 via the network NW. The delivery information providing unit 302 acquires a summary and detailed information of the live streaming specified by the stream ID included in the received delivery request from the stream DB 314 and transmits it to the requesting user terminal via the network NW. In addition, the delivery information providing unit 302 starts providing the live streaming specified by the stream ID included in the received delivery request to the requesting user terminal. The delivery information providing unit 302 updates the stream DB 314 so that the viewer ID of the stream ID includes the user ID of the active user of the requesting user terminal. As a result, the active user becomes a viewer of the selected live streaming.

[0049] The relay unit 304 relays the transmission of video data from the user terminal 20 of the distributor to the user terminal 30 of the viewer in the live distribution started by the distribution information providing unit 302. The relay unit 304 receives a signal indicating a user input by the viewer during the live distribution, i.e., during the playback of the video data, from the viewing side communication unit 204. The signal indicating the user input may be an object designation signal indicating the designation of an object displayed on the display of the user terminal 30, and the object designation signal includes the viewer ID of the viewer, the distributor ID of the distributor performing the live distribution that the viewer is viewing, and an object ID that identifies the object. If the object is a gift icon, the object ID is a gift ID. In that case, the object designation signal is a gift use signal indicating the use of a gift by the viewer for the distributor. Similarly, the relay unit 304 receives a signal indicating a user input by the distributor during the playback of the video data, for example, an object designation signal, from the distribution side communication unit 110 of the distribution unit 100 of the user terminal 20.

[0050] The gift processing unit 308 updates the user DB 318 so as to increase the distributor's reward according to the reward for the gift specified by the gift ID included in the gift use signal. The gift processing unit 308 refers to the gift DB 320 and specifies the reward corresponding to the gift ID included in the received gift use signal. The gift processing unit 308 updates the user DB 318 so as to add the specified reward to the reward corresponding to the distributor ID included in the gift use signal.

[0051] In response to receiving the gift use signal, the payment processing unit 310 processes the payment of the value of the gift by the viewer. The payment processing unit 310 refers to the gift DB 320 and identifies the value points of the gift identified by the gift ID included in the gift use signal. The payment processing unit 310 updates the user DB 318 so as to subtract the identified value points from the points of the viewer identified by the viewer ID included in the gift use signal.

[0052] The summary generation unit 322 periodically acquires time series data and generates summaries. The summary generation unit 322 periodically acquires time series data of ongoing live distribution from the stream DB 314, for example, once every five minutes. The summary generation unit 322 generates a summary of the ongoing live distribution at the time of acquiring the time series data based on the acquired time series data. The summary generation unit 322 inputs the acquired time series data to the summary generation model 326, and acquires the summary output by the summary generation model 326 as the summary at the time of acquiring the time series data. The summary generation unit 322 updates the stream DB 314 with the acquired summary. For example, when the summary generation unit 322 acquires a new summary of a certain live distribution, it overwrites the summary that has been held up to that point in time for that live distribution in the stream DB 314 with the acquired summary.

[0053] The first time series data acquired by the summary generation unit 322 at a first time point during the progress of live streaming is different from the second time series data acquired by the summary generation unit 322 at a second time point during the progress of the live streaming, which is later than the first time point. Therefore, the summary generated based on the first time series data at the first time point and the summary generated based on the second time series data at the second time point may be different.

[0054] Fig. 7 is an explanatory diagram showing the relationship between the progress of a live broadcast and the generated summary. The top part of Fig. 7 shows the timeline of the progress of the live broadcast. In this example, the broadcaster specifies the title before the broadcast starts, so the title is fixed to the text "Tarot readings available!" regardless of the progress of the live broadcast. However, the topic of the live broadcast, which was initially fortune telling, changes to casual conversation at time t3.

[0055] The summary generation unit 322 generates a standard summary at the start of live distribution and registers it in the stream DB 314. The standard summary (also called default) is a text that does not depend on the time series data of the live distribution, but is generated based on information (static information) input by the distributor before the start of distribution, or a fixed text that does not depend on the distributor, such as "distribution has just started". After that, at a time t1 during the progress of the live distribution, the summary generation unit 322 acquires the time series data Dt1 from the stream DB 314. This time series data Dt1 represents the contents of fortune-telling, which is the topic of the live distribution up to the time t1. For example, if the distributor has performed fortune-telling for three viewers by the time t1, the time series data Dt1 includes data that arranges the fortune-telling results of the three viewers in chronological order. The summary generation unit 322 inputs the acquired time series data Dt1 into the summary generation model 326, acquires the summary output by the summary generation model 326 as a summary X1 at the time t1, and registers it in the stream DB 314.

[0056] After that, viewer M joins the live distribution at time t2, which is before the next summary update time. At this time, distribution information providing unit 302 obtains summary X1 at time t1 from stream DB 314 and transmits it to viewer M's user terminal via network NW. This allows viewer M to immediately understand that, as the title indicates, fortune telling is currently taking place in the live distribution.

[0057] After that, at time t3, the topic changes from fortune telling to chatting. After that, at time t4 while the live distribution is in progress, the summary generation unit 322 acquires time series data Dt4 from the stream DB 314. This time series data Dt4 represents the content of the live distribution up to time t4, that is, originally fortune telling, but then chatting began midway through. The summary generation unit 322 inputs the acquired time series data Dt4 to the summary generation model 326, acquires the summary output by the summary generation model 326 as summary X3 at time t4, and registers it in the stream DB 314.

[0058] After that (after time t4), viewer N joins the live distribution at time t5, which is before the next summary update time. At this time, the distribution information providing unit 302 obtains summary X3 at time t4 from the stream DB 314 and transmits it to the user terminal of viewer N via the network NW. This allows viewer N to immediately understand that, unlike the title, casual conversation is taking place immediately in the live distribution.

[0059] Returning to Fig. 3, the detail generation unit 324 generates detailed information about an ongoing live broadcast at the time of acquiring the time series data based on the time series data acquired by the summary generation unit 322. The detail generation unit 324 inputs the acquired time series data to the detail generation model 328, and acquires the detailed information output by the detail generation model 328 as the detailed information at the time of acquiring the time series data. The detail generation unit 324 updates the stream DB 314 with the acquired detailed information. For example, when the detail generation unit 324 newly acquires detailed information about a certain live broadcast, it overwrites the detailed information about the live broadcast that has been held up to that point in time in the stream DB 314 with the acquired detailed information.

[0060] When the candidate comment generating unit 330 receives a distribution request for live distribution, it acquires time-series data of the ongoing live distribution identified by the stream ID included in the distribution request from the stream DB 314. The candidate comment generating unit 330 generates a plurality of different comment candidates based on the acquired time-series data. The candidate comment generating unit 330 transmits the generated plurality of comment candidates to the requesting user terminal via the network NW.

[0061] The candidate comment generating unit 330 inputs the acquired time-series data to a comment candidate generation model and acquires a plurality of different comment candidates output by the comment candidate generation model. The comment candidate generation model is a pre-trained machine learning model that inputs time-series data of live distribution and outputs a plurality of comment candidates suitable for viewers to post at the time of acquiring the time-series data. If the input time-series data changes, the output comment candidates also change, so the contents of the plurality of comment candidates may differ depending on the timing of the distribution request.

[0062] The operation of the live distribution system 1 configured as above will now be described. 8 is a flowchart showing a series of processes for dynamic summary generation of live streaming. The streaming information providing unit 302 receives a streaming start instruction accompanied by streaming setting information including a personality specified by the streamer (S202). The summary generating unit 322 adjusts the summary generation model 326 using the personality specified in the streaming setting information received in step S202 (S204). The streaming information providing unit 302 starts providing live streaming in response to the streaming start instruction received in step S202 (S206). The streaming information providing unit 302 starts registering time-series data of the started live streaming (S208). The summary generating unit 322 updates the stream DB 314 so that a default sentence is registered in the summary of the started live streaming (S210).

[0063] The distribution information providing unit 302 determines whether a new viewer has joined the live distribution started in step S206 (S212). If a new viewer has joined (Y in S212), the distribution information providing unit 302 extracts a summary of the live distribution from the stream DB 314 and transmits it to the user terminal of the viewer who joined the live distribution in accordance with the start of the provision of the video of the live distribution (S214). After that, the process returns to step S212.

[0064] If it is determined in step S212 that the user has not participated (N in S212), the distribution information providing unit 302 determines whether or not a distribution end instruction has been received from the user terminal of the distributor (S216). If it has been received (Y in S216), the distribution information providing unit 302 ends the provision of the live distribution (S218). If it has not been received (N in S216), the summary generating unit 322 determines whether or not a predetermined period of time, for example, 5 minutes or 15 minutes, has elapsed since the previous summary update for the live distribution started in step S206 (S220). If it has not elapsed (N in S220), the process returns to step S212.

[0065] If so (Y in S220), the summary generation unit 322 acquires time-series data of the live distribution started in step S206 from the stream DB 314 (S222). The summary generation unit 322 inputs the acquired time-series data to the summary generation model 326 and acquires a summary output by the summary generation model 326 (S224). The summary generation unit 322 updates or replaces the summary of the live distribution started in step S206 in the stream DB 314 with the summary acquired in step S224 (S226). After that, the process returns to step S212.

[0066] FIG. 9 is a representative screen diagram of a live streaming selection screen 600 displayed on the display of the user terminal of an active user. The live streaming selection screen 600 includes thumbnails 602 showing each live streaming in the list of currently viewable live streaming received from the server 10. The non-streaming UI control unit 402 generates the live streaming selection screen 600 based on the list of live streaming acquired from the server 10, and displays it on the display. When the non-streaming UI control unit 402 accepts the selection of a thumbnail by the active user on the live streaming selection screen 600, it generates a streaming request including the stream ID of the live streaming corresponding to the selected thumbnail, and transmits it to the server 10 via the network NW. The streaming information providing unit 302 acquires a summary and detailed information of the live streaming specified by the stream ID included in the received streaming request from the stream DB 314, and transmits it to the user terminal that originated the request via the network NW. In addition, the streaming information providing unit 302 transmits a plurality of comment candidates generated in response to the streaming request to the user terminal that originated the request via the network NW. The non-broadcast communication unit 404 receives the summary, detailed information, and multiple comment candidates thus transmitted from the server 10 via the network NW. As described above, the summary and detailed information are information about a live broadcast that is currently viewable, i.e., currently in progress, and the contents may differ depending on the timing at which a broadcast request is generated.

[0067] 10 is a representative screen diagram of a live streaming room screen 608 displayed on the display of the viewer's user terminal. When the active user taps the thumbnail 602 on the live streaming selection screen 600 of FIG. 9, the viewer's UI control unit 202 starts displaying the live streaming room screen 608 of FIG. 10 on the display, and starts playing the video related to the live streaming on the live streaming room screen 608. The live streaming room screen 608 displays the video generated on the user terminal 20 of the broadcaster in real time. The live streaming room screen 608 has a video 610 of the broadcaster obtained by playing the video data received from the server 10, a gift object 612, a comment input area 616, a comment display area 618, a viewing end button 620, and a summary display object 622. The viewer UI control unit 202 generates a live streaming room screen 608 by superimposing other objects, namely, a gift object 612, a comment input area 616, a comment display area 618, an end viewing button 620, and a summary display object 622, on a video image 610 obtained by playing back the video data.

[0068] The comment display area 618 may include comments entered by the viewer, comments entered by other viewers, and notifications from the system. The notifications from the system may include information indicating who gave which gift to the broadcaster, and information indicating that a new viewer has joined the live broadcast. The viewer-side UI control unit 202 generates a comment display area 618 including the comments of other viewers received from the server 10 and notifications from the system, and includes the generated comment display area 618 in the live broadcast room screen 608.

[0069] The comment input area 616 accepts input of a comment by a viewer. The viewer-side communication unit 204 generates a comment input signal including the comment input in the comment input area 616, and transmits the signal to the server 10 via the network NW. Additionally, the viewer-side UI control unit 202 updates the comment display area 618 so as to display the comment input in the comment input area 616.

[0070] The viewing end button 620 is an object for receiving an instruction from the viewer to stop viewing the live broadcast. The unnecessary object 624 is an object for stopping the display of the summary display object 622. When a tap on the unnecessary object 624 is detected, the viewer-side UI control unit 202 ends the display of the summary display object 622 and the unnecessary object 624.

[0071] The viewing side UI control unit 202 of the user terminal displays a summary of the live streaming received from the server 10 on the display in synchronization with the start of playback of the video related to the live streaming. In the example of Fig. 10, the viewing side UI control unit 202 first displays a summary display object 622 on the display in synchronization with the start of playback of the video related to the live streaming, and when a designation of the summary display object 622 is accepted (for example, when a tap on the summary display object 622 is detected), a summary area 626 is generated that displays the summary received from the server 10 in response to the streaming request. The viewing side UI control unit 202 displays the generated summary area 626 on the live streaming room screen 608.

[0072] 11 is a representative screen diagram of the live streaming room screen 608 with a summary area 626 superimposed thereon, displayed on the display of the viewer's user terminal. FIG. 11 corresponds to the display of the live streaming room screen when the viewer M participates in the live streaming in FIG. 7. The summary area 626 displays a detail object 628 in conjunction with the provision of the summary. The summary area 626 includes a summary of the participated live streaming and a detail object 628. When the viewer UI control unit 202 receives a designation of the detail object 628, it generates a detail information area (not shown) that displays the detailed information received from the server 10 in response to the streaming request. The viewer UI control unit 202 causes the generated detail information area to be displayed on the live streaming room screen 608.

[0073] In another embodiment, instead of or in addition to providing a summary display area, the summary may be displayed in the comment display area 618 or may be output as audio. Alternatively, if a live streaming assistant using a machine learning model is provided on the live streaming room screen, the assistant may be made to output the summary.

[0074] Fig. 12 is a representative screen diagram of a live streaming room screen 608 displayed on the display of a viewer's user terminal, with a summary area 630 superimposed thereon. Fig. 12 corresponds to the display of the live streaming room screen when viewer N participates in the live streaming in Fig. 7. The summary area 630 displays a detail object 628 in conjunction with the provision of the summary. The summary area 630 includes a summary of the participated live streaming and the detail object 628.

[0075] The viewing side UI control unit 202 of the user terminal displays a plurality of comment candidates received from the server 10 on the display in accordance with the provision of the summary. In the example of FIG. 12, the viewing side UI control unit 202 stops displaying the summary area 630 when a predetermined period of time, for example, 10 seconds, has elapsed since the start of displaying the summary area 630. Alternatively, the viewing side UI control unit 202 stops displaying the summary area 630 when it detects a tap on an area other than the summary area 630. When the viewing side UI control unit 202 stops displaying the summary area 630, it generates a comment candidate display area 636 that displays a plurality of comment candidates received from the server 10. The viewing side UI control unit 202 displays the generated comment candidate display area 636 on the live streaming room screen 608.

[0076] FIG. 13 is a representative screen diagram of the live streaming room screen 608, in which a comment candidate display area 636 is superimposed and displayed on the display of the user terminal of the viewer. FIG. 13 corresponds to a state after the display of the summary area 630 in FIG. 12 has ended. The comment candidate display area 636 includes a first candidate object 632 that displays a first comment candidate in text, and a second candidate object 634 that displays a second comment candidate in text. When the viewer-side communication unit 204 detects a tap on the first candidate object 632, it generates a comment input signal including the first comment candidate and transmits it to the server 10 via the network NW. At the same time, the viewer-side UI control unit 202 updates the comment display area 618 to display the first comment candidate. The same process is performed when the second candidate object 634 is tapped.

[0077] Some viewers are not good at interacting with others, and even if they join a live broadcast, they may find it difficult to make the first comment. In response to this, by providing a system that allows viewers to select from appropriate comment candidates and then comment, the hurdle to making the first comment can be lowered or it can be made easier to comment, thereby revitalizing live broadcasts.

[0078] 14 is a representative screen diagram of a distribution preparation screen 670 displayed on the display of a user terminal of a distributor. The distribution preparation screen 670 is displayed before a distributor starts live distribution, and accepts distribution settings by the distributor. The distribution preparation screen 670 has a video image 672 of the distributor obtained by playing video data to be transmitted by the video transmission unit 106, a distribution setting area 674 for accepting distribution settings, and a distribution start button 676.

[0079] The distribution setting area 674 has an event setting area 682 that accepts input or selection of an event in which the distributor will participate, a tag setting area 684 that accepts input or selection of a distribution content tag that represents the content of live distribution, a personality setting area 678 that accepts input or selection of the personality of the summary generation model 326 or the detail generation model 328, and a title setting area 680 that accepts input of a title. The distributor inputs a desired setting in the distribution setting area 674 and taps the distribution start button 676. When a tap on the distribution start button 676 is detected, the distribution side communication unit 110 of the user terminal of the distributor generates distribution setting information including the event, distribution content tag, personality, and title input in the distribution setting area 674 at that time, and transmits a distribution start instruction accompanied by the generated distribution setting information to the server 10 via the network NW.

[0080] FIG. 15 is a representative screen diagram of a live distribution room screen 650 displayed on the display of the user terminal of the distributor. The live distribution room screen 650 displays a video image generated by the user terminal of the distributor in real time. The live distribution room screen 650 has a video image 652 of the distributor obtained by playing the video data to be transmitted by the video transmission unit 106, a comment display area 654, a distribution end button 656, a summary display area 658, an update object 660, and a hidden object 662. The summary display area 658, the update object 660, and the hidden object 662 are displayed in association with each other. The distribution side UI control unit 108 generates the live distribution room screen 650 by superimposing other objects, i.e., the distribution end button 656, the comment display area 654, the summary display area 658, the update object 660, and the hidden object 662, on the video image 652 obtained by playing the video data.

[0081] The distribution end button 656 is an object for receiving an instruction from the distributor to stop providing live distribution.

[0082] The summary display area 658 displays a summary of the live distribution to be displayed on the display of the user terminal of a viewer who is about to newly join the live distribution. The distribution side communication unit 110 periodically generates a summary provision request, for example, once every five minutes, and transmits it to the server 10 via the network NW. When the summary generation unit 322 receives the summary provision request from the user terminal of the distributor, it acquires a summary of the live distribution performed by the distributor from the stream DB 314. The summary generation unit 322 transmits the acquired summary to the user terminal that made the request via the network NW. The distribution side UI control unit 108 generates a summary display area 658 including the text of the received summary, and displays it on the live distribution room screen 650.

[0083] When a tap on the update object 660 is detected, the delivery-side communication unit 110 generates an update request and transmits it to the server 10 via the network NW. When the summary generation unit 322 receives the update request from the user terminal of the distributor, it acquires time-series data of the live distribution performed by the distributor and generates a summary. The summary generation unit 322 transmits the generated summary to the user terminal that issued the request via the network NW. The delivery-side UI control unit 108 updates the display of the summary display area 658 with the received summary text.

[0084] When a tap on the hidden object 662 is detected, the delivery-side UI control unit 108 stops displaying the summary display area 658, the updated object 660, and the hidden object 662.

[0085] The summary display area 658 allows the distributor to check the summary currently being presented to a viewer newly participating in his / her live distribution, and if it is inappropriate, can change it through the update object 660. Also, if the display of the summary is in the way, the display of the summary can be hidden through the hidden object 662. A rejected object may be displayed in association with the summary display area 658 on the live distribution room screen 650. When a rejected object is designated, the user terminal generates a summary display rejection signal and transmits it to the server 10. For a live distribution for which the server 10 has received a summary display rejection signal, the server 10 does not provide a summary to a newly participating viewer. This allows the distributor to prevent a summary that is not his / her intention from being provided when such a summary is generated.

[0086] In the above-mentioned embodiment, examples of the database (DB) are a hard disk and a semiconductor memory. Furthermore, based on the description in this specification, it will be understood by those skilled in the art who have read this specification that each unit can be realized by a CPU (not shown), an installed application program module, a system program module, a semiconductor memory that temporarily stores the contents of data read from a hard disk, and the like.

[0087] According to the live distribution system 1 according to the present embodiment, a summary of a live distribution is generated based on time-series data of the live distribution. As the live distribution progresses and the time-series data is updated, the content of the summary is also updated. Such a summary is provided to a viewer who joins the live distribution midway. Therefore, by looking at the summary, a viewer who joins midway can quickly grasp what is happening in the live distribution and the flow of the conversation. As a result, the viewer can smoothly enter the circle of communication of the live distribution, and the level of satisfaction is improved. In addition, since the summary more accurately represents the content of the live distribution compared to static information such as a title, the mismatch between the live distribution and the viewer can be reduced or eliminated.

[0088] In the live distribution system 1 according to this embodiment, the summary generation model 326 processes time-series data to generate summaries. This enables real-time generation and updating of summaries, which is difficult to achieve manually. Unlike VOD, live distribution requires real-time summary generation, and applying a machine learning model as in this embodiment makes it possible to achieve such real-time generation.

[0089] In the live distribution system 1 according to this embodiment, the server 10 receives information for adjusting the summary generation model 326 and the detail generation model 328 from the user terminal of the live distributor when starting live distribution. The summary generation model 326 and the detail generation model 328 are adjusted according to this information. Therefore, it becomes possible to generate summaries and detailed information in a form that is more in line with the intentions of the distributor, and each distributor can assert their own unique features.

[0090] The hardware configuration of an information processing device according to this embodiment will be described with reference to Fig. 16. Fig. 16 is a block diagram showing an example of the hardware configuration of an information processing device according to this embodiment. The illustrated information processing device 900 can realize, for example, each of the server 10 and the user terminals 20 and 30 in this embodiment.

[0091] The information processing device 900 includes a CPU 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 925, and a communication device 929. The information processing device 900 further includes an imaging device (not shown) such as a camera. The CPU 901 is an example of a hardware configuration for realizing functions realized by components described in this specification. The functions described in this specification may be realized by a circuit (circuitry) programmed to realize the described functions. The circuit (circuitry) programmed to realize the functions described in this specification includes a CPU (a Central Processing Unit), a DSP (Digital Signal Processor), a general-purpose processor, a specific application processor, an integrated circuit, ASICs (Application Specific Integrated Circuits), and / or a combination thereof. In this specification, a unit that realizes a specific function may be realized as a circuit programmed to realize the function.

[0092] The CPU 901 functions as an arithmetic processing device and a control device, and controls the overall operation or a part of the operation in the information processing device 900 according to various programs recorded in the ROM 902, the RAM 903, the storage device 919, or the removable recording medium 923. For example, the CPU 901 controls the overall operation of each functional unit included in the server 10 and the user terminals 20 and 30 in this embodiment. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change appropriately during the execution. The CPU 901, the ROM 902, and the RAM 903 are mutually connected by a host bus 907 constituted by an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.

[0093] The input device 915 may be, for example, a device operated by a user, such as a mouse, a keyboard, a touch panel, a button, a switch, or a lever, or may be a device that converts a physical quantity into an electric signal, such as a sound sensor such as a microphone, an acceleration sensor, a tilt sensor, an infrared sensor, a depth sensor, a temperature sensor, or a humidity sensor. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an external connection device 927 such as a mobile phone that supports the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user or a sensed physical quantity and outputs the input signal to the CPU 901. The user operates the input device 915 to input various data to the information processing device 900 or to instruct a processing operation.

[0094] The output device 917 is configured with a device capable of visually or audibly notifying the user of acquired information. The output device 917 can be, for example, a display such as an LCD, a PDP, or an OELD, an audio output device such as a speaker or a headphone, or a printer device. The output device 917 outputs the results obtained by the processing of the information processing device 900 as a video such as text or an image, or as a sound such as audio.

[0095] The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900. The storage device 919 is configured, for example, with a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs and various data executed by the CPU 901, and various data acquired from the outside.

[0096] The drive 921 is a reader / writer for a removable recording medium 923 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built into or externally attached to the information processing device 900. The drive 921 reads out information recorded on the attached removable recording medium 923 and outputs the information to the RAM 903. The drive 921 also writes information to the attached removable recording medium 923.

[0097] The connection port 925 is a port for directly connecting a device to the information processing device 900. The connection port 925 can be, for example, a Universal Serial Bus (USB) port, an IEEE 1394 port, a Small Computer System Interface (SCSI) port, or the like. The connection port 925 can also be an RS-232C port, an optical audio terminal, a High-Definition Multimedia Interface (HDMI) (registered trademark) port, or the like. By connecting an external connection device 927 to the connection port 925, various types of data can be exchanged between the information processing device 900 and the external connection device 927.

[0098] The communication device 929 is, for example, a communication interface configured with a communication device for connecting to a network NW. The communication device 929 can be, for example, a communication card for a wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 929 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The communication device 929 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP / IP. The communication network NW connected to the communication device 929 is a network connected by wire or wirelessly, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication. The communication device 929 realizes a function as a communication unit.

[0099] An imaging device (not shown) such as a camera is a device that captures real space and generates a captured image using an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) and various members such as a lens for controlling the formation of a subject image on the imaging element. The imaging device may capture a still image or a moving image.

[0100] The above describes the configuration and operation of the live streaming system 1 according to the embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each process, and that such modifications are also within the scope of the present disclosure.

[0101] In the embodiment, a case where a viewer participates in an ongoing live distribution and is presented with a summary of the live distribution is described, but this is not limited to this. For example, when a viewer starts viewing an archive of a live distribution, a clip (video) cut out from a part of the archive, a VoD (Video On Demand) video, a profile video, or a preview of the live distribution, the summary may be provided to the viewer.

[0102] In the embodiment, the summary generation model 326 outputs the summary in text, but the present invention is not limited to this. Instead of or in addition to text, the summary generation model 326 may generate summary information including marks, effects, audio, still images, and videos (explanatory videos, etc.). Alternatively, in the case of live distribution in an AR application or live walk commentary, the summary generation model 326 may obtain actual videos, live videos, recorded videos, and current locations using a GPS function captured by the camera of the distributor's user terminal, and generate a summary based on the obtained information. Such a summary may include, for example, text that indicates the route of the distributor's walk so far. Such text may include, for example, text to the effect that, as a result of going around such and such places, the distributor is now at such and such a place.

[0103] FIG. 17 is a representative screen diagram of a live streaming room screen 608 displayed on the display of a viewer's user terminal, with a summary area 690 according to a modified example superimposed thereon. When the summary generation model according to this modified example receives time-series data, it outputs a positive index, a negative index, a big customer rate, and a mood as summary information. The summary area 690 displays the summary information received from the server 10 at the start of viewing the live streaming. In the example of FIG. 17, the positive index (P) is 352, the negative index (N) is 120, the big customer rate is 45%, and the mood is "calm." The positive index is an index that increases when there are positive comments or comments by the broadcaster. The negative index is an index that increases when there are negative comments or comments by the broadcaster. In another modified example, the positive index and the negative index may be expressed together as one parameter. For example, the positive rate is expressed as 60% (the negative rate is expressed as 100-60=40%). The big customer rate indicates the proportion of the communication volume of high-value gifters to the total communication volume of the live streaming. For example, if the total number of comments is 1000, and the total number of comments from the high-value gifters (i.e. the viewers who have given the most total amount of gifting in the live stream, the second most amount of gifting, and the third most amount of gifting) is 500, then the big customer rate is 500 / 1000=0.5, or 50%. Mood indicates the mood of the live stream, and in addition to quiet, there are moods such as bright / gloomy / complaining / high-tension / low-tension / everyone encouraging each other.

[0104] In addition to the example of summary information shown in FIG. 17, summary information may be generated such as the number, type, and tendency of gifts used in live streaming, gifts desired by the streamer and their usage status, in the case of music streaming, set list, information on which songs have been played so far and which song is currently being played, in the case of game streaming, information on which games have been played so far, which game is currently being played, and which stage is being played, and what kind of stage it is. In the embodiment, a case where the summary is generated by a machine learning model has been described, but this is not limited to this, and in particular when the above-mentioned marks and parameters are used as summary information, the summary information may be generated using a predetermined formula or lookup table. Alternatively, the server may obtain information obtained from the external Internet regarding the live streaming and the streamer to be summarized, information on the live streaming and the streamer registered in the live streaming platform, and clip videos and archive videos of the streamer, and generate a summary based on the obtained information.

[0105] In the embodiment, the summary generation unit 322 periodically acquires time series data and generates summaries, but the present invention is not limited to this. A summary may be generated when a viewer participates in a live broadcast and provided to the viewer. Referring to the example of FIG. 7, the summary generation unit acquires time series data Dt2 from the stream DB 314 at a time t2 when the viewer M participates in the live broadcast, and generates a summary at the time t2 based on the acquired time series data Dt2. The distribution information providing unit provides the generated summary to the user terminal of the viewer M. The summary generation unit acquires time series data Dt5 from the stream DB 314 at a time t5 when the viewer N participates in the live broadcast, and generates a summary at the time t5 based on the acquired time series data Dt5. The distribution information providing unit provides the generated summary to the user terminal of the viewer N. The summary generation unit may also generate a summary according to the attributes of the viewer. In this case, the summary generation unit acquires desired topic tags of viewers who participated in the live broadcast from the user DB 318 in addition to the time series data, and generates a summary based on the acquired time series data and desired topic tags. The desired topic tags are input to the summary generation model in addition to the time series data. The summary generation model outputs different summaries even for the same time series data if the desired topic tags are different. In addition to desired topic tags, attributes such as gender, age group, region, language used, charged amount, number of years, and ban history may be used to generate summaries.

[0106] In the embodiment, a case where a comment candidate is generated in response to a distribution request has been described, but the present invention is not limited to this, and comment candidates may be generated periodically. In the embodiment, a reason why a comment candidate is generated may be generated in association with the comment candidate, and the comment candidate and the reason may be presented to the viewer in association with each other.

[0107] In the embodiment, a case has been described in which a summary display object is first displayed in conjunction with the start of viewing of a live broadcast, and the summary is displayed when the object is tapped, but this is not limited to this, and the summary may be displayed directly on the live broadcast room screen in conjunction with the start of viewing.

[0108] In the embodiment, the case where the generated summary text is displayed on the live streaming room screen has been described, but the present invention is not limited to this, and for example, the summary generation model 326 may display each character, word, or sentence generated on the live streaming room screen, thereby realizing text streaming that displays the summary as it is being generated. In this case, the summary may be gradually updated based on events that occur during the live streaming after a viewer joins the room midway.

[0109] In the embodiment, a case has been described in which a personality for adjusting the summary generation model 326 and the detailed generation model 328 is received from the user terminal of the live broadcaster when starting live broadcasting, but this is not limited to this, and other adjustment information such as a data acquisition period and a prohibited word list may be received instead of / in addition to the personality, and the model may be adjusted based on them.

[0110] In the embodiment, when a thumbnail is selected on the live streaming selection screen 600, a viewer participates in the live streaming and a summary is also displayed on the live streaming room screen 608. However, this is not limiting. For example, a summary of the live streaming may be displayed on a preview screen of the live streaming.

[0111] In this modified example, when a live broadcast is selected by a viewer, the viewer is not immediately allowed to enter the live broadcast, but is first allowed to view the live broadcast in a preview mode. In the preview mode, the viewer is allowed to view the live broadcast without notifying the distributor or other viewers of the entry, and a summary of the live broadcast is provided to the viewer. The viewer reads the summary in the preview mode, understands the content of the conversation in the live broadcast, and can enter if interested, or can exit without notifying the distributor or other viewers if not interested.

[0112] In this modified example, the user terminal waits for detection of a tap on a thumbnail displayed on the live distribution selection screen. When the tap is detected, the user terminal displays a live distribution room screen in preview mode on the display. FIG. 18 is a representative screen diagram of a live distribution room screen 708 in preview mode displayed on the display of the user terminal. The live distribution room screen 708 in preview mode displays a video generated by the user terminal 20 of the distributor in real time. The live distribution room screen 708 has a video image 710 of the distributor obtained by playing back video data received from the server 10, a comment display area 718, a summary display area 722 that displays a summary of the live distribution received from the server 10, an entry inquiry pop-up 724 for inquiring the viewer whether or not to enter the live distribution, and a preview frame 726. The preview frame 726 indicates that the live distribution is being viewed in preview mode. The preview frame 726 is an object added to distinguish between the live distribution room screen in normal mode as shown in FIG. 10 and the live distribution room screen in preview mode. It can be said that due to the presence of this preview frame 726, the live streaming room screen in normal mode and the live streaming room screen in preview mode are displayed in different formats.

[0113] In the preview mode immediately after a viewer selects a thumbnail, the server is configured not to notify the distributor or other viewers of information about the viewer. For example, the presence or absence of a viewer in preview mode does not affect information accompanying the target live distribution (such as comments and viewer lists). Alternatively, the server may manage viewers in preview mode in a manner different from other viewers. For example, viewers in preview mode may be managed using a viewer list different from the viewer list in which other viewers are registered, or a flag indicating that a viewer in preview mode is in the viewer list may be assigned to the viewer in preview mode. When the server is requested by the distributor to provide a viewer list, the server may exclude viewers with the flag set from the list to be provided.

[0114] In the preview mode, the input of information by the viewer is limited. Specifically, the live streaming room screen 708 in the preview mode does not have a gift object or a comment input area.

[0115] The viewer taps to select the desired option from among "Yes" (i.e., enter the room) or "Leave the broadcast" displayed in the entry inquiry pop-up 724. When "Leave the broadcast", i.e., leave the room, the user terminal performs the exit process. When "Yes", i.e., enter, is selected, the user terminal transitions the display from the live broadcasting room screen in preview mode to a live broadcasting room screen in normal mode (corresponding to the live broadcasting room screen in FIG. 10 excluding the summary display object 622 and unnecessary object 624).

[0116] According to this modification, by reading the summary in addition to viewing the preview, the user can understand the content of the live broadcast more quickly and accurately, and can decide whether or not to subscribe.

[0117] Instead of or in addition to the text summary displayed in the summary display area 722, marks and parameters such as those shown in FIG. 17 may be presented as summary information.

[0118] In an embodiment, the server may change the output format of the summary automatically or as specified by the distributor depending on the type of live streaming. One example is the distributor's specification of the personality. In addition, the server may specify the type of live streaming (music streaming, fortune-telling streaming, chat streaming, game streaming, etc.) from the information of the distributor registered in the user DB 318, and determine the output format of the summary depending on the specified type. For example, if the type is music streaming, the summary is set to include the set list, the name of the song currently being played, and information on which song it is.

[0119] The conversion rates from gift points to given rewards in the embodiments are merely examples, and may be set appropriately by, for example, an administrator of the live distribution system.

[0120] The technical idea according to the embodiment may be applied to virtual live streaming using an avatar that moves in sync with the streamer's movements instead of the streamer's image, or live commerce. In the embodiment, a case has been described in which a server relays video data related to live streaming generated on the streamer's user terminal and transmits it to the viewer's user terminal, but this is not limited to the above. For example, the technical idea according to the embodiment may be applied when a virtual streamer is set instead of an actual streamer. The virtual streamer is, for example, an AI virtual streamer whose appearance is an avatar, whose voice is composed of a TTS (Text-To-Speech) engine, and whose remarks are obtained from a machine learning model that uses the viewer's comments as input. In this case, the streamer's user terminal does not exist, and the streamer's processing is performed by the server.

[0121] In the processing procedures described in this specification, particularly in the processing procedures described using flow diagrams and flowcharts, it is possible to omit some of the steps that make up the processing procedures, to add steps that are not explicitly stated as steps that make up the processing procedures, and / or to rearrange the order of the steps, and processing procedures in which such omissions, additions, or changes in order have been made are included within the scope of this disclosure as long as they do not deviate from the spirit of this disclosure.

[0122] At least a part of the functions realized by the server 10 may be realized by a device other than the server 10, for example, the user terminals 20 and 30. At least a part of the functions realized by the user terminals 20 and 30 may be realized by a device other than the user terminals 20 and 30, for example, the server 10. For example, the superimposition of a predetermined frame image onto an image of video data performed at a viewer's user terminal may be performed at the server 10 or at a distributor's user terminal.

Claims

1. A server including an acquisition means and a generation means, The acquiring means acquires, at a first time point during the progress of the live streaming, first time series data representing the content of the live streaming recorded as the live streaming progresses; The generation means generates summary information at the first time point of the live distribution based on the acquired first time-series data, The acquisition means acquires second time series data representing contents of the live streaming recorded as the live streaming progresses at a second time point during the live streaming, the second time point being later than the first time point; The generation means generates summary information at the second time point of the live distribution based on the acquired second time series data, the first time point is a time point when a viewer joins the live distribution, and the second time point is a time point when another viewer joins the live distribution; The server further comprises: a server comprising a transmitting means for transmitting summary information at the first time point to the terminal of the viewer via a network, and for transmitting summary information at the second time point to the terminal of the other viewer via the network.

2. A server as described in claim 1, wherein summary information of the live stream at the first time point includes a mood of the live stream at the first time point, and summary information of the live stream at the second time point includes a mood of the live stream at the second time point.

3. The server according to claim 1 , wherein the generation means generates summary information at the first time point by inputting the first time series data into a machine learning model, and generates summary information at the second time point by inputting the second time series data into the machine learning model.

4. The server according to claim 3 , further comprising a receiving means for receiving information for adjusting the machine learning model from a terminal of a broadcaster of the live broadcast when the live broadcast is started.

5. A server as described in claim 1, wherein summary information at the first time point of the live broadcast includes a trend of gifts used in the live broadcast up to the first time point, and summary information at the second time point of the live broadcast includes a trend of gifts used in the live broadcast up to the second time point.

6. A server as described in claim 1, wherein summary information of the live stream at the first time point includes feelings of the broadcaster of the live stream at the first time point, and summary information of the live stream at the second time point includes feelings of the broadcaster of the live stream at the second time point.

7. 2. The server according to claim 1, wherein said generating means generates summary information at said first time point according to attributes of said viewer, and generates summary information at said second time point according to attributes of said other viewer.

8. a function of transmitting a request to a server via a network to start viewing an ongoing live broadcast; a function of receiving summary information of the live broadcast, the summary information being generated in response to the request, from the server via the network; A function of starting playback of the video related to the live distribution; a computer program for causing a terminal to realize a function of displaying the summary information on a display in synchronization with the start of playback of the video;

9. The function of displaying the information is a function of displaying an object on the display in synchronization with the start of playback of the video; 9. The computer program product according to claim 8, further comprising a function for displaying the summary information on the display when the designation of the object is accepted.

10. the receiving function includes a function of receiving, together with the summary information, detailed information about the live broadcast that includes more detailed information than the summary information; The function of displaying the information is displaying an object on the display in conjunction with providing the summary information; The computer program product according to claim 8 , further comprising: a function for providing the detailed information when the designation of the object is accepted.

11. a function of receiving from the server via the network a plurality of comment candidates whose contents may vary depending on the timing of occurrence of the request; The computer program product according to claim 8 , further comprising a function of displaying the plurality of comment candidates on the display in conjunction with the provision of the summary information.

12. the function of starting includes a function of starting playback of the video related to the live distribution in a first mode; the function of displaying includes a function of providing the summary information in synchronization with start of playback of the video in the first mode; The computer program further comprises: causing the terminal to realize a function of starting playback of the video in a second mode in response to reception of a predetermined user input in the first mode; 9. The computer program product according to claim 8, wherein disclosure of information about the user of the terminal is more restricted in the first mode than in the second mode.

13. A method comprising: acquiring first time series data representing content of the live streaming, the first time series data being recorded as the live streaming progresses, at a first time point during the live streaming; generating summary information for the first time point of the live broadcast based on the acquired first time-series data; acquiring second time series data representing contents of the live streaming recorded as the live streaming progresses at a second time point during the live streaming, the second time point being later than the first time point; generating summary information at the second time point of the live broadcast based on the acquired second time series data; the first time point is a time point when a viewer joins the live distribution, and the second time point is a time point when another viewer joins the live distribution; The method further comprises: The method includes transmitting summary information at the first time point over a network to a terminal of the viewer, and transmitting summary information at the second time point over a network to a terminal of the other viewer.

Images