Server and computer programs

Abstract

It enables more efficient and flexible editing of live streams. [Solution] The server comprises means for generating multiple different video data, each being a part of the original video data; means for obtaining information from the user's terminal via the network indicating at least one video data selected by the user from among the multiple video data; means for obtaining editing instructions from the terminal via the network; means for obtaining edited video data output by a machine learning model that has the information and editing instructions as inputs; and means for providing the edited video data to the terminal via the network.

Description

Technical Field

[0001] This disclosure relates to a server and a computer program.

Background Art

[0002] With the development of IT technology, the way of information exchange has changed. In the Showa era, one-way information transmission such as newspapers and TV was dominant. In the Heisei era, mobile phones and personal computers became popular, and the communication speed of the Internet was greatly improved. As a result, real-time two-way communication services such as chat services emerged, and on-demand video distribution services were accepted with the reduction of memory costs. And now, in the Reiwa era, with the high functionality of smartphones and the further improvement of network speed represented by 5G, services that realize real-time communication by video, especially live streaming services, are rapidly increasing in popularity. Live streaming services are expanding their users, especially among young people, as services that allow everyone to share the same enjoyable time even when they are in different places.

[0003] Patent Document 1 discloses a video editing technique suitable for the archive of live commerce.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Non-Patent Document 2

[0006] With the increasing demand and diversification of video content, the importance of efficient and flexible editing processes is growing. In particular, selecting the right scenes from long videos and live streams and creating compelling edits is a time-consuming and laborious task. Existing systems fail to adequately address at least one of the following: 1. Efficient scene extraction from large amounts of material. 2. Editorial support that takes into account the context and intent of the content. 3. Real-time editing support for live content 4. Editing should be done with viewer interest and reactions in mind. 5. Dynamic reflection of the editor's intentions and preferences

[0007] This disclosure was made in light of these challenges, and its purpose is to provide technology that enables more efficient and flexible editing of live streams. [Means for solving the problem]

[0008] One aspect of the present invention relates to a server. This server includes means for generating a plurality of different video data, each being a part of an original video data; means for obtaining information from a user's terminal via a network indicating at least one video data selected by the user from among the plurality of video data; means for obtaining editing instructions from the user via a network from the terminal; means for obtaining edited video data output by a machine learning model that has received the information and editing instructions as inputs; and means for providing the edited video data to the terminal via a network.

[0009] Another aspect of the present invention is a computer program. This computer program provides a live streamer's terminal with the following functions: a function to display an object associated with the live stream video on the terminal's display during the live stream; a function to accept object designations from the streamer during the live stream; and a function to transmit the timing of the object designation in the live stream to a server via the network.

[0010] Furthermore, any combination of the above components, or any substitution of components or expressions of the present invention between devices, methods, systems, computer programs, recording media storing computer programs, etc., are also valid embodiments of the present invention. [Effects of the Invention]

[0011] According to the present invention, more efficient and flexible editing of live streams becomes possible. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic diagram showing the configuration of the live streaming system according to this embodiment. [Figure 2] Figure 1 is a block diagram showing the functions and configuration of the user terminal. [Figure 3] This block diagram shows the functions and configuration of the server in Figure 1. [Figure 4] Figure 3 is a data structure diagram showing an example of a stream database. [Figure 5] Figure 3 is a data structure diagram showing an example of a user database. [Figure 6] Figure 3 is a data structure diagram showing an example of a gift database. [Figure 7] Figure 3 is a data structure diagram showing an example of an archive database. [Figure 8] Figure 3 is a data structure diagram showing an example of a clip database. [Figure 9] Figure 3 is a data structure diagram showing an example of an edited video database. [Figure 10]It is a flowchart showing a series of processing flows on an editor's user terminal during editing. [Figure 11] It is a flowchart showing a series of processing flows on a server during editing. [Figure 12] It is a representative screen diagram of an editing screen displayed on the display of an editor's user terminal. [Figure 13] It is a representative screen diagram of a live distribution room screen displayed on the display of a distributor's user terminal during live distribution. [Figure 14] It is a representative screen diagram of a live distribution room screen displayed on the display of a distributor's user terminal during live distribution. [Figure 15] It is a representative screen diagram of an archive viewing screen displayed on the display of an active user's user terminal. [Figure 16] It is a block diagram showing an example of the hardware configuration of an information processing apparatus according to the present embodiment.

Mode for Carrying Out the Invention

[0013] Hereinafter, the same or equivalent components, members, processes, and signals shown in each drawing are denoted by the same reference numerals, and redundant explanations are omitted as appropriate. In addition, some members that are not important for explanation in each drawing are omitted from the display.

[0014] Embodiments according to the present disclosure relate to video analysis, machine learning, content editing support, media production workflow optimization, and real-time distribution technology. Embodiments according to the present disclosure aim to solve at least one of the following problems: ​​​​​​​​5. Integrate multiple video sources to create consistent edited content. 6. Significantly improve the efficiency of editing work and shorten production time. 7. To expand editors' creativity and facilitate collaboration with AI. 8. Provide a mechanism that flexibly reflects user intent and priorities.

[0015] In the live streaming system according to this embodiment, the system automatically generates clips from the live streaming archive. A portion of the live streaming archive or the video data relating to that portion is referred to as a clip of the live streaming. If multiple clips are generated from the archive, the video data of the archive becomes the "original" video data for each clip. The live streaming system presents information about the generated clips to the editor, and the editor inputs the desired clips and editing instructions indicating the editing policy into the system. Based on the specified clips and editing instructions, the live streaming system generates an edited video using an editing machine learning model (hereinafter referred to as the editing ML model).

[0016] This enables us to provide innovative solutions that balance creativity and efficiency in the rapidly changing video production and distribution market. By significantly reducing the burden of traditional editing tasks and optimally combining human creativity with AI processing power, we can support the creation of high-quality content. In particular, real-time processing capabilities and prompt-based flexible control functions open up new possibilities in the field of live streaming editing, enhancing interactivity with editors.

[0017] Figure 1 is a schematic diagram showing the configuration of a live streaming system 1 according to this embodiment. The live streaming system 1 provides a two-way live streaming service that allows broadcasters (also called streamers) LV and viewers (also called audiences) AU (AU1, AU2, ...) to communicate in real time. As shown in Figure 1, the live streaming system 1 comprises a server 10, a user terminal 20 on the broadcaster's side, and user terminals 30 (30a, 30b, ...) on the viewer's side. In addition to broadcasters who are broadcasting live streams and viewers who are watching live streams, there are also users who have logged into the live streaming platform but are neither broadcasting nor watching. Such users are called active users. Broadcasters, viewers, and active users are sometimes collectively referred to as users. The server 10 may be composed of one or more information processing devices connected to a network NW. User terminals 20 and 30 may be mobile terminals such as smartphones, tablet terminals, laptop PCs, recorders, portable game consoles, or wearable devices, or they may be stationary devices such as desktop PCs. Server 10, user terminal 20, and user terminal 30 are connected to each other via various wired or wireless networks (NW) to enable communication.

[0018] The live streaming system 1 involves a streamer (LV), a viewer (AU), and an administrator (not shown) who manages the server 10. A streamer (LV) is a person who broadcasts content in real time by recording or videotaping their own songs, talks, performances, fortune-telling, game commentary, etc., on their user terminal 20 and uploading it directly to the server 10. The administrator provides a platform for live streaming of content on the server 10 and mediates or manages real-time interactions between the streamer (LV) and the viewer (AU). Viewer (AU) accesses the platform on their user terminal 30 to select and watch desired content. During the live streaming of this content, viewer (AU) can perform actions such as commenting, showing support, or requesting fortune-telling via their user terminal 30. The streamer (LV) providing the content responds to such comments, support, and requests, and this response is transmitted to the viewer (AU) via video and / or audio, thus establishing two-way communication.

[0019] In this specification, "live streaming" may mean a data transmission method that enables content recorded on the broadcaster LV's user terminal 20 to be played back and viewed on the viewer AU's user terminal 30 in substantially real time, or it may mean the streaming itself realized by such a transmission method. Live streaming may be realized using existing live streaming technologies such as HTTP Live Streaming, Common Media Application Format, Web Real-Time Communications, Real-Time Messaging Protocol, or MPEG DASH. Live streaming includes a transmission method that enables the viewer AU to view the content with a predetermined delay while the broadcaster LV is recording the content. Regarding the magnitude of the delay, at least a delay large enough for communication between the broadcaster LV and the viewer AU to take place is permitted. However, live streaming is distinguished from so-called on-demand streaming, where the entire data of the recorded content is temporarily stored on a server, and then the data is provided to the user from the server at any time requested by the user.

[0020] In this specification, "video data" refers to data that includes image data (also called video data) generated by the imaging function of user terminals 20 and 30, and audio data (also called audio data) generated by the audio input function of user terminals 20 and 30. The video data is played back on user terminals 20 and 30, enabling users to view the content. In this embodiment, it is assumed that between the time the video data is generated on the distributor's user terminal and the time it is played back on the viewer's user terminal, processing is performed to change the format, size, and specifications of the data, such as compression, decompression, encoding, decoding, and transcoding. Since the content represented by the video data (e.g., moving images and sound) does not substantially change before and after such processing, in this embodiment, the video data after such processing is described as being the same as the video data before such processing. That is, when video data is generated on the distributor's user terminal and then played back on the viewer's user terminal via the server 10, the video data generated on the distributor's user terminal, the video data passing through the server 10, and the video data received and played back on the viewer's user terminal are all the same video data.

[0021] In the example shown in Figure 1, the broadcaster LV is live-streaming a talk. The broadcaster LV's user terminal 20 generates video data by recording the image and sound of the broadcaster LV during the talk, and sends it to the server 10 via the network NW. In addition, the user terminal 20 displays the recorded video VD of the broadcaster LV on its display, allowing the broadcaster LV to confirm the content of the broadcast.

[0022] The user terminals 30a and 30b of viewers AU1 and AU2, who requested the platform to view the live stream of broadcaster LV, respectively receive video data related to the live stream via the network NW. By playing the received video data, they display video images VD1 and VD2 on their displays and output sound from their speakers. The video images VD1 and VD2 displayed on each user terminal 30a and 30b are substantially identical to the video image VD captured by broadcaster LV's user terminal 20, and the sound output by each user terminal 30a and 30b is substantially identical to the sound recorded by broadcaster LV's user terminal 20.

[0023] The recording and video recording on the broadcaster LV's user terminal 20 and the playback of video data on the viewer AU1 and AU2's user terminals 30a and 30b are performed substantially simultaneously. When one viewer AU1 inputs a comment on the content of the broadcaster LV's talk into user terminal 30a, the server 10 displays the comment in real time on the broadcaster LV's user terminal 20, as well as on the user terminals 30a and 30b of each viewer AU1 and AU2. When the broadcaster LV reads the comment and adds a comment over it, the video and audio of that conversation are output to the user terminals 30a and 30b of each viewer AU1 and AU2, and it is recognized that a conversation has taken place between the broadcaster LV and viewer AU1. In this way, the live streaming system 1 realizes live streaming that enables two-way communication rather than one-way communication.

[0024] Server 10 includes a support integration system. The support integration system assists broadcasters in editing archives of their live streams. In particular, the system provides various functions to the broadcaster's user terminal to facilitate editing tasks such as combining and modifying clips generated from the archive.

[0025] Figure 2 is a block diagram showing the functions and configuration of user terminal 20 in Figure 1. User terminal 30 has the same functions and configuration as user terminal 20. Each block shown in Figure 2 and subsequent block diagrams can be realized in hardware terms by elements and mechanical devices such as the CPU of a computer, and in software terms by computer programs, etc., but here we are depicting functional blocks that are realized through the cooperation of these. Therefore, it will be understood by those skilled in the art who have read this specification that these functional blocks can be realized in various ways by combinations of hardware and software.

[0026] The broadcaster (LV) and viewer (AU) download and install the live streaming application program according to this embodiment (hereinafter referred to as the live streaming app) from a download site via the network (NW) to the user terminals 20 and 30. Alternatively, the live streaming app may be pre-installed on the user terminals 20 and 30. When the live streaming app is executed by the user terminals 20 and 30, the user terminals 20 and 30 communicate with the server 10 via the network (NW) and realize various functions. Hereinafter, the functions realized by the execution of the live streaming app by the user terminals 20 and 30 (their CPUs or other processors) will be described as functions of the user terminals 20 and 30. These functions are actually functions that the live streaming app enables on the user terminals 20 and 30. In other embodiments, these functions may be realized by a computer program written in a programming language such as HTML (HyperText Markup Language), which is transmitted from the server 10 to the web browser of the user terminals 20 and 30 via the network (NW) and executed by that web browser.

[0027] The user terminal 20 includes a distribution unit 100 that generates video data recording the user's image and voice and provides it to the server 10, a viewing unit 200 that retrieves and plays the video data from the server 10, and an off-distribution processing unit 400 that processes requests from active users. Users activate the distribution unit 100 when distributing, the viewing unit 200 when viewing, and the off-distribution processing unit 400 when searching for a live stream to watch, viewing a streamer's profile, or viewing an archive. A user terminal with the distribution unit 100 active is the streamer's terminal, i.e., the user terminal that generates the video data; a user terminal with the viewing unit 200 active is the viewer's terminal, i.e., the user terminal that plays the video data; and a user terminal with the off-distribution processing unit 400 active is the user terminal of an active user.

[0028] The distribution unit 100 includes an image capture control unit 102, an audio control unit 104, a video transmission unit 106, a distribution-side UI control unit 108, and a distribution-side communication unit 110. The image capture control unit 102 is connected to a camera (not shown in Figure 2) and controls image capture by the camera. The image capture control unit 102 acquires image data from the camera. The audio control unit 104 is connected to a microphone (not shown in Figure 2) and controls audio input from the microphone. The audio control unit 104 acquires audio data from the microphone. The video transmission unit 106 transmits video data, including image data acquired by the image capture control unit 102 and audio data acquired by the audio control unit 104, to the server 10 via the network NW. The transmission of video data by the video transmission unit 106 is performed in real time. That is, the generation of video data by the image capture control unit 102 and the audio control unit 104, and the transmission of the generated video data by the video transmission unit 106, are performed substantially simultaneously.

[0029] The broadcast-side UI control unit 108 controls the UI for the broadcaster. The broadcast-side UI control unit 108 is connected to a display (not shown in Figure 2) and displays the video on the display by playing the video data that is to be transmitted by the video transmission unit 106. The broadcast-side UI control unit 108 is connected to input means such as a touch panel, keyboard, and display (not shown in Figure 2) and acquires input from the broadcaster through these input means. The broadcast-side UI control unit 108 superimposes a predetermined frame image onto the video. The frame image includes various user interface objects (hereinafter simply referred to as objects) for receiving input from the broadcaster, comments entered by viewers, and information acquired from the server 10. For example, the broadcast-side UI control unit 108 accepts tap input on objects by the broadcaster.

[0030] The broadcasting communication unit 110 controls communication with the server 10 during live streaming. The broadcasting communication unit 110 transmits the content of the input from the broadcaster, acquired by the broadcasting UI control unit 108, to the server 10 via the network NW. The broadcasting communication unit 110 receives various information associated with the live streaming from the server 10 via the network NW.

[0031] The viewing unit 200 includes a viewing-side UI control unit 202 and a viewing-side communication unit 204. The viewing-side communication unit 204 controls communication with the server 10 during live streaming. The viewing-side communication unit 204 receives video data related to live streaming involving broadcasters and viewers from the server 10 via the network NW.

[0032] The viewer-side UI control unit 202 controls the UI for the viewer. The viewer-side UI control unit 202 is connected to a display and speaker (not shown in Figure 2) and plays the received video data to display the video on the display and output the sound from the speaker. The simultaneous output of an image on the display and sound from the speaker can be said to be "playback of video data". The viewer-side UI control unit 202 is connected to input means such as a touch panel, keyboard, and display (not shown in Figure 2) and acquires input from the viewer through these input means. The viewer-side UI control unit 202 superimposes a predetermined frame image onto the video data image acquired from the server 10. The frame image includes various objects for receiving input from the viewer, comments entered by the viewer, and information acquired from the server 10. The viewer-side communication unit 204 transmits the content of the viewer input acquired by the viewer-side UI control unit 202 to the server 10 via the network NW.

[0033] The non-distribution processing unit 400 includes a non-distribution UI control unit 402 and a non-distribution communication unit 404. The non-distribution UI control unit 402 controls the UI for active users. For example, the non-distribution UI control unit 402 generates a live stream selection screen that displays a list of currently available live streams and accepts the selection of a live stream by an active user, and displays it on the display. The non-distribution UI control unit 402 generates a profile screen for any user and displays it on the display. On the profile screen, the non-distribution UI control unit 402 displays a list of currently viewable archives and accepts the selection of an archive by an active user. The non-distribution UI control unit 402 plays the selected archive. The non-distribution UI control unit 402 generates an editing screen that enables the broadcaster to generate an edited video from clips of their own broadcast, and displays it on the display.

[0034] The non-distribution communication unit 404 controls communication with the server 10 outside of live streaming. The non-distribution communication unit 404 receives information from the server 10 via the network NW for generating the live streaming selection screen, information for generating the profile screen, and archive data. The non-distribution communication unit 404 transmits the content of input by the active user to the server 10 via the network NW. During editing, the non-distribution communication unit 404 receives clip information from the server 10 and transmits information identifying the clip selected by the editor and editing instructions to the server 10.

[0035] Figure 3 is a block diagram showing the functions and configuration of server 10 in Figure 1. Server 10 comprises a distribution information provision unit 302, a relay unit 304, a gift processing unit 308, a payment processing unit 310, a support integration system 322, a stream DB 314, a user DB 318, and a gift DB 320.

[0036] The support and integration system 322 assists editors in editing archive clips of live streams. It analyzes live streams in real time and assists broadcasters in marking important scenes. The system generates clips based on viewer reactions (comments, likes, viewing time, etc.). It integrates multiple clips generated from different archives to automatically configure an optimal viewing experience. The system immediately generates edited videos (also known as highlight videos) after the stream ends and automatically posts them to the platform. The system creates personalized digest versions tailored to the audience. The system automatically detects inappropriate content and warns editors. It automatically detects technical problems during live streams (such as poor audio or video glitches) and suggests corrections during the clip editing stage. The system integrates simultaneous streams from multiple broadcasters during a live event, providing a unified editing view. The support integration system 322 has a generation AI model (editing ML model) for removing background music and suggesting and embedding new background music. The support integration system 322 has a prompt-based importance specification function. For example, editors can pre-set instructions such as "emphasize exciting scenes" or "prioritize educational content." Even during broadcasting, real-time instructions such as "emphasize product introduction for the next 5 minutes" are possible. The extraction ML model interprets these instructions and reflects them in scene extraction and highlight generation. The support integration system 322 implements a broadcaster interaction flag system. The support integration system 322 provides broadcasters with flag buttons that can be easily operated on the broadcasting screen (e.g., "exciting moments," "important points," "funny comments," etc.). The support integration system 322 automatically records the moment a flag is assigned as a timestamp and automatically generates live broadcast clips based on the type and frequency of the flag. The support integration system 322 performs flag-linked recommendation optimization. The support integration system 322 prioritizes cutting out and creating clips of sections that the streamer has flagged as "exciting."The support integration system 322 processes the sections marked with the "important points" flag to give them importance when creating educational digest videos. Based on the "interesting comments" flag, the support integration system 322 generates viewer-participation highlight videos.

[0037] These functions of the support integration system 322 allow broadcasters to directly reflect their own feelings and judgments in the editing process, enabling editing that takes into account subtle nuances and contexts that AI alone cannot capture, and making it possible to create highlight videos that retain the sense of realism and unique atmosphere of live broadcasts.

[0038] The support integration system 322 includes an archive generation unit 324, a clip generation unit 326, an edit content acquisition unit 328, an editing processing unit 330, an edited video provision unit 332, an archive DB 334, a clip DB 336, and an edited video DB 338.

[0039] Figure 4 is a data structure diagram showing an example of the Stream DB314 in Figure 3. The Stream DB314 holds information about live streaming that is currently taking place. The Stream DB314 stores the following in association with the Stream ID, which identifies the live streaming on the live streaming platform provided by the Live Streaming System 1; the Streamer ID, which is the user ID that identifies the streamer of the live streaming; the Viewer ID, which is the user ID that identifies the viewers of the live streaming; and the score of the live streaming.

[0040] In the live streaming platform provided by the live streaming system 1 according to this embodiment, when a user conducts a live stream, that user becomes a streamer, and when the same user watches a live stream conducted by another user, that user becomes a viewer. Therefore, the distinction between streamer and viewer is not fixed, and a user ID that was registered as a streamer ID at one time may be registered as a viewer ID at another time.

[0041] The score is an indicator of how popular a live stream is. Live streams with high scores are perceived as "popular" or "exciting." The score fluctuates based on factors such as the number of viewers, streaming time, number of comments, number of shares, number of gifts received, number of viewers who sent gifts, and number of cheers.

[0042] Figure 5 is a data structure diagram showing an example of the user DB318 in Figure 3. The user DB318 holds information about users. The user DB318 includes a user ID that identifies the user, the points the user has, the rewards awarded to the user, and the user's level.

[0043] Points are electronic value that circulates within the live streaming platform. Users purchase points using credit cards or other payment methods. Rewards are electronic value defined within the live streaming platform and are an indicator used to determine the amount of money that streamers receive from the platform's administrators. On the live streaming platform, when viewers send gifts to streamers during or outside of live streams, the viewers' points are consumed, and the streamer's rewards increase accordingly.

[0044] The level is an indicator of a user's performance as a broadcaster on a live streaming platform. In other embodiments, the level may be an indicator of a user's performance as a viewer on a live streaming platform, or an indicator of a user's performance as both a broadcaster and a viewer. The level may increase or decrease based on the number of live streams, the duration of live streams, the total viewing time of live streams, the total viewing time as a viewer of live streams, the number and / or amount of gifts sent, the number and / or amount of gifts received, the number of comments, etc. Alternatively, the level may be evaluated and determined by an administrator based on reviews of the broadcaster, user satisfaction, and comments within the stream. Alternatively, the level may be automatically determined by predetermined rules or an ML model for level determination.

[0045] Figure 6 is a data structure diagram showing an example of the gift DB320 in Figure 3. The gift DB320 holds information about gifts that viewers can use during a live stream. A gift is electronic data or a digital item with the following characteristics: • Can be purchased with points or money, or given away for free. • Something that viewers can send to streamers. Sending a gift to a streamer is also called using a gift or throwing a gift. Some gifts require both purchase and use to occur simultaneously, while others allow viewers to use them at their own discretion after purchase. When viewers send gifts to streamers, the streamers receive appropriate rewards. • When a gift is used, effects associated with the gift may occur. For example, an effect corresponding to the gift may appear on the live streaming room screen.

[0046] Gift DB320 stores a gift ID that identifies a gift, a reward given to the streamer when the gift is sent to the streamer, and consideration points that must be paid when using the gift, in an associated manner. Viewers can send a gift to the streamer by paying the consideration points for the desired gift while watching a live stream. This payment of consideration points may be made by an appropriate electronic payment method, for example, by the viewer paying the consideration points to the administrator. Alternatively, payment may be made by bank transfer or credit card. The relationship between the reward and consideration points can be arbitrarily set by the administrator. For example, the reward may be set to equal the consideration points. Alternatively, the consideration points may be set to the points obtained by multiplying the reward by a predetermined coefficient such as 1.2, or the consideration points may be set to the points obtained by adding a predetermined fee point to the reward.

[0047] Figure 7 is a data structure diagram showing an example of the archive DB334 in Figure 3. The archive DB334 holds data related to archives of live streams that have been or are being performed on the live streaming platform. The archive DB334 stores the following in association: an archive ID that identifies the archive, the broadcaster ID of the broadcaster of the live stream from which the archive originated, the stream ID of the live stream from which the archive originated, the broadcast date and time of the live stream from which the archive originated, the video data of the archive, flag data, comment data, gift data, and viewer count data. In this embodiment, the archive of a live stream is generated simultaneously with the progress of the live stream. Therefore, the archive DB334 also holds archives of live streams that are currently in progress. In the example in Figure 7, the archive "ARC02" is an archive of the live stream "ST92" that is currently in progress, and therefore the end time of the broadcast date and time is "Broadcasting".

[0048] Flag data is data relating to flags assigned by the live streamer during the live stream. In this embodiment, the live streamer can assign flags at any desired timing during the live stream. The type of flag assigned and the timing of the assignment are stored as flag data in the archive DB334. For each flag type or flag ID, the flag data includes the time when the flag was assigned by the streamer. This time is expressed as a time with the start of the archive as zero. In the example in Figure 7, for the archive "ARC01", the flag data records that the streamer "GHK" assigned the flag "FLGA" at times "0:05", "0:13", and "0:44" of the corresponding live stream "ST80", the flag "FLGB" at time "0:33", and the flag "FLGC" at times "0:06", "0:10", and "0:11".

[0049] The comment data stores information about comments entered by participants (broadcasters and viewers) during a live stream. The comment data associates the time a comment was posted with the user ID of the user who posted the comment.

[0050] Gift data stores information about gifts used during live streams. Gift data records which gifts were used, by whom, and when. Gift data associates the time a gift was used with the user ID of the user who used the gift and the gift ID of the gift itself.

[0051] Viewer count data is data that records the number of viewers of a live stream at predetermined time intervals. The viewer count data is stored by associating the time the viewer count was obtained with the number of viewers obtained.

[0052] Figure 8 is a data structure diagram showing an example of the clip DB336 in Figure 3. The clip DB336 holds information about clips generated from archives. The clip DB336 stores the following information in association with each other: a clip ID that identifies the clip, the video data of the clip, an archive ID that identifies the archive from which the clip originated, a stream ID that identifies the live stream corresponding to the archive, the broadcaster ID of the broadcaster of the live stream, the start and end times of the clip in the archive, the tags assigned to the clip, and the reason why the clip was generated. The clip ID may also be a URL.

[0053] The video data of a clip includes video data relating to a portion of the original archive or live stream. This video data may also include video data generated on the broadcaster's user device and data of objects that are superimposed on the video, such as gift effects and comments.

[0054] In the example in Figure 7, clip "CL1" is a clip extracted from the archive "ARC01" of the live stream "ST80" by broadcaster "GHK". It is a 7-second video obtained by extracting the portion between 0:05 (5 seconds from the start) and 0:12 (12 seconds from the start) of archive "ARC01". Clip "CL1" is tagged with "Highlight" and "Action", and it is recorded that this clip was generated because the viewer reaction was positive.

[0055] Figure 9 is a data structure diagram showing an example of the edited video DB 338 in Figure 3. The edited video DB 338 holds data for edited videos generated by the support integration system 322. The edited video DB 338 stores, in association with the following: an edited video ID that identifies the edited video, a creator ID which is the user ID of the user who created the edited video, the video data of the edited video, the original clip ID which is the clip ID of at least one clip from which the edited video was based, the original video ID which is the edited video ID of the edited video from which the variation originated if the edited video is a variation, and the target attributes of the variation if the edited video is a variation.

[0056] In this embodiment, the support integration system 322 receives the clip ID of the clip to be edited and the editing policy from the editor. The support integration system 322 inputs the clip to be edited and the editing policy into the editing ML model and obtains the edited video output from this editing ML model. For this edited video, the clip ID of the clip input into the editing ML model becomes the original clip ID. The editing ML model is also configured to output different versions of the edited video for each target attribute. In the example in Figure 9, the editing ML model generates variations "EV02" for target attribute "levels 0-10", variations "EV03" for target attribute "men in their 20s", and variations "EV04" for target attribute "likes small talk" from the original edited video "EV01".

[0057] Returning to Figure 3, when the distribution information provision unit 302 receives notification from the broadcaster's user terminal 20 via the network NW that a live broadcast will be started, it registers the stream ID that identifies the live broadcast and the broadcaster ID of the broadcaster of the live broadcast in the stream DB 314. When the distribution information provision unit 302 receives a request for information about live broadcasts from the non-distribution communication unit 404 of an active user's terminal via the network NW, it refers to the stream DB 314 and generates a list of live broadcasts that are currently viewable. The distribution information provision unit 302 sends the generated list to the requesting user terminal via the network NW. The non-distribution UI control unit 402 of the requesting user terminal generates a live broadcast selection screen based on the received list and displays it on the user terminal's display.

[0058] When the user terminal's non-distribution UI control unit 402 receives a live stream selection from an active user on the live stream selection screen, it generates a distribution request containing the stream ID of the selected live stream and sends it to the server 10 via the network NW. The distribution information provision unit 302 begins providing the live stream, identified by the stream ID included in the received distribution request, to the requesting user terminal. The distribution information provision unit 302 updates the stream DB 314 so that the viewer ID of the stream ID includes the user ID of the active user on the requesting user terminal. As a result, the active user becomes a viewer of the selected live stream.

[0059] The relay unit 304 relays the transmission of video data from the broadcaster's user terminal 20 to the viewer's user terminal 30 in a live broadcast initiated by the distribution information provision unit 302. The relay unit 304 receives signals from the viewer-side communication unit 204 indicating user input by the viewer during the live broadcast, i.e., during the playback of video data. The signals indicating user input may also be object specification signals indicating the specification of an object displayed on the user terminal 30's display, and such object specification signals include the viewer's viewer ID, the broadcaster ID of the broadcaster conducting the live broadcast that the viewer is watching, and an object ID that identifies the object. If the object is a gift icon, the object ID is the gift ID. In that case, the object specification signal becomes a gift usage signal indicating the viewer's use of a gift to the broadcaster. Similarly, the relay unit 304 receives signals indicating user input by the broadcaster during the playback of video data, such as object specification signals, from the broadcaster-side communication unit 110 of the distribution unit 100 of the user terminal 20. If the object is a flag button, the object ID is the flag ID. In that case, the object designation signal becomes a flagging signal indicating that the broadcaster has assigned a flag.

[0060] The gift processing unit 308 updates the user database 318 to increase the distributor's reward according to the reward granted for the gift identified by the gift ID included in the gift usage signal. The gift processing unit 308 refers to the gift database 320 and identifies the reward granted corresponding to the gift ID included in the received gift usage signal. The gift processing unit 308 updates the user database 318 to add the identified reward granted to the reward corresponding to the distributor ID included in the gift usage signal.

[0061] The payment processing unit 310 processes the payment of the gift by the viewer in response to the receipt of the gift usage signal. The payment processing unit 310 refers to the gift DB 320 and identifies the gift value points identified by the gift ID included in the gift usage signal. The payment processing unit 310 updates the user DB 318 to subtract the identified gift value points from the viewer's points identified by the viewer ID included in the gift usage signal.

[0062] The archive generation unit 324 generates an archive of the live stream in parallel with the live stream in progress and registers it in the archive DB 334. When the archive generation unit 324 receives notification from the streamer's user terminal 20 that a live stream will be started, it begins recording the video data of the live stream provided by the user terminal 20. The archive generation unit 334 registers the archive ID, streamer ID, stream ID, and streaming date and time in association with the recorded video data in the archive DB 334.

[0063] When the archive generation unit 324 detects a comment being posted during a live stream, it registers the information of the posted comment in the comment data of the archive DB 334. When a comment is entered on a user terminal of a participant in a live stream, the user terminal generates a comment input signal that includes the stream ID of the live stream, the participant's user ID, and the entered comment, and sends it to the server 10 via the network NW. When the archive generation unit 324 receives the comment input signal, it registers the user ID included in the signal, the comment included in the signal, and the time the signal was received in the comment data corresponding to the archive ID corresponding to the stream ID included in the signal.

[0064] When the archive generation unit 324 detects the use of a gift during a live stream, it registers information about the used gift in the gift data of the archive DB 334. When the archive generation unit 324 receives a gift usage signal, it registers the gift ID included in the gift usage signal, the time the gift usage signal was received, and the viewer ID included in the gift usage signal in the gift data corresponding to the archive of the live stream in which the gift was used.

[0065] The archive generation unit 324 measures the number of viewers at predetermined time intervals during live streaming and registers the measurement results in the viewer count data of the archive DB 334.

[0066] When the archive generation unit 324 receives a flag assignment signal from the broadcaster's user terminal, it registers the flag ID included in the received flag assignment signal and the time the flag was assigned in the flag data corresponding to the live stream archive that has been flagged.

[0067] The clip generation unit 326 generates multiple different clips, each of which is part of the live stream archive. The clip generation unit 326 generates multiple clips based on the reactions of viewers of the live stream corresponding to the archive and the actions of the live stream broadcaster. The actions of the live stream broadcaster include the broadcaster assigning a flag at a desired timing during the live stream.

[0068] The clip generation unit 326 generates clips from the live stream archives held in the archive DB 334 and registers them in the clip DB 336. The clip generation unit 326 generates clips automatically, that is, regardless of whether there are instructions from the broadcaster or viewers. When the clip generation unit 326 detects the end of a live stream, it may start generating clips from the archive of the finished live stream. Alternatively, the clip generation unit 326 may generate clips from the archive of the live stream up to that point in parallel with the progress of the live stream.

[0069] The clip generation unit 326 acquires video data, flag data, comment data, gift data, and viewer count data from the archive DB 334, and generates multiple clips by processing the acquired data with a predetermined clip generation algorithm. The clip generation algorithm may be an algorithm that determines the range to be clipped within the archive based on at least one of the following: score, flag, comment, gift, and viewer count. The clip generation algorithm may be configured to allow users or administrators to set the above factors and the relative weights between the above factors. The clip generation algorithm may be implemented using a trained extraction ML model or it may be implemented using a rule-based approach. In this example, the clip generation algorithm is configured to identify and determine the range of parts of the archive that received relatively high viewer ratings, parts that received a relatively large number of gifts, and parts that had a relatively high rate of score increase.

[0070] Clip DB336 may determine the corresponding tags by analyzing the video data of the acquired archive and register the determined tags in Clip DB336. A tag determination ML model may be used for this tag determination.

[0071] The editing content acquisition unit 328 acquires the content of the edits that the editor wants to make to multiple clips from the editor's user terminal via the network NW. When the editing content acquisition unit 328 receives an editing start request including the editor's user ID from the editor's user terminal, it extracts clips from the live stream where the editor is the broadcaster, and the information associated with those clips, from the clip DB 336 and sends them to the requesting user terminal. The multiple clips extracted here may be clips from live streams where the editor is the broadcaster, and therefore may include clips generated from different live streams by the same broadcaster. For example, the multiple clips extracted may include a first clip which is part of a first live stream by a certain broadcaster, and a second clip which is part of a second live stream (different from the first live stream) by the same broadcaster. In the example in Figure 8, if the editor is broadcaster "GHK", clips "CL1", "CL2", and "CL3" from live stream "ST80" and clip "CL4" from live stream "ST79" are extracted and provided to the editor "GHK"'s user terminal. In this way, the support integration system 322 enables multi-source editing of highlight videos.

[0072] The editing content acquisition unit 328 acquires selection information from the editor's user terminal via the network NW, indicating at least one clip selected by the editor from among the multiple clips transmitted above, and editing instructions from the editor. This selection information includes the clip ID of the selected clip, the order of the clips adjusted by the editor, and the tags and / or annotations assigned by the editor to each selected clip. The editing content acquisition unit 328 registers the tags included in the selection information in the clip DB 336.

[0073] The editing processing unit 330 acquires edited video data output by an editing ML model that has received selection information and editing instructions as input. The editing processing unit 330 includes a trained editing ML model. This editing ML model takes the clips specified by the selection information and the editing instructions as input. In particular, the editing instructions are included in the prompts of the editing ML model. The editing ML model arranges the clips specified by the selection information in the order specified by the selection information and generates a single video data by performing image processing according to the editing instructions. The editing ML model outputs the generated video data as edited video data. The editing ML model of the editing processing unit 330 may be implemented using, for example, the technology described in Non-Patent Document 1 or Non-Patent Document 2.

[0074] The editing ML model in the editing processing unit 330 generates and outputs multiple versions of video data, each corresponding to a different target attribute, based on the edited video data generated above. For each target attribute, the editing processing unit 330 inputs a description corresponding to that target attribute into the editing ML model in the form of a prompt. The editing ML model re-edits the edited video data according to the description in the prompt and generates and outputs the edited video data for the version corresponding to the target attribute. For example, if the prompt contains a description corresponding to the target attribute "likes small talk," the editing ML model re-edits the edited video to highlight the clips tagged with "small talk" and generates the video data for the version corresponding to the target attribute "likes small talk."

[0075] The edited video provision unit 332 provides edited video data to the editor's user terminal via the network NW. The edited video provision unit 332 registers the edited video data and multiple versions acquired by the editing processing unit 330 in the edited video DB 338.

[0076] The operation of the live streaming system 1 with the above configuration will now be explained. Figure 10 is a flowchart showing the sequence of processes on the editor's user terminal during editing. When the editor's user terminal receives an instruction to start editing from the editor, it generates an editing start request including the editor's user ID and sends it to the server 10 via the network NW (S202). The user terminal receives from the server 10 the clip of the live stream for which the editor is the broadcaster, and the information associated with that clip (S204). The user terminal displays an editing screen on the display that includes a list of thumbnails of the received clips (S206). The editor previews the thumbnails (S208).

[0077] If the previewed thumbnail is not of interest (N in S210), the editor previews the next thumbnail (returns to step S208). If the editor is interested in the previewed thumbnail, i.e., the corresponding clip (Y in S210), the editor specifies the thumbnail of interest (S212). The user terminal displays the details of the clip corresponding to the specified thumbnail (S214) and plays the clip in the preview window of the editing screen (S216).

[0078] If the editor does not select the clip after viewing it in the preview window (N in S218), the editor previews the next thumbnail (returns to step S208). If the editor selects the clip after viewing it in the preview window (Y in S218), the user terminal registers or adds the selected clip to the selection list (S220). If the editor wishes to view other clips (Y in S222), the editor previews the next thumbnail (returns to step S208). If the editor determines that there are no other clips to view (N in S222), the user terminal accepts requests from the editor to adjust the order of the selected clips shown in the selection list (S224). The user terminal accepts the editor's addition of tags and / or annotations to each selected clip shown in the selection list (S226). The user terminal accepts the input of editing instructions (S228). The user terminal performs a confirmation process of the editor's selections and inputs (S230). If the editor does not confirm the selection of clips and the content of the editing instructions (N in S232), the user terminal accepts the editor's correction of the selection and / or input content (S234). The process then returns to step S230. If the editor confirms the selection of clips and the content of the editing instructions (Y in S232), the user terminal sends the clip selection result and editing instructions to the server 10 (S236). The user terminal generates selection information including the selection list, the adjusted order, and the entered tags and / or annotations, and sends this selection information and editing instructions together to the server 10.

[0079] Figure 11 is a flowchart showing the sequence of processes in Server 10 during editing. Server 10 receives an editing start request from the editor's user terminal, which includes the editor's user ID (S302). Server 10 retrieves information about the clips corresponding to the editor, i.e., clips generated from past live broadcasts (archives) that the editor has broadcast as the broadcaster, from Clip DB 336 (S304). The clip information includes the clips from the live broadcasts where the editor is the broadcaster, and the information associated with those clips. Server 10 sends the retrieved clip information to the requesting user terminal (S306). Server 10 receives the editor's clip selection result and editing instructions from the editor's user terminal (S308). Server 10 inputs the selected clips and editing instructions into the editing ML model (S310).

[0080] The editing ML model performs video editing processing (S312). This video editing processing includes interpreting and executing editing instructions (S314), automatically generating transitions between clips (S316), applying effects and background music (S318), and checking and adjusting to ensure overall consistency (S320).

[0081] Server 10 sends the video generated and output by the editing ML model to the editor's user terminal (S322). Server 10 receives the video adjustment results from the editor's user terminal (S324). If the received adjustment results instruct re-editing (Y in S326), Server 10 receives editing instructions for re-editing from the editor's user terminal (S328). The process then returns to step S312. If the received adjustment results do not instruct re-editing (N in S326), Server 10 finalizes the adjusted video as an edited video and registers it in the edited video DB338 (S330). Server 10 generates variations of the edited video corresponding to each target attribute and registers the generated variations in the edited video DB338 (S332).

[0082] Figure 12 is a representative screen view of the editing screen 500 displayed on the editor's user terminal display. When the editor's user terminal sends an editing start request to the server 10 and receives clip information from the server 10, the user terminal generates the editing screen 500 based on that information and displays it on the display. The editing screen 500 includes a thumbnail 502 of the clip included in the clip information, a preview window 504, a clip details display area 506 that displays detailed information of the selected clip, a selected clips display area 508 that displays the clip titles 510 of the clips included in the selection list, an editing instruction input area 512 that accepts editing instructions from the editor in text input format, a confirmation button 514, and a cancel button 516. The thumbnail 502 is a thumbnail of a clip that the editor can select, i.e., a clip from a live stream performed by the editor. In Figure 12, the system suggests that "this area, this area, and this area are exciting" in the archive by generating clips. The streamer then selects some of these clips by tapping, and the selected clips are compressed to create a single video.

[0083] The editor selects a clip from among several thumbnails 502 by tapping the thumbnail 502 of interest. The preview window 504 plays the clip corresponding to the tapped thumbnail 502, and the clip details display area 506 displays the details of that clip. If the editor decides to edit that clip, they tap the corresponding thumbnail 502 again. When the user terminal detects this tap, it adds the corresponding clip to the selection list and also adds the clip title 510 of the corresponding clip to the selected clips display area 508. If the editor wants to deselect a clip, they long-press the clip title 510 of the clip they want to deselect. The user terminal removes the clip of the long-pressed clip title 510 from the selection list. The editor adjusts the display order of clips in the highlight video by dragging and dropping the clip titles 510 within the selected clips display area 508. The editor enters the desired tags and / or annotations in the clip details display area 506.

[0084] When an editor selects a clip, enters editing instructions, and taps the confirmation button 514, the user terminal generates selection information including a selection list, the order adjusted in the selected clips display area 508, and tags and / or annotations entered through the clip details display area 506. This selection information, along with the editing instructions entered in the editing instruction input area 512, is then sent to the server 10.

[0085] Figure 13 is a representative screen view of the live streaming room screen 630 displayed on the streamer's user terminal 20 during live streaming. The live streaming room screen 630 includes a video image 632 of the streamer obtained by playing the video data to be transmitted by the video transmission unit 106, a comment display area 634, a stream end button 636, and objects related to flag assignment and display. The streaming-side UI control unit 108 generates the live streaming room screen 630 by superimposing other objects, namely the stream end button 636, the comment display area 634, and objects related to flag assignment and display, onto the video image 632 obtained by playing the video data. In this way, objects related to flag assignment and display are displayed in association with the streamer's video image 632. Flags may indicate parts that the streamer wants to use later.

[0086] The comment display area 634 may include comments entered by viewers and notifications from the system. Notifications from the system may include information indicating who sent which gifts to the broadcaster. The broadcast-side UI control unit 108 generates a comment display area 634 containing viewer comments and system notifications received from the server 10, and includes the generated comment display area 634 in the live streaming room screen 630.

[0087] The "End Stream" button 636 is an object that receives instructions from the streamer to stop providing the live stream.

[0088] The objects related to flag assignment and display include a time axis object 638, an excitement flag object 640, an important point flag object 642, a funny comment flag object 644, an excitement button 646, an important point button 648, and a funny comment button 650. The time axis object 638, the excitement flag object 640, the important point flag object 642, and the funny comment flag object 644 together constitute a graphical user interface that represents the timing of flag assignment in a live stream. The right end of the time axis object 638 indicates the current time, and the left end indicates the start of the live stream. The excitement flag object 640 and the excitement button 646 are both represented by solid lines to indicate their correspondence. The important point flag object 642 and the important point button 648 are both represented by dashed lines to indicate their correspondence. The funny comment flag object 644 and the funny comment button 650 are both represented by dotted lines to indicate their correspondence. This correspondence is not limited to line type; it may also be represented by other display features such as color and size. The position of each flag object in the time axis object 638 represents the timing at which the flag was assigned. In the example in Figure 13, it is shown that after the live stream started, the important points flag was first assigned by the streamer, then the excitement flag was assigned, and then the funny comments flag was assigned. This makes it easy for the streamer to understand where they placed the flags.

[0089] During a live stream, the streamer taps the excitement button 646 when they feel the stream is getting exciting, the important point button 648 when they feel it's an important point, and the funny comment button 650 when there is a funny comment. When the user terminal's streaming UI control unit 108 detects a tap on any of the buttons during the live stream, it accepts it as a button selection by the streamer. The user terminal's streaming communication unit 110 generates a flag assignment signal that includes information identifying the selected button, i.e., a flag ID indicating whether the selected button is the excitement button 646, the important point button 648, or the funny comment button 650, and the timing or time when the button was selected, and sends it to the server 10 via the network NW. In this embodiment, the excitement button 646 corresponds to the flag ID "FLGA" in Figure 7, the important point button 648 corresponds to the flag ID "FLGB" in Figure 7, and the funny comment button 650 corresponds to the flag ID "FLGC" in Figure 7.

[0090] Figure 14 is a representative screen view of the live streaming room screen 630 displayed on the streamer's user terminal 20 during live streaming. Figure 14 shows the state immediately after the excitement button 646 is tapped in the live streaming room screen 630 of Figure 13. In the live streaming room screen 630 of Figure 14, a new excitement flag object 654 is displayed near the right end of the time axis object 638. This new excitement flag object 654 corresponds to the tapping of the excitement button 646. At the same time, text 652 indicating that the flag button has been tapped and a sticky note has been added is displayed on the live streaming room screen 630.

[0091] Figure 15 is a representative screen view of the archive viewing screen 700 displayed on the user terminal display of an active user. This active user is viewing an archive of their own live stream on the archive viewing screen 700. The archive viewing screen 700 includes an archive playback area 702 that displays the video obtained by playing the archive, a progress bar 704 that indicates the current playback position of the video being played in the archive playback area 702, objects related to flag assignment and display, and a comment display area 720 that displays comments.

[0092] The non-distribution UI control unit 402 of the user terminal plays the archive received from the server 10 and displays the obtained video in the archive playback area 702. At the same time, the non-distribution UI control unit 402 updates the display of the progress bar 704 to indicate the current playback position of the video. The progress bar 704 has a thumb object 722 and a bar object 724. The total length of the bar object 724 represents the entire length of the archive. The position of the thumb object 722 on the bar object 724 indicates the current playback position.

[0093] The objects related to assigning and displaying flags include an excitement flag object 706, an important point flag object 708, a funny comment flag object 710, an excitement button 714, an important point button 716, and a funny comment button 718. The progress bar 704, the excitement flag object 706, the important point flag object 708, and the funny comment flag object 710 together constitute a graphical user interface that indicates when a flag is assigned during live streaming or archive viewing. The correspondence between the display modes of the flag objects and buttons is as described in Figure 13. The position of each flag object on the progress bar 704 indicates when that flag was assigned.

[0094] Active users tap the excitement button 714 when they feel the broadcast is getting exciting while watching their archive, the important point button 716 when they feel it is an important point, and the funny comment button 718 when they see an interesting comment. When the user terminal's off-stream UI control unit 402 detects a tap on any of the buttons while watching the archive, it accepts it as a button selection by the active user. The user terminal's off-stream communication unit 404 generates a flag assignment signal that includes the flag ID corresponding to the selected button and the timing or date when the button was selected, and sends it to the server 10 via the network NW.

[0095] In the archive viewing screen 700 of Figure 15, if an active user operates the cursor 712 to position it at a predetermined location on the progress bar 704, the non-distribution UI control unit 402 may display a portion of the archive video, including the timing indicated by the cursor 712's position, in the archive playback area 702. For example, if the cursor 712 on the progress bar 704 indicates 3 minutes and 35 seconds after the start of the archive, a 30-second portion of the archive video or clip from 3 minutes and 20 seconds to 3 minutes and 50 seconds may be played in the archive playback area 702. In addition, comments posted during those 30 seconds may be displayed in the comment display area 720.

[0096] Alternatively, the system could pick out the most exciting parts of the archive and display them as circular dots on the progress bar 704. Tapping a circular dot would automatically play several tens of seconds before and after that section. In this case, the streamer would also find it easier to spot the most exciting parts.

[0097] In the embodiments described above, examples of DBs include hard disks and semiconductor memory. Furthermore, it will be understood by those skilled in the art who have read this specification that each component can be realized based on the description herein by a CPU (not shown), modules of installed application programs, modules of system programs, and semiconductor memory for temporarily storing the contents of data read from the hard disk.

[0098] According to the live streaming system 1 of this embodiment, the editing process of combining live streaming clips to generate a single highlight video can be performed more efficiently and / or more easily. When an editor selects the target clips and enters editing instructions, the ML model automatically performs the time-consuming video processing, merging, and transition processing, thereby reducing the burden of editing work.

[0099] Furthermore, the live streaming system 1 according to this embodiment allows for the creation of highlight videos by combining clips from different live streams, enabling more flexible and adaptable editing.

[0100] Furthermore, in the live streaming system 1 according to this embodiment, the streamer can assign a flag during live streaming or while viewing the archive. This flag is used for automatic clip generation. This allows the streamer's actions and intentions to be incorporated into the clip generation criteria. As a result, the generated clips will be more in line with the streamer's intentions, improving user satisfaction.

[0101] Furthermore, since there are multiple types of flags, broadcasters can choose the flag that best suits the situation. A graphical user interface is provided that shows when and which flags the broadcaster has assigned, making it easier for broadcasters to keep track of their flag assignments.

[0102] Referring to Figure 16, the hardware configuration of the information processing device according to this embodiment will be described. Figure 16 is a block diagram showing an example of the hardware configuration of the information processing device according to this embodiment. The illustrated information processing device 900 can, for example, realize the server 10 and user terminals 20 and 30 in this embodiment.

[0103] The information processing device 900 includes a CPU 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 925, and a communication device 929. Furthermore, the information processing device 900 may include an imaging device (not shown), such as a camera. The CPU 901 is an example of a hardware configuration for realizing the functions realized by the components described herein. The functions described herein may be realized by circuits programmed to realize such functions. Circuits programmed to realize the functions described herein include a CPU (a Central Processing Unit), a DSP (Digital Signal Processor), a general-purpose processor, an application-specific processor, an integrated circuit, an ASIC (Application Specific Integrated Circuit), and / or a combination thereof. In this specification, a unit that performs a particular function may be implemented as a circuit programmed to perform that function.

[0104] The CPU 901 functions as an arithmetic processing unit and control unit, controlling the overall operation or a part of the operation within the information processing unit 900 according to various programs recorded in the ROM 902, RAM 903, storage device 919, or removable recording medium 923. For example, the CPU 901 controls the overall operation of each functional unit included in the server 10 and user terminals 20 and 30, respectively, in this embodiment. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during its execution. The CPU 901, ROM 902, and RAM 903 are interconnected by a host bus 907, which is composed of internal buses such as the CPU bus. Furthermore, the host bus 907 is connected to an external bus 911, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 909.

[0105] The input device 915 may be a device operated by the user, such as a mouse, keyboard, touch panel, buttons, switches, and levers, or it may be a device that converts physical quantities into electrical signals, such as a sound sensor like a microphone, an acceleration sensor, a tilt sensor, an infrared sensor, a depth sensor, a temperature sensor, or a humidity sensor. The input device 915 may also be a remote control device that uses infrared or other radio waves, or it may be an external connection device 927 such as a mobile phone that is compatible with the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user or a physical quantity sensed and outputs it to the CPU 901. By operating this input device 915, the user inputs various data to the information processing device 900 or instructs it to perform processing operations.

[0106] The output device 917 is comprised of a device capable of notifying the user of acquired information visually or audibly. The output device 917 may be, for example, a display such as an LCD, PDP, or OLED, an audio output device such as a speaker and headphones, or a printer. The output device 917 outputs the results obtained from the processing of the information processing device 900 as text or images, or as sound.

[0107] The storage device 919 is a data storage device configured as an example of the storage unit of the information processing device 900. The storage device 919 is composed of, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 919 stores programs executed by the CPU 901, various data, and various data acquired from external sources.

[0108] The drive 921 is a reader / writer for removable recording media 923, such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory, and is either built into or external to the information processing device 900. The drive 921 reads information recorded on the installed removable recording media 923 and outputs it to the RAM 903. The drive 921 also writes data to the installed removable recording media 923.

[0109] The connection port 925 is a port for directly connecting equipment to the information processing device 900. The connection port 925 may be, for example, a USB (Universal Serial Bus) port, an IEEE1394 port, or a SCSI (Small Computer System Interface) port. Alternatively, the connection port 925 may be an RS-232C port, an optical audio terminal, or an HDMI (High-Definition Multimedia Interface) port. By connecting an external device 927 to the connection port 925, various types of data can be exchanged between the information processing device 900 and the external device 927.

[0110] The communication device 929 is a communication interface composed of, for example, a communication device for connecting to a network NW. The communication device 929 may be, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth®, or WUSB (Wireless USB). Alternatively, the communication device 929 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication. The communication device 929 sends and receives signals, for example, to the Internet or other communication devices using a predetermined protocol such as TCP / IP. The communication network NW connected to the communication device 929 is a network connected by wire or wireless, and may be, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication. The communication device 929 performs the function of a communication unit.

[0111] An imaging device such as a camera (not shown) is a device that captures real space and generates an image using various components such as an image sensor, such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), and a lens for controlling the imaging of a subject onto the image sensor. The imaging device may capture still images or it may capture moving images.

[0112] The configuration and operation of the live streaming system 1 according to this embodiment have been described above. This embodiment is illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of each component and each process, and that such modifications are also within the scope of this disclosure.

[0113] In this embodiment, we have described a case where clips are automatically generated by the system, but we are not limited to this. For example, broadcasters or viewers may be able to manually generate clips from archives or live streams. The live streaming system may be configured to support this manual generation.

[0114] The embodiment described a case where an editor selects desired clips from clips generated from their own live stream archive to generate a highlight video, but it is not limited to this. For example, the system may be configured so that an editor can select desired clips from clips generated from another person's live stream archive to generate a highlight video.

[0115] The conversion rate from gift consideration points to awarded rewards in this embodiment is just one example, and these may be set as appropriate by, for example, the administrator of the live streaming system.

[0116] The technical concept of this embodiment may also be applied to virtual live streaming or live commerce that uses an avatar that moves in sync with the streamer's movements instead of the streamer's image. Furthermore, although this embodiment describes a case where the server relays video data related to the live stream, generated on the streamer's user terminal, to the viewer's user terminal, it is not limited to this. For example, the technical concept of this embodiment may be applied when setting up a virtual streamer instead of an actual streamer. The virtual streamer is, for example, an AI virtual streamer whose appearance uses an avatar, whose voice is composed of a TTS (Text-to-Speech) engine, and whose speech content is obtained from a machine learning model that takes viewer comments as input. In this case, there is no streamer user terminal, and the processing on the streamer's side is performed on the server.

[0117] In this embodiment, the editing ML model may generate a thumbnail corresponding to the target attributes when generating a version corresponding to the target attributes. When a viewer displays the editor's profile screen on their user terminal's display, the thumbnail of the highlight video displayed on that profile screen will be a thumbnail corresponding to that viewer's target attributes. When another viewer with different target attributes accesses the same editor's profile screen, the thumbnail of the highlight video displayed on that profile screen will be different from the thumbnail described above.

[0118] The embodiments relating to this disclosure may include at least one of the following elements: 1. Intelligent Scene Analysis Engine: - Divide long videos into meaningful units (scenes, topics, statements, etc.) - Integrate and use image analysis, speech recognition, and natural language processing technologies. - Evaluate the importance, emotional impact, and technical quality of each scene. 2. Context Understanding Module: - Analyze the overall context and production intent of the video. - Identify genre, target audience, and narrative structure. 3. Editorial suggestion generation engine: - We propose the optimal editing order based on the context of the content and the creative intent. - Generates suggestions based on different editing styles (dynamic, emotional, informative, etc.) 4. Real-time highlight generation module: - Identify important scenes in real time during live streaming. - Automatically generates instantly editable highlight clips 5. Audience reaction analysis engine: - Analyze viewer comments, engagement, and viewing patterns. - Identify popular scenes and topics and incorporate them into the editorial strategy. 6. Multi-source integrated editor: - Extract and integrate relevant scenes from multiple video sources. - Provides multiple perspectives while maintaining a consistent narrative. 7. AI-assisted editing interface: - Enables advanced editing with intuitive drag-and-drop operation. - AI-powered editing suggestions are displayed in real time to support the editor's decision-making. 8. Automatic Transition Effect Generator: - Automatically generates smooth transitions between scenes. - We propose appropriate visual effects that match the atmosphere of the content. 9. Personalized Content Variation Generator: - Automatically generate multiple versions from the same source material for different target audiences. - Length, tone, and focus point are adjustable. 10. Quality Assurance and Consistency Checker: - Automatically checks the technical quality and consistency of edited content. - Point out potential problems (such as audio inconsistencies and visual discontinuities). 11. Prompt-based severity leveling module: - Users can specify their editing intent and priorities through natural language or a GUI. - Customize AI analysis and suggestions based on specified importance levels. - Dynamic prompt adjustment feature allows you to change priority even during the editing process. 12. Driver Interaction Flag System: - Live streamers can add flags such as "excitement level" with a single touch during their broadcast. - Record and analyze the type, timing, and frequency of assigned flags. - Flag data is sent to the AI ​​analysis engine in real time and reflected in editing recommendations. 13. Flag-linked recommendation optimization engine: - Incorporate flags assigned by live streamers into the recommendation algorithm as important indicators. - By combining flag data and viewer reaction data, we can achieve more accurate extraction of important scenes. - Different weightings are applied depending on the type of flag, reflecting the diverse qualities of "excitement."

[0119] According to the live streaming system 1 of this embodiment, at least one of the following effects is achieved. 1. Significantly streamlined and shortened the video editing process. 2. Promoting the creation of high-quality and consistent content. 3. Live content can be edited and distributed instantly. 4. Develop an effective editing strategy based on viewer interests. 5. Simplifying the creation of integrated content from multiple sources. 6. Expand the creativity of editors and explore new possibilities for expression. 7. Promote the personalization and diversification of content. 8. Effectively combine user intent with AI capabilities to achieve more precise editing assistance. 9. Flexibly respond to diverse editing needs and promote a creative editing process. 10. Realizing more accurate and immersive editing support that instantly reflects the intuitive judgment of the live streamer. 11. Highly personalized content generation through collaboration between humans (live streamers) and AI. 12. The editor can include instructions (and intentions) about the parts they deem important, allowing for selection.

[0120] The technical concept of this embodiment can also be applied to the following examples. 1. News production support system: - Automatically extracts the most relevant clips from long-duration news footage to match the context of the news. - Generate real-time highlights during live news coverage, and edit and distribute them immediately. 2. Automatic sports highlight generation system: - Analyzes match footage from multiple cameras and automatically extracts the most impressive moments. - Generate personalized highlight videos while considering viewer reactions. 3. Educational content optimization tools: - Extract key points from long lecture videos and create effective summary videos. - Generate multiple versions with adjusted levels of detail based on the learner's understanding. 4. Marketing video customization system: - Automatically customize the same product introduction video for different target audiences. - Efficiently generate multiple versions for A / B testing 5. Social media content optimization tools: - Automatically generates platform-optimized clips from long-form video content. - Analyze viewer engagement and propose editing strategies that are likely to go viral. 6. Documentary Production Support System: - We propose the optimal scenes from a large amount of footage, aligned with the narrative structure. - Integrate multiple interview videos to create a consistent storyline.

[0121] The technical concept related to this embodiment may be expressed by the following items. 1. A means of analyzing video content and automatically identifying and extracting scenes suitable for editing. 2. A means of understanding the context and creative intent of the content and proposing the optimal editing order. 3. Means for generating highlights in real time during live streaming 4. Means of analyzing viewer reactions and interests and reflecting them in the editing strategy. 5. A means of extracting and integrating relevant scenes from multiple video sources. 6. Means of providing an AI-assisted intuitive editing interface 7. Means for generating automatic transitions and effects between scenes 8. A method for automatically generating multiple versions for different target audiences from the same source material. 9. A means of automatically checking the quality and consistency of edited content. 10. A means for users to specify their editing intent and priorities through natural language or a GUI, and to customize AI analysis and suggestions. 11. A method for streamers to easily add flags during live broadcasts. 12. Means for recording and analyzing data of assigned flags and reflecting it in editing recommendations. 13. An AI-driven integrated video content analysis and editing support system characterized by comprising means for integrating flag data and other analytical data to optimize a recommendation algorithm.

[0122] Examples of applications or uses of this embodiment are listed below. 1. Continuous improvement of deep learning models and expansion of training data. 2. Support for new video formats and platforms 3. Improving real-time processing capabilities on edge devices 4. Examination of legal aspects regarding privacy and content rights 5. Further integration with creator workflows 6. Advanced integrated analysis of multimodal (video, audio, text) data. 7. Customization to meet the specific needs of different genres and industries. 8. Continuous feature improvements and enhancements based on user feedback 9. Development of more advanced prompt interpretation capabilities in line with advancements in natural language processing technology. 10. Learn the user's editing style and preferences and implement a personalized prompt suggestion feature. 11. Support for international content creation through the integration of multilingual support and automatic translation functions. 12. Support for Augmented Reality (AR) and Virtual Reality (VR) content 13. Utilizing blockchain technology to improve content rights management and transparency. 14. Advanced emotion analysis technology enables a more detailed understanding and reflection of viewer reactions. 15. Development of AI-powered creative editing suggestion features (e.g., suggestion of new transition effects and narrative structures) 16. Development of environmentally friendly and efficient processing algorithms (Green AI) 17. Enhanced integration with other creative tools (image editing software, 3D modeling tools, etc.) 18. Addition of automatic music and sound effect selection and generation function. 19. Realizing intuitive editing operations using user gestures and voice commands. 20. Clarifying the rationale behind editorial suggestions by improving the explainability of AI models (Explainable AI) As these diverse implementations demonstrate, this system is applicable to a wide range of industries and content types, and its versatility and scalability are its greatest strengths.

[0123] In this embodiment, comment speed, which is the amount of comments per unit time, may be used as an indicator to determine the most exciting parts of an archive or live stream. Comment speed may measure comments from both viewers and broadcasters, or it may measure comments from viewers only (excluding broadcasters). Comment criteria may be established to select comments that contribute to the calculation of comment speed. For example, comments with fewer than a predetermined minimum number of characters may be set not to contribute to the calculation of comment speed.

[0124] In the processing procedures described herein, particularly those described using flowcharts, it is possible to omit some of the steps constituting the processing procedure, to add steps not explicitly included as constituting the processing procedure, and / or to change the order of such steps. Processing procedures with such omissions, additions, or changes in order are included within the scope of this disclosure, provided they do not deviate from the spirit of this disclosure.

[0125] At least some of the functions implemented by server 10 may be implemented by devices other than server 10, such as user terminals 20 and 30. At least some of the functions implemented by user terminals 20 and 30 may be implemented by devices other than user terminals 20 and 30, such as server 10. For example, the superposition of a predetermined frame image onto an image of video data performed on the viewer's user terminal may be performed on server 10 or on the broadcaster's user terminal.

Claims

1. A means for generating multiple different video data sets, each of which is a part of the original video data, A means for obtaining information indicating at least one video data selected by the user from among the multiple video data from the user's terminal via the network, Means for obtaining editing instructions from the user via the network from the terminal, A means for acquiring edited video data output by a machine learning model that has the aforementioned information and the aforementioned editing instructions as inputs, A server comprising means for providing the edited video data to the terminal via the network.

2. The server according to claim 1, wherein the plurality of video data includes video data which is a part of the original first video data and video data which is a part of the original second video data.

3. The server according to claim 1, wherein the machine learning model outputs multiple edited video data sets, each corresponding to different viewer attributes.

4. The server according to claim 1, wherein the information includes the order of video data adjusted by the user and tags and / or annotations assigned by the user to each selected video data.

5. The aforementioned original video data is video data related to a live stream. The server according to claim 1, wherein the means for generating the data generates the plurality of video data based on the reactions of the live stream viewers and the actions of the live stream broadcaster.

6. The server according to claim 5, wherein the actions of a live streamer include the streamer assigning a flag at a desired timing during the live stream.

7. On the live streamer's device, During the live stream, the terminal's display has a function to display an object associated with the live stream video, During the live stream, a function is provided to accept the designation of the object by the streamer, A computer program for implementing the function of sending the object in the live stream to a server via a network at a specified timing.

8. The aforementioned display function includes a function for displaying multiple different objects, The computer program according to claim 7, wherein the transmission function includes a function of transmitting to the server via the network information that identifies a specified object and the timing at which the object occurs.

9. The computer program according to claim 7, further providing the terminal with a function to display a graphical user interface on the terminal's display that indicates the timing of the object in the live stream.

Images