System and method for streaming

Abstract

To provide a service desired by users in a live streaming service.SOLUTION: A system and a method for responding to comments according to the present invention include the steps of: acquiring a first comment; acquiring a second comment; acquiring the parameter of the first comment; acquiring the parameter of the second comment; determining the first priority score of the first comment on the basis of the parameter of the first comment; determining the second priority score of the second comment on the basis of the parameter of the second comment; and selecting the first comment or the second comment for responding, on the basis of the first priority score and the second priority score.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present invention relates to streaming, and in particular to live streaming. [Background technology]

[0002] As exemplified by live streaming services, real-time communication over the Internet has become a part of everyday life. A variety of platforms and providers offer live streaming services, and competition is fierce. It is important for platforms to provide services that users want.

[0003] China Patent Application Publication No. 116600152 discloses a system for virtual streamers. Summary of the Invention

[0004] A method according to one embodiment of the invention is a method of responding to comments implemented by one or more computers, comprising the steps of: obtaining a first comment; obtaining a second comment; obtaining parameters of the first comment; obtaining parameters of the second comment; determining a first priority score for the first comment based on the parameters of the first comment; determining a second priority score for the second comment based on the parameters of the second comment; and selecting the first comment or the second comment for response based on the first priority score and the second priority score.

[0005] A system according to one embodiment of the present invention is a system for responding to comments including one or more processors, wherein the one or more computer processors execute machine-readable instructions to perform the following steps: obtaining a first comment; obtaining a second comment; obtaining parameters of the first comment; obtaining parameters of the second comment; determining a first priority score for the first comment based on the parameters of the first comment; determining a second priority score for the second comment based on the parameters of the second comment; and selecting the first comment or the second comment for responding based on the first priority score and the second priority score.

[0006] A non-transitory computer-readable medium containing a program for responding to comments, the program causing one or more computers to perform the steps of: obtaining a first comment; obtaining a second comment; obtaining parameters of the first comment; obtaining parameters of the second comment; determining a first priority score for the first comment based on the parameters of the first comment; determining a second priority score for the second comment based on the parameters of the second comment; and selecting the first comment or the second comment for responding based on the first priority score and the second priority score. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram showing a configuration of a live streaming system 1 according to some embodiments of the present invention. [Diagram 2] 2 is a block diagram illustrating the functionality and configuration of user terminal 30 of FIG. 1 in accordance with some embodiments of the present invention. [Diagram 3] 2 is a block diagram illustrating the functionality and configuration of the server of FIG. 1 in accordance with some embodiments of the present invention. [Figure 4]4 is a table showing an example data structure of the stream DB 310 of FIG. 3. [Diagram 5] 4 is a table illustrating an example data structure of the user DB 312 of FIG. 3. [Figure 6] 4 is a table showing an example data structure of the gift DB 314 of FIG. 3. [Figure 7] 1 is a table showing an exemplary data structure of a character DB 324. [Figure 8] 13 is a table showing an exemplary data structure of the comment DB 338. [Figure 9] 13 is an example illustrating comment priority score determination. [Figure 10] 13 is a table showing an exemplary data structure of the comment DB 338. [Figure 11] 1 is an exemplary flow chart according to some embodiments of the present invention. [Figure 12] 1 is a table showing an example data structure of a response DB 340. [Figure 13] 1 is an exemplary flow chart according to some embodiments of the present invention. [Figure 14] 1 is an example of an exemplary AI VLiver system according to some embodiments of the present invention. [Figure 15] 1 is an exemplary virtual character according to some implementations of the present invention. [Figure 16] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the same or similar components, parts, steps, or signals shown in each drawing will be denoted by the same reference numerals in all drawings, and duplicated explanations will be omitted as appropriate. In addition, some parts that are not important in the explanation of each drawing will be omitted.

[0009] A virtual streamer (or virtual liver, virtual live streamer, virtual distributor, virtual anchor, etc.) can interact with viewers in a live streaming room provided by a live streaming platform. It is a virtual character (such as an anime character). In some embodiments, the virtual character automatically interacts with viewers without a real human streamer being behind or controlling the virtual character. The virtual character may digest comments from viewers and respond with actions, voice, or text messages. It is desirable to have a virtual character that can respond to viewers' comments like a real human, rather than responding to each comment one by one like a robot. A virtual streamer that utilizes AI or machine learning technology can be called an AI VLiver.

[0010] FIG. 1 is a schematic diagram showing a configuration of a live streaming system 1 according to some embodiments of the present invention. The live streaming system 1 provides a live streaming service for streaming streamers (also called live streamers, anchors, distributors, and live streamers) LV and viewers (also called audiences) AU (AU1, AU2, ...) to interact or communicate in real time. As shown in FIG. 1, the live streaming system 1 includes a server 10, a user terminal 20, and a user terminal 30 (30a, 30b, ...). In some embodiments, the streamers and viewers may be collectively referred to as users. The server 10 may include one or more information processing devices connected to a network NW. The user terminals 20 and 30 may be, for example, mobile terminal devices such as smartphones, tablets, notebook PCs, recorders, portable game consoles, and wearable terminals, or stationary devices such as desktop PCs. The server 10, the user terminals 20, and the user terminals 30 are connected to each other so as to be able to communicate with each other via various wired or wireless networks NW.

[0011] The live streaming system 1 includes a distributor LV, a viewer AU, and an administrator (or an app provider, not shown) who manages the server 10. The distributor LV is a person who distributes the content in real time by recording the content on his / her user terminal 20 and uploading it directly or indirectly to the server 10. Examples of the content may be the distributor's own song, talk, performance, game play, or any other content. The administrator provides a platform for live streaming the content on the server 10, and mediates or manages real-time interactions between the distributor LV and the viewer AU. The viewer AU accesses the platform with his / her user terminal 30, selects and watches desired content. During the live streaming of the selected content, the viewer AU performs operations such as sending comments, cheers, and gifts via the user terminal 30. The distributor LV who distributes the content may respond to those comments, cheers, or gifts. The response is transmitted to the viewer AU by video and / or audio, and two-way communication is established.

[0012] The term "live streaming" may refer to a data transmission mode that allows the content recorded by the user terminal 20 of the distributor LV to be played back and viewed by the user terminal 30 of the viewer AU in substantially real time, or may refer to a live broadcast realized by such a transmission mode. The live streaming may be realized by utilizing existing live streaming technologies such as HTTP live streaming, CMAF (Common Media Application Format), WebRTC (Web Real-Time Communications), RTMP (Real-Time Messaging Protocol), and the like. Live streaming includes a transmission mode that allows the viewer AU to view the content with a certain delay at the same time as the content is recorded by the distributor LV. The length of the delay may be such that the distributor LV and the viewer AU can communicate with each other. Note that the live streaming is distinguished from so-called on-demand distribution, in which all recorded data of the content is stored once on the server and then provided to the user from the server upon the user's request.

[0013] The term "video data" as used herein refers to data including image data (also called video data) generated using the imaging function of the user terminal 20 or 30, and audio data generated using the audio input function of the user terminal 20 or 30. The video data is played on the user terminal 20, 30 so that the user can view the content. In some embodiments, it is assumed that the video data is subjected to a process such as compression, decompression, encoding, decoding, transcoding, etc., to change its format, size, or data specifications between the generation of the video data on the distributor's user terminal and the playback of the video data on the viewer's user terminal. However, since the content (e.g., video and audio) represented by the video data does not substantially change before and after such a process, the video data after such a process is expressed in this specification as being the same as the video data before such a process. In other words, when video data is generated at the distributor's user terminal and then played at the viewer's user terminal via the server 10, the video data generated at the distributor's user terminal, the video data passing through the server 10, and the video data received and played at the viewer's user terminal are all the same video data.

[0014] In the example shown in Fig. 1, the distributor LV provides live streaming data. The user terminal 20 of the distributor LV records video and audio of the distributor LV to generate the streaming data, and the generated data is transmitted to the server 10 via the network NW. At the same time, the user terminal 20 displays the recorded video VD of the distributor LV on the display of the user terminal 20, allowing the user to check the live streaming content currently being provided by the distributor LV.

[0015] The user terminals 30a, 30b of the viewers AU1, AU2 who have requested the platform to view the live streaming of the distributor LV receive video data related to the live streaming (hereinafter, may be referred to as "live streaming video data") via the network NW, play the received video data, display the videos VD1, VD2 on the display, and output audio from the speaker. The videos VD1, VD2 displayed on the user terminals 30a, 30b are substantially the same as the video VD captured by the user terminal 20 of the distributor LV, and the audio output from the user terminals 30a, 30b is substantially the same as the audio recorded by the user terminal 20 of the distributor LV.

[0016] The recording of video and audio on the user terminal 20 of the distributor LV and the playback of video data on the user terminals 30a and 30b of the viewers AU1 and AU2 are performed substantially simultaneously. When the viewer AU1 inputs a comment on the content provided by the distributor LV to the user terminal 30a, the server 10 displays the comment on the user terminal 20 of the distributor LV in real time, and also displays the comment on the user terminals 30a and 30b of the viewers AU1 and AU2. When the distributor LV reads the comment and develops a talk corresponding to the comment, the video and audio of the talk are displayed on the user terminals 30a and 30b of the viewers AU1 and AU2, respectively. This interactive operation is recognized as a conversation being established between the distributor LV and the viewer AU1. As a result, the live streaming system 1 realizes live streaming that enables two-way communication, not one-way communication.

[0017] 2 is a block diagram showing the functions and configuration of the user terminal 30 of FIG. 1 according to some embodiments of the present invention. The user terminal 20 has the same or similar functions and configuration as the user terminal 30. Each block of FIG. 2 and the following block diagrams may be realized by hardware elements such as a computer CPU or a mechanical device, or by software elements such as a computer program. The functional blocks may be realized by the cooperation of these elements. Therefore, it will be understood by those skilled in the art that these functional blocks may be realized in various forms by combining hardware and software.

[0018] The distributor LV and the viewer AU may download a live streaming application program (hereinafter, referred to as a live streaming application) from a download site via the network NW and install it on the user terminals 20 and 30. Alternatively, the live streaming application may be pre-installed on the user terminals 20 and 30. When the live streaming application is executed on the user terminals 20 and 30, the user terminals 20 and 30 communicate with the server 10 via the network NW and implement or execute various functions. Hereinafter, functions implemented by the user terminals 20 and 30 (processors such as CPUs) on which the live streaming application is executed will be described as functions of the user terminals 20 and 30. These functions are actually realized by the live streaming application on the user terminals 20 and 30. In some embodiments, these functions may be realized by a computer program written in a programming language such as HTML (HyperText Markup Language), transmitted from the server 10 to a web browser of the user terminals 20 and 30 via the network NW, and executed by the web browser.

[0019] The user terminal 30 includes a distribution unit 100 and a viewing unit 200. The distribution unit 100 generates video data in which the video and audio of the user (or the user side) are recorded, and provides the video data to the server 10. The viewing unit 200 receives the video data from the server 10 and plays the video data. The user starts the distribution unit 100 when performing live streaming, and starts the viewing unit 200 when the user watches a video. The user terminal on which the distribution unit 100 is started is the terminal of the distributor, i.e., the user terminal that generates the video data. The user terminal on which the viewing unit 200 is started is the terminal of the viewer, i.e., the user terminal on which the video data is reproduced and played.

[0020] The delivery unit 100 includes an imaging control unit 102, an audio control unit 104, a video transmission unit 106, and a distributor-side UI control unit 108. The imaging control unit 102 is connected to a camera (not shown in FIG. 2) and controls imaging performed by the camera. The imaging control unit 102 acquires image data from the camera. The audio control unit 104 is connected to a microphone (not shown in FIG. 2) and controls audio input from the microphone. The audio control unit 104 acquires the audio data from the microphone. The video transmission unit 106 transmits the image data acquired by the imaging control unit 102 and the video data including the audio data acquired by the audio control unit 104 to the server 10 via the network NW. The video data is transmitted in real time by the video transmission unit 106. That is, the generation of the video data by the imaging control unit 102 and the audio control unit 104 and the transmission of the generated video data by the video transmission unit 106 are performed substantially simultaneously. The distributor's UI control unit 108 controls the UI (user interface) of the distributor. The distributor's UI control unit 108 may be connected to a display (not shown in FIG. 2) and displays a video on the display by playing the video data transmitted by the video transmission unit 106. The distributor's UI control unit 108 may display an operation object or an instruction permission object on the display and accept an input from the distributor who taps on the object.

[0021] The viewing unit 200 includes a viewer-side UI control unit 202, an overlay information generating unit 204, and an input information transmitting unit 206. The viewing unit 200 receives video data related to live streaming in which the distributor, the viewer who is the user of the user terminal 30, and other viewers participate from the server 10 via the network NW. The viewer-side UI control unit 202 controls the UI of the viewer. The viewer-side UI control unit 202 is connected to a display and a speaker (not shown in FIG. 2), plays the received video data, displays the video on the display, and outputs the audio from the speaker. The state in which the video is output to the display and the audio is output from the speaker can be called a "video data is being played" state. The viewer-side UI control unit 202 is also connected to an input means (not shown in FIG. 2) such as a touch panel, a keyboard, a display, etc., and acquires a user's input via the input means. The overlay information generating unit 204 overlays a predetermined frame image on an image generated from video data from the server 10. The frame image includes various user interface objects (hereinafter simply referred to as "objects") for receiving input from the user, comments entered by the viewer, and / or information acquired from the server 10. The input information transmitting unit 206 transmits the user input acquired by the viewer-side UI control unit 202 to the server 10 via the network NW.

[0022] 3 is a block diagram showing the functions and configurations of the server 10 of FIG. 1 according to some embodiments of the present invention. The server 10 includes a distribution information providing unit 302, a relay unit 304, a gift processing unit 306, a payment processing unit 308, a stream DB 310, a user DB 312, a gift DB 314, a video generating unit 320, a voice generating unit 322, a character DB 324, a language model DB 326, an acquisition unit 330, a processing unit 332, a determination unit 334, a sending unit 336, a comment DB 338, and a response DB 340.

[0023] When a notification or request to start live streaming is received from the user terminal 20 on the distributor side via the network NW, the distribution information providing unit 302 registers in the stream DB 310 a stream ID for identifying the live streaming and the distributor ID of the distributor performing the live streaming.

[0024] When the distribution information providing unit 302 receives a request for providing information about live streams from the viewing unit 200 of the user terminal 30 on the viewer side via the network NW, the distribution information providing unit 302 obtains or checks currently available live streams from the stream DB 310, and creates a list of available live streams. The distribution information providing unit 302 transmits the created list to the requesting user terminal 30 via the network NW. The viewer side UI control unit 202 of the requesting user terminal 30 generates a live stream selection screen based on the received list, and displays it on the display of the user terminal 30.

[0025] When the input information sending unit 206 of the user terminal 30 receives the selection result of the viewer on the live stream selection screen, the input information sending unit 206 generates a delivery request including the stream ID of the selected live stream and sends the request to the server 10 via the network NW. The delivery information providing unit 302 starts providing the live stream specified by the stream ID included in the received delivery request to the requesting user terminal 30. The delivery information providing unit 302 updates the stream DB 310 so that the viewer ID of (or the corresponding) the stream ID includes the user ID of the viewer of the requesting user terminal 30.

[0026] The relay unit 304 relays the video data from the user terminal 20 on the distributor side to the user terminal 30 on the viewer side in the live streaming started by the distribution information providing unit 302. The relay unit 304 receives a signal representing a user input by a viewer from the input information transmitting unit 206 during the live streaming or during playback of the video data. The signal representing the user input may be an object designation signal that designates an object displayed on the display of the user terminal 30. The object designation signal may include a viewer ID of the viewer, a distributor ID of the distributor of the live stream that the viewer is viewing, and an object ID that identifies the object. When the object is a gift, the object ID is a gift ID. Similarly, the relay unit 304 receives a signal representing a user input made by a distributor during playback (or live streaming) of the video data from the distribution unit 100 of the user terminal 20. The signal may be an object designation signal.

[0027] Furthermore, the signal representing the user input may be a comment input signal including a comment input by a viewer to the user terminal 30 and the viewer ID of the viewer. Upon receiving the comment input signal, the relay unit 304 transmits the comment and the viewer ID included in the signal to the user terminal 20 of the distributor and the user terminals 30 of other viewers. In these user terminals 20, 30, the viewer-side UI control unit 202 and the overlay information generation unit 204 display the received comment on a display associated with the viewer ID that has also been received.

[0028] The gift processing unit 306 increases the points of the distributor based on the points of the gift specified by the gift ID included in the object designation signal, and updates the user DB 312. Specifically, the gift processing unit 306 refers to the gift DB 314 to specify the points to be given to the gift ID included in the received object designation signal. Then, the gift processing unit 306 updates the user DB 312 and adds the specified points to the points of the distributor ID (or corresponding to the distributor ID) included in the object designation signal.

[0029] In response to receiving the object designation signal, the payment processing unit 308 processes the payment of the gift from the viewer. Specifically, the payment processing unit 308 refers to the gift DB 314 to determine the price points of the gift identified by the gift ID included in the object designation signal. Then, the payment processing unit 308 updates the user DB 312 and subtracts the determined price points from the points of the viewer identified by the viewer ID included in the object designation signal.

[0030] FIG. 4 is a table showing an exemplary data structure of the stream DB 310 of FIG. 3. The stream DB 310 holds information about a live stream currently being performed. The stream DB 310 stores a stream ID, a broadcaster ID, and a viewer ID in a mutually associated manner. The stream ID is an ID for identifying a live stream in a live streaming platform provided by the live streaming system 1. The broadcaster ID is a user ID for identifying a broadcaster providing the live stream. The viewer ID is a user ID for identifying a viewer of the live stream. In a live streaming platform provided by the live streaming system 1 according to some embodiments, when a user starts a live stream, the user becomes a broadcaster, and when the same user watches a live stream broadcasted by another user, the user also becomes a viewer. Therefore, the distinction between a broadcaster and a viewer is not fixed, and a user ID registered as a broadcaster ID at one time may be registered as a viewer ID at another time.

[0031] FIG. 5 is a table showing an exemplary data structure of the user DB 312 of FIG. 3. The user DB 312 holds information about users. The user DB 312 stores the user ID and the points in a mutually associated manner. The user ID identifies a user. The points correspond to points held by the corresponding user. The points are electronic values ​​circulating within the live streaming platform. In some embodiments, when a broadcaster receives a gift from a viewer during a live broadcast, the broadcaster's points increase by a value corresponding to the gift. The points are used, for example, to determine the amount of compensation (such as money) that the broadcaster receives from the administrator of the live streaming platform. In some embodiments, when the broadcaster receives a gift from a viewer, an amount corresponding to the gift may be given instead of the points.

[0032] FIG. 6 is a table showing an exemplary data structure of the gift DB 314 of FIG. 3. The gift DB 314 holds information about gifts available to the viewer during the live streaming. Gifts are electronic data. Gifts may be purchased with points or money, or may be provided free of charge. Gifts may be given to the broadcaster by the viewer. Giving a gift to the broadcaster is also called using a gift, sending a gift, throwing a gift, etc. Some gifts can be used immediately after purchase, while others can be used at any time after purchase by the viewer who purchased the gift. When a viewer gives a gift to a broadcaster, the broadcaster is given an amount of points corresponding to the gift. When a gift is used, an effect related to the gift may occur as a result of the use. For example, an effect (such as a visual effect or an auditory effect) corresponding to the gift is displayed on the live streaming screen.

[0033] The gift DB 314 stores a gift ID, an assignment point, and a price point in association with each other. The gift ID is for identifying a gift. The assignment point is the amount of points that is assigned to a broadcaster when the gift is given to the broadcaster. The price point is the amount of points paid for the use (purchase) of the gift. A viewer can give a desired gift to a broadcaster by paying the price points of the desired gift while watching a live stream. The payment of the price points can be made by an appropriate electronic payment means. For example, the viewer may pay the price points to the administrator. Alternatively, payment by bank transfer or credit card may be used. The administrator can arbitrarily set the relationship between the assignment points and the price points. For example, it may be set as assignment points = price points. Alternatively, it may be set as the price points obtained by multiplying the assignment points by a predetermined coefficient such as 1.2, or as the price points obtained by adding a predetermined fee point to the assignment points.

[0034] 7 is a table showing an exemplary data structure of the character DB 324. The character DB 324 stores character IDs, status tags, video data, and audio data in association with each other.

[0035] The character ID identifies the virtual character. A plurality of characters may be stored in the character DB 324.

[0036] The status tag indicates the status (or mood) of the character. For example, there may be a general (calm, collected, etc.) status, a positive (happy, excited, etc.) status, or a negative (sad, bored, etc.) status.

[0037] The video data stores video sources (or video data) corresponding to different status tags of the character. For example, URL1v is a URL corresponding to a video source of the character CR1 with a general status. URL2v is a URL corresponding to a video source of the character CR1 with a positive status.

[0038] The voice data stores audio sources (or voice data, or audio texture data, or intonation data) corresponding to different status tags of the character. For example, URL1a is a URL corresponding to an audio source of character CR1 with a general status. URL2a is a URL corresponding to an audio source of character CR1 with a positive status.

[0039] The video generation unit 320 is configured to generate streaming video data based on the character video data from the character DB 324. For example, the video generation unit 320 may combine the video of the character with a background image (or video) to generate the streaming video data. For example, the video generation unit 320 may combine the video of the character with live video of a real human streamer to generate the streaming video data.

[0040] In some embodiments, the image generation unit 320 may refer to the response content of the character (in the response DB 340) and adjust the image data of the character based on the response content. For example, the image generation unit 320 may adjust the facial expression and mouth movement of the character according to the response content so that the character speaks in a manner similar to a real human. In some embodiments, the image generation unit 320 may select the image data in the character DB 324 based on the status tag in the character DB 324 that corresponds to the response status of the response content in the response DB 340.

[0041] The voice generation unit 322 is configured to generate streaming voice data based on the character voice data from the character DB 324. In some embodiments, the voice generation unit 322 may select the voice data in the character DB 324 based on the status tag in the character DB 324 that corresponds to a response status of a response content in the response DB 340. The voice generation unit 322 converts the response content (which is text data) in the response DB 340 into the streaming voice data using a text-to-speech (TTS) function of the voice data and voice.

[0042] 8 is a table showing an exemplary data structure of the comment DB 338. The comment DB 338 stores the timing of a comment, a viewer ID, a comment content, a comment length, a time length since the last chat, a topic similarity score, a viewer attribute score, and a priority score in association with each other.

[0043] The comment timing may be the timing of acquiring the comment. The ID identifies the viewer who made the comment. The comment content stores the content of the comment. The comment length is the length of the comment. In this embodiment, the number of words is used as the comment length. In some embodiments, various calculation methods may be used for the comment length, such as using the total number of characters in the comment.

[0044] "Length since last chat" identifies how close or recent the viewer who made the comment was to the character last chatting (or last responding). In some embodiments, the "Length since last chat" is the length of time between the current timing and the timing of the last chat from the character to the viewer who made the comment. The calculation is within the same live stream. Thus, for comments by viewers that the character has not yet spoken to (or responded to) in the stream, the value for that comment will be "NA" (not applicable) or zero (in some embodiments).

[0045] For example, the value is "NA" because the character has not responded to viewer V3 prior to the comment "OK, I'll send you a rocket" made by viewer V3. For example, there are 6 seconds between the current timing and the last chat from the character to viewer V1 who commented "How do you want to send a rocket?" The last chat may have been a response made by the character between t1 and t4 to the comment "What's your favorite gift?" from viewer V1. The length of time since the last chat may continue to change over time.

[0046] In some embodiments, the length of time since last chat is the length of time between the time of capture of the comment and the timing of the last chat from the character to the viewer who made the comment, in which case the value does not change over time.

[0047] The topic similarity score specifies the similarity of the comment to the current topic from the character's perspective, and its details and calculation are described below.

[0048] The viewer attribute score identifies an attribute of the viewer making the comment. In this embodiment, the viewer attribute score is a contribution score of the viewer with respect to the character. The contribution score may identify how much the viewer contributed to the character. The contribution score may be a total number of gifts or a total value of gifts made by the viewer to the character. In some embodiments, the viewer attribute score may be a contribution score of the viewer with respect to other contributors or other characters. In some embodiments, the viewer attribute score may be points (or billing points) for the viewer on the streaming platform.

[0049] The priority score specifies the priority of responding to a comment from the character's point of view.

[0050] In some embodiments, the priority score increases the more recent the comment is, a mechanism that makes the character resemble a real human broadcaster who tends to remember and respond to recent comments.

[0051] In some embodiments, the priority score increases as the comment length increases, a mechanism that makes the character resemble a real human broadcaster who is attracted to and tends to respond to longer comments.

[0052] In some embodiments, the priority score increases the shorter the "length of time since last chat." A shorter "length of time since last chat" indicates a shorter length of time between the current moment and the last time the character spoke or responded to the viewer who made the comment. This mechanism makes the character more similar to a real human streamer who tends to respond first to viewers who they recently spoke with. In some embodiments, a shorter "length of time since last chat" means that the viewer and the character are "on the same page."

[0053] In some embodiments, the priority score increases as the "topic similarity score" becomes higher. A higher topic similarity score may indicate that a comment is more in line with the current topic of the character's "thinking". This mechanism makes the character more similar to a real human broadcaster, who tends to respond first to comments that are close to what the broadcaster is thinking. More details are provided below.

[0054] In some embodiments, the priority score increases as the "viewer attribute score" increases (e.g., viewer V2). For example, the priority score may be higher for comments made by viewers who have contributed more to the character. In some embodiments, the priority score may be higher for comments made by viewers who have contributed more to other streamers. In some embodiments, the priority score may be higher for comments made by viewers with high billing points. This mechanism makes the character resemble a real human streamer, who tends to respond first to comments made by viewers with higher potential contributions to the streamer.

[0055] An example showing the priority score determination of a comment is shown in Figure 9. As shown in the figure, the score "S" is determined based on the parameters TL, CL, TS, TC, and CS.

[0056] TL is determined by the current timing, “Tcurrent”, and the timing of receiving the comment, “Treceived”. As shown in the figure, the larger the TL, the more recent the comment.

[0057] CL is determined by the comment length. For example, "len(comment)" may be the number of words or characters in the comment. In this example, a comment with a length greater than 50 and a comment with a length of 50 have the same CL value.

[0058] TS indicates a topic similarity score. The higher the TS, the more similar (or relevant) the comment is to the topic that the character is currently "thinking" about. The calculation is explained below.

[0059] TC is determined by the current timing, "Tcurrent," and the timing, "Tlast," of when the character last spoke with the commenter. As shown, the more recently the character responded to the commenter, the higher the TC value. In some embodiments, TC is the inverse of "time since last chat."

[0060] CS is the contribution score of the commenting viewer.

[0061] The weight values ​​w1, w2, w3, w4, and w5 may be determined by a streaming platform operator according to actual operation. The operator may assign higher weight values ​​to factors that the operator considers more important. In some embodiments, the calculation of the priority score determination may be performed by the processing unit 332.

[0062] 10 is a table showing an exemplary data structure of the comment DB 338. The comment DB 338 stores timing, response content (or chat content), simulation comment, simulation comment vector, viewer comment, viewer comment vector, and topic similarity score in association with each other.

[0063] The response contents are responses (or chats / responses / comments) from the character at different times. The response contents may be accessed from the response DB 340.

[0064] The simulated comments may be referred to as expected comments, which are potential (or possible) comments from a simulated audience regarding the character's response (or the character's chat). The simulated comments may be generated by inputting the content of the character's response into a language model (such as ChatGPT). The language model may be included in the language model DB 326. In some embodiments, the language model may be implemented separately from the language model DB 326 and may be much lighter than the language model DB 326. The language model then generates a simulated comment for the response. For example, in the embodiment shown in FIG. 10, a simulated comment of "Why are you in a bad mood?" is generated in relation to the character's chat of "I'm in a bad mood today." In some embodiments, the execution of the simulated comment generation may be performed by the processing unit 332.

[0065] The simulation comment vectors are generated by inputting the simulation comments into a text-to-vector model (or converter). In some embodiments, the model may be or include a Bidirectional Encoder Representations from Transformers (BERT) or Sentence-BERT (SBERT) word embedding model. The model may be included in the language model DB 326. In some embodiments, the model may be different from the language model DB 326 and may be implemented outside the server 10. In some embodiments, the conversion process may be performed by the processing unit 332.

[0066] The viewer comments store comments from viewers at different times.

[0067] The viewer comment vector is generated by inputting the viewer comments into a text-to-vector model (or converter). In some embodiments, the conversion process may be performed by the processing unit 332.

[0068] The topic similarity score indicates how similar the viewer comment is to the simulation comment (or the most recent simulation comment). Various methods can be used to calculate the similarity score. For example, a dot product between the viewer comment and the simulation comment vector may be performed. For example, a correlation coefficient between the viewer comment and the simulation comment vector may be calculated. In this example, the viewer comment "Why are you in a bad mood?" corresponds to the highest topic similarity score because it is the most similar comment to the simulation comment "Why are you in a bad mood?". In some embodiments, the calculation may be performed by the processing unit 332. In some embodiments, a higher topic similarity score indicates that the comment (e.g., "Why are you in a bad mood?") is more related to the character's last chat / response (e.g., "I'm in a bad mood today").

[0069] For example, in the embodiment shown in Figure 8, the comment "OK, I'll send you a rocket" has the highest topic similarity score. This may be because the character responds with a response such as "I like rockets" (see Figure 15) in response to the comment "What's your favorite gift?". A simulated comment such as "I'll send you a rocket" (not shown) is generated in response, and subsequent viewer comments that are more relevant to the simulated comment have a higher topic similarity score.

[0070] FIG. 11 illustrates an exemplary flow chart according to some embodiments of the present invention.

[0071] In step S1100, the determining unit 334 determines a representative response (which may be a previous chat) from the character that represents the current topic. In some embodiments, the determining unit 334 may select the most recent response to represent the current topic. In some embodiments, the determining unit 334 utilizes the language model DB 326 to determine which response (e.g., among the last five responses) is most representative of the current topic. In some embodiments, a rule-based method (such as keyword matching) may be used to select the representative response.

[0072] In step S1102, the processing unit 332 generates simulated comments (or simulated viewer comments) about the representative responses.

[0073] In step S1104, the processing unit 332 generates a simulation comment vector Vsim from the simulation comments.

[0074] In step S1106, the determining unit 334 determines subsequent viewer comments (comments obtained from the viewer after the representative response).

[0075] In step S1108, the processing unit 332 generates viewer comment vectors Vv1, Vv2, . . . from these subsequent viewer comments.

[0076] In step S1110, the processing unit 332 calculates the similarity score for each of the viewer comments to the simulation comment, which is then stored in the comment DB 338.

[0077] 12 is a table showing an exemplary data structure of the response DB 340. The response DB 340 stores priority scores, viewer comment contents, response contents, response statuses, response video data, and response audio data in mutually associated manner.

[0078] The priority score and the viewer comment content may be obtained from the comment DB 338 .

[0079] The response content is a response (or chat) of the character to the viewer comment, and may be generated by inputting the comment into the language model DB 326. The language model DB 326 then generates the response content for the comment. The response status is a status (or mood) corresponding to the response content, and may be generated simultaneously with the response content. For example, the language model DB 326 may be used to analyze the response content and provide the corresponding response status when generating the response content. The above process may be executed by the processing unit 332, for example.

[0080] The response video data is stream video data generated by the video generation unit 320 based on the response content and the response status. The response video data includes the visual reaction (or visual response) of the character to the corresponding viewer comment. The response audio data is stream audio data generated by the audio generation unit 322 based on the response content and the response status. The response audio data includes the audio reaction (or audio response) of the character to the corresponding viewer comment.

[0081] The language model DB 326 may include one or more large-scale language models (LLMs), such as Generative Pre-trained Transformer (GPT), Language Model for Many Applications (LLaMA), BigScience Large Open-science Open-access Multilingual Language Model (BLOOM), etc. Other language models may also be implemented in the language model DB 326.

[0082] The acquisition unit 330 may be configured to acquire viewer comments and acquire / extract parameters of the viewer comments, such as the comment timing, the viewer ID of the comment, the comment content, the comment length, the “time since last chat”, the topic similarity score, and the viewer attribute score.

[0083] The determining unit 334 may be configured to select (or determine) the comment to respond to based on the priority score of the comment. The determining unit 334 may determine to respond to the comment with a higher priority score first. For example, when determining that a first priority score of a first comment is higher than a second priority score of a second comment, the determining unit 334 then determines to send a first response to the first comment and then send a second response to the second comment.

[0084] The sending unit 336 is configured to send the response video data generated by the video generation unit 320 and the response audio data generated by the audio generation unit 322 to the user terminals of the viewers in the character's live stream. In some embodiments, the distribution information providing unit 302 or the relay unit 304 may be involved in the transmission.

[0085] FIG. 13 illustrates an exemplary flow chart according to some embodiments of the present invention.

[0086] In step S1300, the capture unit 330 captures comments from viewers in the character's live stream.

[0087] In step S1302, the obtaining unit 330 extracts parameters of the comment, such as parameters in the comment DB 338.

[0088] In step S1304, the processing unit 332 calculates the priority score of each comment based on the parameters of each comment, and stores the priority score in the comment DB 338.

[0089] In step S1306, the determining unit 334 determines the order in which to respond to the comments based on the priority scores.

[0090] In step S1308, the processing unit 332 generates the response content for the comment to which the response is made. Then, the response video data and the response audio data are generated based on the response content. The response video data and the response audio data may be called response data.

[0091] In step S1310, the responsive video data and the responsive audio data are transmitted to the viewer in the character's stream.

[0092] In some embodiments, before or after generating the response data, the determining unit 334 may determine whether the viewer to whom the response data is to be sent is still in the streaming room. For example, before generating the response data, if the determining unit 334 detects that the viewer has already left the streaming room, the determining unit 334 may decide not to generate the response data to save processing resources. For example, after generating the response data but before sending the response data, if the determining unit 334 detects that the viewer has already left the streaming room, the determining unit 334 may decide not to send the response data to save processing resources.

[0093] In some embodiments, if the priority score of a comment is lower than a threshold, the determining unit 334 may decide to ignore the comment and not respond.

[0094] In some embodiments, the processing unit 332 may utilize the language model DB 326 to detect whether a comment contains harmful intent, and the determining unit 334 may then decide to ignore the comment with harmful intent, so that the harmful topic will not continue in the streaming room.

[0095] The present disclosure allows the character to respond to comments in the same way as a real human broadcaster. The present disclosure may be implemented in situations other than live streaming. For example, the functionality of the present disclosure may be implemented in non-live streaming platforms, answering services on e-commerce sites, automated messaging services, any automated customer service system, etc.

[0096] FIG. 14 illustrates an example of an exemplary AI VLiver system according to some embodiments of the present invention.

[0097] As shown in the figure, the system includes an AI VLiver server, a skin module, a voice synthesis module, an emotion module, a memory module, and a reasoning module.

[0098] The emotion module is configured to provide emotions to the character, which may be influenced by daily news and accumulated information. The memory module is configured to provide memories to the character, and includes a short-term memory module and a long-term memory module.

[0099] The reasoning module is configured to generate a response for the character and includes a re-ranking module, a reasoning system, a moderator for practical adjustments, and an LLM model. The re-ranking module may be referred to as a reordering module and may use the present disclosure to determine an order in which to respond to comments from viewers. The reasoning system and the LLM model may be used to generate the response content.

[0100] An exemplary virtual character according to some embodiments of the present invention is shown in FIG 15. The conversation between the viewer and the character is similar to the embodiment shown in FIG 8.

[0101] By the time t1, the character acquires comments from viewers V3 and V1.

[0102] At time t2, the character responds to viewer V1 and ignores the comment from viewer V3 because the priority score of the comment "Hello" is too low.

[0103] At time t4, the character gets new comments from viewers V3 and V1. The character responds to viewer V3 because the comment "OK, let's send a rocket" is more similar to the comment expected in the character's mind.

[0104] The hardware configuration of the information processing device will be described with reference to Fig. 16. Fig. 16 is a block diagram showing an example of the hardware configuration of an information processing device based on some embodiments of the present invention. The information processing device 900 shown in the figure can realize, for example, the server 10 and / or the user terminals 20 and 30 in some embodiments.

[0105] The information processing device 900 includes a CPU 901, a ROM (read only memory) 903, and a RAM (random access memory) 905. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 925, and a communication device 929. The information processing device 900 further includes an imaging device such as a camera (not shown). The information processing device 900 may also include a DSP (digital signal processor) or an ASIC (application specific integrated circuit) in addition to or instead of the CPU 901.

[0106] The CPU 901 functions as an arithmetic processing device and a control device, and controls all or part of the operation of the information processing device 900 according to various programs stored in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 923. For example, the CPU 901 controls the overall operation of each functional unit included in the server 10 and the user terminals 20 and 30 in some embodiments. The ROM 903 stores programs and arithmetic parameters used by the CPU 901. The RAM 905 functions as a main storage device that stores programs used when the CPU 901 is executed, parameters that change appropriately during execution, and the like. The CPU 901, the ROM 903, and the RAM 905 are interconnected by a host bus 907, which may be an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.

[0107] The input device 915 may be a device operated by a user, such as a mouse, a keyboard, a touch panel, a button, a switch, or a lever, or a device that converts a physical quantity into an electric signal, such as a sound sensor represented by a microphone, an acceleration sensor, a tilt sensor, an infrared sensor, a depth sensor, a temperature sensor, or a humidity sensor. The input device 915 may be, for example, a remote control device that uses radio waves such as infrared rays, or an external connection device 927 such as a mobile phone that supports the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user or a detected physical quantity and outputs the input signal to the CPU 901. The user operates the input device 915 to input various data to the information processing device 900 and instruct the information processing device 900 to operate.

[0108] The output device 917 is a device that can visually or audibly notify the user of acquired information. The output device 917 may be, for example, a display such as an LCD, a PDP, or an OLED, an audio output device such as a speaker or a headphone, or a printer. The output device 917 outputs the processing result by the information processing device 900 as a video such as text or an image, or an audio such as a voice.

[0109] The storage device 919 is a data storage device and is configured as an example of a storage unit of the information processing device 900. The storage device 919 is, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores the programs executed by the CPU 901, various data, various data acquired from the outside, and the like.

[0110] The drive 921 is a reader / writer of a removable recording medium 923 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built into or externally attached to the information processing device 900. The drive 921 reads out information recorded on the attached removable recording medium 923, and outputs it to the RAM 905. Furthermore, the drive 921 writes information to the attached removable recording medium 923.

[0111] The connection port 925 is a port for directly connecting a device to the information processing device 900. The connection port 925 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. Furthermore, the connection port 925 may be an RS-232C port, an optical audio terminal, an HDMI (High Definition Multimedia Interface) port, or the like. By connecting the external connection device 927 to the connection port 925, various data can be exchanged between the information processing device 900 and the external connection device 927.

[0112] The communication device 929 is, for example, a communication interface formed by a communication device for connecting to the network NW. The communication device 929 may be, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (trademark), or wireless USB (WUSB). Furthermore, the communication device 929 may be a router for optical communication, a router for ADSL (asymmetric digital subscriber line), or a modem for various communications. The communication device 929 transmits and receives signals, etc., on the Internet or between other communication devices using a predetermined protocol such as TCP / IP. The communication network NW connected to the communication device 929 is a network connected by wire or wirelessly, for example, the Internet, a home LAN, infrared communication, radio wave communication, satellite communication, etc. The communication device 929 realizes a function as a communication device.

[0113] The imaging device (not shown) is a device that captures an image of real space using an imaging element such as a CCD (charge-coupled device) or a CMOS (complementary metal-oxide semiconductor) and various members such as a lens for controlling the formation of a subject image on the imaging element, and generates a captured image. The imaging device may capture still images or moving images.

[0114] The configuration and operation of the live streaming system 1 according to this embodiment have been described above. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible with respect to the combination of each component and each process, and that such modifications are also within the scope of the present invention.

[0115] The processes and procedures described in the present invention can be realized by software, hardware, or any combination thereof, in addition to those explicitly described. For example, the processes and procedures described herein can be realized by implementing logic corresponding to the processes and procedures in a medium such as an integrated circuit, a volatile memory, a non-volatile memory, a non-transitory computer-readable medium, a magnetic disk, etc. Furthermore, the processes and procedures described herein can be realized as a computer program corresponding to the processes and procedures, and can be executed by various computers.

[0116] Furthermore, the system or method described in the above embodiments may be integrated into a program stored in a non-transitory computer-readable medium such as a solid-state storage device, an optical disk storage device, a magnetic disk storage device, etc. Alternatively, the program may be downloaded from a server via the Internet and executed by a processor.

[0117] Although the technical contents and features of the present invention have been described above, those skilled in the art can still make many variations and modifications without departing from the teachings and disclosure of the present invention. Therefore, the scope of the present invention is not limited to the embodiments already disclosed, but is within the scope of the claims, including other variations and modifications that do not depart from the present invention. [Explanation of symbols]

[0118] 1. Communication Systems 10 Server 20 User terminal 30, 30a, 30b User terminal LV Streamer AU1, AU2 viewers VD, VD1, VD2 Video NW Network 30 User terminals 100 Delivery Units 102 Imaging control unit 104 Audio control unit 106 Video transmission unit 108 Provider UI Control Unit 200 viewing units 202 Viewer side UI control unit 204 Superposition Information Generation Unit 206 Input information transmission unit 302 Distribution Information Unit 304 Relay unit 306 Gift Processing Unit 308 Payment Processing Unit 310 Stream DB 312 User DB 314 Gift DB 320 Image Generation Unit 322 Speech Generation Unit 324 Character DB 326 Language Model DB 330 Acquired Units 332 Processing Unit 334 Judgment Unit 336 Transmitting Unit 338 Comments DB 340 Response DB 900 Information processing device 901 CPU 903 ROM 905 RAM 907 Host Bus 909 Bridge 911 External Bus 913 Interface 915 Input Device 917 Output Device 919 Storage Device 921 Drive 923 Removable Recording Media 925 Connection Port 927 External connection device 929 Communication Equipment

Claims

1. 1. A method for responding to comments implemented by a server, comprising: obtaining a first comment; obtaining a second comment; obtaining parameters of the first comment; obtaining parameters of the second comment; determining a first priority score for the first comment based on the parameters of the first comment; determining a second priority score for the second comment based on the parameters of the second comment; selecting the first comment or the second comment for response based on the first priority score and the second priority score; 16. A method of responding to a comment, comprising:

2. further determining that the first priority score is higher than the second priority score; sending a first response to the first comment and then a second response to the second comment; 2. The method of claim 1, comprising:

3. 2. The method of responding to comments as described in claim 1, wherein the parameters of the first comment include a first comment length, and the first priority score is determined to be higher when the first comment length is longer.

4. 2. The method of responding to comments as described in claim 1, characterized in that the parameters of the first comment include a time length between a current timing and a last timing of a last response to a first viewer who made the first comment, and the first priority score is determined to be higher when the time length is shorter.

5. 2. The method of responding to comments as described in claim 1, characterized in that the parameters of the first comment include a topic similarity score of the first comment, the first priority score is determined to be higher when the topic similarity score is high, the topic similarity score increases when the first comment is more relevant to a last response, and the last response is sent before retrieving the first comment.

6. further comprising generating a simulation comment regarding said final response; calculating a first correlation coefficient between the simulation comments and the first comments; determining the topic similarity score to be higher when the first correlation coefficient is larger; 6. A method for responding to comments as recited in claim 5, comprising:

7. 7. The method of claim 6, wherein the step of generating the simulation comment regarding the final response includes inputting the final response into an LLM model.

8. 2. The method of responding to comments as described in claim 1, wherein the method is for a virtual character to respond to the comment in a live stream, and the first comment and the second comment are from different viewers in the live stream.

9. 1. A system for responding to comments, comprising: one or more processors, the one or more processors executing machine-readable instructions to: obtaining a first comment; obtaining a second comment; obtaining parameters of the first comment; obtaining parameters of the second comment; determining a first priority score for the first comment based on the parameters of the first comment; determining a second priority score for the second comment based on the parameters of the second comment; selecting the first comment or the second comment for response based on the first priority score and the second priority score; A system for responding to comments, comprising:

10. 1. A non-transitory computer readable medium containing a program for responding to comments, the program being configured to: obtaining a first comment; obtaining a second comment; obtaining parameters of the first comment; obtaining parameters of the second comment; determining a first priority score for the first comment based on the parameters of the first comment; determining a second priority score for the second comment based on the parameters of the second comment; selecting the first comment or the second comment for response based on the first priority score and the second priority score; A non-transitory computer-readable medium having a program for executing the program.

Images