Server, method, and computer program

Abstract

To provide a server, a method, and a computer program for improving communication between a viewer and an AI virtual live streamer.SOLUTION: In a live streaming system, an application startup process performed by a server includes: a step of generating a virtual chatbot via a machine learning model; a step of setting the virtual chatbot in a live streaming room; a step S502 of receiving comments from users within the live streaming room; a step S508 of storing, in response to detection of a keyword associated with a user in the comments, YES at a step S504, the keyword into a first database; and a step of supplying information on the first database to the machine learning model. The keyword is related to information on the user.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present disclosure relates to information and communication technology, and in particular to a server, a method, and a computer program for live streaming. [Background technology]

[0002] Some apps and platforms offer live streaming services that allow live streamers and viewers to interact with each other, such as by making performances to support their viewers, or by donating or sending gifts to support the live streamer.

[0003] With the advancement of technology, AI models are now being applied to live streaming, giving birth to AI virtual live streamers. Patent Document 1 discloses a method for processing interactions with an AI virtual live streamer from viewers.

[0004] However, viewers may not be attracted to interactions from AI virtual live streamers who do not behave like humans. This may lead to a poor user experience. Therefore, how to improve the user experience is important. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US Patent Publication No. 20230061778 Summary of the Invention

[0006] A server according to one embodiment of the present disclosure includes a circuit configured to perform the steps of generating a virtual chatbot through a machine learning model, setting the virtual chatbot in a live streaming room, receiving comments from a user in the live streaming room, and in response to a keyword associated with the user being detected from the comment, storing the keyword in a first database, and providing information of the first database to the machine learning model, wherein the keyword relates to information about the user.

[0007] Another embodiment of the present disclosure relates to a method for providing a live stream on a live streaming platform, the method including: generating a virtual chatbot via a machine learning model; setting the virtual chatbot in a live streaming room; receiving comments from a user in the live streaming room; and in response to detecting a keyword associated with the user from the comment, storing the keyword in a first database; and providing information of the first database to the machine learning model, the keyword relating to information about the user.

[0008] Yet another embodiment of the present disclosure relates to a computer program that causes a server to perform the following functions: generating a virtual chatbot through a machine learning model; setting the virtual chatbot in a live streaming room; receiving comments from a user in the live streaming room; and in response to detecting a keyword associated with the user from the comment, storing the keyword in a first database; and providing information of the first database to the machine learning model, wherein the keyword is related to information about the user.

[0009] According to the present disclosure, the communication between viewers and AI virtual live streamers may be improved. Furthermore, the quality of the live streaming platform on which the AI ​​virtual live streamers are present may also be improved. Therefore, the user experience may also be improved. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic diagram showing a configuration of a live streaming system 1 based on some embodiments of the present disclosure. [Diagram 2] FIG. 2 is a block diagram of a user terminal 20 according to some embodiments of the present disclosure. [Diagram 3] FIG. 1 is a block diagram of a server 10 according to some embodiments of the present disclosure. [Figure 4] 4 is a table showing an example data structure of the stream DB 320 of FIG. 3. [Diagram 5] 4 is a table showing an example data structure of the user DB 322 of FIG. 3. [Figure 6] 4 is a table showing an example data structure of the virtual liver DB 324 of FIG. 3. [Figure 7] 4 is a table illustrating an example data structure of the short-term memory DB 326 of FIG. 3. [Figure 8] 4 is a table illustrating an example data structure of the long-term storage DB 328 of FIG. 3. [Figure 9] 4 is a table illustrating an example data structure for the sentiment analysis model 330 of FIG. 3. [Figure 10] 4 is a table illustrating an example data structure for the action lookup table 332 of FIG. 3. [Figure 11] 1 illustrates an exemplary functional structure of a live streaming system 1 according to some embodiments of the present disclosure. [Figure 12] 6 is a simplified functional structure of a live streaming system 1 with an exemplary screen image of a live streaming room screen 600 according to some embodiments of the present disclosure. [Figure 13]1 is a flowchart showing steps of an application startup process in a live streaming system 1 according to some embodiments of the present disclosure. [Figure 14] 1 illustrates an exemplary hardware configuration of an information processing device based on some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the same or similar components, parts, steps, or signals shown in each drawing will be denoted by the same reference numerals in all drawings, and duplicated explanations will be omitted as appropriate. In addition, some parts that are not important in the explanation of each drawing will be omitted.

[0012] The live streaming system 1 according to some embodiments of the present disclosure provides enhanced functionality to facilitate communication and interaction between users, and more specifically, to entertain viewers and streamers in a technological manner.

[0013] FIG. 1 is a schematic diagram showing a configuration of a live streaming system 1 according to some embodiments of the present disclosure. The live streaming system 1 provides a live streaming service for streaming streamers (also called live distributors or streamers) LV and viewers (also called audiences) AU (AU1, AU2, ...) to interact with each other in real time. As shown in FIG. 1, the live streaming system 1 may include a server 10, a user terminal 20, and a user terminal 30 (30a, 30b, ...). The user terminal 20 may be a streamer, and the user terminal 30 may be a viewer. In some embodiments, the streamer and the viewer may be called users. The server 10 may include one or more information processing devices connected via a network NW. The user terminals 20 and 30 may be, for example, mobile terminals such as smartphones, tablets, notebook PCs, recorders, portable game consoles, and wearable terminals, or stationary computers such as desktop PCs. The server 10, the user terminals 20, and the user terminals 30 may be communicably connected by any type of wired or wireless network NW.

[0014] The live streaming system 1 involves the streamer LV, the viewer AU, and an app provider (not shown) that provides the server 10. The streamer LV can record content such as his / her own singing, talking, performance, game streaming, etc. on his / her user terminal 20 and upload it to the server 10, thereby distributing the content in real time. In some embodiments, the streamer LV can interact with the viewer AU through the live streaming.

[0015] The app provider may provide a platform for content to be live streamed on the server 10. In some embodiments, the app provider may be a media or manager that manages real-time communication between the streamer LV and the viewer AU. The viewer AU may access the platform through the user terminal 30 and select and view the content that the viewer AU wants to view. The viewer AU may perform operations to interact with the streamer, such as commenting on the streamer or cheering on the streamer, through the user terminal 30. The streamer providing the content may respond to the comments or cheers. The streamer's response may be transmitted to the viewer AU by video and / or audio. Thus, mutual communication between the streamer and the viewer may be achieved.

[0016] The term "live streaming" as used herein may refer to data transmission that allows the viewer AU to substantially play and view the content recorded by the streamer LV through the user terminal 20 via the user terminal 30. In some embodiments, "live streaming" may refer to streaming realized by the above-mentioned data transmission. The live streaming may be realized by known technologies such as HTTP live streaming, CMAF (Common Media Application Format), WebRTC (Web Real-Time Communications), RTMP (Real-Time Messaging Protocol), and MPEG DASH. The live streaming may further include an embodiment in which the viewer AU can play or view the content with a certain delay while the streamer is recording the content. The degree of the delay is preferably small enough to at least allow the streamer LV and the viewer AU to communicate with each other. However, live streaming is different from so-called on-demand distribution. More specifically, the on-demand distribution may refer to storing all data of the recorded content in a server and providing the data to the user from the server at random times in response to a user's request.

[0017] In this specification, "streaming data" may refer to data including image data and audio data. More specifically, the image data (which may be referred to as video data) may be generated by an image capture function of the user terminals 20 and 30. The audio data (which may be referred to as audio data) may be generated by an audio input function of the user terminals 20 and 30. The streaming data may be played on the user terminals 20 and 30 to enable viewing of content related to the user. In some embodiments, processing such as compression, expansion, encoding, decoding, transcoding, and the like to change the format, size, and standard of data is assumed between the time when the streaming data is generated on the user terminal of the streamer and the time when it is played on the user terminal of the viewer. Before and after such processing, the content (video and audio) is not substantially changed, and therefore, in the current embodiment of the present disclosure, the streaming data before processing and the streaming data after processing are described as being the same. In other words, when the streaming data generated by the user terminal of the streamer is played back on the user terminal of the viewer via the server 10, the streaming data generated by the user terminal of the streamer, the streaming data that has passed through the server 10, and the streaming data received and played back by the user terminal of the viewer are all the same streaming data.

[0018] As shown in Fig. 1, a streamer LV provides live streaming. A user terminal 20 of the streamer generates streaming data by recording the video and / or audio of the streamer, and transmits the streaming data to a server 10 via a network NW. At the same time, the user terminal 20 displays the video VD on the display of the user terminal 20, so that the user can check the streaming content of the streamer LV.

[0019] The streamer LV may be a human or an AI model. The streaming data of the streamer LV may be generated or rendered by the server 10, or via a user terminal 20, etc. The communication between the streamer LV and the viewers AU1, AU2 may be realized via a network NW. In some embodiments, the AI ​​model may be trained internally or provided by a third-party service such as Google PaLM, ChatGPT, or other LLMs (large-scale language models).

[0020] The viewers AU1, AU2 of the user terminals 30a, 30b that request the platform to provide live streaming of the streamer can receive streaming data corresponding to the live streaming via the network NW, play the received streaming data to display the videos VD1, VD2 on a display, and output the audio from a speaker, etc. The videos VD1, VD2 displayed on the user terminals 30a, 30b, respectively, are substantially the same as the videos VD recorded by the user terminals of the streamer LV, and the audio output from the user terminals 30a, 30b is substantially the same as the audio recorded by the user terminals of the streamer LV.

[0021] The recording at the user terminal 20 of the streamer may be simultaneous with the playback of the streaming data at the user terminals 30a, 30b of the viewers AU1, AU2. When the viewer AU1 inputs a comment on the content of the streamer LV to the user terminal 30a, the server 10 displays the comment in real time on the user terminal 20 of the streamer, and also on the user terminals 30a, 30b of the viewers AU1, AU2. When the streamer LV responds to the comment, the response is output as text, image, video, or audio from the user terminals 30a, 30b of the viewers AU1, AU2, thereby realizing communication between the streamer LV and the viewers AU1, AU2. Thus, the live streaming system can realize two-way live streaming.

[0022] 2 is a block diagram showing the functions and configuration of the user terminal 20 shown in FIG. 1 according to an embodiment of the present disclosure. The user terminal 30 has the same functions and configuration as the user terminal 20. The blocks depicted in the block diagrams in this specification are implemented by hardware such as a device or mechanical part, such as a computer CPU, and software such as a computer program that represents functional blocks implemented by the cooperation of these elements. Therefore, it will be understood by those skilled in the art that the functional blocks can be implemented in various ways by combining hardware and software.

[0023] The streamer LV and the viewer AU can download and install the live streaming application (live streaming app) of the present disclosure from a download site to the user terminals 20 and 30 via the network NW. Alternatively, the live streaming app may be pre-installed in the user terminals 20 and 30. By executing live streaming by the user terminals 20 and 30, the user terminals 20 and 30 can communicate with the server 10 via the network NW and realize a plurality of functions. The functions realized by executing the live streaming app by the user terminals 20 and 30 (more specifically, a processor such as a CPU) are described below as functions of the user terminals 20 and 30. The functions are basically functions that the live streaming app causes the user terminals 20 and 30 to realize. In some embodiments, these functions may be realized by transmitting them from the server 10 to the web browsers of the user terminals 20 and 30 via the network NW and executing them by a computer program of the web browser. The computer program may be written in a programming language such as HTML (Hyper Text Markup Language).

[0024] The user terminal 20 includes a streaming unit 100 and a viewing unit 200. In some embodiments, the streaming unit 100 is configured to record a user's audio and / or video data and generate streaming data to send to the server 10. The viewing unit 200 is configured to receive and play streaming data from the server 10. In some embodiments, a user can activate the streaming unit 100 at the time of a broadcast or activate the viewing unit 200 when viewing a stream. In some embodiments, the user terminal that activates the streaming unit 100 can be referred to as a streamer or the user terminal that generates streaming data. The user terminal that activates the viewing unit 200 can be referred to as a viewer or the user terminal that plays the streaming data.

[0025] The streaming unit 100 may include a video control unit 102, an audio control unit 104, a distribution unit 106, and a UI control unit 108. The video control unit 102 may be connected to a camera (not shown), and the video is controlled by the camera. The video control unit 102 can obtain the video data from the camera. The audio control unit 104 may be connected to a microphone (not shown), and the audio is controlled by the microphone. The audio control unit 104 can obtain the audio data from the microphone.

[0026] The delivery unit 106 receives streaming data including video data from the video control unit 102 and audio data from the audio control unit 104 and transmits the streaming data to the server 10 via the network NW. In some embodiments, the delivery unit 106 transmits the streaming data in real time, i.e., the generation of the streaming data from the video control unit 102 and the audio control unit 104 and the delivery by the delivery unit 106 are performed simultaneously.

[0027] The UI control unit 108 controls the UI of the streamer. The UI control unit 108 is connected to a display (not shown) and configured to generate the streaming data for the distribution unit 106 to transmit, play and display on the display. The UI control unit 108 is configured to display an object to be operated or to receive an instruction on the display and to receive a tap input from the streamer.

[0028] The viewing unit 200 may include a UI control unit 202, a rendering unit 204, an input sending unit 206, and a processing unit 208. The viewing unit 200 is configured to receive streaming data from the server 10 via a network NW. The UI control unit 202 controls the UI of the viewer. The UI control unit 202 is connected to a display (not shown) and / or a speaker (not shown) and configured to display a video on the display and output a sound from the speaker by playing the streaming data. In some embodiments, outputting a video on the display and outputting a sound from the speaker can be referred to as "playing streaming data". The UI control unit 202 is connected to an input unit such as a touch panel, a keyboard, a display, etc., and can obtain input from a user.

[0029] The rendering unit 204 may be configured to render streaming data from the server 10 and frame images. The frame images may include user interface objects for accepting input from a user, comments entered by a viewer, and data received from the server 10. The input sending unit 206 is configured to receive the user input from the UI control unit 202 and send it to the server 10 via the network NW.

[0030] In some embodiments, the user input may be clicking an object on the screen of the user terminal, such as selecting a live broadcast, entering a comment, sending a gift, following or unfollowing a user, voting in an event, playing a game, etc. For example, the input sending unit 206 may generate gift information and send it to the server 10 via the Internet NW when the user terminal of a viewer clicks a gift object on the screen to send a gift to a streamer.

[0031] 3 is a block diagram of a server 10 according to some embodiments of the present disclosure. The server 10 may include a streaming information unit 302, a relay unit 304, a processing unit 306, a stream DB 320, a user DB 322, a virtual live streamer DB 324, a short-term memory DB 326, a long-term memory DB 328, a sentiment analysis model 330, and an action lookup table 332.

[0032] The streaming information unit 302 receives a request for live streaming from the user terminal 20 of the streamer via the network NW. Upon receiving the request, the streaming information unit 302 registers information of the live streaming in the stream DB 320. In some embodiments, the information of the live streaming may be a stream ID of the live streaming and / or a streamer ID of the streamer corresponding to the live streaming.

[0033] When a request for the information of the live streaming is received from the viewer via the network NW from the viewing unit 200 of the user terminal 30, the streaming information unit 302 refers to the stream DB 320 and generates a list of available live streaming.

[0034] Then, the streaming information unit 302 transmits the list to the user terminal 30 via the network NW. The UI control unit 202 of the user terminal 30 generates a live streaming selection screen based on the list, and displays the list on the display of the user terminal 30.

[0035] When the input sending unit 206 of the user terminal 30 receives the selection of live streaming by the viewer on the live streaming selection screen, it generates a delivery request including the stream ID of the selected live streaming and sends it to the server 10 via the network. The streaming information unit 302 can start providing the live streaming specified by the stream ID in the delivery request to the user terminal 30. The streaming information unit 302 can update the stream DB 320 and add the viewer ID of the viewer of the user terminal 30 to the streamer ID of the stream ID.

[0036] The relay unit 304 can relay the transmission of live streaming from the user terminal 20 of the streamer to the user terminal 30 of the viewer in the live streaming started by the streaming information unit 302. The relay unit 304 can receive a signal indicating a user input from the viewer from the input transmission unit 206 during playback of streaming data. The signal indicating the user input may be an object designation signal indicating designation of an object to be displayed on the display of the user terminal 30. The object designation signal may include a viewer ID of the viewer, a streamer ID of a streamer distributing the live streaming that the viewer is watching, and an object ID designated by the object. If the object is a gift or the like, the object ID may be a gift ID or the like. Similarly, the relay unit 304 can receive a signal indicating a user input of the streamer, such as the object designation signal, from the streaming unit 100 of the user terminal 20 during playback of streaming data.

[0037] The processing unit 306 may be configured to handle communication and data processing within or between various parts of the live streaming system 1. For example, the processing unit 306 may manage information in the short-term memory DB 326 and the long-term memory DB 328, perform emotion recognition to trigger corresponding actions, and convert text data or voice data to and from other data.

[0038] In some embodiments, after receiving the conversation in the live streaming room, the processing unit 306 may store the message in the short-term memory DB 326. The processing unit 306 may store a certain amount of the latest messages in the live streaming room. In some embodiments, the processing unit 306 may delete old messages in the short-term memory DB 326 and keep the latest messages in the short-term memory DB 326. For example, the latest 10 messages may be kept in the short-term memory DB 326, and old messages may be deleted periodically, etc.

[0039] In some embodiments, the processing unit 306 may further provide the information in the short-term memory DB 326 to a machine learning model of an AI V-Liver. According to these embodiments, the AI ​​V-Liver may have information about the latest conversation with the viewer, making the conversation between the AI ​​V-Liver and the viewer smoother. Furthermore, since old conversations are not kept in the short-term memory DB 326, storage is freed and the AI ​​V-Liver can act like a human, such as forgetting things. Thus, the user experience may be improved.

[0040] In some embodiments, the processing unit 306 may detect keywords in comments from the viewer and then store the keywords associated with the viewer in the long-term storage DB 328. The processing unit 306 may store keywords related to the viewer. The keywords may be information about the viewer, such as birthday, star sign, nickname, etc.

[0041] In some embodiments, the processing unit 306 may detect keywords related to the topic of the AI ​​V-Liver, associate them with the viewer, and store them in the long-term memory DB 328. For example, if the topic of the AI ​​V-Liver is astrology, the processing unit 306 may associate a horoscope with the viewer from the viewer's comments, and store it in the long-term memory DB 328. In some embodiments, the processing unit 306 may further provide the information in the long-term memory DB 328 to a machine learning model of the AI ​​V-Liver.

[0042] According to these embodiments, the AI ​​V-Liver can have information about the viewer with whom it has spoken, facilitating smooth conversation between the AI ​​V-Liver and the viewer. Furthermore, the information about the viewer does not need to be provided again, facilitating smooth conversation, and the AI ​​V-Liver can act like a human, such as remembering things. Thus, the user experience may be improved.

[0043] In some embodiments, the processing unit 306 may determine an emotion for the response generated from a machine learning model of the AI ​​V-Liver. In some embodiments, the processing unit 306 may feed the response to a machine learning model, such as emotion recognition model 550, to determine the emotion of the response. The processing unit 306 may determine whether the emotion of the response is positive, neutral, negative, such as happy, calm, angry, etc.

[0044] In some embodiments, the processing unit 306 may further determine an action or combination of actions corresponding to the emotion. For example, if the emotion is positive or "joy", a smile, dance, or combination of actions may be determined. In some embodiments, the action may be determined via a lookup table, machine learning techniques, or the like. In some embodiments, the processing unit 306 may further trigger an action of the AI ​​V-Liver via an API, or the like. For example, if the response emotion is positive, a file of an action in an AI V-Liver model, such as Live2D, may be triggered and displayed on the AI ​​V-Liver.

[0045] According to these embodiments, the AI ​​V-Liver can have emotions and actions while interacting with the viewer, which may facilitate smoother conversations between the AI ​​V-Liver and the viewer. Furthermore, the actions on the AI ​​V-Liver can be changed based on the emotion of the response, which facilitates smooth conversations and allows the AI ​​V-Liver to act human-like, such as acting happy or angry. Thus, the user experience may be improved.

[0046] In some embodiments, the processing unit 306 may convert the response from text data to voice data, etc. In some embodiments, the processing unit 306 may provide the response to a text-to-speech unit, such as a TTS unit 530, to convert the response to an audio file. The processing unit 306 may convert the response to an audio file, such as a .wav file. In some embodiments, the processing unit 306 may further transmit the voice data, text data, etc. to a broadcast unit 520. In some embodiments, the voice data may be played while the AI ​​V-Liver is performing a corresponding operation. In some embodiments, the text data may be displayed as subtitles in the live streaming room.

[0047] In some embodiments, the response, the voice data of the response, the emotion of the response, and the corresponding action may be sent to the live streaming room. In some embodiments, the action may be triggered on the virtual chatbot while the voice data is being played. In some embodiments, the action may be triggered on the virtual chatbot while the response is being displayed. In some embodiments, the response may be displayed as subtitles in the live streaming room while the voice data is being played. In some embodiments, the rendering and generation of the streaming data in the live streaming room may be flexibly realized according to actual needs.

[0048] In some embodiments, a speech-to-text unit may be applied, for example, when the viewer's comments are voice data. In some embodiments, the voice of the AI ​​V-Liver may be any available resource or may be trained via machine learning techniques. In some embodiments, the voice may be trained via machine learning models from users, celebrities, people with specific personalities, genres, etc. in the live streaming platform. For example, the voice may be trained from a cute Japanese girl, a rugged Indian man, etc.

[0049] According to these embodiments, the response from the AI ​​V-Liver may have subtitles while interacting with the viewer, which may facilitate smoother conversation between the AI ​​V-Liver and the viewer. Furthermore, the AI ​​V-Liver may respond to the viewer via voice data, which may facilitate smoother conversation and allow the AI ​​V-Liver to act like a human, such as by speaking. Thus, the user experience may be improved.

[0050] Fig. 4 shows a table illustrating an exemplary data structure of the stream DB 320 in Fig. 3. The stream DB 320 holds information about live broadcasts currently being performed. The stream DB 320 stores a stream ID for identifying a live broadcast on a live streaming platform provided by the live streaming system 1, a streamer ID for identifying a streamer providing the live broadcast, and a viewer ID for identifying a viewer of the live broadcast, in association with each other.

[0051] 5 is a table showing an example data structure of the user DB 322 of FIG. 3. The user DB 322 holds information about users. The user DB 322 stores a user ID for identifying a user, a point for identifying points accumulated by the user, a level for identifying the level of the user, and a status for identifying the status of the user, in association with each other. The point is an electronic value circulating within the live streaming platform. The level may be an indicator of the amount of activity or engagement of the user on the live streaming platform. The status may be the identity or membership status of the user on the live streaming platform.

[0052] Fig. 6 shows an example data structure of the AI ​​virtual liver DB 324 in Fig. 3. The AI ​​virtual liver DB 324 holds information about a live stream of an AI virtual liver that is currently running. The AI ​​virtual liver DB 324 stores an AI virtual liver ID that identifies an AI virtual liver, a topic that identifies the topic of the AI ​​virtual liver, a name that identifies the name of the AI ​​virtual liver, a motion ID that identifies the motion of the AI ​​virtual liver, a motion description that identifies a description of the motion, and a URL that identifies the location of the AI ​​virtual liver model, in mutual association.

[0053] In some embodiments, the topic of the AI ​​virtual liver may be a conversation that the AI ​​virtual liver primarily focuses on. For example, if the AI ​​virtual liver is set to be an astrologer, the topic may be horoscopes, and if the AI ​​virtual liver is set to be a politician, the topic may be politics. In some embodiments, the name of the AI ​​virtual liver may be displayed in a live streaming room, such as object 602 in FIG. 12. The motion ID and motion description may correspond to the AI ​​virtual liver's facial expression, body movement, or a combination, such as smiling, dancing, running, etc. The URL may indicate the location of the AI ​​virtual liver model, which may include the AI ​​virtual liver's files, such as a Live2D model, a VRM model, etc.

[0054] In some embodiments, the live broadcast from the AI ​​virtual live streamer may be stored in the stream DB 320 or a separate DB. In some embodiments, the information of the AI ​​virtual live streamer may be stored as a user in the user DB 322 or in a separate DB. In some embodiments, the details of each database may be flexibly determined according to actual needs.

[0055] Fig. 7 is a table showing an exemplary data structure of the short-term memory DB 326 of Fig. 3. The short-term memory DB 326 holds information about a conversation in a live streaming room. The short-term memory DB 326 stores a message ID for identifying a message in a conversation between the viewer and the AI ​​V-Liver and a description for identifying the content of the message in the conversation, in association with each other.

[0056] In some embodiments, the short-term memory DB 326 may store conversations from the viewer side, the AI ​​V-Liver side, or both. In some embodiments, the short-term memory DB 326 may store a certain amount of conversations, such as the latest 10 messages in the live streaming room. In some embodiments, the information stored in the short-term memory DB 326 may be flexibly determined according to actual needs.

[0057] Figure 8 is a table showing an exemplary data structure of the long-term memory DB 328 of Figure 3. The long-term memory DB 328 holds information with keywords associated with viewers in a live streaming room. The long-term memory DB 328 stores a memory ID that identifies a long-term memory associated with a viewer, a user ID that identifies a viewer associated with the keyword, a data type that identifies a data type of the keyword, data information that identifies information of the data, and a format that identifies the association between the keyword and the viewer, in association with each other.

[0058] In some embodiments, the keyword stored in the long-term memory DB 328 may be information related to the viewer. For example, the keyword may be the viewer's birthday, star sign, nickname. In some embodiments, the keyword may be information related to the topic of the AI ​​V-Liver. For example, the keyword may be the viewer's star sign if the topic of the AI ​​V-Liver is an astrologer, etc.

[0059] FIG. 9 is a table showing an exemplary data structure of the sentiment analysis model 330 of FIG. 3. The sentiment analysis model 330 may be an internal language model or a language model provided by a third party such as Google, OpenAI, etc. For example, the sentiment analysis model 330 may be BERT, such as bert-base-chinese, GPT, etc. Training data for the sentiment analysis model 330 may be past data in the server 10, or a dataset provided by a third party, etc. For example, a dataset such as MPDD (Multi-Party Dialogue Dataset) or CPED (Chinese personalized and emotional dialogue dataset) may be used to train the sentiment analysis model 330.

[0060] In some embodiments, the trained model 330 may be used to determine the emotion of the response. The response from the AI ​​V-Liver may be provided to the emotion analysis model 330. As shown in FIG. 9, the emotion of the response may be generated based on the emotion analysis model 330. For example, if the viewer sends an image of a cockroach, the AI ​​V-Liver may respond, "Cockroaches are really gross, go away." The response may be provided to the emotion analysis model 330, and an emotion of "anger" may be inferred from the response.

[0061] Fig. 10 is a table showing an example data structure of the action lookup table 332 of Fig. 3. The action lookup table 332 holds information on emotions and corresponding actions. The action lookup table 332 stores emotions that specify the emotions of the AI ​​V-Liver and action IDs that specify the actions triggered by the emotions in association with each other.

[0062] In some embodiments, the emotion may be positive, neutral, or negative, such as joy, calm, anger, etc. In some embodiments, the action may be a facial expression or a body action, such as smiling or waving. In some embodiments, the emotion may be one emotion or a combination of two or more emotions, and the action may be one action or a combination of two or more actions. For example, if the emotion is joy and excitement, the AI ​​V-Liver may laugh and dance.

[0063] 11 is a schematic diagram showing an example of a functional configuration of a live streaming system 1 according to some embodiments of the present disclosure. The configuration of the AI ​​virtual live streamer in the server 10 may include a conversation server unit 500, a broadcast unit 520, a TTS unit 530, an information unit 540, an emotion recognition unit 550, and a long-term memory unit 560.

[0064] The conversation server unit 500 is configured to generate a conversation between the AI ​​virtual live streamer and the viewer. The broadcast unit 520 is configured to display the AI ​​virtual live streamer's broadcast to the viewer. The TTS unit 530 is configured to read out text as the AI ​​virtual live streamer speaks. The information unit 540 is configured to collect information for the AI ​​virtual live streamer to interact with the viewer. The emotion recognition unit 550 is configured to recognize the emotion of the AI ​​virtual live streamer and trigger a corresponding action. The long-term memory unit 560 is configured to extract keywords from the conversation between the AI ​​virtual live streamer and the viewer and store them in the long-term memory DB 328 for further processing.

[0065] In some embodiments, the conversation server unit 500 may include an inference system 502. The inference system 502 is configured to infer a response to a comment from the viewer based on a number of parameters, such as the viewer's interaction. In some embodiments, the response may be triggered, for example, when the viewer enters a live streaming room, when the viewer interacts with the AI ​​virtual live streamer, etc. The interaction from the viewer may include comments from the viewer, gifts, snacks, etc. In some embodiments, the inference system 502 may infer a mood or emotion to be set for the AI ​​virtual live streamer based on input from the information unit 540.

[0066] In some embodiments, the inference system 502 may also receive input from a moderator 504, generate responses, etc. Here, the "moderator" may refer to an administrator in a live streaming room who is responsible for managing the live streaming and the overall interaction within the stream. The moderator 504 may be a human or an AI virtual live streamer. The moderator 504 may be used to ensure orderly interaction among viewers, address inappropriate behavior, provide event guidelines, etc.

[0067] In some embodiments, information collected by the inference system 502 may also be fed into a machine learning model, such as an LLM 506, to generate a response to the viewer. In some embodiments, the processing unit 306 may store information from the LLM 506 in a short-term memory unit 508. For example, the short-term memory unit 508 may be configured to store the last 10 conversations between the AI ​​virtual live streamer and a viewer.

[0068] Output such as a response may be sent to the broadcast unit 520 for the AI ​​virtual liver to interact with an audience. In some embodiments, the response may also be sent to the broadcast unit 520 via the emotion recognition unit 550. The emotion recognition unit 550 may recognize the emotion of the response and determine one or more corresponding actions of the AI ​​virtual liver.

[0069] In some embodiments, the response may be sent to the broadcast unit 520 via the TTS unit 530. The TTS unit 530 may read out the text of the response. In some embodiments, the response in the form of text data and audio data may be sent to the broadcast unit 520. The text data may be generated as a comment in a message zone 606 of the live streaming room or as a subtitle in a subtitle zone 613, and the audio data may be played based on the action of the AI ​​virtual live streamer. Thus, the AI ​​virtual live streamer may be able to act like a human being, and the user experience may be improved.

[0070] In some embodiments, the broadcast unit 520 may include a client 522 and a custom skin 524. The client 522 may be an AI virtual live streamer entity displayed in the live streaming room. In some embodiments, the streaming data of the AI ​​virtual live streamer may be rendered by the client 522, and the viewer may retrieve and play the streaming data on the user terminal 30 to watch the live streaming. The custom skin 524 may include a virtual skin or virtual avatar of the AI ​​virtual live streamer. The virtual skin may be created in the server 10. Or it may be a 2D or 3D character model provided by a third party service, such as a Live2D model, a VRM model, etc.

[0071] The custom skin 524 may also include an action file of the AI ​​virtual live streamer. The action file may be an image, video, or animation, and the action may be facial features, body movements, such as smiling, jumping, waving, dancing, etc. Each action may correspond to one or more emotions, and multiple actions may be combined to show one or more emotions. For example, the AI ​​virtual live streamer may smile and wave when a viewer enters the live streaming room.

[0072] In some embodiments, the TTS unit 530 may include a text-to-speech unit 532. The text-to-speech unit 532 is configured to convert text input into speech output. In some embodiments, the speech output may be a .wav file or the like. In some embodiments, machine learning techniques may be applied to generate the AI ​​virtual live streamer's voice. In some embodiments, the voice may be from a user in the live streaming platform, such as a live streamer. The voice may be trained in a particular genre, such as a cute girl or a muscular man. In some embodiments, the TTS unit 530 may also include a speech-to-text unit to convert text input into speech output.

[0073] In some embodiments, the information unit 540 may include a daily information collector 542. The daily information collector 542 is configured to collect information about the AI ​​virtual live streamer, such as to generate daily emotions. The information may be, for example, the current date, day of the week, current time, day or night, weather, stock prices, top news, topics of interest to viewers, and the like. In some embodiments, an information storage (not shown) may be connected to the information unit 540 to store the information. The information from the information unit 540 may be sent to the inference system 502 to generate daily emotions, responses, and the like.

[0074] In some embodiments, output from the LLM 506 may be further sent to the long-term storage 560. The long-term storage 560 may include a machine learning model, such as an LLM 562. Comments from viewers and responses from the AI ​​may form a conversation, which may be provided to the LLM 562. The LLM 562 may extract keywords associated with the user from the conversation.

[0075] In some embodiments, the keyword may be related to the AI ​​virtual liver's topic. For example, if the AI ​​virtual liver is an astrologer, "horoscope" may be one of the AI ​​virtual liver's topics. The viewer may ask the AI ​​virtual liver, "What's my fortune today?" The AI ​​virtual liver may generate a response, "What's your zodiac sign?" and ask the viewer's zodiac sign to tell their fortune. The viewer may further respond to the AI ​​virtual liver, such as, "I'm a Taurus." The LLM 562 may analyze the conversation between the AI ​​virtual liver and the viewer and associate the keyword "Taurus" with the viewer.

[0076] The long-term storage unit 560 may further include a long-term memory DB, such as DB 564, for storing the association between the keyword and the viewer. In some embodiments, DB 564 may be a database, such as Redis. Redis is an open source in-memory database management system that is classified as a NoSQL (Not Only SQL) database. In some embodiments, other suitable databases may be used to store information as long-term memory. In some embodiments, the conversation including the keyword may be stored in the DB 564 after being converted into a specific data format, such as a json format.

[0077] The long-term storage section 560 may further include a long-term storage unit 566. The long-term storage unit 566 may obtain information from the DB 564 and transmit the information to the inference system 502. For example, the long-term storage unit 566 may obtain information {"username":"Chris", "star sign":"Taurus"} from the DB 564 and transmit the information to the inference system 502. Thus, the inference system 502 may infer a response to the comment based on the information in the DB 564.

[0078] For example, the viewer Chris told the AI ​​virtual live streamer yesterday that his or her zodiac sign is "Taurus," and the next day the viewer Chris enters the live streaming room and asks, "What's my fortune today?" The AI ​​virtual live streamer may respond based on the information {"username":"Chris", "zodiac sign":"Taurus"}, or may respond, "Hello, Chris. Taurus' fortunes today are pretty good." In other words, the AI ​​virtual live streamer can remember certain information related to the viewer. Thus, conversation between the AI ​​virtual live streamer and the viewer may be promoted, improving the user experience.

[0079] In some embodiments, the communication between the conversation server unit 500 and the broadcast unit 520 may be realized through a backend 570. In some embodiments, the communication between each unit may be realized through a message communication service 572 such as Pub / Sub. Here, Pub / Sub stands for "publish / subscribe", which is a messaging communication pattern. Messages such as comments, gifts, chat history, audio data, subtitles, etc. may be communicated in real time between different units through the Pub / Sub pattern, etc. In some embodiments, other suitable messaging communication patterns may be flexibly applied.

[0080] In some embodiments, the status in the live streaming room may be provided to the conversation server unit 500 via the backend 570 and the message communication service 572. The "status" in the live streaming room may be, for example, the AI ​​virtual liver's received points, the length of broadcast time, the number of followers or subscribers, etc. A response from the AI ​​virtual liver may further be generated based on the "status" in the live streaming room. For example, if the AI ​​virtual liver is participating in an event and has few received points, a response such as "Please give me a gift to win the event" may be generated.

[0081] In some embodiments, a chat history data warehouse 574 may also be provided to store chat history of conversations between the AI ​​virtual live streamer and the viewers in the live streaming room. In some embodiments, the chat history data warehouse 574 may be an in-house data management and analytics warehouse or may be provided by a third party such as BigQuery. In some embodiments, the chat history in the chat history data warehouse 574 may be used in a more flexible manner.

[0082] According to this embodiment, the AI ​​virtual live streamer may be trained by the information unit 540, the conversation server unit 500, and the long-term memory unit 560. Thus, a unique AI virtual live streamer with different emotion prompts can be obtained. According to this embodiment, the interaction between the AI ​​virtual live streamer and the viewer may be different based on the emotion of the AI ​​virtual live streamer. Even if the interaction from the viewer is the same, the AI ​​virtual live streamer may generate different responses based on different emotions. Thus, the fun in the live streaming room may be increased, and the user experience may be improved.

[0083] 12 shows a simplified functional configuration of an exemplary screen image of a live streaming room screen 600 in the live streaming system 1 according to some embodiments of the present disclosure. As shown in the screen 600 in FIG. 12, when the viewer selects and enters the live streaming room, the live streaming room screen 600 of the AI ​​virtual liver may be displayed on the display. The live streaming room screen 600 may include an AI virtual liver information object 602, an AI virtual liver image 604, a message zone 606, a message input box 608, a gift object 610, a sharing object 612, and the like.

[0084] In some embodiments, the AI ​​virtual live streamer may start greeting the viewer when the viewer enters the live streaming room. For example, the AI ​​virtual live streamer may say, "Welcome to my live streaming room. What is your zodiac sign?" when the viewer enters the live streaming room. In some embodiments, the AI ​​virtual live streamer may respond to the viewer when the viewer enters a comment or the like. For example, if the viewer says, "I'm a Taurus," the AI ​​virtual live streamer may generate a response related to Taurus horoscope or the like.

[0085] In some embodiments, the AI ​​virtual live streamer may also spontaneously generate conversations or topics in response to the absence of interaction in the live streaming room within a certain period of time. For example, if there is no interaction in the live streaming room for more than five minutes, the AI ​​virtual live streamer may spontaneously generate conversations such as "I'll tell you a joke" or "I'll tell you today's Taurus fortune." According to this embodiment, the popularity of the live streaming room may be enhanced and the user experience may be improved.

[0086] In some embodiments, the conversation 614, including comments from the viewer and / or responses from the AI ​​virtual live streamer, may be sent to an AI brain 620. The AI ​​brain 620 may refer to the AI ​​virtual live streamer's brain and may be configured to generate emotions and infer responses. In some embodiments, the AI ​​brain 620 may refer to the conversation server unit 500 in FIG. 11.

[0087] The AI ​​brain 620 may receive information 622, such as environmental information, emotion algorithms, etc., as input. In some embodiments, the conversation may be stored in short-term memory 624. In some embodiments, certain information may be stored in long-term memory 626. The AI ​​brain 620 may generate responses to comments from the viewers and send the responses to a TTS 628. The TTS 628 may further convert the responses from text data to audio data and display them in the live streaming room.

[0088] Next, the operation of the live streaming system 1 having the above configuration will be described. FIG. 13 is a flowchart showing the steps of an application activation process in the live streaming system 1 according to some embodiments of the present disclosure. When a user enters a live streaming room, the user may perform operations on the user terminal to interact with the AI ​​virtual live streamer. In some embodiments, the operations may include sending a message, sending a gift, following, unfollowing, voting, playing a game, and the like.

[0089] The server 10 may receive a comment from a user terminal of a user (S502). The processing unit 306 may monitor the message by detecting a keyword contained in the message. For example, the processing unit 306 may scan the message to detect whether it contains a particular keyword. In some embodiments, the keyword may be information related to the viewer, such as a birthday, astrological sign, or a nickname. In some embodiments, the keyword may be information related to the AI ​​V-Liver and the viewer. For example, if the topic of the AI ​​V-Liver is a politician, the keyword may be the political party supported by the viewer, etc.

[0090] If the message contains a particular keyword (S504: YES), the processing unit 306 may associate the keyword with the user (S506). More specifically, the processing unit 306 may extract the user ID, data type, and data information and convert them into a particular format for storage. For example, the processing unit 306 may extract "Chris", "zodiac sign", and "Taurus", and then convert them into a key-value format, such as a .json format, and store them in a database. In some embodiments, the format may be any possible type of format that can be stored in or retrieved from a database.

[0091] In some embodiments, the association between the keyword and the user may be detected, such as by a machine learning model. The machine learning model may be used to determine whether the keyword is associated with the viewer. For example, the viewer may say, "I'm a Taurus" or "My friend is a Taurus." Although both messages contain the keyword "Taurus," the former may indicate that the viewer is a Taurus, while the latter may not provide information associated with the user. The machine learning model may distinguish this difference and determine whether there is an association.

[0092] In some embodiments, the processing unit 306 may further store (S508) the keyword information and the user in a long-term memory database 328. In some embodiments, the information in the long-term memory database 328 may further be fed to a machine learning model of the AI ​​V-Liver, and the response may be generated based on the information in the long-term memory database 328.

[0093] The comment from the user terminal of the user may further be fed into a machine learning model to generate a response to the comment (S510). In some embodiments, the response may be generated based on multiple parameters, such as the AI ​​V-Liver's daily emotions, the short-term memory database 326, the long-term memory database 328, comments from the viewers, and the status in the live streaming room.

[0094] The processing unit 306 may further determine an action for the response. In some embodiments, the response may include a positive, neutral, or negative emotion, such as happy, calm, or anger. The AI ​​V-Liver's behavior, such as facial expressions or body movements, may correspond to different emotions. In some embodiments, the processing unit 306 may feed the response to a machine learning model to determine the emotion of the response. The processing unit 306 may further determine the action of the AI ​​V-Liver based on the emotion of the response.

[0095] The processing unit 306 may further generate audio data of the response via the TTS unit 530 (S514). The processing unit 306 may further transmit the response, the action, and the audio data to the live streaming room (S516). In some embodiments, the response may be displayed as subtitles in the live streaming room. The action may be displayed through the virtual skin of the AI ​​V-Liver, and the audio data may be played in sync with the action to indicate interaction with the viewers in the live streaming room.

[0096] Fig. 14 is a schematic block diagram of a system configuration and computer hardware for executing processing according to some embodiments of the present disclosure. The information processing device 900 shown in Fig. 14 is configured to realize, for example, the server 10 and the user terminals 20 and 30 according to some embodiments of the present disclosure.

[0097] The information processing device 900 includes a CPU 901, a read-only memory (ROM) 903, and a random access memory (RAM) 905. The information processing device 900 may further include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input unit 915, an output unit 917, a storage unit 919, a drive 921, a connection port 925, and a communication unit 929. The information processing device 900 may include an imaging device (not shown) such as a camera. The CPU 901 is an example of a hardware configuration for implementing various functions performed by the components described herein. The functions described herein may be implemented by a circuit programmed to implement the functions described herein. The circuit programmed to implement the functions described herein may include a central processing unit (CPU), a digital signal processor (DSP), a general-purpose processor, a dedicated processor, an integrated circuit, an application specific integrated circuit (ASIC), and / or a combination thereof. Various units configured to implement specific functions described herein, including but not limited to the streaming unit 100, the viewing unit 200, the video control unit 102, the audio control unit 104, the delivery unit 106, the UI control unit 108, the UI control unit 202, the rendering unit 204, the input sending unit 206, the streaming information unit 302, the relay unit 304, the processing unit 306, the stream DB 320, the user DB 322, the AI ​​virtual live streamer DB 324, the short-term memory DB 326, and the long-term memory DB 328, may be embodied as a circuit programmed to implement the functions.

[0098] The CPU 901 functions as an arithmetic processing device and a control device, and controls the overall operation of the information processing device 900 or a part of the operation according to various programs recorded in the ROM 903, the RAM 905, the storage unit 919, or the removable recording medium 923. For example, the CPU 901 controls the overall operation of each functional unit included in the server 10 and the user terminals 20 and 30 of the above-mentioned embodiment. The ROM 903 stores programs and operation parameters used by the CPU 901. The RAM 905 transiently stores programs used when the CPU 901 executes programs and parameters that change appropriately when the programs are executed. The CPU 901, the ROM 903, and the RAM 905 are connected to each other via a host bus 907 consisting of an internal bus such as a CPU bus. The host bus 907 is connected to an external bus 911 such as a peripheral component interconnect / interface (PCI) bus via the bridge 909.

[0099] The input unit 915 is a device operated by a user, such as a mouse, a keyboard, a touch panel, a button, a switch, or a lever. The input unit 915 may be a device that converts a physical quantity into an electric signal, such as an audio sensor (such as a microphone), an acceleration sensor, a tilt sensor, an infrared sensor, a depth sensor, a temperature sensor, or a humidity sensor. The input unit 915 may be, for example, a remote control device that uses infrared rays or another type of radio wave. Alternatively, the input unit 915 may be an external connection terminal 927 such as a mobile phone that supports the operation of the information processing device 900. The input unit 915 includes an input control circuit that generates an input signal based on information input by a user and outputs the generated input signal to the CPU 901. A user operates the input unit 915 to input various data and instruct the information processing device 900 to perform a processing operation.

[0100] The output unit 917 includes a device that can visually or audibly notify the user of the acquired information. The output unit 917 may be, for example, a display device such as an LCD, a PDP, or an OLED, an audio output device such as a speaker or a headphone, a printer, etc. The output unit 917 outputs the results obtained by the processing executed by the information processing device 900 in the form of text, video such as an image, or sound such as voice.

[0101] The storage unit 919 is a data storage device and is an example of a storage unit of the information processing device 900. The storage unit 919 includes, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc. The storage unit 919 stores programs and various data executed by the CPU 901, and various data acquired from the outside.

[0102] The drive 921 is a reader / writer of a removable recording medium 923 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built into or externally attached to the information processing device 900. The drive 921 reads out information recorded on the attached removable recording medium 923, and outputs it to the RAM 905. The drive 921 writes information to the attached removable recording medium 923.

[0103] The connection port 925 is a port used for directly connecting a device to the information processing device 900. The connection port 925 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. The connection port 925 may be an RS-232C port, an optical audio terminal, an HDMI (High Definition Multimedia Interface (registered trademark)) port, or the like. By connecting an external connection terminal 927 to the connection port 925, various data can be exchanged between the information processing device 900 and the external connection terminal 927.

[0104] The communication unit 929 is, for example, a communication interface including a communication device for connecting to a communication network NW. The communication unit 929 may be, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (registered trademark), or Wireless USB (WUSB).

[0105] The communication unit 929 may be, for example, a router for optical communication, a router for ADSL (asymmetric digital subscriber line), or a modem for various types of communication. For example, the communication unit 929 transmits and receives signals on the Internet and transmits and receives signals with other communication devices using a predetermined protocol such as TCP / IP. The communication network NW to which the communication unit 929 is connected is a network established by a wired connection or a wireless connection. The communication network NW is, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0106] The imaging device (not shown) is a device that captures real space and generates a captured image using an imaging element such as a CCD (charge-coupled device) or a CMOS (complementary metal-oxide semiconductor) and various members such as a lens for controlling the formation of a subject image on the imaging element. The imaging device may capture still images or moving images.

[0107] The live streaming system 1 of the present disclosure has been described above with reference to the embodiments. The above-mentioned embodiments are described merely for the purpose of explanation. Rather, those skilled in the art can easily conceive of various combinations of the above-mentioned components and processes of the embodiments and various modifications, which are also included in the technical scope of the present disclosure.

[0108] The steps described in this specification, particularly the steps described using flow charts or flow charts, may include some of the steps constituting the steps, may include steps not explicitly included in the steps constituting the steps, and / or may rearrange the order of the steps. Such omissions, additions, and rearrangements of steps are also included in the scope of the present disclosure as long as they do not deviate from the gist of the present disclosure.

[0109] In some embodiments, at least a portion of the functions performed by the server 10 may be performed by a device other than the server 10, for example, the user terminal 20 or 30. In some embodiments, at least a portion of the functions performed by the user terminal 20 or 30 may be performed by a device other than the user terminal 20 or 30, for example, the server 10. In some embodiments, rendering of frame images may be performed by the user terminal of a viewer, server, streamer, etc.

[0110] Furthermore, the system or method described in the above embodiments may be provided in a non-transitory computer-readable storage device such as a solid-state storage device, an optical disk storage device, a magnetic disk storage device, or a computer program product. Alternatively, the program may be downloaded from a server via the Internet.

[0111] Although the technical contents and features of the present disclosure have been described above, those skilled in the art can still make many variations and modifications without departing from the teachings and disclosure of the present disclosure. Therefore, the scope of the present disclosure is not limited to the embodiments already disclosed, but is within the scope of the following claims, including other variations and modifications that do not depart from the present disclosure. [Explanation of symbols]

[0112] 1. Live Streaming System 10 Server 20 User terminal 100 Streaming Units 102 Video control unit 104 Audio control unit 106 Delivery Unit 108 UI control unit 200 viewing units 202 UI control unit 204 Rendering Units 206 Input Unit 30, 30a, 30b User terminal 302 Streaming Information Unit 304 Relay unit 306 Processing Unit 320 Stream DB 322 User DB 324 Virtual Liver DB 326 Short-term memory DB 328 Long-term memory DB 330 Sentiment Analysis Model 332 Action Lookup Table 500 Conversation Server 502 Inference Systems 504 Moderator 506 LLM 508 Short-Term Memory Unit 520 Broadcast Division 522 clients 524 Custom Skin 530 TTS Department 532 Text to Speech Unit 540 Information Department 542 Daily Information Collector 550 Emotion recognition section 560 Long-term memory 562 LLM 564 DB 566 Long-Term Memory Unit 570 Backend 572 Message Communication Service 574 Chat History Data Warehouse 600 screens 602 Objects 604 images 606 Message Zone 608 Message input box 610 Gift Object 612 Shared Objects 613 Subtitle Zone 614 Gift List 620 AI Brain 622 Information 624 Short-term memory 626 Long-term memory 628 TTS 900 Information processing device 901 CPU 903 ROM 905 RAM 907 Host Bus 909 Bridge 911 External Bus 913 Interface 915 Input Unit 917 Output Unit 919 Storage Unit 921 Drive 923 Removable Recording Media 925 Connection Port 927 External connection terminal 929 Communication Unit LS Live Streaming LV Live Streamer NW Network SP Specific part AU1, AU2 viewers S502~S512 process VD, VD1, VD2 Video

Claims

1. A server including a circuit, the circuit comprising: generating a virtual chatbot such that the machine learning model is included in the virtual chatbot; setting up said virtual chatbot in a live streaming room; receiving comments from users in the live streaming room; in response to detecting keywords associated with the user from the comments, storing the keywords in a first database; providing information from the first database to the machine learning model; wherein the keyword is related to information of the user.

2. and further setting a topic on the virtual chatbot; the keywords are further related to the topic of the virtual chatbot; The keywords related to the topic are further associated with the user and then stored in the first database.

2. The server according to claim 1 .

3. storing the comments in a second database; providing information from the second database to the machine learning model; 2. The server of claim 1, further comprising: a second database for storing a certain amount of recent conversations in the live streaming room.

4. generating a response to the comment via the machine learning model; sending said response to said live streaming room; The server of claim 1 further comprising:

5. determining an action in response; converting the response into audio data; triggering the action on the virtual chatbot while the audio data is being played; The server of claim 4 further comprising:

6. determining an action in response; triggering the action on the virtual chatbot while the response is being displayed; The server of claim 4 further comprising a display feature.

7. converting the response into audio data; displaying the responses as subtitles in the live streaming room while the audio data is being played; The server of claim 4 further comprising:

8. 2. The server of claim 1, further comprising: generating a response in response to the user entering the live streaming room, or in response to the user submitting the comment, or in response to no interaction in the live streaming room for more than a predetermined period of time.

9. 1. A method of providing a live stream on a live streaming platform, comprising: generating a virtual chatbot such that the machine learning model is included in the virtual chatbot; setting up said virtual chatbot in a live streaming room; receiving comments from users in the live streaming room; in response to detecting keywords associated with the user from the comments, storing the keywords in a first database; providing information from the first database to the machine learning model; wherein the keywords are related to information of the user.

10. A computer program comprising: generating a virtual chatbot such that the machine learning model is included in the virtual chatbot; The function of setting the virtual chatbot in a live streaming room; receiving comments from users in the live streaming room; in response to detecting a keyword associated with the user in the comment, storing the keyword in a first database; providing information from the first database to the machine learning model; and wherein the keyword is related to information of the user.

Images