Server, method, and computer program

Abstract

SOLUTION: Receiving a video clip from a first user end of a first user, inputting the video clip into a first machine-learning model for S506, generating a first virtual character, displaying the first virtual character in a live streaming room, receiving a comment from the live streaming room, generating a response to the comment through the first machine-learning model, and displaying the response through the first virtual character in the live streaming room.EFFECT: The communication between the audience and the AI virtual character is improved, the quality of the live streaming platform deployed with the AI virtual character is improved, and the user experience is improved.SELECTED DRAWING: Figure 18

Description

[Technical Field]

[0001] The present disclosure relates to information and communication technology, and in particular to a server, a method, and a computer program for live streaming. [Background technology]

[0002] Some apps and platforms offer live streaming services that allow live streamers and viewers to interact with each other, with live streamers sometimes performing cheering performances for their viewers, and viewers sometimes donating or sending gifts to support the live streamers.

[0003] With the advancement of technology, AI models are now being applied to live streaming, giving birth to AI virtual live streamers. Patent Document 1 discloses a method for processing viewers' interactions with an AI virtual live streamer.

[0004] However, viewers may not be attracted to interacting with AI virtual live streamers who do not behave like humans, which may lead to a poor user experience. Therefore, how to improve the user experience is important. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US Patent Application Publication No. 20230061778 Summary of the Invention

[0006] One embodiment of the present invention relates to a method for managing a stream in a live streaming platform, the method including steps of receiving a video clip from a first user terminal of a first user, inputting the video clip into a first machine learning model to generate a first virtual character, displaying the first virtual character in a live streaming room, receiving comments from the live streaming room, generating a response to the comment via the first machine learning model, and displaying the response via the first virtual character in the live streaming room.

[0007] Another embodiment of the present invention relates to a server including a circuit configured to perform the following functions: receive a video clip from a first user terminal of a first user; input the video clip into a first machine learning model to generate a first virtual character; display the first virtual character in a live streaming room; receive comments from the live streaming room; generate a response to the comment via the first machine learning model; and display the response in the live streaming room via the first virtual character.

[0008] Another embodiment of the present invention relates to a computer program that causes a server to perform the following functions: receive a video clip from a first user terminal of a first user; input the video clip into a first machine learning model to generate a first virtual character; display the first virtual character in a live streaming room; receive comments from the live streaming room; generate a response to the comment through the first machine learning model; and display the response in the live streaming room through the first virtual character.

[0009] According to the book, this could improve communication between viewers and AI virtual characters, as well as the quality of live streaming platforms featuring AI virtual characters, thereby improving user experience. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram illustrating the configuration of a live streaming system 1 based on some embodiments of the present disclosure. [Figure 2] FIG. 2 is a block diagram of a user terminal 20 according to some embodiments of the present disclosure. [Figure 3] 1 is a block diagram of a server 10 according to some embodiments of the present disclosure. [Figure 4] 4 is a table illustrating an exemplary data structure of the stream DB 320 of FIG. 3. [Figure 5] 4 is a table illustrating an exemplary data structure of the user DB 322 of FIG. 3. [Figure 6] 4 is a table illustrating an example data structure of the AI ​​virtual character DB 324 of FIG. 3. [Figure 7] 4 is a table illustrating an example data structure of the game DB 326DB of FIG. 3. [Figure 8] 4 is a table illustrating an exemplary data structure of the video clip DB 328 of FIG. 3. [Figure 9] 4 is a table illustrating an example data structure of the AI ​​virtual character emotion DB 332 of FIG. 3. [Figure 10] 6 is an exemplary screen image of a live streaming room screen 600 displayed on the display of a live streamer's user terminal 20 or a viewer's user terminal 30. [Figure 11] 6 is an exemplary screen image of a live streaming room screen 600 displayed on the display of a live streamer's user terminal 20 or a viewer's user terminal 30. [Figure 12]6 is an exemplary screen image of a live streaming room screen 600 displayed on the display of a live streamer's user terminal 20 or a viewer's user terminal 30. [Figure 13] 6 is an exemplary screen image of a live streaming room screen 600 displayed on the display of a live streamer's user terminal 20 or a viewer's user terminal 30. [Figure 14] 6 is an exemplary screen image of a live streaming room screen 600 displayed on the display of a live streamer's user terminal 20 or a viewer's user terminal 30. [Figure 15] 6 is an exemplary screen image of a live streaming room screen 600 displayed on the display of a live streamer's user terminal 20 or a viewer's user terminal 30. [Figure 16] FIG. 1 is a block diagram of a server 10 with details regarding AI virtual characters according to some embodiments of the present disclosure. [Figure 17] FIG. 1 is a block diagram of a server 10 with details regarding AI virtual characters according to some embodiments of the present disclosure. [Figure 18] 1 illustrates an exemplary functional structure of a live streaming system 1 according to some embodiments of the present disclosure. [Figure 19] 1 illustrates an exemplary hardware configuration of an information processing device based on some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, identical or similar components, parts, steps, or signals shown in each drawing will be denoted by the same reference numerals in all drawings, and redundant explanations will be omitted as appropriate. In addition, some parts that are not important in the explanation of each drawing will be omitted.

[0012] A live streaming system 1 according to some embodiments of the present disclosure provides enhanced functionality to facilitate communication and interaction between users, and more specifically, to entertain viewers and streamers in a technologically advanced manner.

[0013] FIG. 1 is a schematic diagram illustrating the configuration of a live streaming system 1 according to some embodiments of the present disclosure. The live streaming system 1 provides a live streaming service for real-time interaction between streaming streamers (also referred to as live broadcasters or streamers) LV and viewers (also referred to as audiences) AU (AU1, AU2, ...). As shown in FIG. 1, the live streaming system 1 may include a server 10, a user terminal 20, and user terminals 30 (30a, 30b, ...). The user terminal 20 may be a streamer, and the user terminal 30 may be a viewer. In some embodiments, the streamer and the viewer may be referred to as users. The server 10 may include one or more information processing devices connected via a network NW. The user terminals 20 and 30 may be, for example, mobile terminals such as smartphones, tablets, laptops, recorders, portable game consoles, and wearable devices, or stationary computers such as desktop PCs. The server 10, the user terminals 20, and the user terminals 30 may be communicatively connected via any type of wired or wireless network NW.

[0014] The live streaming system 1 involves the streamer LV, the viewer AU, and an app provider (not shown) that provides the server 10. The streamer LV can record content such as his / her own singing, talking, performance, game streaming, etc. on his / her user terminal 20 and upload it to the server 10, thereby distributing the content in real time. In some embodiments, the streamer LV can interact with the viewer AU through the live streaming.

[0015] The app provider may provide a platform for live-streamed content on the server 10. In some embodiments, the app provider may be a mediator or manager that manages real-time communication between the streamer LV and the viewer AU. The viewer AU can access the platform using the user terminal 30 and select and view the content they want to watch. The viewer AU can use the user terminal 30 to perform operations to interact with the streamer, such as commenting on the streamer or cheering on the streamer. The streamer providing the content can respond to the comments or cheers. The streamer's response can be transmitted to the viewer AU via video and / or audio. Thus, mutual communication between the streamer and the viewer can be achieved.

[0016] As used herein, "live streaming" may refer to data transmission that enables a viewer AU to virtually play or view content recorded by a streamer LV via the user terminal 20 via the user terminal 30. In some embodiments, "live streaming" may also refer to streaming achieved by the above-described data transmission. Live streaming may be achieved using known technologies, such as HTTP live streaming, Common Media Application Format (CMAF), WebRTC (Web Real-Time Communications), RTMP (Real-Time Messaging Protocol), and MPEG DASH. Live streaming may also include an embodiment in which a viewer AU plays or views content with a specific delay while the streamer is recording the content. The degree of delay is preferably small enough to at least allow communication between the streamer LV and the viewer AU. However, live streaming differs from so-called on-demand delivery. More specifically, on-demand delivery may refer to storing all recorded data of the content on a server and providing the data to the user from the server at random times in response to user requests.

[0017] As used herein, "streaming data" may refer to data including image data and audio data. More specifically, the image data (which may be referred to as video data) may be generated by an image capture function of the user terminals 20 and 30. The audio data (which may be referred to as audio data) may be generated by an audio input function of the user terminals 20 and 30. The streaming data may be played on the user terminals 20 and 30 to enable viewing of content related to the user. In some embodiments, processing such as compression, expansion, encoding, decoding, and transcoding to change the format, size, or standard of the data may be performed between the time the streaming data is generated at the streamer's user terminal and the time it is played at the viewer's user terminal. The content (video and audio) remains substantially unchanged before and after such processing. Therefore, in the current embodiment of the present disclosure, the streaming data before processing and the streaming data after processing are described as being the same. In other words, when the streaming data generated by the user terminal of the streamer is played on the user terminal of the viewer via the server 10, the streaming data generated by the user terminal of the streamer, the streaming data that has passed through the server 10, and the streaming data that is received and played by the user terminal of the viewer are all the same streaming data.

[0018] As shown in Figure 1, a streamer LV provides live streaming. The streamer's user terminal 20 generates streaming data by recording the streamer's video and / or audio, and transmits the data to a server 10 via a network NW. At the same time, the user terminal 20 displays the video VD on its display, allowing users to check the streaming content of the streamer LV.

[0019] The streamer LV may be a human or an AI model. The streaming data of the streamer LV may be generated or rendered by the server 10, via a user terminal 20, or the like. Communication between the streamer LV and the viewers AU1 and AU2 may be achieved via a network NW. In some embodiments, the AI ​​model may be trained internally or provided by a third-party service such as Google PaLM, ChatGPT, or other LLMs (large-scale language models).

[0020] Viewers AU1 and AU2 of user terminals 30a and 30b that request the platform to provide live streaming from the streamer receive streaming data corresponding to the live streaming via the network NW, play the received streaming data, and display videos VD1 and VD2 on a display and output audio from speakers, etc. The videos VD1 and VD2 displayed on the user terminals 30a and 30b, respectively, are substantially the same as the videos VD recorded by the user terminals of the streamer LV, and the audio output from the user terminals 30a and 30b is substantially the same as the audio recorded by the user terminals of the streamer LV.

[0021] Recording of the streamer on the user terminal 20 may occur simultaneously with playback of the streaming data on the user terminals 30a and 30b of the viewers AU1 and AU2. When viewer AU1 inputs a comment about the content of the streamer LV into the user terminal 30a, the server 10 displays the comment in real time on the user terminal 20 of the streamer and also on the user terminals 30a and 30b of the viewers AU1 and AU2. When the streamer LV responds to the comment, the response is output as text, image, video, or audio from the user terminals 30a and 30b of the viewers AU1 and AU2, enabling communication between the streamer LV and the viewers AU1 and AU2. Therefore, the live streaming system enables two-way live streaming.

[0022] 2 is a block diagram showing the functions and configuration of the user terminal 20 shown in FIG. 1 according to an embodiment of the present disclosure. The user terminal 30 has the same functions and configuration as the user terminal 20. The blocks depicted in the block diagrams in this specification are implemented by hardware, such as devices and mechanical components, such as a computer CPU, and software, such as a computer program, that represents functional blocks implemented by the cooperation of these elements. Therefore, it will be understood by those skilled in the art that the functional blocks can be implemented in various ways using a combination of hardware and software.

[0023] The streamer LV and the viewer AU can download and install the live streaming application (live streaming app) of the present disclosure to the user terminals 20 and 30 from a download site via the network NW. Alternatively, the live streaming app may be pre-installed on the user terminals 20 and 30. When the user terminals 20 and 30 execute live streaming, the user terminals 20 and 30 communicate with the server 10 via the network NW and can realize multiple functions. The functions realized by the execution of the live streaming app by the user terminals 20 and 30 (more specifically, a processor such as a CPU) are described below as functions of the user terminals 20 and 30. These functions are basically functions that the live streaming app causes the user terminals 20 and 30 to realize. In some embodiments, these functions may be realized by transmitting them from the server 10 to a web browser of the user terminals 20 and 30 via the network NW and executing them as a computer program in the web browser. The computer program may be written in a programming language such as HTML (Hyper Text Markup Language).

[0024] The user terminal 20 includes a streaming unit 100 and a viewing unit 200. In some embodiments, the streaming unit 100 is configured to record a user's audio and / or video data and generate streaming data to transmit to the server 10. The viewing unit 200 is configured to receive and play the streaming data from the server 10. In some embodiments, a user can activate the streaming unit 100 during a broadcast or activate the viewing unit 200 when viewing a stream. In some embodiments, the user terminal that activates the streaming unit 100 can be referred to as a streamer or the user terminal that generates the streaming data. The user terminal that activates the viewing unit 200 can be referred to as a viewer or the user terminal that plays the streaming data.

[0025] The streaming unit 100 may include a video control unit 102, an audio control unit 104, a distribution unit 106, and a UI control unit 108. The video control unit 102 may be connected to a camera (not shown), and the video is controlled by the camera. The video control unit 102 can acquire the video data from the camera. The audio control unit 104 may be connected to a microphone (not shown), and the audio is controlled by the microphone. The audio control unit 104 can acquire the audio data from the microphone.

[0026] The distribution unit 106 receives streaming data including video data from the video control unit 102 and audio data from the audio control unit 104 and transmits the streaming data to the server 10 over the network NW. In some embodiments, the distribution unit 106 transmits the streaming data in real time, i.e., the generation of the streaming data from the video control unit 102 and the audio control unit 104 and the distribution by the distribution unit 106 are performed simultaneously.

[0027] The UI control unit 108 controls the UI of the streamer. The UI control unit 108 is connected to a display (not shown) and is configured to generate the streaming data for the recipient to whom the distribution unit 106 transmits the streaming data, play it, and display it on the display. The UI control unit 108 is configured to display an object to be operated or an object to be instructed on the display, and to receive tap inputs from the streamer.

[0028] The viewing unit 200 may include a UI control unit 202, a rendering unit 204, an input sending unit 206, and a processing unit 208. The viewing unit 200 is configured to receive streaming data from the server 10 via a network NW. The UI control unit 202 controls the UI of the viewer. The UI control unit 202 is connected to a display (not shown) and / or a speaker (not shown) and configured to display video on the display and output audio from the speaker by playing the streaming data. In some embodiments, outputting video on the display and outputting audio from the speaker can be referred to as "playing streaming data." The UI control unit 202 is connected to an input unit such as a touch panel, a keyboard, or a display and can obtain input from a user.

[0029] The rendering unit 204 may be configured to render streaming data from the server 10 and frame images. The frame images may include user interface objects for accepting input from a user, comments entered by a viewer, and data received from the server 10. The input sending unit 206 is configured to receive the user input from the UI control unit 202 and send it to the server 10 via the network NW.

[0030] In some embodiments, the user input may be clicking an object on the screen of the user terminal, such as selecting a live broadcast, entering a comment, sending a gift, following or unfollowing a user, voting in an event, playing a game, etc. For example, when the user terminal of a viewer clicks a gift object on the screen to send a gift to a streamer, the input sending unit 206 may generate gift information and send it to the server 10 via the Internet NW.

[0031] In some embodiments, the user terminal 20 may include a terminal-side gift DB (not shown). The terminal-side gift DB is configured to store the gift information of the user terminal 20 or 30. The terminal-side gift DB may include a gift ID, an icon ID, an effect data ID, and a last update date. The gift ID may identify the gift. The icon ID may be icon data of the gift object. The effect data ID may identify data that realizes an effect corresponding to the gift. The last update date may identify the date the gift was last updated.

[0032] In some embodiments, the viewer may download the gift in advance or download the gift when sending the gift. In some embodiments, gift information may be downloaded in advance and stored in a terminal-side gift DB. In some embodiments, the terminal-side gift DB may include a gift list corresponding to the gift list provided by the server 10. In some embodiments, the gift list in the terminal-side gift DB may be synchronized with the server 10 periodically or upon user input.

[0033] 3 is a block diagram of a server 10 according to some embodiments of the present disclosure. The server 10 may include a streaming information unit 302, a relay unit 304, a processing unit 306, a stream DB 320, a user DB 322, an AI virtual character DB 324, a game DB 326, video clips 328, a machine learning model 330, and an AI virtual character emotion DB 332.

[0034] The streaming information unit 302 receives a request for live streaming from the user terminal 20 of the streamer via the network NW. Upon receiving the request, the streaming information unit 302 registers information about the live streaming in the stream DB 320. In some embodiments, the information about the live streaming may be a stream ID of the live streaming and / or a streamer ID of the streamer corresponding to the live streaming.

[0035] When a request for providing information about the live streaming is received from the viewer via the network NW from the viewing unit 200 of the user terminal 30, the streaming information unit 302 refers to the stream DB 320 and generates a list of available live streaming.

[0036] The streaming information unit 302 then transmits the list to the user terminal 30 via the network NW. The UI control unit 202 of the user terminal 30 generates a live streaming selection screen based on the list and displays the list on the display of the user terminal 30.

[0037] When the input sending unit 206 of the user terminal 30 receives the selection of live streaming by the viewer on the live streaming selection screen, it generates a delivery request including the stream ID of the selected live streaming and sends it to the server 10 via the network. The streaming information unit 302 can start providing the live streaming specified by the stream ID in the delivery request to the user terminal 30. The streaming information unit 302 can update the stream DB 320 and add the viewer ID of the viewer of the user terminal 30 to the streamer ID of the stream ID.

[0038] The relay unit 304 can relay transmission of the live streaming from the user terminal 20 of the streamer to the user terminal 30 of the viewer in the live streaming started by the streaming information unit 302. The relay unit 304 can receive a signal indicating a user input from the viewer from the input transmission unit 206 during playback of the streaming data. The signal indicating the user input may be an object designation signal indicating designation of an object to be displayed on the display of the user terminal 30. The object designation signal may include the viewer ID of the viewer, the streamer ID of the streamer transmitting the live streaming being viewed by the viewer, and an object ID designated by the object. If the object is a gift or the like, the object ID may be a gift ID or the like. Similarly, the relay unit 304 can receive a signal indicating a user input from the streamer, such as the object designation signal, from the streaming unit 100 of the user terminal 20 during playback of the streaming data.

[0039] In some embodiments, the server 10 may further include a gift information unit (not shown) and a gift processing unit (not shown). The gift information unit provides gift information to corresponding streamers and viewers. The gift information unit transmits gift information for the user terminal to the streamer in response to a request from the user terminal. The gift information unit obtains effect data corresponding to the gift ID included in the gift information signal received by the relay unit 304 from the gift DB. The gift information unit transmits the obtained effect data to other user terminals 20, 30 in response to a gift transmission request signal.

[0040] The gift processing unit updates the user DB 322 to update the points of the streamer and the viewer according to the points of the gift identified by the gift ID included in the gift use signal. Specifically, the gift processing unit references the gift DB to identify the points to be awarded for the gift ID included in the received gift use signal. The gift processing unit then updates the user DB 322 to add the determined points to the points of the streamer ID included in the gift use signal. In some embodiments, the gift processing unit may update the user DB 322 to subtract the determined points from the points of the viewer ID included in the gift use signal.

[0041] Figure 4 shows a table illustrating an example data structure of the stream DB 320 in Figure 3. The stream DB 320 holds information about live broadcasts currently being performed. The stream DB 320 stores a stream ID for identifying a live broadcast on a live streaming platform provided by the live streaming system 1, a streamer ID for identifying a streamer providing the live broadcast, and a viewer ID for identifying a viewer of the live broadcast, in association with each other.

[0042] FIG. 5 is a table illustrating an exemplary data structure of the user DB 322 of FIG. 3. The user DB 322 holds information about users. The user DB 322 stores a user ID for identifying a user, points for specifying points accumulated by the user, a level for identifying the user's level, and a status for identifying the user's status, all of which are associated with one another. The points are electronic value circulated within the live streaming platform. The level may be an indicator of the amount of activity or engagement of the user on the live streaming platform. The status may be the user's identity or membership status on the live streaming platform.

[0043] Figure 6 shows an example data structure of the AI ​​virtual character DB 324 in Figure 3. The AI ​​virtual character DB 324 holds information about current live streams of AI virtual characters. The AI ​​virtual character DB 324 stores an AI virtual character ID that identifies an AI virtual character, a topic that identifies the topic of the AI ​​virtual character, a name that identifies the name of the AI ​​virtual character, a motion ID that identifies the motion of the AI ​​virtual character, a motion description that identifies a description of the motion, and a URL that identifies the location of the AI ​​virtual character model, all of which are associated with one another.

[0044] In some embodiments, the topic of the AI ​​virtual character may be a conversation that the AI ​​virtual character primarily focuses on. For example, if the AI ​​virtual character's setting is an astrologer, the topic may be horoscopes, or if the AI ​​virtual character's setting is a politician, the topic may be politics. In some embodiments, the AI ​​virtual character's name may be displayed in a live streaming room. The motion ID and motion description may correspond to a facial expression, a body movement, or a combination of the AI ​​virtual character, such as smiling, dancing, or running. The URL may indicate the location of the AI ​​virtual character model, which may include files for the AI ​​virtual character, such as a Live2D model or a VRM model.

[0045] In some embodiments, live broadcasts from the AI ​​virtual character may be stored in the stream DB 320 or a separate DB. For example, a live broadcast of an AI virtual character or a group call between an AI virtual character and other live broadcasters may be stored in the stream DB 320 for reference. In some embodiments, information about the AI ​​virtual character may be stored as a user in the user DB 322 or in a separate DB. In some embodiments, the details of each database may be flexibly determined according to actual needs.

[0046] 7 is a table showing an example data structure of the game DB 326 in FIG. 3. The game DB 326 holds information about games. The game DB 326 stores a game ID that identifies a game on the live streaming platform provided by the live streaming system 1, a title that specifies the game title, and a URL that specifies the location of the game, in association with each other.

[0047] In some embodiments, the game may be an online game or an offline game. The game may be an interactive game with live streamers or other users. The game may be pre-downloaded or played in real time over a network. In some embodiments, the implementation of the gameplay may be flexibly determined according to actual needs.

[0048] 8 is a table illustrating an exemplary data structure of the clip DB 328 of FIG. 3. The clip DB 328 holds information about clip content. The clip DB 328 stores a clip ID that identifies the clip content, a duration that identifies the duration of the clip content, and a clip URL that specifies the location of the clip content, in association with one another. In some embodiments, the clip DB 328 may further store information about, for example, the type of clip, the creator of the video clip (which may be a game player if the video is a game interaction), the person uploading the video, and the like, in association with one another.

[0049] In some embodiments, the video clip may be from a live stream. For example, the live streamer or the viewer may play a game in the live streaming room, and the video clip may be a recording of the gameplay. In some embodiments, the video clip may be uploaded by the user. For example, the user may generate the video clip from another website, such as Twitch or any other available resource. In some embodiments, the video clip may be flexibly generated and provided according to actual needs.

[0050] In some embodiments, the machine learning model 330 may be any possible machine learning model, such as a content-based model, a behavior-based model, a language model such as ChatGPT or Google Gemini, or an image generation model such as Midjourney. In some embodiments, the machine learning model may infer possible outputs based on input, and the summarization step may refer to providing the inferences from the model to a user. In some embodiments, the machine learning model may generate images or videos based on input from the live streaming room, etc.

[0051] 10 to 15 are exemplary screen images of a live streaming room screen 600 displayed on the display of a live streamer's user terminal 20 or a viewer's user terminal 30. When a viewer selects and enters a live streaming room, the live streaming room screen 600 of the live streamer may be displayed on the display. The live streaming room screen 600 may include a live streamer information object 602, a live streamer image 604, a message zone 606, a message input box 608, a gift object 610, a sharing object (not shown), etc.

[0052] The live streamer information object 602 displays information about the streamer. The live streamer image zone 604 shows an image or video of the streamer, which may be obtained by playing video data. The message zone 606 displays messages from viewers, streamers, or the server 10. The message input zone 608 is an input zone for the streamer or viewers to input messages, etc. The gift object 610 provides the user with the ability to give gifts to the live streamer, and the share object provides the user with the ability to share the live streaming room. In some embodiments, the UI and related functions within the live streaming room may be flexibly designed.

[0053] In some embodiments, a game interaction button 612 may be provided on the screen 600. The button 612 may include an icon associated with the game and indicate access to game interaction. In some embodiments, the user may play a game on the screen 600 within the live streaming room. In some embodiments, the user may have a single player game or a multiplayer game with other live streamers, viewers, computer players, AI players, etc.

[0054] In some embodiments, the user may upload a video resource as an input for game interaction. The video resource may be an online game or an offline archive from a website such as Twitch or another app or program. In some embodiments, the video resource may be a local video on the user device, and the video may be received by any available means. In some embodiments, access to upload the video clip may be via the button 612 or another UI object on the screen 600.

[0055] In some embodiments, the video resource may be game-related content. For example, the video may be a clip of online or offline gameplay, such as "Super Mario Bros." In some embodiments, the video resource may be any possible content, such as animation, drama, variety show, short video, selfie video, etc. For example, the video resource may be a clip of animation, such as "One Piece," and the clip may be a scene of an argument between the characters "Zoro" and "Sanji." In some embodiments, the video may relate to any entertainment-related content, etc.

[0056] In some embodiments, the live streamer or the viewer may play a game on the screen 600 within the live streaming room. As shown in FIG. 11 , for example, a game zone 614 may be displayed. The game may be an adventure game such as "Super Mario," a car racing game, a tower strategy game, or the like. The user may control a character and play the game using a physical controller or a virtual controller. In some embodiments, the viewer may gift a user an item to help the user complete the game. For example, the viewer may use their points to gift a user a sword to kill monsters.

[0057] In some embodiments, information about the game may be input into a machine learning model. The information about the game may be real-time or offline. In some embodiments, the information about the game may include, for example, streaming data of the game, a game log, etc. The game log may include, for example, background settings of the game, objects in the game, input instructions from the user, movement of the objects in the game, etc. In some embodiments, the game log may include any possible information related to the game.

[0058] In some embodiments, the machine learning model may output hints to assist the user in playing the game. For example, if there is a gap and falling into the gap would kill the character, a hint such as "jump over the gap" may be displayed on the screen 600 to assist the user in completing the game. In some embodiments, the machine learning model may be a deep learning model, and the learning data may be any real-time or past data related to the game.

[0059] As shown in FIG. 12 , hint information HT may be displayed around the portion the user is attempting to complete to assist the user. The machine learning model may infer an output (such as the hint information HT) to assist the user and display it on the screen 600. In some embodiments, the hint information HT may be displayed in the game zone or in a zone separate from the game zone (such as a message zone or a hint zone). In some embodiments, the hint information HT may be fixedly displayed within the game or may be automatically displayed when the user approaches that portion. This embodiment may help the user complete the game more smoothly, improving the user experience.

[0060] After the game interaction is over, information about the game may be further input into a machine learning model. The machine learning model may be the same or a different model from the previous one. The machine learning model may operate as an AI virtual character in the live streaming room, such as an AI virtual live streamer or an AI assistant. In some embodiments, the input to the machine learning model may be prompts describing the personality of the AI ​​virtual character. In some embodiments, the input may be from information about the game, such as streaming data or the game logs described above. In some embodiments, the input may be any online or offline game interaction video, entertainment-related video, or the like.

[0061] In some embodiments, the machine learning model may operate as an AI virtual live streamer, and the learning data may be game or entertainment-related information obtained internally or externally from the Internet, etc. As shown in FIG. 13, the live streamer may enter into a group call with the AI ​​virtual live streamer after completing a game interaction or uploading the video clip. The AI ​​virtual live streamer may have a character, personality, or gaming experience based on the character in the video resource, for example, since the game information is input as a prompt for the AI ​​virtual live streamer.

[0062] In some embodiments, the livestreamer and viewers may discuss the game interactions, and the machine learning model of the AI ​​virtual livestreamer may receive voice or text comments and infer an output as a response to the livestreamer and viewers. For example, the livestreamer or the viewers may ask the AI ​​virtual livestreamer via voice or comment whether they enjoyed the game, and the AI ​​virtual livestreamer may respond via voice, comment, etc., about their impressions of the game.

[0063] In some embodiments, the machine learning model of the AI ​​virtual livestreamer may generate feedback about the game interactions as the livestreamer controls the character. For example, if the livestreamer controls the character and causes the character to fall into a gap three or more times, the AI ​​virtual livestreamer may respond, "Falling into a gap hurts. You're not very good at games." In some embodiments, the response may be generated while the user is playing the game, after the AI ​​virtual character is generated, etc.

[0064] In some embodiments, the AI ​​virtual live streamer may be the only live streamer in the live streaming room. For example, the viewer may create an AI virtual live streamer by playing an online game or uploading a video resource. As shown in FIG. 14 , the AI ​​virtual live streamer is created and displayed on the screen 600. The AI ​​virtual live streamer may act as a character in a game or entertainment-related video and interact with the viewer. For example, the viewer may upload a video clip of their gameplay and create the AI ​​virtual live streamer as a character in the game to discuss the content of the game.

[0065] In some embodiments, the video resource may be an entertainment-related video, such as an animation or drama clip. For example, the viewer or the live streamer may upload a drama video clip, such as a love story about an actor and an actress. The character settings of the actor or actress may be input into the machine learning model to generate an AI virtual live streamer with the actor's or actress's characteristics. In some embodiments, the character settings may be obtained from the video clip, the Internet, or the like. The viewer or the live streamer may talk about the storyline or ask the actor or actress to act out a similar scene together. This embodiment may make interactions within the live streaming room more interesting and improve the user experience.

[0066] In some embodiments, two or more AI virtual live streamers may be generated and interact with each other and with the live streamer and viewers in the live streaming room. As shown in FIG. 15, two AI virtual live streamers may interact with multiple users in the live streaming room, such as through PK or group calls. In some embodiments, the number of AI virtual live streamers may be flexibly determined. In some embodiments, the interaction with multiple users may involve only one or more AI virtual live streamers, or may involve live streamers including human live streamers, virtual live streamers, audio live streamers, etc.

[0067] In some embodiments, one or more AI virtual live streamers may be generated from video resources of game interactions. For example, if the game is the Super Mario series, characters Mario and Luigi may be generated as AI virtual live streamers and displayed on the screen 600 in the live streaming room. In some embodiments, a microphone object 616 may be displayed above a person speaking in the live streaming room to clarify who is speaking. This embodiment may make interactions in the live streaming room more interesting and improve the user experience.

[0068] In some embodiments, the AI ​​virtual characters may be generated from the same or different video clips. For example, virtual characters of Sanji and Zoro may be generated from a clip of "One Piece" and interact with each other and other users. A user may also upload one or more video clips and generate different AI virtual characters to interact with each other. For example, a user may upload a video clip of "Doraemon" and "Pikachu" and generate AI virtual characters such as "Doraemon" and "Pikachu" within the live streaming room.

[0069] 16-17 are block diagrams of the server 10 with details regarding an AI virtual character according to some embodiments of the present invention. The configuration of the AI ​​virtual character in the server 10 may further include a prompt intelligence unit 500, a perception engine unit 520, an engagement engine unit 530, an emotion engine unit 540, a persona engine unit 550, an interactive model engine unit 560, and an AI model engine unit 570.

[0070] The prompt intelligence unit 500 is configured to merge inputs from the above units. The inputs may be current perceptions, character characteristics, feedback and actions, criteria and engagement, rendered content, current emotions, etc. The prompt intelligence unit 500 may merge them together to generate a basic prompt as a setting for the AI ​​virtual character. In some embodiments, the basic prompt may be input to the machine learning model to generate the AI ​​virtual character. When a viewer in the live streaming room asks the AI ​​virtual character a question, the question may be input to the machine learning model to generate a response to the question.

[0071] In some embodiments, the prompt intelligent unit 500 may be referred to as the brain of the AI ​​virtual character and may be configured to send and receive information to and from a corresponding unit. In some embodiments, the prompt intelligent unit 500 may process prompts, emotions, responses, etc. based on criteria and engagement within the live streaming room. In some embodiments, the prompt intelligent unit 500 may receive feedback from and provide actions to an AI model of the AI ​​virtual character.

[0072] In some embodiments, the prompt intelligence unit 500 may generate prompts including emotions and traits and infer responses. The prompt intelligence unit 500 may receive information such as current perceptions from the perception engine unit 520, criteria and engagement from the engagement engine unit 530, emotion tags from the emotion engine unit 540, character traits from the persona engine unit 550, rendering content from the interaction model engine unit 560, and feedback and actions from the AI ​​model engine unit 570.

[0073] In some embodiments, the criteria and engagement may be corresponding parameters in the live streaming room that indicate the degree of objectivity or subjectivity of interactions in the live streaming room. For example, the criteria may be the user's level, the amount of donations from the viewers, the user's status, etc. The engagement may be, for example, the number of comments from the viewers, the degree of engagement from the viewers, etc. In some embodiments, the current emotion or the emotion tag may include emotions from the AI ​​virtual character, the human live streamer, the viewers, etc. In some embodiments, the emotion tag may be determined by an internally pre-trained machine learning model or a third-party machine learning model from Google, etc.

[0074] In some embodiments, the characteristics or emotions of the AI ​​virtual character may be obtained from an uploaded video or obtained from the internet based on the context or setting of the video. In some embodiments, the characteristics or emotions of the AI ​​virtual character may be manually entered by an administrator or automatically entered, such as by another machine learning model. The characteristics or emotions may be saved as prompts for use in generating prompts for the AI ​​virtual character. In some embodiments, the prompt intelligence unit 500 may include a corresponding database for each parameter. The prompt intelligence unit 500 may determine a prompt as a setting for the AI ​​virtual character based on the parameters.

[0075] The perception engine unit 520 is configured to generate and send a perception to the prompt intelligence unit 500. In some embodiments, the perception may be an emotion such as sadness, anger, or any other emotional change. In some embodiments, the perception may be the current situation of the streaming room. Based on the perception engine unit 520, the user's emotion and the atmosphere of the live streaming room may be understood by the AI ​​virtual live streamer.

[0076] In some embodiments, the emotion or the perception of the current situation may be inferred by a machine learning model. In some embodiments, input to the machine learning model may be visual or audio data from the live streaming room, such as visions or voices within the live streaming room. In some embodiments, input to the machine learning model may be external input (e.g., comments) from the live streamer or viewers.

[0077] In some embodiments, the machine learning model may infer a perception based on the inputs described above. In some embodiments, the perception may be an emotion, such as sadness or happiness, or a topic in the live streaming room, such as travel, food, or any other possible content. In some embodiments, the corresponding database for storing the input data, such as snapshots, clips, interaction logs, etc., may be provided in the perception engine unit 520.

[0078] In some embodiments, the perception engine unit 520 may send a push notification to the user based on the perception. That is, the perception engine unit 520 may send a notification sharing an emotion to a user who is not on the live streaming platform. More specifically, the perception engine unit 520 may send the notification in response to detecting a perception indicating a specific event that occurred in the live streaming room. For example, if an interesting topic is being discussed in the live streaming room, the perception engine unit 520 may send a notification to a user who is not on the live streaming room so that the user can know that a celebration is underway in the live streaming room and join in, thereby enjoying the time with the live streamer and viewers.

[0079] In some embodiments, the perception engine unit 520 may send the push notification in response to a change in perception in the live streaming room. For example, if the perception in the live streaming room changes from "gloomy" to "happy," the perception engine unit 520 may send a push notification containing information about a happy atmosphere to a user who is not in the live streaming room. According to this embodiment, the viewer can join the live streaming room when the perception of the atmosphere in the live streaming room is good, which may improve the user experience.

[0080] In some embodiments, the perception engine unit 520 may detect a highlight moment in the live streaming room based on the perception, and then create a clip of the highlight moment. For example, when a joyful perception indicating a celebratory moment is detected in the live streaming room, the perception engine unit 520 may clip the moment as a highlight video of the live streaming room. The perception engine unit 520 may further transmit the highlight video to the live streamer and the viewers in the live streaming room. In some embodiments, the perception engine unit 520 may generate a post of the celebratory moment including the highlight video for viewing by users in the live streaming platform.

[0081] The engagement engine unit 530 is configured to evaluate engagement in the live streaming room. The engagement engine unit 530 may evaluate the engagement according to multiple parameters or business matrices. For example, the parameters may be the performance of the live streamer, such as popularity, gift rate, voice volume, changes in revenue such as received points, comments from viewers, topics in the live streaming room, etc. For example, if the tone of the live streamer's voice is not flat, but is high-pitched, low-pitched, or both, it may indicate that the live streamer is conversing with viewers.

[0082] In some embodiments, the engagement engine unit 530 may include engagement criteria for evaluating the performance of the current prompt for the AI ​​virtual character. The criteria may be, for example, the level of interaction in the live streaming room, the increase or decrease in viewers, comments in the live streaming room, etc. Based on the criteria, the prompt may be changed or rolled back to an older version of the prompt. In some embodiments, the machine learning model may include, for example, a deep learning machine learning model for training the AI ​​virtual character to maintain more positive interactions in the live streaming. That is, the prompt for the AI ​​virtual character may change in real time to ensure the AI ​​virtual character better supports the live streamer.

[0083] In some embodiments, certain other features may affect the engagement in the live streaming room. For example, features such as translation, recommendations, and a rich profile page may also lead to high engagement in the live streaming room. For example, if the live streaming room provides accurate translation, viewers from various countries may be able to interact with the live streamer, which may increase engagement rates.

[0084] The emotion engine unit 540 is configured to generate emotions for the AI ​​virtual character. In some embodiments, the input may be a prompt-like sentence that indicates one or more emotions for the AI ​​virtual character. In some embodiments, the emotions may be directed toward all other users or a specific target user. For example, if a viewer angers the AI ​​virtual character, the AI ​​virtual character may have an angry emotion toward the viewer and a neutral emotion toward other users.

[0085] 9 shows an example data structure of the AI ​​virtual character emotion DB332 of FIG. 3. The AI ​​virtual character emotion DB332 may store an AI virtual character ID that identifies an AI virtual character, a user ID that identifies a user, and an emotion that identifies the AI ​​virtual character's emotion toward the user, in association with one another. In some embodiments, the user ID may indicate a specific user or a group of users in the live streaming room. For example, the AI ​​virtual character AI017 may have an angry emotion toward the user SS5K and a happy emotion toward other users in the live streaming room.

[0086] In some embodiments, the emotion engine unit 540 may infer emotions based on psychological phenomena such as the halo effect. More specifically, the emotion engine unit 540 may infer emotions from the AI ​​virtual character toward user B based on the emotion from the AI ​​virtual character toward user A and the value of interactions between viewer A and viewer B. For example, if the AI ​​virtual live streamer has anger or negative emotions toward viewer A, and viewer A and viewer B have a good relationship based on their interactions in the live streaming room, the emotion engine unit 540 may infer that the AI ​​virtual live streamer may also have anger or negative emotions toward viewer B.

[0087] In some embodiments, there may be a parameter indicating the interaction between users in the live streaming room. The parameter may be, for example, an interaction value, which may range from -100 to 100. Values ​​below -80, between -80 and -40, between -40 and 40, between 40 and 80, and above 90 may indicate very bad, bad, average, good, and very good, respectively. In some embodiments, the interaction value may be determined based on the user's interaction in the live streaming room. For example, if viewer A interacts with viewer B in a positive way, such as tagging viewer B, clicking "like" for viewer B, or sending a gift to viewer B, the interaction value between viewer A and viewer B may be increasingly higher, and vice versa.

[0088] In some embodiments, the similarity of emotions from the AI ​​virtual character to user A and user B may be determined based on an interaction value between user A and user B. More specifically, the emotions from the AI ​​virtual character to user A and user B may be the same or similar if the interaction value between user A and user B exceeds a threshold. For example, if the interaction value is higher than 80, the emotions may be the same. If the interaction value is between 40 and 80, the emotions may be similar. Based on emotion inference, interactions within the live streaming room may be more interesting, improving the user experience.

[0089] In some embodiments, the emotion engine unit 540 may include an emotion enhancement function and an emotion suppression function. The emotion enhancement function and the emotion suppression function are configured to determine whether to enhance or suppress the emotion from the AI ​​virtual live streamer to other users. For example, if the emotion from the AI ​​virtual live streamer to viewer A is anger and the viewer says something nice to the AI ​​virtual live streamer, the emotion from the AI ​​virtual live streamer to viewer A may become less angry or more happy based on inference from the emotion engine unit 540.

[0090] In some embodiments, the emotion inference by the emotion engine unit 540 may be realized, for example, by a fuzzy inference system. The fuzzy inference system is a computational framework based on fuzzy logic designed to simulate human reasoning processes and may include processes such as fuzzification, rule-based, inference engine, and defuzzification. For example, the fuzzy inference system may be used to classify the degree of emotion, such as very angry, angry, not angry, neutral, etc.

[0091] The persona engine unit 550 is configured to generate basic characteristics of the AI ​​virtual live streamer. For example, if the AI ​​virtual live streamer is "Super Mario," the persona engine unit 550 extracts "Mario's" characteristics, story background, etc. In some embodiments, the persona engine unit 550 may obtain information from the video clip or any available resource (e.g., the Internet). In some embodiments, the character's characteristics may be generated as a simple description, such as prompt-like content.

[0092] In some embodiments, the persona engine unit 550 may analyze characteristics of the live streamer or the viewer. For example, if a viewer consistently behaves badly toward other viewers in the live streaming room, the persona engine unit 550 may associate the viewer's characteristics as negative. In some embodiments, characteristics of other users may also be sent to the prompt intelligence unit 500 to generate responses to comments from the viewer. For example, if the viewer consistently behaves badly in the live streaming room, when the viewer asks a question, the AI ​​virtual live streamer may respond negatively to the viewer's question.

[0093] The interactive model engine unit 560 is configured to process the generation of the virtual character. In some embodiments, the interactive model engine unit 560 may perform rendering of interactions between the AI ​​virtual character and other users in the live streaming room. The virtual character may be used to become a virtual live streamer, an AI virtual live streamer, or the like. In some embodiments, the interactive model engine unit 560 may generate avatars for the live streamer, viewers, AI live streamers, or the like. In some embodiments, the generation of the virtual character may be achieved via a third-party service such as Preferred Network, LIVE 2D, or the like. In some embodiments, the virtual character may be rendered and displayed on the screen 600.

[0094] In some embodiments, the interactive model engine unit 560 may include a model engine that processes the virtual character model, a model controller that determines the movement and behavior of the virtual character, and a model renderer that renders the virtual character model on the screen 600. In some embodiments, the interactive model engine unit 560 may further include materials for rendering background materials, audio materials, emotion materials, decoration materials, property materials, interaction materials, etc.

[0095] In some embodiments, the model engine, model controller, and model rendering may determine the interactions and rendering of the virtual character based on the above-described materials. In some embodiments, the materials for generating a virtual character may be provided by default by server 10, uploaded by a user, generated by a machine learning model, etc. For example, the user may upload images or videos of "Mario" to generate a "Mario" virtual character.

[0096] In some embodiments, the interactive model engine unit 560 may include a decoration unit. The decoration unit is configured to render decorations, such as background effects and special effects, in the live streaming room. In some embodiments, the decoration unit may decorate the live streaming room based on the perception. For example, if a topic in the live streaming room is related to Christmas, a snowy background effect may be rendered and displayed on the screen 600. In another example, the decoration unit may render a fireworks effect if the viewer in the live streaming room comments a keyword such as "fireworks."

[0097] The AI ​​model engine unit 570 is configured to generate and manage AI models. The AI ​​model engine unit 570 may include multiple machine learning models, such as LLM, for generating interactions within the live streaming room. For example, the AI ​​model may receive feedback from the live streaming room and trigger interactions. In some embodiments, the AI ​​model engine unit 570 may manage information about co-host live streamers. The co-host live streamers may be AI virtual characters, human live streamers, virtual live streamers, etc. In some embodiments, the AI ​​model engine unit 570 may include necessary parameters, such as emotions, responses, prompts, and corresponding databases.

[0098] 18 is a schematic diagram illustrating an example functional configuration of a live streaming system 1 related to an AI virtual character based on some embodiments of the present invention. The live streamer or the viewer may play a game on the screen 600 in the live streaming room. Streaming data of the gameplay may be pushed to the server 10, more specifically, to a streaming server (S502). In some embodiments, instead of streaming data of real-time gameplay, video clips related to the game interaction or entertainment may be uploaded by the user.

[0099] In some embodiments, image or video processing may be performed on the video clip (S504). For example, the video clip may be a series of screenshots, or information such as a background setting of the video may be extracted from the video, and audio data such as the voice of the character may be extracted from the video and generated as the voice of the character in the live streaming room.

[0100] Once the image or video processing is complete, information from the video may be input into a machine learning model to generate information for the AI ​​virtual character (S506). In some embodiments, game logs and other necessary information, such as the information described above, may also be input into the machine learning model (S508). In some embodiments, the machine learning model may be a self-trained or pre-trained machine learning model, or a third-party machine learning model, such as Gemini or ChatGPT. In some embodiments, the information input into the machine learning model may be a prompt-like description containing the necessary information. A description of the information is provided below. Machine learning model 1: Here's a video of the streamer playing the Princess Rescue game. ## Game Overview #### Main character: Mako #### Objective: Rescue the princess from the monster kingdom ## Game Features - Heart: Make Mako bigger #### enemy: - Monsters: Basic enemies that can be killed by jumping on them. #### Possible scenarios for the game: - If you hit an enemy, you will be injured. - Falling into a hole or gap will kill you.

[0101] In some embodiments, the machine learning model may generate the AI ​​virtual character or information for the AI ​​virtual character based on the inputs. In some embodiments, some instructions to the machine learning model may be input as statements, such as prompts, instructing the AI ​​virtual character to perform certain actions. For example, a prompt such as "Please provide a brief summary of your gameplay based on the content of the video and tell us how you felt about the process" may generate a summary of a game video clip.

[0102] In some embodiments, the prompts may include, for example, "Tell me how you feel from a first- or third-person perspective," with summaries from the main character himself or the viewer. In some embodiments, a baseline or scenario may be provided to the machine learning model, and a corresponding output may be generated. For example, a prompt such as "You really hate falling into holes, and it makes you very angry" may be input to the machine learning model to teach the AI ​​virtual character that falling into gaps in a game is a bad idea.

[0103] In some embodiments, the input may include a statement asking the machine learning model to provide an evaluation of the player's performance in the game. For example, if the live streamer controls the virtual character and plays the game, the input may include, "Do you think the player performed well or poorly?" to ask the machine learning model whether the player is good or bad at the game. For example, if the player repeatedly falls into gaps, the machine learning model may output, "That was poor because the player died or ran into enemies a lot."

[0104] Based on the above inputs, the machine learning model may generate a description that summarizes the above (S510). Below is an example of output from the machine learning model: Example output: As Mako, I began my journey to rescue the princess in the Monster Kingdom. Things were going well until the player caused me to fall into a pit and I lost a life. …We continued to move forward, but the player kept bumping into enemies and falling into pits. I died two more times! …I'm still stuck in the Monster Kingdom, and the princess is waiting to be rescued. …

[0105] Based on the above output, a perception concept of the AI ​​virtual character may be extracted by the perception engine unit 520, and the perception information may be stored in a corresponding database (S512). The perception information may be further sent to the prompt intelligence unit 500 for prompt generation (S514). In some embodiments, the perception information may also be sent to the emotion engine unit 540 for emotion generation (S516).

[0106] In some embodiments, the basic characteristics of the AI ​​virtual character generated by the persona engine unit 550 may be further transmitted to the prompt intelligence unit 500 (S518). In some embodiments, the basic characteristics of the AI ​​virtual character may be the same as the background setting of the video clip, etc. In some embodiments, the basic characteristics may be provided manually by an administrator or automatically from the internet or a machine learning model. For example, to indicate the occupation and place of birth as the basic characteristics of the AI ​​virtual character, an engineer may enter "You are a fortune teller from Paris." In some embodiments, if the AI ​​virtual character is a famous character, the basic characteristics may be obtained from the internet or a pre-trained machine learning model.

[0107] Upon receiving information from the perception engine unit 520 and the persona engine unit 550, the prompt intelligence unit 500 may generate prompts for the AI ​​virtual character (S520). In some embodiments, the prompt intelligence unit 500 may perform self-adaptive prompt generation (S522). More specifically, the self-adaptive prompt generation is a type of natural language processing (NLP) prompt. For example, the AI ​​virtual character's responses in the live streaming room may be adjusted or personalized based on interactions in the live streaming room, etc.

[0108] Once the prompt for the AI ​​virtual character is generated, it may be used to generate or dynamically update the AI ​​virtual character. In some embodiments, the AI ​​virtual character may be generated via a machine learning model. More specifically, the generated prompt may be input into the machine learning model to generate the AI ​​virtual character. In some embodiments, the machine learning model may be a pre-trained or third-party machine learning model, such as ChatGPT. The following is an example of a prompt for generating the AI ​​virtual character: Prompt from the AI ​​virtual character You are the game character "Mako," a hero of the Monster Kingdom, and you are currently interacting with players who control you on a streaming platform. The following [game progression] is your experience, and you have to interact with the player based on this progression, and you have fun making fun of the player's mistakes. [Game Progression]: {Output from machine learning model in step S510} Chat History: {history}

[0109] In some embodiments, the description of the "game progress" may be obtained from the machine learning model in step S510. In some embodiments, the "chat history" may be obtained from voice or text comments from the user in the live streaming room. In some embodiments, the machine learning model in step S510 may generate the "game progress" once. In some embodiments, the machine learning model in step S524 may dynamically generate or update the AI ​​virtual character while the AI ​​virtual character is displayed in the live streaming room. In some embodiments, the machine learning model in step S510 may be the same as or different from the machine learning model in step S524. In some embodiments, the machine learning models in steps S510 and S524 may be machine learning models pre-trained on the server 10 or third-party machine learning models such as Google Gemini and ChatGPT.

[0110] Once the AI ​​virtual character is created or updated (S524), the AI ​​model engine unit 570 may generate the AI ​​virtual character and display it in the live streaming room (S526). In some embodiments, the virtual character model's construction, such as the virtual character's visual and audio context, may be generated internally or applied by a third-party service, such as Preferred Network or Live2D. As shown on the right side of FIG. 18 , a human live streamer may enter into a group call with the AI ​​virtual character to, for example, discuss game interactions.

[0111] Furthermore, interactions within the live streaming room may be fed back to the prompt intelligence unit 500. In some embodiments, the interactions may be, for example, engagements between the viewers or the live streamer and the AI ​​virtual character. The engagement engine unit 530 may acquire the engagements within the live streaming room (S528). In some embodiments, information of the engagements may be stored in a corresponding database (S530).

[0112] In some embodiments, the engagement information may be used by the emotion engine unit 540 to dynamically generate emotions for the AI ​​virtual character (S532). The emotions of the AI ​​virtual character may change based on the user's interactions and engagement in the live streaming room. In some embodiments, the emotions of the AI ​​virtual character toward other users may be different for the same or different users. In some embodiments, the emotion information may be stored in a corresponding database (S534).

[0113] Further, the engagement and emotion information may be sent to the prompt intelligence unit 500 to generate or update the prompt for the AI ​​virtual character (S536 and S538). In some embodiments, the prompt intelligence unit 500 may further perform steps S520, S522, S524, and S526 to generate or update the prompt for the AI ​​virtual character (S520). This embodiment allows the live streamer and viewers to interact with the AI ​​virtual character more smoothly and efficiently.

[0114] According to the book, this could improve communication between viewers and AI virtual characters, as well as the quality of live streaming platforms featuring AI virtual characters, thereby improving user experience.

[0115] 19 is a schematic block diagram of a system configuration and computer hardware for executing processing according to some embodiments of the present disclosure. The information processing device 900 shown in FIG. 19 is configured to realize, for example, the server 10 and the user terminals 20 and 30 according to some embodiments of the present disclosure.

[0116] The information processing device 900 includes a CPU 901, a read-only memory (ROM) 903, and a random access memory (RAM) 905. The information processing device 900 may further include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input unit 915, an output unit 917, a storage unit 919, a drive 921, a connection port 925, and a communication unit 929. The information processing device 900 may also include an imaging device (not shown), such as a camera. The CPU 901 is an example of a hardware configuration for implementing functions realized by the components described herein. The functions described herein may be implemented by a circuit programmed to implement the described functions. The circuit programmed to implement the functions described herein may include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a general-purpose processor, an application-specific processor, an integrated circuit, an ASIC (Application Specific Integrated Circuit), and / or a combination thereof. In this specification, a unit for realizing a specific function may be realized as a circuit programmed to realize the function, including, but not limited to, the streaming unit 100, the viewing unit 200, the video control unit 102, the audio control unit 104, the delivery unit 106, the UI control unit 108, the UI control unit 202, the rendering unit 204, the input sending unit 206, the streaming information unit 302, the relay unit 304, the processing unit 306, the stream DB 320, the user DB 322, the AI ​​virtual character DB 324, the game DB 326, the video clip 328, the machine learning model 330, the AI ​​virtual character emotion DB 332, the prompt intelligence unit 500, the cognition engine unit 520, the engagement engine unit 530, the emotion engine unit 540, the personality engine unit 550, the interaction model engine unit 560, the AI ​​model engine unit 570, etc.

[0117] The CPU 901 functions as an arithmetic processing unit and control device, controlling the overall operation of the information processing device 900 or a portion of the operation thereof in accordance with various programs stored in the ROM 903, RAM 905, storage unit 919, or removable recording medium 923. For example, the CPU 901 controls the overall operation of each functional unit included in the server 10 and the user terminals 20 and 30 of the above-described embodiments. The ROM 903 stores programs and operating parameters used by the CPU 901. The RAM 905 transiently stores programs used by the CPU 901 when executing the programs and parameters that change as appropriate when the programs are executed. The CPU 901, the ROM 903, and the RAM 905 are connected to one another via a host bus 907, which is an internal bus such as a CPU bus. The host bus 907 is connected to an external bus 911, such as a Peripheral Component Interconnect / Interface (PCI) bus, via the bridge 909.

[0118] The input unit 915 is a device operated by a user, such as a mouse, keyboard, touch panel, button, switch, or lever. The input unit 915 may be a device that converts a physical quantity into an electrical signal, such as an audio sensor (such as a microphone), an acceleration sensor, a tilt sensor, an infrared sensor, a depth sensor, a temperature sensor, or a humidity sensor. The input unit 915 may be, for example, a remote control device that uses infrared or another type of radio wave. Alternatively, the input unit 915 may be an external connection terminal 927, such as a mobile phone, that supports the operation of the information processing device 900. The input unit 915 includes an input control circuit that generates an input signal based on information input by a user and outputs the generated input signal to the CPU 901. The user operates the input unit 915 to input various data and instruct the information processing device 900 to perform processing operations.

[0119] The output unit 917 includes a device that can visually or audibly notify the user of the acquired information. The output unit 917 may be, for example, a display device such as an LCD, PDP, or OLED, an audio output device such as a speaker or headphones, a printer, etc. The output unit 917 outputs the results obtained by the processing performed by the information processing device 900 in the form of text, video such as an image, or sound such as voice.

[0120] The storage unit 919 is a data storage device and is an example of a storage unit of the information processing device 900. The storage unit 919 includes, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc. The storage unit 919 stores programs and various data executed by the CPU 901, as well as various data acquired from the outside.

[0121] The drive 921 is a reader / writer of a removable recording medium 923 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the information processing device 900. The drive 921 reads information recorded on the attached removable recording medium 923 and outputs the information to the RAM 905. The drive 921 writes information to the attached removable recording medium 923.

[0122] The connection port 925 is a port used to directly connect a device to the information processing device 900. The connection port 925 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. The connection port 925 may also be an RS-232C port, an optical audio terminal, an HDMI (High-Definition Multimedia Interface (registered trademark)) port, or the like. Connecting an external connection terminal 927 to the connection port 925 enables the exchange of various data between the information processing device 900 and the external connection terminal 927.

[0123] The communication unit 929 is, for example, a communication interface including a communication device for connecting to a communication network NW, and may be, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (registered trademark), or Wireless USB (WUSB).

[0124] The communication unit 929 may be, for example, a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communication purposes. For example, the communication unit 929 uses a predetermined protocol such as TCP / IP to transmit and receive signals over the Internet or with other communication devices. The communication network NW to which the communication unit 929 connects is a network established by a wired or wireless connection. The communication network NW is, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0125] The imaging device (not shown) is a device that captures real space and generates a captured image using an imaging element such as a CCD (charge-coupled device) or a CMOS (complementary metal-oxide semiconductor) and various components such as a lens for controlling the formation of a subject image on the imaging element. The imaging device may capture either still images or moving images.

[0126] The live streaming system 1 of the present disclosure has been described above with reference to the embodiments. The above-described embodiments have been described merely for the purpose of explanation. Rather, those skilled in the art will readily appreciate that the above-described components and processes of the embodiments can be combined in various ways and modified in various ways, and these are also within the technical scope of the present disclosure.

[0127] The steps described in this specification, particularly the steps described using flowcharts or flow charts, may include some of the steps constituting the steps, may include steps not explicitly included in the steps constituting the steps, and / or may be rearranged in order. Steps that are the subject of such omission, addition, or rearrangement are also included within the scope of the present disclosure as long as they do not deviate from the gist of the present disclosure.

[0128] In some embodiments, at least some of the functions performed by the server 10 may be performed by a device other than the server 10, for example, by the user terminal 20 or 30. In some embodiments, at least some of the functions performed by the user terminal 20 or 30 may be performed by a device other than the user terminal 20 or 30, for example, by the server 10. In some embodiments, rendering of frame images may be performed by the user terminal of a viewer, server, streamer, or the like.

[0129] Furthermore, the system or method described in the above embodiments may be provided in a non-transitory computer-readable storage device such as a solid-state storage device, an optical disk storage device, or a magnetic disk storage device, or in a computer program product, etc. Alternatively, the program may be downloaded from a server via the Internet.

[0130] Although the technical contents and features of the present disclosure have been described above, those skilled in the art can still make many variations and modifications without departing from the teachings and disclosure of the present disclosure. Therefore, the scope of the present disclosure is not limited to the embodiments already disclosed, but is within the scope of the appended claims, including other variations and modifications that do not depart from the present disclosure. [Explanation of symbols]

[0131] 1. Live Streaming System 10 Servers 20 User terminal 100 Streaming Units 102 Video control unit 104 Audio control unit 106 Delivery Unit 108 UI control unit 200 viewing units 202 UI control unit 204 rendering units 206 Input Unit 30, 30a, 30b User terminal 302 Streaming Information Unit 304 Relay unit 306 Processing Unit 320 Stream DB 322 User DB 324 Virtual Character DB 326 Game DB 328 Video Clip DB 330 Machine Learning Model 332 AI Virtual Character Emotion DB 600 screens 602 objects 604 images 606 Message Zone 608 Message input box 610 Gift Object 612 Button 614 Game Zone 900 Information Processing Equipment 901 CPU 903 ROM 905 RAM 907 Host Bus 909 Bridge 911 External Bus 913 Interface 915 Input Unit 917 Output Unit 919 Storage Unit 921 Drive 923 Removable Recording Media 925 connection port 927 External connection terminal 929 Communication Unit LS Live Streaming LV Live Streamer NW Network AU1, AU2 viewers S502~S536 process VD, VD1, VD2 video

Claims

1. 1. A method for managing streams in a live streaming platform, comprising: receiving a video clip from a first user terminal of a first user; inputting the video clip into a first machine learning model to generate a first virtual character; displaying the first virtual character in a live streaming room; receiving comments from the live streaming room; generating a response to the comment via the first machine learning model; displaying the response via the first virtual character in the live streaming room; A method comprising:

2. the video clip is generated from a real-time game played by the first user in the live streaming room or an offline video uploaded by the first user; The method of claim 1 , wherein the virtual character is a machine learning model trained with information about characters in the video clip.

3. determining a first emotion from the first virtual character to a second user in the live streaming room; determining a second emotion from the first virtual character to a user other than the second user in the live streaming room; 10. The method of claim 1, wherein the first emotion is different from the second emotion.

4. determining a first emotion from the first virtual character to a second user in the live streaming room; determining a second emotion from the first virtual character to a third user in the live streaming room; 2. The method of claim 1, wherein the first emotion is the same as the second emotion in response to an interaction between the second user and the third user being above or below a threshold.

5. determining a perception within the live streaming room; and in response to a detected particular perception or a changed perception in the live streaming room, sending a push notification to a user who is not in the live streaming room, the push notification including information of the perception.

6. obtaining information from the video clip; inputting the information into the first machine learning model; 2. The method of claim 1, wherein the information includes a story setting for the video clip, a character trait or an emotion of the character in the video clip.

7. generating a collaborative broadcast stream including the first virtual character and the first user of the first user terminal; and displaying the collaborative broadcast delivery within the live streaming room.

8. inputting the video clip into a second machine learning model to generate a second virtual character; generating a collaborative broadcast stream including the first virtual character and the second virtual character; and displaying the collaborative broadcast delivery within the live streaming room.

9. 1. A server including circuitry, the circuitry comprising: receiving a video clip from a first user terminal of a first user; inputting the video clip into a first machine learning model to generate a first virtual character; Displaying the first virtual character in a live streaming room; A function of receiving comments from the live streaming room; generating a response to the comment via the first machine learning model; displaying the response via the first virtual character in the live streaming room; A server configured to execute the

10. A computer program, comprising: receiving a video clip from a first user terminal of a first user; inputting the video clip into a first machine learning model to generate a first virtual character; Displaying the first virtual character in a live streaming room; A function of receiving comments from the live streaming room; generating a response to the comment via the first machine learning model; displaying the response via the first virtual character in the live streaming room; A computer program characterized by realizing the above.

Images