Interaction system, program and interaction method

The dialogue system addresses the mismatch between AI response and speaker emotions by estimating and aligning the AI's response characteristics with the speaker's emotional state, enhancing interaction comfort and efficiency.

JP2024164644APending Publication Date: 2024-11-27LY CORP

Patent Information

Application Number
JP2023080278
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-15
Publication Date
2024-11-27

AI Technical Summary

Technical Problem

Conventional dialogue systems using AI agents fail to consider the response characteristics in relation to the speaker's emotions, leading to a mismatch between the AI's response and the speaker's tension and atmosphere, causing discomfort.

Method used

A dialogue system that includes an estimation unit to determine the speaker's emotion based on expression information and a feature determining unit to set response characteristics for the AI agent, ensuring the AI's response aligns with the speaker's emotions.

Benefits of technology

The system reduces the discomfort felt by speakers interacting with AI agents by matching the AI's response characteristics to their emotional state, improving interaction quality and reducing computational load.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164644000001_ABST
    Figure 2024164644000001_ABST
Patent Text Reader

Abstract

To provide an interaction system, a program and an interaction method, which allow a speaker to interact with an AI agent without feeling discomfort.SOLUTION: An interaction system controls an AI agent interacting with a speaker and includes: an estimation unit for estimating emotion of the speaker based on expression information on expression of the speaker; and a feature determination unit for determining a response feature when the AI agent makes a response based on emotion of the speaker, which the estimation unit estimates. The response feature when the AI agent makes a response is determined based on emotion of the speaker. Thus, the speaker can interact with the AI agent without feeling discomfort.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a dialogue system, a program, and a dialogue method. [Background technology]

[0002] Patent Document 1 discloses a technology that, when a customer and a person in charge have a voice dialogue, proposes a conversation text to the person in charge based on voice information uttered by the customer. In recent years, voice dialogue between a speaker and an AI agent that uses an AI agent that converses with a person has been performed. The technology disclosed in Patent Document 1 can also be applied to voice dialogue between a speaker and an AI agent. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-157419 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional dialogue response using an AI agent to which the technology disclosed in Patent Document 1 is applied, no consideration was given to determining the response features (response features) that are the manner in which the AI ​​agent's response content is transmitted. As a result, a gap was created between the AI ​​agent's response features and the speaker's tension and atmosphere, which sometimes made the speaker feel uncomfortable. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, the dialogue system is a system for controlling an AI agent that dialogues with a speaker, and includes an estimation unit that estimates the speaker's emotions based on expression information related to the speaker's expression, and a feature determination unit that determines response features when the AI ​​agent responds based on the speaker's emotions estimated by the estimation unit.

[0006] According to a second aspect of the present disclosure, a program is executed by a computer to control an AI agent that engages in a dialogue with a speaker, the program executing on the computer includes estimating the speaker's emotions based on expression information related to the speaker's expression, and determining response features for the AI ​​agent to respond based on the estimated speaker's emotions.

[0007] According to a third aspect of the present disclosure, a dialogue method is a method performed by a computer for controlling an AI agent engaging in a dialogue with a speaker, the method including: estimating an emotion of the speaker based on expression information related to the speaker's expression; and determining response features for the AI ​​agent to respond based on the estimated emotion of the speaker. [Brief description of the drawings]

[0008] [Figure 1] 1 is a schematic diagram showing an overall overview of a dialogue system in a first embodiment. [Diagram 2] FIG. 1 is a diagram showing the device configuration of an interactive system according to a first embodiment. [Diagram 3] 1 is a functional block diagram of a main part of the dialogue system according to the first embodiment. [Figure 4] 4 is a flowchart showing an example of the operation of the dialogue system in the first embodiment. [Diagram 5] FIG. 4 is a functional block diagram of a modified example of the dialogue system in the first embodiment. [Figure 6] FIG. 11 is a functional block diagram of a main part of a dialogue system according to a second embodiment. [Figure 7] 11 is a flowchart showing an example of the operation of the dialogue system in the second embodiment. [Figure 8] FIG. 11 is a functional block diagram of a main part of a dialogue system according to a third embodiment. [Figure 9] 11 is a flowchart showing an example of the operation of the dialogue system in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Please note that the disclosure described in this disclosure is subject to compliance with legal matters of the country of implementation necessary for the implementation of this disclosure, such as secrecy of communications. The embodiments for implementing the dialogue system, program, and dialogue method according to the present disclosure will be described with reference to the attached drawings. In each drawing, the same or corresponding parts are given the same reference numerals, and duplicated explanations are appropriately simplified or omitted. Note that this disclosure is not limited to the following embodiments, and each embodiment can be freely combined, any component of each embodiment can be modified, or any component of each embodiment can be omitted, within the scope of the gist of this disclosure.

[0010] Embodiment 1 A first embodiment of the present disclosure will be described with reference to Fig. 1 to Fig. 4. Fig. 1 is a schematic diagram showing an overall overview of a dialogue system. Fig. 2 is a diagram showing the device configuration of the dialogue system. Fig. 3 is a functional block diagram of the main parts of the dialogue system. Fig. 4 is a flowchart showing an example of the operation of the dialogue system.

[0011] 1 and 2, a dialogue system 400 according to this embodiment includes a server 100 and a user terminal 200. In the dialogue system 400, the server 100 and the user terminal 200 are connected via a network 300 so as to be able to communicate with each other.

[0012] In the configuration example described here, the server 100 provides a live streaming service to a user terminal 200 owned by a user via a network 300. The live streaming service is a service in which a distributor shoots video and distributes the shot video to viewers in real time. Note that only audio may be distributed, not video. Viewers can participate in the live streaming service by posting reactions such as comments.

[0013] The user terminals 200 include a distributor terminal 200a and a viewer terminal 200b. The distributor terminal 200a and the viewer terminal 200b may be collectively referred to as the user terminal 200 without distinction. The number of each of the distributor terminal 200a and the viewer terminal 200b connected to the network 300 is not limited to one, and may be two or more.

[0014] The network 300 serves to connect the user terminal 200 and one or more servers 100. In other words, the network 300 refers to a communication network that provides a connection path so that the user terminal 200 can transmit and receive data after connecting to the server 100. One or more parts of the network 300 may or may not be a wired network or a wireless network.

[0015] The network 300 may include, for example, an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a public switched telephone network (PSTN), a mobile phone network, an integrated service digital network (ISDN), a wireless LAN, a long term evolution (LTE), a code division multiple access (CDMA), Bluetooth (registered trademark), or a satellite communication, or a combination of two or more of these. The network 300 may include one or more networks 300.

[0016] The user terminal 200 may be any terminal that can realize the functions of the embodiments of the present disclosure. The user terminal 200 includes, for example, a smartphone, a mobile phone (feature phone), a computer (e.g., a desktop PC, a laptop PC, a tablet PC, etc.), a media computer platform (e.g., a cable, a satellite set-top box, a digital video recorder), a handheld computer device (e.g., a PDA (Personal Digital Assistant), an email client, etc.), a wearable terminal (a glasses-type device, a watch-type device, etc.), or other types of computers or communication platforms. The user terminal 200 may also be expressed as an information processing terminal.

[0017] The server 100 has a function of providing a predetermined service to the user terminal 200. The server 100 may be any device that can realize the functions of the embodiments of the present disclosure. The server 100 includes, for example, a server device, a computer (e.g., a desktop PC, a laptop PC, a tablet PC, etc.), a media computer platform (e.g., a cable or satellite set-top box, a digital video recorder), a handheld computer device (e.g., a PDA, an email client, etc.), or other types of computers or communication platforms. The server 100 may also be expressed as an information processing device. When there is no need to distinguish between the server 100 and the user terminal 200, the server 100 and the user terminal 200 may or may not be expressed as an information processing device.

[0018] Next, the configuration of the user terminal 200 will be described with reference to FIG. 2. The user terminal 200 includes a terminal control means 210, a terminal storage means 220, a terminal communication means 230, an input / output means 240, a display means 250, a microphone 260, a speaker 270, and a camera 280. The hardware components of the user terminal 200 are connected to each other via a bus, for example. It is not essential that the hardware configuration of the user terminal 200 includes all the components described here. For example, the user terminal 200 may or may not be configured such that individual components such as the camera 280 or multiple components are detachable.

[0019] The terminal control means 210 is capable of controlling the overall operation of the user terminal 200 by using various information such as information stored in the terminal storage means 220 .

[0020] The terminal communication means 230 is an I / F (interface) that transmits and receives various data via the network 300. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as the communication between the terminals can be performed. The terminal communication means 230 has a function of performing communication with the server 100 via the network 300. The terminal communication means 230 transmits various information to the server 100 according to an instruction from the terminal control means 210. The terminal communication means 230 receives various information transmitted from the server 100 and transmits it to the terminal control means 210. The terminal communication means 230 may be simply referred to as a communication means. When the terminal communication means 230 is configured by a physically structured circuit, it may be referred to as a communication circuit.

[0021] The input / output means 240 includes an input means and an output means. The input means is a device for inputting various user operations to the user terminal 200. The output means is a device for outputting the results of processing performed by the user terminal 200. In the input / output means 240, the input means and the output means may be integrated, or the input means and the output means may be separate.

[0022] The input means is realized by any one or a combination of all kinds of devices that can receive input from a user and transmit information related to the input to the terminal control means 210. For example, the input means can include hardware keys such as a touch panel, a touch display, and a keyboard, and a pointing device such as a mouse.

[0023] The output means is realized by any one or a combination of all kinds of devices capable of outputting the processing results processed by the terminal control means 210. The output means may include, for example, a touch panel, a touch display, a lens (for example, 3D (Three Dimensions) output or hologram output), a printer, etc.

[0024] The display means 250 is realized by any one of all kinds of devices capable of displaying according to the display data written in the frame buffer, or a combination thereof. That is, the display means 250 may be included in the output means. The display means 250 may include, for example, a touch panel, a touch display, a monitor (for example, a liquid crystal display or an OELD (Organic Electroluminescence Display)), a head mounted display (HDM), a projection mapping, a hologram, a device capable of displaying images, text information, etc. in air (which may or may not be a vacuum). Note that these display means 250 may or may not be capable of displaying display data in 3D.

[0025] When the input / output means 240 has a touch panel, the input / output means 240 and the display means 250 may be disposed facing each other and have approximately the same size and shape.

[0026] The microphone 260 is included in the input means. The microphone 260 is used to input audio information, which is expression information. The speaker 270 is included in the output means. The speaker 270 is used to output audio information. The camera 280 is included in the input means. The camera 280 is used to acquire still image information, which is expression information, and / or video information, which is continuous still image information.

[0027] The terminal control means 210 has a circuit physically structured to execute the functions realized by the codes or instructions included in the program, and is realized by, for example, a data processing device built into the hardware. Therefore, the terminal control means 210 may or may not be expressed as a control circuit.

[0028] The terminal control means 210 includes, for example, a central processing unit (CPU), a microprocessor, a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), and the like.

[0029] The terminal storage means 220 has a function of storing various programs and various data required for the operation of the user terminal 200. The terminal storage means 220 includes various storage media such as, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a random access memory (RAM), and a read only memory (ROM). Furthermore, the terminal storage means 220 may or may not be expressed as a memory.

[0030] The user terminal 200 stores a program in the terminal storage means 220, and executes this program, causing the terminal control means 210 to execute processing as each unit included in the terminal control means 210. That is, the program stored in the terminal storage means 220 causes the user terminal 200 to realize each function executed by the terminal control means 210. In other words, the processor executes the program stored in the memory in the user terminal 200, and the hardware and software of the user terminal 200 work together to realize the function of each unit included in the user terminal 200. Note that this program may or may not be expressed as a program module.

[0031] Next, a description will be given of the configuration of the server 100. The server 100 includes a server control means 110, a server storage means 120, and a server communication means 130. The hardware components of the server 100 are connected to each other via a bus, for example.

[0032] The server control means 110 has a physically structured circuit for executing functions realized by codes or instructions included in a program, and is realized, for example, by a data processing device built into hardware. The server control means 110 is typically a central processing unit (CPU), and may or may not be a microprocessor, a processor core, a multiprocessor, an ASIC, an FPGA, or the like. In the present disclosure, the server control means 110 is not limited to these.

[0033] The server storage means 120 has a function of storing various programs and various data required for the operation of the server 100. The server storage means 120 is realized by various storage media such as HDD, SSD, flash memory, etc. However, in this disclosure, the server storage means 120 is not limited to these. Also, the server storage means 120 may or may not be expressed as memory.

[0034] The server communication means 130 transmits and receives various data via the network 300. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as the communication between the two is possible. The server communication means 130 has a function of communicating with the user terminal 200 via the network 300. Specifically, the server communication means 130 transmits various pieces of information to the user terminal 200 in accordance with instructions from the server control means 110. The server communication means 130 receives various pieces of information transmitted from the user terminal 200 and transmits them to the server control means 110.

[0035] The server communication means 130 may be expressed as a server communication I / F (interface). When the server communication means 130 is configured with a physically structured circuit, it may be expressed as a server communication circuit.

[0036] The server 100 may include an input / output means and a display as a hardware configuration. The input / output means is realized by a device that inputs various operations to the server 100. The input / output means is realized by any of all types of devices that can receive input from a user and transmit information related to the input to the server control means 110, or a combination thereof. The display is typically realized by a monitor (e.g., a liquid crystal display, an OELD, etc.). In this case, for example, the hardware of the server 100 may or may not be configured to have a detachable display.

[0037] Server 100 stores a program in server storage means 120, and executes this program, causing server control means 110 to execute processing as each unit included in server control means 110. That is, the program stored in server storage means 120 causes server 100 to realize each function executed by server control means 110. In other words, a processor executes the program stored in memory in server 100, and the hardware and software of server 100 work together to realize the function of each unit included in server 100. Note that this program may or may not be expressed as a program module.

[0038] The terminal control means 210 of the user terminal 200 and / or the server control means 110 of the server 100 may or may not realize each process by a CPU having a control circuit, but may also realize each process by a logic circuit (hardware) formed in an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)) or a dedicated circuit. These circuits may be realized by one or more integrated circuits, and the multiple processes shown in the embodiments of the present disclosure may or may not be realized by one integrated circuit. LSIs may also be called VLSIs, super LSIs, ultra LSIs, etc., depending on the degree of integration.

[0039] In addition, the program (e.g., a software program, a computer program, or a program module) of the embodiment according to the present disclosure may or may not be provided in a state stored in a computer-readable storage medium. The storage medium can store the program in a "non-transitory tangible medium." In addition, the program may or may not be for realizing part of the functions of the embodiment according to the present disclosure. Furthermore, the program may or may not be a so-called difference file (difference program) that can realize the functions of the embodiment according to the present disclosure in combination with a program already recorded in the storage medium.

[0040] The storage medium may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs) or application specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid state drives (SSDs), RAM drives, secure digital cards, or drives, any other suitable storage media, or any suitable combination of two or more of these. The storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate. It should be noted that the storage medium is not limited to these examples and may be any device or medium capable of storing a program. In addition, the storage medium may or may not be referred to as a memory.

[0041] Furthermore, the program of the present disclosure may or may not be provided to the server 100 and / or the user terminal 200 via any transmission medium capable of transmitting the program (such as a communication network or broadcast waves). When the program is provided via a transmission medium, the server 100 and / or the user terminal 200 can realize the functions of the multiple functional units shown in each embodiment by executing a program downloaded via the Internet, for example.

[0042] Also, by way of example and not by way of limitation, when the user terminal 200 is a smartphone or a PC, an application received from the server 100 (or a server device other than the server 100) may be incorporated (installed) into the smartphone or PC, and some of the elements (processing / operations) according to the claims of the present disclosure may be executed via the application. Also, by way of example and not by way of limitation, when the user terminal 200 is a smartphone or a PC, some of the elements (processing / operations) according to the claims of the present disclosure may be executed via a website accessible from the smartphone or PC, without an application being incorporated (installed) into the smartphone or PC.

[0043] Moreover, the embodiment according to the present disclosure may be realized in the form of a data signal embedded in a carrier wave in which a program is embodied by electronic transmission. Moreover, at least a part of the processing in the server 100 and / or the user terminal 200 may or may not be realized by cloud computing configured by one or more computers. Moreover, at least a part of the processing in the user terminal 200 may or may not be configured to be performed by the server 100. In this case, at least a part of the processing of each functional unit of the terminal control means 210 of the user terminal 200 may or may not be configured to be performed by the server 100. Moreover, at least a part of the processing in the server 100 may or may not be configured to be performed by the user terminal 200. In this case, at least a part of the processing of each functional unit of the server control means 110 of the server 100 may or may not be configured to be performed by the user terminal 200.

[0044] The program of the present disclosure may be implemented using, for example, a scripting language such as ActionScript or JavaScript (registered trademark), an object-oriented programming language such as Objective-C or Java (registered trademark), or a markup language such as HTML5.

[0045] Next, the function of the server 100 will be described with reference to FIG.

[0046] 3, the server 100 includes a storage unit 10, a delivery interface unit 11, an input unit 12, an output unit 13, a response generation unit 14, an estimation unit 15, a feature determination unit 16, an integration unit 17, and an addition unit 18. For example, the delivery interface unit 11, the input unit 12, the output unit 13, the response generation unit 14, the estimation unit 15, the feature determination unit 16, the integration unit 17, and the addition unit 18 are realized by a server control means 110.

[0047] For example, the storage unit 10 is realized by the server storage means 120 shown in Fig. 2. The storage unit 10 records various types of information required for control.

[0048] The distribution interface unit 11 provides a distribution interface (hereinafter also referred to as "distribution I / F") for a live distribution service. The distribution I / F allows a distributor to live-distribute video with audio or audio only from the distributor terminal 200a. The distribution I / F allows a viewer to view the live-distributed video or audio from the viewer terminal 200b. The viewer can post comments from the viewer terminal 200b in a comment section provided in the distribution I / F.

[0049] The distribution I / F may be capable of providing a distributor screen and a viewer screen. The distributor screen is displayed on the distributor terminal 200a and displays content related to live distribution. The viewer screen is displayed on the viewer terminal 200b and displays content related to live distribution. The viewer screen is at least partially different from the distributor screen.

[0050] The server 100 has an AI agent function. The AI ​​agent function includes the functions of an input unit 12, an output unit 13, a response generation unit 14, an estimation unit 15, a feature determination unit 16, an integration unit 17, and an addition unit 18. The AI ​​agent function may include at least the response generation unit 14, the estimation unit 15, and the feature determination unit 16. The AI ​​agent function controls the AI ​​agent. The AI ​​agent generates a voice of a dialogue with a broadcaster who is a speaker, and has a dialogue with the speaker in real time using the voice. The AI ​​agent may have a dialogue with the speaker in real time using only text or video, not voice. For example, the dialogue between the AI ​​agent and the broadcaster is live-streamed via a streaming I / F. A viewer can watch the dialogue between the AI ​​agent and the broadcaster on the streaming I / F.

[0051] The input unit 12 accepts input of expression information related to an expression made by a distributor received from the distributor terminal 200a. The expression information may be information of a voice uttered by the distributor, or information of a video that is an image or a series of images showing the distributor. For example, the expression information is information of a video with audio or only audio that is distributed via a distribution I / F.

[0052] The output unit 13 outputs output information to the distribution I / F as an output related to the dialogue of the AI ​​agent. For example, the output information includes information related to the voice or text uttered by the AI ​​agent. That is, the dialogue between the distributor and the AI ​​agent is distributed to the viewer by outputting the voice or text of the AI ​​agent from the output unit 13 to the distribution I / F. In this embodiment, the output information includes at least information on the voice of the AI ​​agent.

[0053] The output unit 13 may output information for controlling an avatar of an AI agent as output information. In this case, an image or video of the avatar is displayed on the distribution I / F. The output information may include information for controlling the movement, facial expression, etc. of the avatar.

[0054] The response generation unit 14 generates a dialogue text based on the expression information. The dialogue text is a text of a sentence uttered by an AI agent. For example, the response generation unit 14 extracts semantic information uttered by the distributor from the expression information. When the expression information includes the voice of the distributor, the semantic information may be information indicating the meaning of a sentence included in the voice. When the expression information includes an image or video, the semantic information may be information indicating a meaning corresponding to the movement of the distributor appearing in the image or video. The movement of the distributor may be any movement such as gestures or sign language. The response generation unit 14 generates a dialogue text corresponding to the semantic information based on the extracted semantic information. For example, when the semantic information indicates the meaning of "live distribution will start," the response generation unit 14 generates a dialogue text of "What is today's theme?" in order to host the live distribution as a response to the semantic information. In addition, the response generation unit 14 may use a known dialogue response model when generating a dialogue text from the semantic information.

[0055] The estimation unit 15 estimates the emotion of the broadcaster, who is the speaker, based on the expression information. The estimation unit 15 generates emotion information indicating the estimated emotion of the broadcaster and outputs it. When the expression information includes the voice of the broadcaster, the estimation unit 15 may estimate the emotion of the broadcaster from the voice. When the expression information includes an image or video of the broadcaster, the estimation unit 15 may estimate the emotion of the broadcaster from the image of the broadcaster shown in the image. Any technology may be applied as a method for estimating the emotion of the broadcaster and information included in the emotion information. Note that, although several examples will be shown hereinafter, the method and emotion information are not limited to these examples.

[0056] For example, in a first example of the method in the case where the expression information includes a voice, the estimation unit 15 extracts a feature from the voice. For example, the extracted feature includes pitch, energy, speed, MFCC (Mel Frequency Coefficient), etc. The estimation unit 15 estimates the emotion of the distributor by using a machine learning model that estimates emotion from the feature.

[0057] In a second example of the method when the expression information includes a voice, the estimation unit 15 generates emotion information directly from the voice represented in the expression information. That is, in the second example of the method, an end-to-end model is used. In the second example of the method, unlike the first example, no process for extracting features is involved.

[0058] In a third example of the method in which the expression information includes voice, the estimation unit 15 calculates the difference between the expression information of the same distributor input in a live distribution performed at another date and time and the currently input expression information, and estimates the current emotion of the distributor. In this case, the estimation unit 15 may calculate the difference between the feature amount extracted from the expression information at the other date and time and the feature amount extracted from the current expression information. The expression information at the other date and time is stored in advance in the storage unit 10. Note that the expression information at the other date and time may be labeled with an emotion by the distributor. In this case, the estimation unit 15 may estimate the current emotion of the distributor based on the emotion label of the expression information at the other date and time.

[0059] In a first example of the method when the expression information includes an image or a video, the estimation unit 15 recognizes the facial expression of the distributor reflected in the expression information, and estimates the emotion indicated by the facial expression.

[0060] In a second example of the method when the expression information includes an image or video, the estimation unit 15 further recognizes the meaning indicated by the distributor's gestures (hand gestures / hand-over-face), sign language, etc. shown in the expression information in addition to the distributor's facial expressions, and estimates the distributor's emotions.

[0061] In the first example of the emotion information, the emotion information includes information that classifies the emotion of the distributor into two categories: negative and positive. In this case, the emotion information may include only the two values ​​of negative and positive, or may include an index value indicating the degree of negativity and an index value indicating the degree of positivity.

[0062] In the second example of the emotion information, an existing emotion estimation model is used in the emotion information, and thus the emotion information includes information that classifies the emotion of the distributor in more detail than the first example. For example, the emotion information may include emotion information classified based on a valence-arousal model.

[0063] In the third example of the emotion information, the emotion information includes a plurality of parameters that indicate the emotion of the distributor. Unlike the first and second examples, the parameters included in the emotion information of the third example do not need to be provided with a label for classifying the emotion, such as that used in a model for classifying the emotion.

[0064] In addition, when the expression information includes voice, the estimation unit 15 may further use the ASR (automatic speech recognition) result of the voice signal indicated in the expression information as an auxiliary input to the machine learning model of the first, second or third example of the method. That is, as an example, the estimation unit 15 may extract semantic information from the expression information and use it as an auxiliary input for estimating emotions. For example, the estimation unit 15 may pick up a word indicated in a database such as a Japanese emotion expression dictionary from the semantic information and use the emotion corresponding to the picked word as an auxiliary input. For example, the estimation unit 15 may use the result of processing the semantic information with a specific processing model that handles NLP (natural language processing) as an auxiliary input.

[0065] The estimation unit 15 processes the expression information at each specified input interval. For example, the input interval is a value between 10 milliseconds and 100 milliseconds. The estimation unit 15 estimates the emotion of the distributor at each specified output interval. For example, the output interval is one phrase, one utterance, etc. Specifically, the estimation unit 15 extracts features from the voice information included in the expression information at each 10 milliseconds, and estimates the emotion of the distributor when uttering each phrase.

[0066] The feature determination unit 16 determines the response feature of the AI ​​agent based on the emotion information. The response feature is a feature of the AI ​​agent when it responds to the expression information of the distributor who is the speaker, and in particular, is a response feature that corresponds to the emotion of the distributor. That is, the feature determination unit 16 determines the response feature of the AI ​​agent when it responds based on the emotion of the speaker estimated by the estimation unit 15.

[0067] In this embodiment, the feature determination unit 16 determines, as the response feature, a set value of the response feature of the voice uttered when the AI ​​agent responds. For example, the response feature includes at least one of the volume, voice quality, and voice tone of the voice uttered by the AI ​​agent. The volume of the voice may mean the tension of the voice. The voice quality is a parameter that is mainly adjusted when expressing differences in the subject uttering the voice, such as voice characteristics that differ from person to person. The voice tone is a parameter that is mainly adjusted when expressing a whisper voice, an angry voice, etc. The feature determination unit 16 determines the response feature corresponding to the emotion of the distributor indicated in the emotion information. The value to which the response feature is set in response to the emotion of the distributor, that is, the tendency of the AI ​​agent to emit a voice with certain characteristics, may be set in advance by the distributor or other persons.

[0068] As an example, the feature determination unit 16 determines the response feature so that the response feature is the same as or similar to the emotion indicated in the emotion information. In this case, the feature determination unit 16 may determine the response feature so as to be close to the emotion indicated in the emotion information. Specifically, when the estimation unit 15 estimates that the streamer is in low spirits or has negative emotions, the feature determination unit 16 reduces the volume to a specified value and determines the set values ​​of the voice quality and tone of voice so that the listener feels kindness. In this case, the response feature is determined to be a volume, voice quality, and tone of voice that encourages or sympathizes. In addition, when the estimation unit 15 estimates that the streamer is in high spirits or has positive emotions, the feature determination unit 16 increases the volume to a specified value and determines the set values ​​of the voice quality and tone of voice so that the listener feels cheerful.

[0069] As another example, the feature determining unit 16 determines a response feature that indicates an emotion opposite to the emotion indicated by the emotion information. Specifically, when the estimation unit 15 estimates that the streamer is in low spirits or has negative emotions, the feature determining unit 16 increases the volume to a value higher than the specified value and determines a voice quality and tone of voice that make the listener feel cheerful. When the estimation unit 15 estimates that the streamer is in high spirits or has positive emotions, the feature determining unit 16 decreases the volume to a value lower than the specified value and determines a voice quality and tone of voice that make the listener feel depressed.

[0070] The storage unit 10 may store a basic setting value (preset) for each value of the response feature. In this case, the feature determining unit 16 may determine an increase or decrease based on the basic setting value as the response feature. The storage unit 10 may store a setting range for each value of the response feature. In this case, the feature determining unit 16 may determine each value of the response feature so that the value falls within the setting range.

[0071] In addition, the storage unit 10 may store a plurality of patterns of combinations of response feature determination tendency, basic setting value, and response feature. When a broadcaster selects any of the combinations from the broadcaster terminal 200a, the feature determination unit 16 determines the response feature based on the determination tendency, basic setting value, and setting range corresponding to the selected combination. For example, each pattern may be set with characteristic labels such as "AI Akari-chan who is always cheerful," "Alisa-san who is easygoing but sympathetic in important situations," and "Achilles-kun who always has a nice voice and is calm and collected." For example, in the case of "Alisa-san," the following combinations are obtained. The basic setting value is set with a response feature that gives a cheerful impression. The response feature determination tendency is set to be the same or similar to the emotion indicated in the emotion information. The setting range is set relatively wide for each value so that a range of voices is created, from cheerfulness to seriousness when being sympathetic.

[0072] Furthermore, if an avatar is set for the AI ​​agent, the basic setting values ​​and setting ranges of the response features may be associated with the avatar, which can prevent the voice tone and quality from being different from what is expected from the avatar image.

[0073] The feature determining unit 16 may further use semantic information as auxiliary input to be added to the emotion information. In this case, for example, the feature determining unit 16 may determine the response feature based on the emotion information and the semantic information by performing the same process as when the estimation unit 15 uses the semantic information as auxiliary input.

[0074] The feature determination unit 16 may further use the dialogue text generated by the response generation unit 14 as an auxiliary input to be added to the emotion information. In this case, for example, the feature determination unit 16 may determine the response feature based on the emotion information and the dialogue text so as to have an emotion similar to the content indicated by the dialogue text.

[0075] The integration unit 17 creates output information by integrating the dialogue text generated by the response generation unit 14 and the response features determined by the feature determination unit 16. The output information includes at least information on the voice uttered by the AI ​​agent.

[0076] In this embodiment, the integration unit 17 generates output information for reading out the dialogue text in a voice based on the response features. This output information is output by the output unit 13, and as a result, in live streaming, the AI ​​agent utters the dialogue text in a voice indicated by the response features. That is, the AI ​​agent utters a dialogue voice corresponding to the expression information, which is the distributor's utterance. The viewer can watch the distributor's utterance and the AI ​​agent uttering the voice of the response features, which is a voice of a sentence that corresponds to the distributor's utterance and that corresponds to the distributor's tension.

[0077] Note that the output information may be associated with response features for each defined output unit, such as a phrase or a sentence of the dialogue text.

[0078] The addition unit 18 adds further information to the output information generated by the integration unit 17. For example, the addition unit 18 may select background music based on emotion information and add information on the selected background music to the output information. In this case, the output unit 13 plays back the background music selected by the addition unit 18 at the same time as voicing the dialogue text shown in the output information.

[0079] When an image or video of an avatar of an AI agent is set, the addition unit 18 may generate at least one of facial expressions and gestures of the avatar based on the emotion information and add the generated information to the output information. In this case, the output unit 13 reflects the generated facial expressions and gestures on the avatar when uttering the voice of the dialogue text shown in the output information.

[0080] For example, the adding unit 18 may generate information indicating a time transition of the emotion estimated by the estimation unit 15 and add the information to the output information. In this case, the output unit 13 displays a time chart indicating the time transition on at least one of the distributor screen and the viewer screen.

[0081] The flowchart shown in Fig. 4 is an example of a basic operation of a dialogue with a distributor performed by an AI agent function. For example, the flowchart shown in Fig. 4 starts when the distributor speaks.

[0082] In step S01, the input unit 12 acquires expression information corresponding to an utterance of a distributor.

[0083] Then, in step S02, the response generation unit 14 generates a dialogue text based on the expression information. After step S01, the operation of step S03 is performed in parallel with step S02. In step S03, the estimation unit 15 estimates the emotion of the distributor based on the expression information and generates emotion information.

[0084] After steps S02 and S03, in step S04, the feature determining unit 16 determines a response feature amount as a response feature based on the emotion information.

[0085] After that, in step S05, the integration unit 17 integrates the dialogue text and the response features to generate output information.

[0086] After that, in step S06, the output unit 13 outputs audio to the distribution I / F based on the output information. Then, the operation of the flowchart ends.

[0087] According to the above-described first embodiment, the dialogue system 400 includes the estimation unit 15 and the feature determination unit 16. The dialogue system 400 may be realized by the server 100, which is a computer, executing a program. That is, the executed program causes the computer to execute the dialogue method of the present disclosure. In the conventional technology as shown in Patent Document 1, the response content is recommended based on the estimated emotion of the speaker. However, in the conventional dialogue with the AI ​​agent, the emotion of the speaker is not taken into consideration for the response feature, which is the mode in which the response content of the AI ​​agent is transmitted. Therefore, a gap occurs between the response feature of the AI ​​agent and the tension and atmosphere of the speaker. On the other hand, in the dialogue system 400, the response feature when the AI ​​agent responds is determined based on the emotion of the distributor, who is the speaker. Therefore, the speaker can dialogue with the AI ​​agent without feeling uncomfortable. As a result, the number of questions about the discomfort felt by the speaker to the AI ​​agent can be reduced, and the computation load of the server 100 required for dialogue can be reduced.

[0088] The expression information is information on the voice uttered by the speaker. The estimation unit 15 estimates the emotion of the speaker based on the voice information. Therefore, for example, even in a dialogue in which only voice is exchanged, the dialogue system 400 can estimate the emotion of the speaker.

[0089] The expression information is information on an image or video in which the speaker is shown. The estimation unit 15 estimates the emotion of the speaker based on the image or video information. Therefore, for example, even in a dialogue in which the speaker cannot make a sound or the speaker's voice information cannot be obtained, the dialogue system 400 can estimate the emotion of the speaker.

[0090] In addition, the feature determining unit 16 determines the setting of the response feature of the voice uttered by the AI ​​agent as the response feature. In the conventional technology, the feature of the voice uttered by the AI ​​agent could not be made to conform to the emotion of the speaker. For this reason, a communication gap occurs between the speaker and the AI ​​agent, and the speaker may feel uncomfortable. The dialogue system 400 can change the feature of the voice uttered by the AI ​​agent in real time according to the emotion of the speaker. As a result, the speaker can converse with the AI ​​agent without feeling uncomfortable.

[0091] In addition, the feature determining unit 16 generates response features that indicate emotions opposite to those of the estimated speaker, thereby enabling a greater variety in dialogue with the AI ​​agent.

[0092] The dialogue system 400 further includes a response generation unit 14. The AI ​​agent reads out the dialogue text generated by the response generation unit 14 based on the expression information in a voice based on the settings included in the response features. The response features are determined based on the expression information. That is, the dialogue text and the response features are generated or determined by the same expression information by the speaker. Therefore, the AI ​​agent can utter a voice with contents and features corresponding to the contents of the speaker's utterance.

[0093] Furthermore, the feature determining unit 16 determines the response feature based on the dialogue text in addition to the emotion of the speaker, so that the AI ​​agent can utter a voice with features that are consistent with the content of the dialogue text.

[0094] The response generation unit 14 may generate a dialogue text using the emotion of the speaker indicated in the emotion information as auxiliary input in addition to the meaning indicated in the expression information. Therefore, a dialogue text corresponding to the emotion of the speaker can be generated. When generating a dialogue text, the response generation unit 14 may use the expression information and the emotion information generated by the estimation unit 15 from the expression information, or may use the expression information and the emotion information generated by the estimation unit 15 from expression information other than the expression information.

[0095] The AI ​​agent function does not have to be premised on being made public through live distribution. Fig. 5 is a functional block diagram of a modified example of the dialogue system in the first embodiment. For example, as shown in Fig. 5, a speaker dialogues with an AI interface via a user terminal 200. Here, compared to the example shown in Fig. 2, the speaker is not a distributor, and a distribution interface unit 11 is not provided. There is no viewer watching the dialogue between the speaker and the AI ​​interface.

[0096] The output unit 13 outputs the dialogue content of the AI ​​agent to an interface for dialogue, not to a distribution I / F. For example, the interface for dialogue is a control interface of a smart speaker, etc. In this way, the AI ​​agent function according to this embodiment is applicable to a system that dialogues with a speaker, such as a smart speaker.

[0097] As another example of a case where a dialogue with an AI agent is not live-streamed, the invention of the present disclosure may be adopted in an application capable of recording, editing, etc., video or audio relating to a dialogue with an AI agent. In this case, in the video or audio recording of a dialogue between a speaker and an AI agent, the response features of the AI ​​agent when responding may be edited by the application to be changed to those corresponding to the emotional information of the speaker.

[0098] In the present disclosure, the output information generated by the integration unit 17 includes at least voice information, but is not limited thereto. For example, the present invention may be adopted in a generation AI such as a chatbot that responds in text format to a question in text format from a questioner who is a speaker. In this case, since the output information is text information, the response features determined by the feature determination unit 16 may include some or more of the following: font format and size of the text information, speed of the response (text reply), and wording of the text expression in the response (for example, whether it is a polite sentence such as "is. Masu." or a concise expression such as a noun-ending sentence). In this case, the emotion information estimated by the estimation unit 15 may be estimated from the voice or image as described above, or may be estimated by other methods. For example, the estimation unit 15 may estimate the emotion information of the questioner from the text information input by the questioner. As a specific example, emotional information may be inferred from text information based on some or all of the following information: the content and expressions of the input text (for example, the presence or absence of overbearing expressions), the speed at which the text information is input, the time between when the generation AI outputs the text and when the questioner inputs the next question, inconsistencies in the content, expressions, and wording of the input text, and the presence or absence of typos.

[0099] Embodiment 2 Fig. 6 is a functional block diagram of the main part of the dialogue system. Fig. 7 is a flowchart showing an example of the operation of the dialogue system. Note that the same reference numerals are used to designate parts that are the same as or equivalent to those in the first embodiment. Explanation of these parts will be omitted.

[0100] In the second embodiment, the AI ​​agent reads out parts of comments posted by viewers in a comment section of the distribution I / F.

[0101] The feature determination unit 16 selects a first comment corresponding to the emotion of the distributor as a response feature based on the emotion information. In addition, the feature determination unit 16 determines the selected first comment as a response feature as a candidate for a comment to be read out by the AI ​​agent. At this time, the feature determination unit 16 analyzes each of one or more comments posted in the comment section, and selects a first comment based on the emotion information and the analysis result. Any method may be adopted for the analysis and selection. Examples of the method are described below, but the method is not limited to these examples.

[0102] In a first example of a method for analyzing comments posted in a comment section, the feature determining unit 16 estimates the emotion indicated in the text of the comment. In this case, the feature determining unit 16 may estimate the emotion indicated in the text of the comment by performing the same process as the process in which the estimation unit 15 estimates the emotion from the semantic information. In this case, the feature determining unit 16 may select, as the first comment, a comment indicating an emotion that is the same as or similar to the emotion indicated in the emotion information, or may select, as the first comment, a comment indicating an emotion opposite to the emotion indicated in the emotion information.

[0103] In a second example of the method for analyzing comments posted in a comment section, the feature determining unit 16 calculates a relevance score between emotion information and a comment. The stronger the relevance between the emotion indicated in the emotion information and the topic, content, etc. indicated in the comment, the larger the relevance score becomes. The feature determining unit 16 selects the comment with the highest relevance score as the first comment from among the multiple comments posted in the comment section. Note that the feature determining unit 16 may select the multiple comments with high relevance scores as the first comment. Note that the feature determining unit 16 may select the comment with a relevance score within a specified score range as the first comment from among the multiple comments posted in the comment section.

[0104] The feature determining unit 16 may adjust the frequency of selecting a first comment, i.e., the interval between selecting a first comment and selecting the next first comment, based on the emotion information. Specifically, for example, when the emotion indicated in the emotion information is a negative emotion, the feature determining unit 16 may lengthen or shorten the interval until selecting a first comment.

[0105] The integration unit 17 generates output information of a voice that integrates the dialogue text and the first comment determined by the feature determination unit 16. The output information includes the dialogue text and the first comment in a distinguished manner.

[0106] The output unit 13 outputs the output information to the distribution I / F. In this case, the AI ​​agent distinguishes between the dialogue text and the first comment when reading them out. Specifically, the AI ​​agent does not read out the dialogue text and the first comment at the same time. The AI ​​agent may read out the dialogue text and the first comment so that the voice reading out the dialogue text and the voice reading out the first comment have different features. The AI ​​agent may also read out the dialogue text as usual, and when reading out the first comment, add a sentence indicating that it is a posted comment.

[0107] In addition, the output unit 13 may display the first comment included in the output information only on the distributor screen of the distribution I / F in an operable state before outputting the output information so that the first comment is read by the AI ​​agent. When the first comment is operated on the distributor terminal 200a, the AI ​​agent may read out the operated first comment. An operation to not read out the comment may be possible on the distributor screen. In addition, the first comment displayed on the distributor screen may be a plurality of comments. In this case, one or more first comments operated on the distributor terminal 200a may be read out by the AI ​​agent. Furthermore, the feature determination unit 16 may select the first comment and infer the expected development of the story from the first comment. In this case, the distributor screen displays the predicted development of the story together with the first comment. Therefore, the distributor can operate the distributor terminal 200a so that an appropriate comment is read out after predicting the future development of the story.

[0108] Note that the integration unit 17 may generate output information that includes the first comment but does not include the dialogue text, without waiting for the dialogue text to be generated by the response generation unit 14. In this case, the AI ​​agent may read out only the first comment.

[0109] The flowchart shown in Fig. 7 is an example of a basic operation of a dialogue with a distributor performed by an AI agent function. For example, the flowchart shown in Fig. 7 starts when the distributor speaks.

[0110] In step S11, the input unit 12 acquires expression information corresponding to the utterance of the distributor.

[0111] After that, in step S12, the response generation unit 14 generates a dialogue text based on the expression information. After step S11, the operation of step S13 is performed in parallel with step S12. In step S13, the estimation unit 15 estimates the emotion of the distributor based on the expression information and generates emotion information.

[0112] After steps S12 and S13, in step S14, the feature determining unit 16 determines the first comment as a response feature based on the emotion information.

[0113] After that, in step S15, the integration unit 17 integrates the dialogue text and the first comment to generate output information.

[0114] After that, in step S16, the output unit 13 outputs voice to the distribution I / F based on the output information. At this time, the AI ​​agent reads out the dialogue text and the first comment separately. After that, the operation of the flowchart ends.

[0115] 7 may start at a specified cycle. In this case, the operations from steps S11 to S13 may not be performed. In this case, in step S14, the feature determining unit 16 may determine the first comment based on the emotion information generated immediately before.

[0116] According to the above-described second embodiment, the dialogue between the broadcaster, who is the speaker, and the AI ​​agent is performed in the distribution interface. The distribution interface is provided with a comment section in which viewers can post comments. In live distribution, such comments and exchanges between the broadcaster and the broadcaster are frequently performed. In the conventional technology, such comments were sometimes read out in the order in which they were posted, in parallel with the dialogue with the AI ​​agent. However, even when the speaker is emotional, the comments were only read out in order, so that a gap was created between the comments read out by the AI ​​agent and the speaker's tension and atmosphere. In the second embodiment, the feature determination unit 16 of the dialogue system 400 selects a first comment corresponding to the speaker's emotion from among the comments posted in the comment section as a response feature of the AI ​​agent, and determines it as the comment to be read out. Therefore, the speaker can proceed with the distribution without feeling any discomfort about the read-out comment, which is part of the dialogue with the AI ​​agent.

[0117] The dialogue system 400 further includes a response generation unit 14. In the function of controlling the AI ​​agent, a process of generating a dialogue text based on expression information and a process of selecting a first comment based on emotion information are performed in parallel. The AI ​​agent reads out the dialogue text and the first comment while distinguishing them. This allows the speaker and the viewer to recognize the dialogue text by the AI ​​agent and the first comment separately. As a result, the speaker can smoothly perform live distribution, and the viewer can watch the live distribution without misunderstanding.

[0118] The feature determining unit 16 may further generate response features from emotion information, etc., as in the first embodiment. In this case, the integrating unit 17 generates output information for reading out at least one of the dialogue comment and the first comment in a voice based on the response features.

[0119] In addition, when selecting the first comment, the feature determination unit 16 may use the dialogue text as an auxiliary input. That is, the feature determination unit 16 may select the first comment based on the emotion information and the dialogue text. In addition, when selecting the first comment, the feature determination unit 16 may extract semantic information from the expression information. The feature determination unit 16 may use the semantic information as an auxiliary input. That is, the feature determination unit 16 may select the first comment based on the emotion information and the semantic information.

[0120] Selection conditions for the feature determining unit 16 to select comments in the comment field may be set in advance. For example, conditions for comments that are likely to be selected may be set. Specifically, conditions may be set such that aggressive comments are likely to be selected, calm comments are likely to be selected, etc. In addition, conditions may be set such that NG words are set and comments containing NG words are not selected.

[0121] If the selected first comment is written language, a literary style, or the like, the output unit 13 may correct the first comment to spoken language, a colloquial style, or the like, and output the corrected first comment as the output information. In this case, the AI ​​agent reads out the first comment corrected to spoken language, a colloquial style, or the like.

[0122] Embodiment 3 Fig. 8 is a functional block diagram of the main parts of the dialogue system. Fig. 9 is a flowchart showing an example of the operation of the dialogue system. Note that the same reference numerals are used to denote parts that are the same as or equivalent to those in the first or second embodiment. Explanation of these parts will be omitted.

[0123] In the third embodiment, the AI ​​agent reads out a portion of a comment posted by a viewer in a comment section of the distribution I / F. At this time, the AI ​​agent generates a comment based on the distributor's emotions. The AI ​​agent then posts the generated comment in the comment section as if it were a comment from a viewer, and reads it out loud. Note that the AI ​​agent may post the comment in the comment section and read it out loud so that it is clear that it was generated by the AI ​​agent.

[0124] Specifically, the feature determining unit 16 generates a second comment corresponding to the emotion of the distributor as a response feature based on the emotion information. The feature determining unit 16 determines the generated second comment as a response feature, as a candidate for a comment to be read out by the AI ​​agent. The feature determining unit 16 may generate a second comment indicating an emotion identical to or similar to an emotion indicated in the emotion information, or may generate a second comment indicating an emotion opposite to an emotion indicated in the emotion information. Any method may be adopted as a method by which the feature determining unit 16 generates the second comment.

[0125] As an example, the feature determining unit 16 may generate a generative model of a comment by machine learning using the comments posted to the distribution I / F in the current live distribution as training data. The feature determining unit 16 may generate or update a second comment by inputting information including at least emotion information to the generative model.

[0126] Furthermore, the storage unit 10 may store information on past comments posted in the comment section in past live broadcasts made by the same broadcaster. Also, the storage unit 10 may store other broadcast comments posted in the comment section in past live broadcasts made by another broadcaster. The feature determination unit 16 may generate or update a comment generation model by machine learning using at least one of the past comments and other broadcast comments as training data in addition to the comments posted in the current live broadcast.

[0127] The feature determining unit 16 may calculate a generation index value for generating a second comment based on the number of most recently posted comments, the most recently generated emotion information, etc., and may calculate the sum of the generation index values ​​calculated in the past. When the sum of the generation index values ​​exceeds a specified generation threshold, the feature determining unit 16 may generate a second comment.

[0128] The feature determining unit 16 may adjust the frequency of generating the second comment, i.e., the interval between generating a second comment and generating the next second comment, based on the emotion information. Specifically, for example, when the emotion indicated in the emotion information is a negative emotion, the feature determining unit 16 may lengthen or shorten the interval until the second comment is generated.

[0129] The integration unit 17 generates output information of a voice that integrates the dialogue text and the second comment determined by the feature determination unit 16. The output information includes the dialogue text and the second comment in a distinguished manner.

[0130] The output unit 13 outputs the output information to the distribution I / F. In this case, the second comment is posted in a comment section of the distribution I / F. The AI ​​agent also distinguishes between the dialogue text and the second comment when reading them out. Specifically, the AI ​​agent does not read out the dialogue text and the second comment at the same time. The AI ​​agent may read out the dialogue text and the second comment so that the voice reading them has different features. The AI ​​agent may also read out the dialogue text as usual, and when reading out the second comment, add a sentence indicating that it is a posted comment.

[0131] Note that the output unit 13 may display the second comment included in the output information in an operable state only on the distributor screen of the distribution I / F before outputting the output information so that the second comment can be read by the AI ​​agent. When the second comment is operated on the distributor terminal 200a, the AI ​​agent may read out the operated second comment. At this time, the second comment is posted in the comment section of the distribution I / F.

[0132] Also, the second comment generated by the feature determining unit 16 may be multiple. In this case, the second comment displayed on the broadcaster screen may be multiple comments. After that, one or more second comments operated on the broadcaster terminal 200a may be posted in the comment section and read out by the AI ​​agent.

[0133] Furthermore, when generating the second comment, the feature determining unit 16 may use the dialogue text as auxiliary input. That is, the feature determining unit 16 may generate the second comment based on emotion information and the dialogue text.

[0134] In addition, when generating the second comment, the feature determination unit 16 may extract semantic information from the expression information. The feature determination unit 16 may use the semantic information as auxiliary input. That is, the feature determination unit 16 may generate the second comment based on the emotion information and the semantic information. In this case, if the semantic information includes a question from the distributor, the feature determination unit 16 may generate a second comment including an answer to the question. For example, during a live broadcast, the distributor utters a question about something he or she does not understand. In this case, the feature determination unit 16 searches for or generates information that is an answer to the question on the Internet or the like, and generates a second comment including the answer. At this time, if the second comment including the answer has more characters than a specified number, the feature determination unit 16 may newly generate a second comment summarizing the content of the answer, and use it as a candidate for a comment to be posted in the comment section.

[0135] Furthermore, the feature determining unit 16 may infer the expected development of the story from the second comment together with the second comment. In this case, the predicted development of the story is displayed together with the second comment on the distributor screen. Therefore, the distributor can operate the distributor terminal 200a so that appropriate comments are read out after predicting the future development of the story.

[0136] Note that the integration unit 17 may generate output information including the second comment but not including the dialogue text, without waiting for the dialogue text to be generated by the response generation unit 14. In this case, the second comment may be posted in the comment field, and the AI ​​agent may read out only the second comment.

[0137] The flowchart shown in Fig. 9 is an example of a basic operation of a dialogue with a distributor performed by an AI agent function. For example, the flowchart shown in Fig. 9 starts when the distributor speaks.

[0138] In step S21, the input unit 12 acquires expression information corresponding to the utterance of the distributor.

[0139] After that, in step S22, the response generation unit 14 generates a dialogue text based on the expression information. After step S21, the operation of step S23 is performed in parallel with step S22. In step S23, the estimation unit 15 estimates the emotion of the distributor based on the expression information and generates emotion information.

[0140] After steps S22 and S23, in step S24, the feature determining unit 16 determines the second comment as a response feature based on the emotion information.

[0141] After that, in step S25, the integration unit 17 integrates the dialogue text and the second comment to generate output information.

[0142] Then, in step S26, the output unit 13 outputs voice to the distribution I / F based on the output information. At this time, the AI ​​agent reads out the dialogue text and the second comment while distinguishing between them. Also, the second comment is posted in the comment field. Then, the operation of the flowchart ends.

[0143] 9 may start at a specified cycle. In this case, the operations from steps S21 to S23 may not be performed. In this case, in step S24, the feature determining unit 16 may determine the second comment based on the emotion information generated immediately before.

[0144] According to the third embodiment described above, the dialogue between the broadcaster, who is the speaker, and the AI ​​agent is performed in the distribution interface. The distribution interface is provided with a comment section in which viewers can post comments. In live distribution, such comments and exchanges between the broadcaster and the broadcaster are frequently performed. In the conventional technology, such comments were sometimes read out in the order in which they were posted in parallel with the dialogue with the AI ​​agent. However, even when the speaker's emotions were high, the comments posted in the comment section were only read out in order. In this case, if there were only comments in the comment section that were unrelated to the speaker's emotions, a gap was created between the comment read out by the AI ​​agent and the speaker's tension and atmosphere. In the third embodiment, the feature determination unit 16 of the dialogue system 400 generates a second comment corresponding to the speaker's emotions as a response feature of the AI ​​agent, and determines it as the comment to be read out. Therefore, the speaker can proceed with the distribution without feeling any discomfort about the read-out comment, which is part of the dialogue with the AI ​​agent.

[0145] The dialogue system 400 further includes a response generation unit 14. In the function of controlling the AI ​​agent, a process of generating a dialogue text based on expression information and a process of generating a second comment based on emotion information are performed in parallel. The AI ​​agent reads out the dialogue text and the second comment while distinguishing them. This allows the speaker and the viewer to recognize the dialogue text by the AI ​​agent and the second comment separately. As a result, the speaker can smoothly perform live distribution, and the viewer can watch the live distribution without misunderstanding.

[0146] It should be noted that the feature determining unit 16 may generate a second comment having the same content as the other delivered comment based on the emotion information.

[0147] The feature determining unit 16 may further generate response features from emotion information, etc., as in the first embodiment. In this case, the integrating unit 17 generates output information for reading out at least one of the dialogue comment and the second comment in a voice based on the response features.

[0148] Note that the feature determining unit 16 may select the first comment from the comment field based on emotion information, etc., as in the second embodiment. In this case, the integrating unit 17 may generate output information that reads out the first comment and the second comment.

[0149] In the second or third embodiment, the output unit 13 may display a button on the broadcaster screen for controlling the timing at which the AI ​​reads out the comment. The broadcaster can adjust the timing at which the AI ​​reads out the first comment or the second comment by operating the button. This prevents the AI ​​from reading out the comment at the timing at which the broadcaster is about to speak. The button may also include a button indicating that the comment should not be read out. This can prevent the AI ​​from reading out the comment in an undesirable situation. As a result, the broadcaster can more accurately control the progress of the live broadcast.

[0150] In the first to third embodiments, the server 100 may further include a role setting unit. For example, the role setting unit is realized by the server control means 110. The role setting unit is included in the AI ​​agent function. The role setting unit sets the role of the AI ​​agent when the AI ​​agent converses with the broadcaster. For example, the role of the AI ​​agent is a role of a person who mainly listens to the conversation, a role of a person who mainly presents new topics and asks questions to the broadcaster, a role of a person who adds some humor to the conversation, etc. The response generation unit 14 generates a dialogue text according to the role set by the role setting unit based on the expression information. The feature determination unit 16 determines a response feature according to the role set by the role setting unit based on the emotion information.

[0151] Specifically, in the first embodiment, the feature determination unit 16 may determine a response feature amount according to a basic setting value and a setting range according to the role. In the second embodiment, the feature determination unit 16 may select a first comment according to a criterion according to the role. In the third embodiment, the feature determination unit 16 may generate a second comment according to a criterion according to the role. Since a dialogue is conducted according to the role set in this way, the dialogue between the distributor and the AI ​​agent becomes more consistent. In addition, the distributor can more easily control the dialogue with the AI ​​agent.

[0152] The role setting unit may also change the role based on emotion information. For example, when it is estimated that the broadcaster is feeling depressed, the role of the AI ​​agent is changed to a role of listening to the broadcaster. This allows a dialogue appropriate to the broadcaster's emotion to take place.

[0153] In the first to third embodiments, when the live streaming ends, the output unit 13 may display a feedback field on at least one of the distributor screen and the viewer screen. The distributor or viewer can input an evaluation of the response features determined in the current live streaming in the feedback field via the distributor terminal 200a or the viewer terminal 200b. The feature determination unit 16 may reflect the evaluation in the process of determining the response features.

[0154] Although the invention according to the present disclosure has been described based on various drawings and examples, it should be noted that a person skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these modifications and corrections are included in the scope of the invention according to the present disclosure. For example, the functions included in each part, each means, each step, etc. can be rearranged so as not to be logically contradictory, and multiple means, steps, etc. can be combined into one or divided. In addition, the configurations shown in the embodiments described above may be appropriately combined.

[0155] The dialogue system, program, and dialogue method disclosed herein can be used in a system that provides a live video distribution service in which an AI agent and a broadcaster interact with each other.

[0156] To summarize the above explanation, possible configurations of the technology according to the present disclosure include the configurations listed below as appendices. (Appendix 1) A system for controlling an AI agent that has a dialogue with a speaker, comprising: An estimation unit that estimates an emotion of the speaker based on expression information related to an expression of the speaker; a feature determination unit that determines a response feature for the AI ​​agent to respond based on the emotion of the speaker estimated by the estimation unit; A dialogue system with (Appendix 2) the expression information is information of a voice uttered by the speaker, 2. The dialogue system according to claim 1, wherein the estimation unit estimates an emotion of the speaker based on a voice uttered by the speaker. (Appendix 3) The expression information is information on an image or video in which the speaker is shown, The dialogue system according to claim 1, wherein the estimation unit estimates the emotion of the speaker based on an image of the speaker captured in the image or the video. (Appendix 4) the response features determined by the feature determination unit are response features related to a voice uttered by the AI ​​agent, and include a setting of at least one response feature of the volume, voice quality, and voice tone of the voice uttered by the AI ​​agent; 4. The dialogue system according to claim 1, wherein the feature determination unit determines, as the response feature, the response feature amount corresponding to the emotion of the speaker estimated by the estimation unit. (Appendix 5) The dialogue system according to claim 4, wherein the feature determination unit generates the response features that indicate an emotion opposite to the emotion of the speaker estimated by the estimation unit. (Appendix 6) a response generation unit for generating a dialogue text, which is a text of a voice uttered by the AI ​​agent and is a response to the meaning indicated in the expression information; Further comprising: The dialogue system of claim 4 or 5, wherein the AI ​​agent reads out the dialogue text in a voice based on the settings of the response features. (Appendix 7) The dialogue system according to claim 6, wherein the feature determination unit determines the response features based on the emotion of the speaker estimated by the estimation unit and the dialogue text. (Appendix 8) The dialogue system of claim 6 or 7, wherein the response generation unit generates the dialogue text based on the meaning indicated in the expression information and the emotion of the speaker estimated by the estimation unit. (Appendix 9) a role setting unit that sets a role of the AI ​​agent when the AI ​​agent has a dialogue with the speaker; Further comprising: 9. The dialogue system according to any one of Supplementary Note 1 to Supplementary Note 8, wherein the feature determination unit determines the response feature in accordance with the role set by the role setting unit. (Appendix 10) 10. The dialogue system according to claim 9, wherein the role setting unit changes the role of the AI ​​agent in response to the emotion of the speaker estimated by the estimation unit. (Appendix 11) a dialogue between the speaker and the AI ​​agent is carried out in a distribution interface that is viewable by viewers and has a comment section in which comments from the viewers can be posted; 11. The dialogue system according to any one of claims 1 to 10, wherein the feature determination unit selects, as the response feature, a first comment corresponding to the emotion estimated by the estimation unit from among the comments posted in the comment section, and determines the first comment as a comment to be read out by the AI ​​agent. (Appendix 12) a response generation unit for generating a dialogue text, which is a text of a voice uttered by the AI ​​agent and is a response to the meaning indicated in the expression information; Further comprising: The dialogue system of claim 11, wherein the AI ​​agent distinguishes between the dialogue text and the first comment and reads them out. (Appendix 13) a dialogue between the speaker and the AI ​​agent is carried out in a distribution interface that is viewable by viewers and has a comment section in which comments from the viewers can be posted; The dialogue system according to any one of Supplementary Note 1 to Supplementary Note 12, wherein the feature determination unit generates, as the response feature, a second comment corresponding to the emotion of the speaker estimated by the estimation unit, and determines the second comment as the comment to be posted in the comment section and read out by the AI ​​agent. (Appendix 14) a response generation unit for generating a dialogue text, which is a text of a voice uttered by the AI ​​agent and is a response to the meaning indicated in the expression information; Further comprising: The dialogue system of claim 13, wherein the AI ​​agent distinguishes between the dialogue text and the second comment and reads them out. (Appendix 15) A program for controlling an AI agent that is executed by a computer and that engages in a dialogue with a speaker, comprising: Estimating the emotion of the speaker based on expression information related to an expression of the speaker; determining response features for the AI ​​agent to respond based on the estimated emotion of the speaker; and A program executed on the computer. (Appendix 16) 1. A computer-implemented method for controlling an AI agent that interacts with a speaker, comprising: Estimating the emotion of the speaker based on expression information related to an expression of the speaker; determining response features for the AI ​​agent to respond based on the estimated emotion of the speaker; and Interaction methods including. (Appendix 17) A system for controlling an AI agent that has a dialogue with a speaker, comprising: An estimation unit that estimates an emotion of the speaker based on expression information related to an expression of the speaker; a feature determination unit that determines a response feature amount when the AI ​​agent responds based on the emotion of the speaker estimated by the estimation unit; Equipped with The response features determined by the feature determination unit are response features related to the voice uttered by the AI ​​agent, and include settings for at least one of the volume, voice quality, and tone of voice of the voice uttered by the AI ​​agent. (Appendix 18) A system for controlling an AI agent that has a dialogue with a speaker in a distribution interface that is viewable by viewers and has a comment section in which comments from the viewers can be posted, comprising: An estimation unit that estimates an emotion of the speaker based on expression information related to an expression of the speaker; a feature determination unit that selects, from among the comments posted in the comment section, a first comment that corresponds to the emotion estimated by the estimation unit and is to be read out by the AI ​​agent; A dialogue system with (Appendix 19) A system for controlling an AI agent that has a dialogue with a speaker in a distribution interface that is viewable by viewers and has a comment section in which comments from the viewers can be posted, comprising: An estimation unit that estimates an emotion of the speaker based on expression information related to an expression of the speaker; a feature determination unit that generates a second comment corresponding to the emotion of the speaker estimated by the estimation unit, the second comment being posted to the comment field and read out by the AI ​​agent; A dialogue system with [Explanation of symbols]

[0157] 10 memory unit, 11 distribution interface unit, 12 input unit, 13 output unit, 14 response generation unit, 15 estimation unit, 16 feature determination unit, 17 integration unit, 18 addition unit, 100 server, 110 server control means, 120 server storage means, 130 server communication means, 200 user terminal, 200a distributor terminal, 200b viewer terminal, 210 terminal control means, 220 terminal storage means, 230 terminal communication means, 240 input / output means, 250 display means, 260 microphone, 270 speaker, 280 camera, 300 network, 400 dialogue system

Claims

1. A system for controlling an AI agent that has a dialogue with a speaker, An estimation unit that estimates an emotion of the speaker based on expression information related to an expression of the speaker; A feature determination unit that determines a response feature when the AI ​​agent responds based on the emotion of the speaker estimated by the estimation unit; A dialogue system with

2. the expression information is information of a voice uttered by the speaker, The dialogue system according to claim 1 , wherein the estimation unit estimates the emotion of the speaker based on a voice uttered by the speaker.

3. The expression information is information on an image or video in which the speaker is shown, The dialogue system according to claim 1 , wherein the estimation unit estimates the emotion of the speaker based on an image of the speaker captured in the image or the video.

4. The response feature determined by the feature determination unit is a response feature related to the voice uttered by the AI ​​agent, and includes a setting of at least one response feature of the volume, voice quality, and voice tone of the voice uttered by the AI ​​agent; The dialogue system according to claim 1 , wherein the feature determining unit determines, as the response feature, the response feature amount corresponding to the emotion of the speaker estimated by the estimating unit.

5. The dialogue system according to claim 4 , wherein the feature determining unit generates the response features that indicate an emotion opposite to the emotion of the speaker estimated by the estimating unit.

6. a response generation unit that generates a dialogue text, which is a text of the voice uttered by the AI ​​agent and is a response to the meaning indicated in the expression information; Further comprising: The dialogue system according to claim 4 , wherein the AI ​​agent reads out the dialogue text in a voice based on settings of the response features.

7. The dialogue system according to claim 6 , wherein the feature determining unit determines the response feature based on the emotion of the speaker estimated by the estimation unit and the dialogue text.

8. The dialogue system according to claim 6 , wherein the response generation unit generates the dialogue text based on a meaning indicated in the expression information and the emotion of the speaker estimated by the estimation unit.

9. A role setting unit that sets a role of the AI ​​agent when the AI ​​agent has a dialogue with the speaker; Further comprising: The dialogue system according to claim 1 , wherein the feature determining unit determines the response feature in accordance with the role set by the role setting unit.

10. The dialogue system according to claim 9 , wherein the role setting unit changes the role of the AI ​​agent in response to the emotion of the speaker estimated by the estimation unit.

11. A dialogue between the speaker and the AI ​​agent is carried out in a distribution interface that is viewable by viewers and has a comment section in which comments from the viewers can be posted; 2. The dialogue system according to claim 1, wherein the feature determination unit selects, as the response feature, a first comment corresponding to the emotion estimated by the estimation unit from among the comments posted in the comment section, and determines the first comment as the comment to be read out by the AI ​​agent.

12. a response generation unit that generates a dialogue text, which is a text of the voice uttered by the AI ​​agent and is a response to the meaning indicated in the expression information; Further comprising: The dialogue system according to claim 11 , wherein the AI ​​agent distinguishes between the dialogue text and the first comment and reads them out.

13. A dialogue between the speaker and the AI ​​agent is carried out in a distribution interface that is viewable by viewers and has a comment section in which comments from the viewers can be posted; The dialogue system according to claim 1 , wherein the feature determination unit generates a second comment corresponding to the emotion of the speaker estimated by the estimation unit as the response feature, and determines the second comment as the comment to be posted in the comment section and read out by the AI ​​agent.

14. a response generation unit that generates a dialogue text, which is a text of the voice uttered by the AI ​​agent and is a response to the meaning indicated in the expression information; Further comprising: The dialogue system according to claim 13 , wherein the AI ​​agent distinguishes between the dialogue text and the second comment and reads them out.

15. A program for controlling an AI agent that is executed by a computer and that interacts with a speaker, Estimating the emotion of the speaker based on expression information related to an expression of the speaker; Determining response features for the AI ​​agent to respond based on the estimated emotion of the speaker; A program executed on the computer.

16. 1. A computer-implemented method for controlling an AI agent that interacts with a speaker, comprising: Estimating the emotion of the speaker based on expression information related to an expression of the speaker; Determining response features for the AI ​​agent to respond based on the estimated emotion of the speaker; Interaction methods including.

Citation Information

Patent Citations

  • Interactive business support system and interactive business support method

    JP2021157419A

Cited By

  • Information processing device, content generation method, and content generation program

    JP2026106878A

  • Information processing system, information processing device, information processing method, and program

    JP2026150107A

  • Post distribution system, post distribution method, program, and computer-readable recording medium

    JP7814599B1