Agent management device
The agent management device addresses the challenge of facilitating multi-user conversations by intelligently intervening to enhance interaction smoothness and engagement through state evaluation and behavior control.
Patent Information
- Application Number
- JP2023075484
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-01
- Publication Date
- 2024-11-14
AI Technical Summary
Conventional agents are ineffective in assisting the progress of conversations between multiple users, particularly in remote communication scenarios where finding time to converse is difficult and users may feel burdened.
An agent management device that acquires conversation information, evaluates the conversation state, and determines whether to intervene, controlling the agent's behavior to facilitate smoother interactions among multiple users by addressing factors like speaker bias, positivity, and inappropriate comments.
The agent management device enhances conversation flow by encouraging participation, guiding topics, and managing inappropriate remarks, thereby improving the overall smoothness and engagement of multi-user discussions.
Smart Images

Figure 2024160467000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an agent management device. [Background technology]
[0002] Conventionally, the use of "agents," which are computer programs that autonomously communicate with humans using natural language, has been increasing. For example, a multi-person thought-provoking dialogue device disclosed in the following Patent Document 1 stores dialogue control rules for selecting a dialogue act of an agent according to a dialogue situation, and utterance generation rules for selecting an utterance expression of an agent according to the dialogue situation, etc. The user utterance understanding unit converts a user utterance into a dialogue act and writes it into a utterance queue and a dialogue state. The dialogue control unit refers to the dialogue act at the top of the utterance queue and the history of the dialogue acts stored in the dialogue state, applies the dialogue control rules to select a dialogue act related to the agent, and removes the dialogue act at the top of the utterance queue. When acquiring a dialogue act, the agent utterance generation unit refers to the history of the dialogue acts, generates an utterance expression of the agent based on the utterance generation rules, presents it to the user, stores this dialogue act in the dialogue state, and adds it to the utterance queue. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2010-186237 A Summary of the Invention [Problem to be solved by the invention]
[0004] With the development of remote communication technology, there are more opportunities for multiple users in distant locations to converse with each other. While remote communication is convenient, it is difficult to take the appropriate timing in a conversation compared to a face-to-face conversation, and some users feel burdened by leading the conversation. In such cases, it may be possible to use an agent as an observer and have the agent assist the progress of the conversation. However, conventional agents are designed for the purpose of the agent itself to converse with humans, and there is a problem in that it is difficult to assist the progress of a conversation between multiple users.
[0005] An object of the present invention is to provide an agent that assists in the progress of a conversation between a plurality of users. [Means for solving the problem]
[0006] An agent management device according to one embodiment of the present invention comprises an acquisition unit that acquires conversation information including a plurality of elements related to a conversation between a plurality of users, a judgment unit that evaluates a state of the conversation based on the plurality of elements included in the conversation information and judges whether or not an agent should intervene in the conversation depending on a result of the evaluation, a decision unit that decides an action of the agent when a judgment result of the judgment unit is positive, and an execution unit that controls the agent to execute the action decided by the decision unit. Effect of the Invention
[0007] According to one aspect of the present invention, an agent that assists in the progress of a conversation between a plurality of users can be provided. [Brief description of the drawings]
[0008] [Figure 1] 1 is a block diagram showing a configuration of a system 1 according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing the configuration of a user terminal 20. [Diagram 3] FIG. 2 is a block diagram showing the configuration of a server 10. [Figure 4A] FIG. 2 is a schematic diagram showing an example of an agent E. [Figure 4B] FIG. 2 is a schematic diagram showing an example of an agent E. [Diagram 5] FIG. 2 is a diagram showing an outline of processing in an agent control unit 110. [Figure 6] 4 is a flowchart showing the operation of the processing device 103. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] A. First embodiment A-1. System configuration 1 is a block diagram showing a configuration of a system 1 according to an embodiment. The system 1 includes a server 10 and a plurality of user terminals 20A to 20C. The server 10 and the user terminals 20A to 20C are connected via a network N.
[0010] The user terminal 20A is owned by a user U1. The user terminal 20B is owned by a user U2. The user terminal 20C is owned by a user U3. Hereinafter, when there is no need to distinguish between the multiple user terminals 20A to 20C, they will simply be referred to as "user terminal 20." Furthermore, when there is no need to distinguish between the multiple users U1 to U3, they will simply be referred to as "user U."
[0011] The system 1 enables the user U to have various experiences in the virtual space VS by displaying the virtual space VS (see FIG. 4A, etc.) on the user terminal 20. In this embodiment, a case where the user U converses with another user U in the virtual space VS will be mainly described.
[0012] A user U acts as an avatar A in the virtual space VS. The avatar A corresponding to user U1 is referred to as avatar A1, the avatar A corresponding to user U2 is referred to as avatar A2, and the avatar A corresponding to user U3 is referred to as avatar A3. When there is no need to distinguish between the multiple avatars A1 to A3, they are simply referred to as "avatar A."
[0013] For example, an image of the virtual space VS (hereinafter referred to as the "virtual space image") displayed on the user terminal 20A shows the field of view of an avatar A1 corresponding to the user U1. The viewpoint of the virtual space image may be positioned near the avatar A1 so that the avatar A1 appears in the virtual space image. By operating the avatar A1 using the user terminal 20A, the user U1 can move within the virtual space VS and communicate, including have conversations, with other users U2 and U3.
[0014] The server 10 manages the virtual space VS. The server 10 also manages an agent E (see FIG. 4) that assists the user U in the virtual space VS. The server 10 is an example of an agent management device. More specifically, the server 10 manages virtual space data VD for displaying the virtual space VS on the user terminal 20. The server 10 also generates display data for displaying a virtual space image on each user terminal 20 based on the virtual space data VD, and transmits the display data to each user terminal 20.
[0015] The user terminal 20 is an example of a terminal used by a user U. The user terminal 20 is, for example, an information processing terminal such as a smartphone, a tablet terminal, a personal computer, or a head-mounted display. Although three user terminals 20A to 20C are illustrated in FIG. 1, the number of user terminals 20 may be four or more, or may be two.
[0016] A-2. User terminal 20 2 is a block diagram showing the configuration of a user terminal 20. The user terminal 20 includes a communication device 201, an input device 202, a display device 203, a microphone 204, a speaker 205, an imaging device 206, a storage device 207, a processing device 208, and a bus 220 that interconnects these devices.
[0017] The communication device 201 communicates with the server 10 using wireless communication or wired communication. In this embodiment, the communication device 201 has an interface connectable to a network N, and communicates with the communication device 101 of the server 10 (see FIG. 3) via the network N.
[0018] The input device 202 is an input device (e.g., a keyboard, a mouse, a switch, a button, or a sensor) that receives input from the outside. The display device 203 is a display device (e.g., various display panels such as a liquid crystal display panel or an organic EL display panel) that displays information to the outside. The display device 203 and the input device 202 may be integrated into one (e.g., a touch panel).
[0019] It should be noted that, in the case where the user terminal 20 is a head-mounted display, a projection device that projects an image onto lenses disposed in front of both eyes of the user U is provided instead of the display device 203.
[0020] The microphone 204 collects sounds generated around the user terminal 20 and outputs them as audio data. In this embodiment, the sounds collected by the microphone 204 are mainly the speech of the user U. The speaker 205 outputs sounds that can be heard around the user terminal 20 based on the input audio data. In this embodiment, the sounds output by the speaker 205 are mainly the speech of another user U or agent E.
[0021] The imaging device 206 generates an image of the user U (hereinafter referred to as a "user image"). When the user terminal 20 is a smartphone, a tablet terminal, or a personal computer, the imaging device 206 is, for example, an in-camera whose imaging direction faces the user U. Also, when the user terminal 20 is, for example, a head-mounted display, the imaging device 206 is, for example, a camera for detecting a line of sight.
[0022] The storage device 207 is a recording medium readable by the processing device 208. The storage device 207 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), and an Electrically Erasable Programmable Read Only Memory (EEPROM). The volatile memory is, for example, a Random Access Memory (RAM). The storage device 207 stores a program PG2. The program PG2 is a program for operating the user terminal 20.
[0023] The processing device 208 includes one or more central processing units (CPUs). The one or more CPUs are an example of one or more processors. Each of the processor and the CPU is an example of a computer.
[0024] The processing device 208 reads the program PG2 from the storage device 207. The processing device 208 executes the program PG2 to function as an input control unit 211 and an output control unit 212. At least some of the functions of the input control unit 211 and the output control unit 212 may be configured by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).
[0025] The input control unit 211 transmits information input from the input device 202 (hereinafter referred to as “input information”) to the server 10. The input information is, for example, information instructing the movement direction of the avatar A in the virtual space VS, the spoken voice of the user U, or a captured image of the user U.
[0026] The output control unit 212 outputs information (hereinafter referred to as "output information") transmitted from the server 10 from the display device 203. The output information is, for example, a virtual space image, or a speech voice of another user U or agent E, or the like.
[0027] A-3. Server 10 3 is a block diagram showing the configuration of the server 10. The server 10 includes a communication device 101, a storage device 102, a processing device 103, and a bus 150 that interconnects these devices.
[0028] The communication device 101 communicates with the user terminal 20 using wireless communication or wired communication. In this embodiment, the communication device 101 has an interface connectable to a network N, and communicates with a communication device 201 (see FIG. 2) of the user terminal 20 via the network N.
[0029] The storage device 102 is a recording medium readable by the processing device 103. The storage device 102 includes, for example, a non-volatile memory and a volatile memory. The non-volatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 102 stores a program PG1 and virtual space data VD.
[0030] The program PG1 is a program for operating the server 10. The virtual space data VD is data for displaying the virtual space VS on the user terminal 20. The virtual space data VD includes, for example, information indicating the shapes and appearances of objects placed in the virtual space VS, information indicating the positions of the objects, information indicating the shape and appearance of an avatar A corresponding to a user U, and information indicating the shape and appearance of an agent E.
[0031] The processing unit 103 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of the processor and the CPU is an example of a computer.
[0032] The processing device 103 reads the program PG1 from the storage device 102. The processing device 103 executes the program PG1 to function as an agent control unit 110, an avatar control unit 120, a virtual space control unit 130, and a transmission data generation unit 140. The agent control unit 110 also includes an acquisition unit 111, a generation unit 112, a judgment unit 113, a determination unit 114, and an execution unit 115. At least some of the functions of the agent control unit 110 (the acquisition unit 111, the generation unit 112, the judgment unit 113, the determination unit 114, and the execution unit 115), the avatar control unit 120, the virtual space control unit 130, and the transmission data generation unit 140 may be configured by circuits such as a DSP, an ASIC, a PLD, and an FPGA.
[0033] A-4. Details of the processing device 103 Next, the agent control unit 110 (acquisition unit 111, generation unit 112, judgment unit 113, decision unit 114 and execution unit 115), avatar control unit 120, virtual space control unit 130 and transmission data generation unit 140 realized by the processing device 103 will be described.
[0034] The agent control unit 110 controls the behavior of the agent E in the virtual space VS. The agent control unit 110 will be described in detail later.
[0035] The avatar control unit 120 controls the behavior of the avatar A in the virtual space VS. For example, in the case of the user terminal 20A, the avatar control unit 120 determines the position of the avatar A1 in the virtual space VS based on information transmitted from the user terminal 20A indicating the moving direction of the avatar A1 in the virtual space VS. The avatar control unit 120 also determines the facial expression of the avatar A1 based on the captured image (the facial expression of the user U1) transmitted from the user terminal 20A. The avatar control unit 120 also controls the speech of the avatar A1 based on the speech of the user U1 transmitted from the user terminal 20A. The speech of the avatar A1 is basically the speech of the user U1 itself, but the data of the speech may be processed by, for example, a voice changer. The avatar control unit 120 also determines the movement of the mouth of the avatar A1 based on the speech of the user U1 transmitted from the user terminal 20A.
[0036] Virtual space control unit 130 controls the virtual space VS. Controlling the virtual space VS includes reflecting the behavior of avatar A and the behavior of agent E determined by avatar control unit 120 on the virtual space data VD. Controlling the virtual space VS also includes reflecting changes in the virtual space VS caused by, for example, the behavior of avatar A or agent E on the virtual space data VD.
[0037] The transmission data generation unit 140 generates transmission data to be transmitted to the user terminals 20A to 20C. The transmission data is generated individually for each of the user terminals 20A to 20C and transmitted to each of the user terminals 20A to 20C. The virtual space data VD mainly includes image data and audio data. The image data causes the display device 203 of the user terminal 20 to display a virtual space image. The audio data causes the speaker 205 of the user terminal 20 to output sound in the virtual space VS.
[0038] Taking the transmission data to be transmitted to the user terminal 20A as an example, the transmission data generation unit 140 identifies a portion of the virtual space VS included in the visual field of the avatar A1 based on the position (including the line of sight direction) of the avatar A1 in the virtual space VS. The transmission data generation unit 140 extracts image data showing the portion of the virtual space VS from the virtual space data VD. When other avatars A2, A3 and agent E are located in the portion of the virtual space VS, the images of the other avatars A2, A3 and agent E are also included in the image data. In addition, the transmission data generation unit 140 identifies sounds occurring around the position of the avatar A1 in the virtual space VS, and generates audio data showing the sounds. The transmission data generation unit 140 adjusts the volume of each sound based on the volume of the sounds and the distance between the source of the sounds and the avatar A1, and then generates the audio data. In this embodiment, the audio data mainly includes the speech of the other avatars A and the speech of the agent E with whom the avatar A1 (user U1) is talking in the virtual space VS.
[0039] Next, the details of the agent control unit 110 will be described. In this embodiment, the behavior of the agent E will be described in detail with respect to the contents of the utterances of the agent E. As described above, the agent E assists the user U (avatar A) in the virtual space VS. For example, one agent E may be attached to one avatar A (user U), or multiple agents E may be attached to one avatar A (for example, different agents E assist depending on the contents of the action), or one agent E may be attached to multiple avatars A (for example, one agent E assists a group of multiple avatars A with the same purpose). In addition, a user U who does not need an agent E may not attach an agent E to his or her avatar A. In this embodiment, it is assumed that an agent E is attached only to the avatar A1 among the avatars A1 to A3.
[0040] FIG. 4A and FIG. 4B are schematic diagrams showing an example of an agent E. FIG. 4A and FIG. 4B show virtual space images VI1, VI2 and audio that the user U1 views using the user terminal 20A. In FIG. 4A and FIG. 4B, text surrounded by dotted lines indicates that the text is output as audio. In the virtual space VS, the avatar A1 (user U1) can create, for example, his own room (not shown), arrange virtual objects (furniture or decorations, etc.) that suit his preferences, and invite an avatar A corresponding to another user U. Outside the room, a streetscape as exemplified in FIG. 4A spreads out, and the avatar A1 can interact with other avatars A and go out to places of interest. The outside of the room is not limited to a streetscape, and may be a park, a forest, or the sea.
[0041] When the user U accesses the virtual space VS, an agent E appears and assists the actions of the avatar A1 (user U1) in the virtual space VS. The agent E is represented in the shape of a character so as to be distinguishable from the human-shaped avatar A, for example. The agent E introduces events in the virtual space VS and teaches how to participate in the events, for example, based on the preferences of the user U1. In the example of FIG. 4A, the agent E (strictly speaking, the agent control unit 110) knows that the user U1 is a fan of the baseball team X, based on the input of preference information from the user U1 or the history of the user U1's actions. Therefore, the agent E introduces a fan meeting of the baseball team X to the user U1.
[0042] FIG. 4B shows avatar A1 attending a fan meeting for baseball team X. Avatar A2 and avatar A3 are seated at the same table as avatar A1. Avatars A1 to A3 are all fans of baseball team X, and it is expected that the conversation will center around topics related to baseball team X.
[0043] In a one-to-one conversation between avatar A1 and agent E as shown in Fig. 4A, agent E actively speaks to avatar A1. On the other hand, in a scene in which avatar A1 is conversing with other avatars A2 and A3 as shown in Fig. 4B, agent E does not actively speak, but only speaks when necessary to help the conversation between avatars A1 to A3 proceed.
[0044] The following mainly describes the control of agent E in a case where a conversation is being held between a plurality of users U1 to U3 corresponding to a plurality of avatars A1 to A3, as shown in Fig. 4B. As described above, agent control unit 110 includes an acquisition unit 111, a generation unit 112, a judgment unit 113, a decision unit 114, and an execution unit 115.
[0045] The acquisition unit 111 acquires conversation information including multiple elements related to conversations between multiple users U1 to U3. In this embodiment, the acquisition unit 111 acquires, as the conversation information, voice data collected by the microphone 204 of each of the user terminals 20A to 20C. The acquisition unit 111 also acquires, as the conversation information, user images captured by the imaging device 206 of each of the user terminals 20A to 20C. For example, when the user terminals 20A to 20C are linked to wearable devices (e.g., smart watches, etc.) worn by the users U1 to U3, the acquisition unit 111 may acquire, as the conversation information, biological information (heart rate, respiratory rate, body temperature, etc.) of the users U1 to U3 measured by the wearable devices.
[0046] The generation unit 112 generates candidates for response sentences to be uttered by the agent E based on the utterance contents of the users U1 to U3 in the conversation. The generation unit 112 corresponds to a dialogue engine in a general agent system. Typically, the generation unit 112 generates a plurality of response sentence candidates for the utterance contents of the users U1 to U3. Each response sentence candidate is assigned a score indicating the degree of match with the utterance contents.
[0047] The contents of the utterances of the users U1 to U3 can be detected by performing a voice recognition process on the voice data acquired by the acquisition unit 111. The voice recognition process on the voice data may be performed by the server 10 or the user terminal 20.
[0048] The generation unit 112 may select candidates for motions (emotes) to be performed by the agent E based on the contents of the utterances of the users U1 to U3. The motions may be performed together with the utterance of a response sentence, or may be performed independently without the utterance of a response sentence.
[0049] Depending on the result of the determination by the determination unit 113 described below, agent E may not take action (such as uttering a response sentence or performing a motion). The reason why the response sentence is generated, etc., despite the possibility that agent E may not take action is to enable output of a response sentence candidate that takes into account the conversation history and context. However, in order to reduce computational resources, the generation unit 112 may generate a response sentence, etc., only when an action by agent E is required.
[0050] The determination unit 113 evaluates the state of the conversation based on multiple elements included in the conversation information, and determines whether or not the agent E should intervene in the conversation depending on the evaluation result. The determination unit 113 estimates an index indicating the state of the conversation based on, for example, multiple elements, and evaluates the state of the conversation based on the index. The state of the conversation is, for example, the smoothness of the conversation, the excitement of the conversation, or the breadth of the conversation. In the following, the smoothness of the conversation will be taken as an example of the state of the conversation. The smoothness of the conversation may be rephrased as the degree of smoothness of the conversation. When the evaluation value of the smoothness of the conversation falls below the intervention threshold, the determination unit 113 determines that the agent E should intervene in the conversation.
[0051] In this embodiment, the "elements included in the conversation information" are items that are directly derived by analyzing the conversation information. Also, in this embodiment, the "indicator of the conversation situation" is an item that is estimated using the elements included in the conversation information.
[0052] The elements included in the conversation information are classified into, for example, non-verbal information and linguistic information. In this embodiment, at least one of the multiple elements included in the conversation information includes non-verbal information related to the conversation of at least one of the multiple users U1 to U3. The non-verbal information is, for example, at least one of the facial expression, voice intonation, time length between statements (time length between statements), gaze direction, heart rate, respiratory rate, or body temperature of at least one user U. Of the non-verbal information exemplified above, the facial expression and gaze direction of the user U can be detected from a user image. Furthermore, the voice intonation and time length between statements can be detected from voice data. Furthermore, the heart rate, respiratory rate, or body temperature can be identified by acquiring biological information measured by the wearable device.
[0053] At least one of the multiple elements included in the conversation information may include language information related to the conversation of at least one of the multiple users U1 to U3. The language information is, for example, the utterance content of at least one user U. As described above, the utterance content of the user U is used to generate a response sentence in the generation unit 112. In addition, the utterance content of the user U may be used as material for the judgment in the judgment unit 113.
[0054] As an example, an index showing the state of a conversation is the bias of speakers in the conversation. The bias of speakers can be estimated, for example, by detecting the number of times each of users U1 to U3 who participate in the conversation speaks or the cumulative total time of speaking. The number of times each of users speaks or the cumulative total time of speaking can be detected, for example, by individually analyzing the voice data acquired from the user terminals 20A to 20C and tallying up the number of times and the duration of time that a voice is detected in the voice data.
[0055] When there is a bias in the speakers in the conversation, the determination unit 113 determines that the agent E should intervene in the conversation. The determination unit 113 evaluates that the smoothness of the conversation is higher when the number of utterances or the cumulative total time of utterances of each of the users U1 to U3 participating in the conversation is closer to the same value. On the other hand, the determination unit 113 evaluates that the smoothness of the conversation is lower when the variation in the number of utterances or the cumulative total time of utterances of each of the users U1 to U3 participating in the conversation is larger. For example, when the number of utterances or the cumulative total time of utterances of the user U3 among the users U1 to U3 is smaller than the number of utterances or the cumulative total time of utterances of the other users U1 and U2, it is predicted that the user U3 is in a situation where it is difficult to join the conversation. Also, when the number of utterances or the cumulative total time of utterances of the user U1 among the users U1 to U3 is significantly larger than the number of utterances or the cumulative total time of utterances of the other users U2 and U3, it is predicted that the user U1 will develop his / her own opinion, and the other users U2 and U3 will have difficulty speaking up. Therefore, in these cases, the smoothness of the conversation can be said to be low.
[0056] When agent E is made to intervene based on the bias of speakers in a conversation, the intervention threshold may be changed according to the number of participants in the conversation. Specifically, the more the number of participants in a conversation, the lower the intervention threshold may be (the less likely intervention occurs). This is because, in general, the more the number of participants in a conversation, the more difficult it becomes to make the degree of participation in the conversation uniform.
[0057] Also, as an example, the index indicating the conversation situation is the positivity of each of the multiple users U1 to U3 toward the conversation. In this embodiment, the positivity is a general term for the favorable attitude of the user U. The positivity can be estimated in a composite manner based on multiple elements. For example, the determination unit 113 can estimate the emotion of the user U based on the facial expression of the user U, and estimate the positivity of the user U based on the emotion of the user U. There are various known methods for classifying emotions, and for example, taking "joy, anger, sorrow, and happiness" ("joy," "anger," "sorrow," and "fun") as an example, a state in which at least one of "joy" or "fun" is high can be said to have a high positivity. Conversely, a state in which at least one of "anger" or "sorrow" is high can be said to have a low positivity.
[0058] The determination unit 113 may also estimate the positivity of the user U based on, for example, the intonation of the user U's voice. For example, it may be estimated that the positivity is high when the voice intonation is loud, and that the positivity is low when the voice intonation is soft. The determination unit 113 may also estimate the positivity of the user U based on, for example, the direction of the user U's gaze. For example, it may be estimated that the positivity is low when the gaze direction changes a lot (when the gaze direction is restless and looking around), and that the positivity is high when the gaze direction changes a little.
[0059] The positivity of the user U may be restated as, for example, the user U's emotion or the degree of interest in the conversation.
[0060] The determination unit 113 evaluates that the smoothness of the conversation is higher as the cumulative value of the positivity of the users U1 to U3 participating in the conversation is higher. On the other hand, the determination unit 113 evaluates that the smoothness of the conversation is lower as the cumulative value of the positivity of the users U1 to U3 participating in the conversation is lower. Alternatively, the determination unit 113 may evaluate the smoothness of the conversation based on the value of the positivity of the user U who has the lowest positivity among the users U1 to U3 participating in the conversation.
[0061] In addition, for example, the presence or absence of inappropriate remarks may be used as an indicator of the state of the conversation. Inappropriate remarks include, for example, abusive remarks against other users U participating in the conversation or other targets, or remarks that affirm or encourage crime. Whether or not an inappropriate remark has been made in the conversation is determined based on the content of the user U's speech.
[0062] When an inappropriate remark is made, the determination unit 113 determines that the agent E should intervene in the conversation. When, for example, a remark determined to be inappropriate is made, the determination unit 113 immediately determines that the smoothness of the conversation has fallen below the intervention threshold. Alternatively, the determination unit 113 may determine the degree of smoothness of the conversation according to, for example, the content of the inappropriate remark. Specifically, it may be determined that the smoothness of the conversation is lower as the degree of inappropriateness of the remark is greater.
[0063] Other indicators of the conversation status may include, for example, (1) the length of silent time from the end of the previous utterance, or (2) the length of time until the predicted value of the next utterance timing. The determination unit 113 determines that the longer these times are, the less smooth the conversation is. The determination unit 113 may also determine whether or not agent E should intervene in the conversation based on (3) the duration from the start of the dialogue. In this case, the determination unit 113 determines that agent E should intervene in the conversation when the duration from the start of the dialogue is equal to or greater than a threshold value.
[0064] The determination unit 113 may determine whether the agent E should intervene in the conversation using multiple elements included in the conversation information, regardless of the smoothness of the conversation. For example, if the users U1 to U3 participating in the conversation call out to or ask the agent E a question, the determination unit 113 determines that the agent E should intervene in the conversation. Conversely, if the users U1 to U3 make a statement to the agent E requesting that the agent E refrain from intervening, the determination unit 113 may raise the intervention threshold or set a limit on the number of times that the agent E is allowed to intervene in a certain time period, and based on this, the determination unit 113 determines whether the agent E should intervene in the conversation.
[0065] The determination unit 114 determines the behavior of agent E when the determination result of the judgment unit 113 is positive. In this embodiment, the determination unit 114 determines the behavior of agent E based on an indicator indicating the conversation situation. When the behavior of agent E is to utter a response sentence, the determination unit 114 selects a response sentence to be uttered by agent E from among the response sentence candidates generated by the generation unit 112. In addition, when the behavior of agent E is to perform a motion, the determination unit 114 selects a motion to be performed by agent E from among the motion candidates selected by the generation unit 112.
[0066] For example, when the indicator that causes the judgment result to be positive is a bias in the speaker in the conversation, the determination unit 114 determines that the agent E performs an action to encourage the user U, who has a lower frequency of speaking compared to other users U, to speak. For example, the user U who has a low frequency of speaking is user U3 (Mr. Y). The action of encouraging the user U3 to speak may be, for example, making a statement that directly encourages the user U3 to speak, such as "What do you think, Mr. Y?". In addition, the action of encouraging the user U3 to speak may be, for example, leading the conversation to a topic that the user U3 is comfortable talking about. The topic that the user U3 is comfortable talking about is, for example, a topic that matches the hobbies or preferences of the user U3, or a topic that was talked about when the user U3 had a high frequency of speaking. In this case, the determination unit 114 may determine, as the speech content of the agent E, a candidate response sentence generated by the generation unit 112 at a timing when the user U3 had a high frequency of speaking.
[0067] Also, for example, when the index causing the positive judgment result is the positivity of each of the multiple users U1 to U3 toward the conversation, the determination unit 114 determines that the agent E takes an action to encourage a change in the content of the conversation. The action to encourage a change in the content of the conversation may be, for example, guiding the conversation to a topic that the multiple users U are commonly interested in (in the above example, a topic related to baseball team X), or guiding the conversation to a topic that the user U with the lowest positivity among the multiple users U is likely to talk about.
[0068] If the positivity of any of the users U1 to U3 does not improve even after the agent E takes action to prompt a change in the content of the conversation, the problem may not be the content of the conversation, but the compatibility between the users U1 to U3 participating in the conversation. Therefore, in this case, the determination unit 114 may determine that the agent E takes action to prompt the end of the conversation.
[0069] Also, for example, when the indicator that is the cause of the positive judgment result is an inappropriate statement, the determination unit 114 determines that the agent E takes an action to prompt the agent E to end the conversation or change the content of the conversation. The action to prompt the agent E to end the conversation may be, for example, a statement that directly ends the conversation, such as "Shall we call it a day for today?" or "I want to go home soon," or a statement that makes it easier to leave the conversation, such as "It's almost time for the next appointment," to any of the users U1 to U3 participating in the conversation. In addition, the determination unit 114 may execute a motion in which the agent E makes a facial expression showing discomfort, for example.
[0070] The execution unit 115 controls the agent E to execute the action determined by the determination unit 114. The execution unit 115 generates voice data indicating the speech of the agent E and image data of the agent E performing a motion. The voice data and image data are transmitted to the user terminal 20 as part of the transmission data.
[0071] FIG. 5 is a diagram showing an overview of processing in the agent control unit 110. The agent control unit 110 acquires conversation information from the user terminal 20. The conversation information is, for example, voice data, a user image, or biometric information. The agent control unit 110 detects elements contained in the conversation information. The elements include linguistic information and non-linguistic information. The linguistic information is, for example, the content of the user U's utterance. The non-linguistic information is, for example, the user U's facial expression, intonation of voice, the length of time between utterances (indicated as "length of time between utterances" in the figure), the direction of gaze, heart rate, respiratory rate, body temperature, etc.
[0072] The agent control unit 110 estimates an index indicating the state of the conversation based on the elements. The index is, for example, a bias in speech frequency between users U, positivity, emotion, degree of interest, presence or absence of inappropriate remarks, etc. The agent control unit 110 judges whether or not the agent E should intervene based on the index, and determines the behavior of the agent E based on the index. The behavior of the agent E is, for example, encouraging a specific user U to speak, changing the topic of the conversation, encouraging the end of the conversation, etc.
[0073] A-5. Operation of the processing device 103 Fig. 6 is a flowchart showing the operation of the processing device 103. The flowchart in Fig. 6 shows the processing related to the agent control unit 110. The processing device 103 of the server 10 functions as an acquisition unit 111 and acquires conversation information from the user terminal 20 (step S11). The processing device 103 functions as a generation unit 112 and generates response sentence candidates based on the utterance contents of the users U1 to U3 included in the conversation information (step S12).
[0074] The processing device 103 functions as the determination unit 113 and detects a plurality of elements included in the conversation information (step S13). The processing device 103 functions as the determination unit 113 and estimates an index indicating the conversation situation based on the plurality of elements detected in step S13 (step S14). The processing device 103 functions as the determination unit 113 and evaluates the state of the conversation, specifically, the smoothness of the conversation, based on the index indicating the conversation situation, and determines whether or not to intervene in the conversation (step S15).
[0075] When it is determined that the conversation should be intervened (step S15: YES), the processing device 103 functions as a decision unit 114 and decides the action of agent E (step S16). The processing device 103 functions as an execution unit 115 and causes agent E to execute the action decided in step S16 (step S17), and returns the process to step S11. On the other hand, when it is determined in step S15 that the conversation should not be intervened (step S15: NO), the processing device 103 returns the process to step S11.
[0076] A-6. Summary of embodiments As described above, the server 10 according to the embodiment evaluates the state of the conversation based on multiple elements included in the conversation information, and judges whether the agent E should intervene in the conversation. By using multiple elements for the judgment, it is possible to more accurately judge whether the agent E should intervene in the conversation.
[0077] In addition, the server 10 uses non-verbal information related to the conversation as at least one of the multiple elements. Therefore, the state of the user U can be more accurately reflected in the judgment compared to the case where only linguistic information such as the contents of the utterance is used. The non-verbal information is at least one of the facial expression, intonation of the voice, the time between utterances, the direction of the gaze, the pulse rate, the respiratory rate, and the body temperature of the user U. Therefore, the state of the user U that is not reflected in the linguistic information can be grasped.
[0078] In addition, the server 10 estimates an index indicating the conversation situation based on a plurality of elements, evaluates the state of the conversation based on the index, and determines the behavior of the agent E based on the index. Thus, the behavior of the agent E that more accurately reflects the conversation situation can be determined.
[0079] For example, when there is a bias in who speaks in a conversation, the server 10 determines that the agent E performs an action to encourage a user U who is a speaker less frequently than other users U to speak. This reduces the bias in the frequency of speaking among the users U participating in the conversation.
[0080] Furthermore, when the positivity of at least one of the multiple users U1 to U3 participating in the conversation is equal to or lower than a threshold, the server 10 determines that the agent E takes an action to prompt a change in the content of the conversation. Thus, the conversation can be guided to a content that will interest all of the users U1 to U3 participating in the conversation.
[0081] Furthermore, when an inappropriate remark is made in the conversation, the server 10 determines that the agent E should end the conversation or take an action to prompt a change in the content of the conversation. This avoids a situation in which some users U are offended by an inappropriate remark or the conversation becomes unpleasant.
[0082] B: Variation The following are modified aspects of the above-described embodiment. Two or more aspects selected from the following modified aspects may be combined as appropriate within the scope of not being mutually contradictory.
[0083] B1: First modified example In the above embodiment, the communication between the users U1 to U3 in the virtual space VS has been described as an example. However, the present invention is not limited to this. For example, when the users U1 to U3 hold a WEB conference by connecting the user terminals 20A to 20C, the agent E may be used as an observer.
[0084] This embodiment is also applicable to the case where the audience views, for example, a panel discussion by guests (which may be video distribution by individuals, etc.) using the user terminal 20. In this case, for example, the agent E also participates in the conversation between the guests. The server 10 acquires conversation information (mainly non-verbal information such as facial images of the audience) from the user terminal 20 used by the audience, and estimates the positivity of the audience. If the positivity of the audience is decreasing, that is, if the content of the discussion is no longer of interest to the audience, the server 10 causes the agent E to take an action to prompt a change in the content of the conversation. Also, if an inappropriate remark is made by the guest, the server 10 causes the agent E to take an action to prompt a change in the content of the conversation or an action to prompt an end to the conversation.
[0085] As shown in the first modified example, the system 1 according to the present embodiment is applicable to various situations.
[0086] C:Other (1) In the above-described embodiment, ROM and RAM, etc. are given as examples of storage devices 102 and 205, but storage devices 102 and 205 may also be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disc), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium.
[0087] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
[0088] (3) In the above-described embodiment, the input / output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input / output information, etc. may be overwritten, updated, or added. The output information, etc. may be deleted. The input information, etc. may be transmitted to another device.
[0089] (4) In the above-described embodiments, the determination may be made based on a value represented by 1 bit (0 or 1), a Boolean (true or false) value, or a numerical comparison (e.g., comparison with a predetermined value).
[0090] (5) The order of the process steps, sequences, flow charts, etc. illustrated in the above-described embodiments may be changed without causing any inconsistency. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.
[0091] (6) Each function illustrated in Figures 2 and 3 is realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by using wires, wirelessly, etc.). The functional block may be realized by combining the one device or the multiple devices with software.
[0092] (7) The programs exemplified in the above-described embodiments should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.
[0093] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.
[0094] (8) In each of the above embodiments, the terms “system” and “network” are used interchangeably.
[0095] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a predetermined value, or corresponding other information.
[0096] (10) In the above-described embodiments, the portable device may be a mobile station (MS). A mobile station may be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other suitable term. In this disclosure, the terms "mobile station", "user terminal", "user equipment (UE)", "terminal", etc. may be used interchangeably.
[0097] (11) In the above-mentioned embodiments, the terms "connected" and "coupled" or any variation thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "accessed." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.
[0098] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."
[0099] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.
[0100] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this disclosure is not intended to be an exclusive or.
[0101] (15) In this disclosure, where articles have been added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural.
[0102] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."
[0103] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information). A combination of a plurality of aspects selected from the above may be used. [Explanation of symbols]
[0104] 1...system, 10...server, 20 (20A to 20C)...user terminal, 110...agent control unit, 111...acquisition unit, 112...generation unit, 113...judgment unit, 114...decision unit, 115...execution unit, 120...avatar control unit, 130...virtual space control unit, 140...transmission data generation unit, A (A1 to A3)...avatar, E...agent, N...network, U (U1 to U3)...user, VD...virtual space data, VS...virtual space.
Claims
1. An acquisition unit that acquires conversation information including a plurality of elements related to a conversation between a plurality of users; a determination unit that evaluates a state of the conversation based on the plurality of elements included in the conversation information, and determines whether or not an agent should intervene in the conversation depending on a result of the evaluation; a decision unit that decides an action of the agent when a result of the decision made by the decision unit is affirmative; an execution unit that controls the agent to execute the action determined by the determination unit; An agent management device comprising:
2. At least one of the plurality of elements includes non-verbal information of at least one of the plurality of users regarding the conversation.
2. The agent management device according to claim 1.
3. The non-verbal information is at least one of a facial expression, a voice intonation, a time length between utterances, a gaze direction, a pulse rate, a respiratory rate, and a body temperature of the at least one user.
3. The agent management device according to claim 2.
4. The determination unit estimates an index indicating a state of the conversation based on the plurality of elements, and evaluates the state of the conversation based on the index; The determination unit determines the action based on the index.
2. The agent management device according to claim 1.
5. The indicator is a bias of a speaker in the conversation, the determination unit determines that the agent should intervene in the conversation when there is a bias in the speaker; the determination unit determines that the agent should perform an action to encourage the user who has a lower frequency of speaking compared to other users to speak; 5. The agent management device according to claim 4.
6. the index is a positivity level of each of the plurality of users toward the conversation, the determination unit determines that the agent should intervene in the conversation when the positivity of at least one of the plurality of users is equal to or less than a threshold value; The determination unit determines that the agent should take an action to prompt a change in the content of the conversation.
5. The agent management device according to claim 4.
7. the indicator being the presence or absence of inappropriate remarks in the conversation; the determination unit determines that the agent should intervene in the conversation when the inappropriate remark is made, The determination unit determines that the agent should take an action to prompt the end of the conversation or a change in the content of the conversation.
5. The agent management device according to claim 4.
Citation Information
Patent Citations
Speech control system in remote conference system
JP2003304337A
Interactive device
JP2018045118A
Information processor, communication server, and game system
JP2018187209A
Information processing device and program
JP2021090668A
Conference support device, conference support method, and conference support program
JP2022068905A
Cited By
system
JP2026140308A
system
JP7934880B2
