Conversation management system and conversation management method

Abstract

To determine a speech production destination in conversation among a plurality of participants.SOLUTION: A conversation management system manages conversation in which a plurality of participants are present. The conversation management system stores past speech production history information of each of the participants in the conversation. The conversation management system acquires a text feature amount of first current speech production of a speaker. The conversation management system acquires a text feature amount of past speech production of each of a plurality of listeners. The conversation management system calculates a speech production destination probability of each listener on the basis of the text feature amount of the first current speech production and the text feature amount of the past speech production of each listener. The conversation management system determines a speech production destination of the first current speech production on the basis of the speech production probabilities of the listeners.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] TECHNICAL FIELD This disclosure relates to techniques for managing multi-participant conversations. [Background technology]

[0002] There is technology related to agents that support conversations in which multiple people participate. In everyday life, it is common for multiple people to have conversations. Here, the people participating in the conversation are referred to as users. In a conversation that uses a system in which an agent that supports the conversation exists, multiple users not only converse with each other, but also with the system's agent. That is, each user converses with one or more participants. A participant here means a human user or an agent. The users coordinate the conversation among themselves so that the purpose of the conversation can be achieved efficiently and effectively.

[0003] An agent may react to the previous utterance of the user, may spontaneously talk about new ideas, or may interrupt the user's speech as necessary. In order to achieve the purpose of a conversation, it is important that the conversation between users can continue seamlessly. Therefore, one possible means of realizing this is for the agent to support the seamless continuation of the conversation between users by speaking appropriate content at the right time. For an agent to efficiently support a conversation, it is essential that it has the function of identifying which participant the speaker is speaking to. However, in a conversation involving multiple users, it is not self-evident how to detect who the speaker is speaking to.

[0004] For example, US Patent No. 9,761,247 discloses a system for supporting conversations involving multiple users, who converse with each other and with an agent of the system, and which uses prosodic and lexical features in the conversations of the multiple users to identify whether a given speech is intended for the agent. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] U.S. Patent No. 9,761,247 Summary of the Invention [Problem to be solved by the invention]

[0006] Previous technologies for detecting the destination of an utterance have focused on choosing between two options: whether the user's utterance is directed to an agent or to a human user. However, to truly communicate, conventional technologies alone are insufficient. For example, the system's agent needs to predict which user in a conversation is likely to speak next, and depending on the prediction result, decide whether the agent should speak next or wait. For this reason, it is important to properly predict which user the speaker is speaking to.

[0007] There are two types of target detection: agent-directed and human-directed. Agent-directed means that the target belongs to an agent of the system. However, inferring agent-directed utterances is not enough. In a group conversation, a speaker can address any of the other participants (including human users and agents) or address the whole group. Therefore, when detecting who a speaker is addressing, it is important to distinguish between addressing a specific user and addressing a group to continue the conversation seamlessly. [Means for solving the problem]

[0008] One aspect of the present disclosure is a conversation management system including one or more storage devices and one or more arithmetic devices. The one or more storage devices store past utterance history information of each of a plurality of participants in a conversation. The one or more arithmetic devices acquire text features of a first current utterance of a speaker. The one or more arithmetic devices acquire text features of past utterances of each of a plurality of listeners from the past utterance history information. The one or more arithmetic devices calculate a destination probability of each listener based on the text features of the first current utterance and the text features of the past utterances of each listener. The one or more arithmetic devices determine a destination of the first current utterance based on the destination probabilities of the plurality of listeners. Effect of the Invention

[0009] According to one aspect of the present disclosure, it is possible to estimate the addressee of a speaker in a conversation involving multiple participants. [Brief description of the drawings]

[0010] [Figure 1A] 1 illustrates an example of a usage form of a conversation management system according to an embodiment of the present specification. [Figure 1B] 4 shows another application form of the conversation management system of the present specification. [Diagram 2] 2 shows an example of the hardware configuration of a conversation management system. [Diagram 3] 2 shows an example of the hardware configuration of a user terminal. [Figure 4] 1 illustrates an example of a functional configuration of a conversation management system according to an embodiment of the present specification. [Diagram 5] 13 shows an example of the configuration of participant specific feature management information. [Figure 6] 13 shows an example of the configuration of utterance history management information. [Figure 7] 13 is a flowchart of an example of an operation of the conversation management system. [Figure 8] Shown are samples of utterances collected during a conversation. [Figure 9]13 shows examples of inputs to and outputs from a recurrent neural network that can be used by the speech destination determination unit to determine the speech destination. [Figure 10] 1 illustrates a schematic example of semantic relatedness of words. [Figure 11] 13 shows an example of a multitasking process for determining a speech destination. [Figure 12A] Indicates one of a series of utterances that make up a conversation. [Figure 12B] Indicates one of a series of utterances that make up a conversation. [Figure 12C] Indicates one of a series of utterances that make up a conversation. [Figure 12D] Indicates one of a series of utterances that make up a conversation. [Figure 12E] Indicates one of a series of utterances that make up a conversation. [Figure 12F] Indicates one of a series of utterances that make up a conversation. [Figure 12G] Indicates one of a series of utterances that make up a conversation. [Figure 12H] Indicates one of a series of utterances that make up a conversation. [Figure 13] This shows an RNN that distinguishes whether a speaker is speaking to a group or an individual. [Figure 14A] The details of the processing in LSTM are shown below. [Figure 14B] Here are details on how words are input to the word embedding layer and LSTM. [Figure 15] When the target of the utterance is an individual, we show an RNN that determines which listener the utterance is targeted to. [Figure 16] The configuration of MaLSTM is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] In the following, when necessary for convenience, the description will be divided into multiple sections or embodiments, but unless otherwise specified, they are not unrelated to each other, and one is related to the other as a partial or complete modification, detail, supplementary explanation, etc. Furthermore, in the following, when the number of elements, etc. (including the number, numerical value, amount, range, etc.) is mentioned, it is not limited to the specific number, and may be more than or less than the specific number, unless otherwise specified or clearly limited in principle to a specific number, etc.

[0012] This system may be a physical computer system (one or more physical computers) or a system built on a group of computing resources (multiple computing resources) such as a cloud infrastructure. The computer system or the group of computing resources includes one or more interface devices (including, for example, a communication device and an input / output device), one or more storage devices (including, for example, a memory (main memory) and an auxiliary storage device), and one or more arithmetic devices.

[0013] When a function is realized by a program being executed by a computing device, the defined processing is performed using a storage device and / or an interface device, etc. as appropriate, and therefore the function may be at least a part of the computing device. Processing described with a function as the subject may be processing performed by a system having a computing device. The program may be installed from a program source. The program source may be, for example, a program distribution computer or a computer-readable storage medium (for example, a computer-readable non-transitory storage medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0014] An embodiment of the present specification provides a mechanism for estimating to whom a speaker is speaking (speech destination) when multiple users are having a conversation. An embodiment of the present specification is a system for supporting a conversation involving multiple participants. A "participant" refers to a participant in a conversation. A "participant" may be a "user" or an "agent." A "user" is a human being. An "agent" is, for example, a robot equipped with a voice recognition function, a computer system that executes an application program, or a program thereof. An agent may further be equipped with a voice synthesis function and be capable of responding by voice.

[0015] In one embodiment of the present specification, a group participating in a conversation may be composed of three or more participants and may include one or more agents. However, it is also possible that there is no agent and the conversation management system described in this embodiment simply estimates the next speaker in a conversation between human users. In a conversation in which multiple participants are participating, the conversation management system listens to utterances by the participants and estimates which participant is the intended speaker.

[0016] In one embodiment of the present specification, the conversation management system performs an N-free classification, where N is a symbol representing the number of participants, meaning that the system for detecting the destination can operate independently of the number of participants. The conversation management system can operate in conversations with any number of participants. In one embodiment of the present specification, the conversation management system calculates a destination probability score for each participant based on a calculation using the input utterance and the speech history of each listener and other information. The participant with the highest probability is presumed to be the destination of the current utterance.

[0017] A conversation management system according to an embodiment of the present specification estimates a conversation destination by executing the process described below. Fig. 1A shows an example of a usage form of a conversation management system according to an embodiment of the present specification. The system includes a conversation management system 101 and a plurality of user terminals 104. In the example of Fig. 1A, the conversation management system 101 is configured with one server, and four users are participating in a conversation.

[0018] 1A, the user terminal 104 accesses the access point 103 wirelessly or via a cable. The conversation management system 101 and the user terminal 104 communicate with each other via the network 102 and the access point 103. Note that the configuration of the communication network between the conversation management system 101 and the user terminal 104 is arbitrary.

[0019] The conversation management system 101 provides a conversation service, such as a conference or meeting, via a network to the user terminal 104. Furthermore, the conversation management system 101 executes an agent program and supports conversations between users. The conversation management system 101 provides a conversation service in which multiple users can participate, and supports the conversation.

[0020] The conversation management system 101 can be configured with one or more computers, or can be configured with computer resources on a cloud. Thus, the conversation management system 101 can include one or more computing devices and one or more storage devices.

[0021] A plurality of users can use user terminals 104 to log in to the conversation management system 101 and hold conversations, including meetings of various sizes, on the platform provided by the conversation management system 101. A user can transmit his / her own image to other participants using a camera implemented in the user terminal 104, and can transmit his / her own speech to other participants using a microphone. A user can view images of other users on a display device implemented in the user terminal 104, and can hear the speech of other participants through a speaker.

[0022] 1B shows another application form of the conversation management system 101 of the present specification. The conversation management system 101 can be implemented in, for example, a desktop or laptop computer system. A plurality of users directly converse face-to-face, and the conversation management system 101 present in the conversation site supports the conversation between the users. In the example of FIG. 1B, four users 105 participate in the conversation.

[0023] Each participant can talk with other participants without using a device. The conversation management system 101 can obtain voice data of the speech of the user 105 by a built-in microphone, and can make the speech to the participants by a speaker. The conversation management system 101 may be equipped with a camera so that an image of each user 105 can be obtained.

[0024] 2 shows an example of the hardware configuration of the conversation management system 101. The conversation management system 101 includes a main storage device 202 configured with a volatile storage element such as a RAM, and an auxiliary storage device 203 configured with an appropriate non-volatile storage element such as an SSD (Solid State Drive) or a hard disk drive.

[0025] The conversation management system 101 further includes a calculation device 201 such as a CPU that executes a program 206 stored in an auxiliary storage device 203 or the like by reading it into a main storage device 202, and performs overall control of the system itself as well as various judgments, calculations, and control processes. The conversation management system 101 includes a communication interface 204 for connecting to a network and exchanging data.

[0026] The conversation management system 101 includes input devices 208, such as a keyboard, mouse, touch panel, microphone, and camera, that accept input operations, and output devices 305, such as a display that displays processing results and a speaker that outputs sound. The components of the conversation management system 101 can communicate with each other via the internal bus 205. Just as the output device 209 and the input device 208 can be omitted in the configuration example shown in Fig. 1A, some of the components of the conversation management system 101 may be omitted.

[0027] The auxiliary storage device 203 stores a program 206 for implementing the functions required for the conversation management system 101 of this embodiment, as well as an information database (DB) 207 that stores data required for various processes. Figure 2 shows the program 206 loaded from the auxiliary storage device 203 to the main storage device 202.

[0028] 3 shows an example of the hardware configuration of the user terminal 104. The user terminal 104 includes a main memory device 302 configured with a volatile memory element such as a RAM, and an auxiliary memory device 303 configured with an appropriate non-volatile memory element such as an SSD or a hard disk drive. The user terminal 104 further includes a calculation device 301 such as a CPU that reads out a program 308 stored in the auxiliary memory device 303 or the like into the main memory device 302 and executes it, performs overall control of the system itself, and performs various judgments, calculations, and control processes. The program 308 includes a program that enables the user terminal 104 to participate in a conversation via a network.

[0029] The user terminal 104 includes a communication interface 306 for connecting to a network and exchanging data, an input device 304 such as a keyboard, mouse, touch panel, microphone, camera, etc. for accepting input operations from the user, and an output device 305 such as a display for showing the processing results to the user and a speaker for outputting sound, etc. These components of the user terminal 104 can communicate with each other via an internal bus 307. In the auxiliary storage device 303, necessary information (data) is stored in addition to a program 308 for implementing necessary functions.

[0030] Next, a description will be given of functions of the conversation management system 101. Fig. 4 shows an example of a functional configuration of the conversation management system 101 according to an embodiment of the present specification. The functions described below are implemented by, for example, the computing device 201 of the conversation management system 101 executing the program 206.

[0031] The program 206 of the conversation management system 101 includes a voice recognition unit 401, a participant identification unit 403, a voice feature extraction unit 405, a video feature extraction unit 407, a speech destination determination unit 411, and an agent 413. The database 207 includes participant specific feature management information 421 and speech history management information 423. Details of the participant specific feature management information 421 and the speech history management information 423 will be described later with reference to Figs. 5 and 6, respectively.

[0032] The voice recognition unit 401 recognizes speech by a user. The voice is received via a microphone of the conversation management system 101 (example of FIG. 1B) or via a network from the user terminal 104 (example of FIG. 1A). For example, when a voice is followed by a predefined silence period, for example, silence of 200 milliseconds or more, the voice recognition unit 401 can extract the voice as a speech.

[0033] The speech recognition unit 401 further converts the user's voice data into text. The result of conversion from the speech is the text feature of the speech. When the speech recognition unit 401 receives speech as input, it converts it into vocabulary or words and outputs it. The text of the speech is stored in the speech history management information 423.

[0034] In the situation shown in Fig. 1B, the participant identification unit 403 determines the speech position of the participant and associates the determined speech position of the participant with the participant ID. As shown in Fig. 1A, in the case of a conversation via a network using the user terminal 104, the participant identification unit 403 can determine the participant ID by the participant's login ID without determining the position of the participant.

[0035] The speech feature extraction unit 405 extracts speech features from speech data of an utterance by a predetermined processing method. Examples of the speech features include MFCC (Mel-Frequency Cepstrum Coefficients), PLP (perceptual linear prediction), pitch, etc. The speech feature extraction unit 405 stores the extracted speech features in the speech history management information 423.

[0036] The video feature extraction unit 407 extracts video features from the video data of the speaker by a predetermined processing method. The video features are generated from pixel values ​​of the video data using, for example, a convolutional neural network. The video data is received from the camera of the conversation management system 101 (example of FIG. 1B) or from the user terminal 104 via a network (example of FIG. 1A). The video feature extraction unit 407 stores the extracted video features in the speech history management information 423. In addition to the video features of the speaker, the video feature extraction unit 407 may also extract video features of other users and store them in the speech history management information 423.

[0037] The destination determination unit 411 estimates the destination of the current utterance based on the speaker's current utterance and the listener's past utterance history. As described later, the destination may be estimated based on the listener's unique feature amount, and the feature amounts of the voice data, text data, and video data at the time of the utterance.

[0038] The agent 413 encourages the progress of the conversation by interacting with the participants of the conversation through voice or voice and video. The agent 413 can determine whether or not an utterance is permitted based on the recognized utterance destination, and can determine the utterance content and utterance destination to be output based on the recognized utterance and utterance destination.

[0039] 5 shows an example of the configuration of participant-specific feature management information 421. The participant-specific feature management information 421 manages features unique to each participant taking part in a conversation. The participant-specific feature management information 421 is set and registered in advance before a conversation starts. In the example configuration shown in FIG. 5, the participant-specific feature management information 421 includes a participant ID column 451, a name column 452, a role column 453, a soft skill column 454, and a hard skill column 455.

[0040] The participant ID column 451 indicates the ID of each participant in the conversation. The name column 452 indicates the name of each participant. The role column 453 indicates the role assigned to each participant. The role column 453 can indicate the role of the participant in the organization or conversation. In the example of FIG. 5, the role column 453 indicates the position in the company. The soft skills column 454 indicates the participant's general (non-specialized) skills that are not based on specialized knowledge. The hard skills column 455 indicates the participant's skills (specialized skills) based on specialized knowledge. Note that the participant-specific features shown in FIG. 5 are merely examples, and features different from these may be registered.

[0041] Fig. 6 shows an example of the configuration of the speech history management information 423. The speech history management information 423 manages the history of utterances in a conversation. The speech history management information 423 is updated as the conversation progresses. In the example configuration shown in Fig. 6, the speech history management information 423 includes an utterance ID field 471, a speaker ID field 472, an utterance text field 473, an utterance time field 474, an audio feature field 475, and a video feature field 476.

[0042] The utterance ID field 471 indicates the ID of the utterance. The speaker ID field 472 indicates the ID of the participant (speaker) who made the utterance. The utterance text field 473 indicates the text of the utterance (text feature). As described above, the utterance text is generated by the speech recognition unit 401 and stored in the utterance history management information 423. The utterance time field 474 indicates the time of the utterance, for example, the start time.

[0043] The audio feature field 475 indicates the audio feature of the utterance. As described above, the audio feature is extracted by the audio feature extraction unit 405 and stored in the utterance history management information 423. The video feature field 476 indicates the video feature of the speaker who made the utterance. As described above, the video feature is extracted by the video feature extraction unit 407 and stored in the utterance history management information 423. Video features of other users other than the speaker may also be stored together.

[0044] The speech history management information 423 may store information on all utterances in a conversation, or may store only information on a specified number of utterances or less for each speaker ID. The speech history management information 423 may manage paths indicating files of utterance texts and features instead of storing these themselves. Note that the speech history management information 423 may manage only a part of the information shown in FIG. 6.

[0045] The operation of the conversation management system 101 will be described below. Fig. 7 is a flowchart of an example of the operation of the conversation management system 101. The conversation management system 101 estimates the speaker's utterance destination in a conversation involving three or more participants. Participants can include agents 413 of the conversation management system 101 in addition to human users.

[0046] First, the conversation management system 101 receives audio data and video data of a conversation (S11). The audio data can be received from a microphone of the conversation management system 101 or a user terminal 104 via a network, and the video data can be received from a camera of the conversation management system 101 or a user terminal 104 via a network.

[0047] Next, the speech recognition unit 401 analyzes the speech data, recognizes one utterance, and extracts the speech data (S12). Specifically, the speech is converted into text, and text features are extracted. When a speech is followed by silence for a predefined silence period or longer, the speech recognition unit 401 can extract the speech as a speech, convert it into text, and output it.

[0048] Next, the participant identification unit 403 analyzes the speech audio data to determine the corresponding speaker (S13). The participant identification unit 403 can determine the speaker's ID from the speech audio data using machine learning (e.g., diarization technology). Diarization takes a voice as input and outputs who the speaker is.

[0049] As a method that does not use machine learning, the participant identification unit 403 can also calculate and obtain the relative positional relationship between the speaker and the microphone using a specific microphone device including a multi-channel microphone array. Then, the speaker can be identified by the participant ID associated with the speaker's position information. If video information is also available, person identification based on a face image may be used. Furthermore, the speaker may be estimated based on the movement of the lips of the face, or the speaker may be estimated using both image information and audio information.

[0050] Next, features are extracted from the voice data of the utterance and the video data of the speaker at the time of the utterance (S14). Specifically, the voice recognition unit 401 extracts text features, the voice feature extraction unit 405 extracts features of the voice data, and the video feature extraction unit 407 extracts video features from the video data of the speaker. The voice recognition unit 401 can use a known automatic voice recognition technique. The voice recognition unit 401, the voice feature extraction unit 405, and the video feature extraction unit 407 can be configured using, for example, a neural network. The voice recognition unit 401, the voice feature extraction unit 405, and the video feature extraction unit 407 each store the extracted features in a record of the utterance history management information 423 (S15).

[0051] Next, the utterance destination determination unit 411 determines the utterance destination of the current utterance (S16). The method of determining the utterance destination will be described in detail later. In one embodiment of this specification, the utterance destination determination unit 411 acquires information on the current utterance and information on past utterances from the utterance history management information 423, and further acquires the unique feature of each participant from the participant unique feature management information 421. The utterance destination determination unit 411 determines the utterance destination based on the acquired information.

[0052] In one embodiment of the present specification, the speech destination determination unit 411 determines the type of speech destination (S17). Specifically, it is determined whether the speech destination is all listeners of the participants (the entire group), a specific listener user, or an agent executed by the conversation management system 101. If a specific user is the speech destination, the speech destination determination unit 411 determines the user ID of that user.

[0053] The output of destination determination unit 411 is one of the following: "User ID", "Agent", or "Group". If the destination is "Agent", agent 413 generates a response (S18) and outputs it (S19). If the destination is a specific "User ID" or "Group", agent 413 determines whether feedback is permitted (S20). If it is permitted (S20: YES), agent 413 generates a response (S18) and outputs it (S19). If it is not permitted, the flow returns to step S11.

[0054] There are cases where the agent 413 of the conversation management system 101 is given feedback even if the output of the speech destination determination unit 411 is not an utterance directed to the agent 413. Examples of such conditions are shown below.

[0055] There are cases where the destination determination unit 411 returns the "user ID" of the destination, and the agent 413 cannot detect the feedback of a particular participant for a certain period of time. In this case, the agent 413 may, for example, call out the corresponding user ID by voice to prompt the person to respond.

[0056] When the destination decision unit 411 returns that the destination is a "group", this group includes an agent 413 of the conversation management system 101. Therefore, the agent 413 may give feedback based on the knowledge it has.

[0057] The conversation management system 101 may detect the emotion of the participants as additional information by using features (visual features), audio features, or text features of the video data. This emotion estimation may estimate the emotion of the participants, such as whether the listener understands the talk, by using a neural network. The agent 413 may return feedback according to the emotion of the participants. For example, if it is determined that the listener does not understand the talk, the agent 413 may prompt the speaker to repeat the explanation.

[0058] The process performed by the utterance destination determination unit 411 will be described in detail below. The utterance destination determination unit 411 determines the destination of the current utterance based on the past utterance history of the participants. This enables a more appropriate utterance destination to be determined. As described above, the utterance history management information 423 stores feature amount information of past utterances of each speaker. The utterance history management information 423 accumulates not only the immediately preceding utterance but also the history of multiple past utterances.

[0059] In the speech history of each participant, for example, up to k latest input features are stored. k is a natural number, for example, a natural number equal to or greater than 2. The past feature information means k speech histories, that is, k past input features collected from each participant during the conversation. The extracted speech history is regarded as the past feature information.

[0060] For example, when only text features are considered and there are four participants in a conversation, the collected utterances may look like the sample shown in FIG. 8. The current utterance is an utterance by JAMES. FIG. 8 shows the current utterance text of JAMES. As examples of past utterance texts, the utterance texts of ELENA, ROBY, and agent 413 are shown. The utterance text is the text feature of the utterance. The utterance destination determination unit 411 determines the utterance destination based on the relationship between the text feature of the utterance of the current speaker and the text feature of the past utterances of the other participants.

[0061] The destination decision unit 411 can generate the probability that each listener is a destination, for example, using a neural network-based architecture. Figure 9 shows an example of input and output to a neural network that the destination decision unit 411 can use to decide the destination.

[0062] The neural network 501 outputs the probability 521 that one listener is a possible destination. The destination determination unit 411 executes an inference calculation of the neural network for each listener to calculate the probability that each listener is a destination. The configuration of this neural network is not limited, but for example, a Transformer, a recurrent neural network, or a convolutional neural network can be used.

[0063] As shown in Fig. 9, the inputs of the neural network 501 are inputted as input feature 511 which is the speech information of the speaker, text feature (Listener Utterance) 512 of the listener's past speech, and listener's unique feature 513. In the configuration example shown in Fig. 9, the speech destination determination unit 411 determines the speech destination based on the relationship between the listener's unique feature and the text feature of the speaker's current speech, in addition to the listener's past speech history. The input feature 511 which is the speech information of the speaker may include text feature obtained from the text uttered by the speaker, voice feature obtained from the voice of the speaker, and speaker's unique feature.

[0064] Another embodiment of the present invention will be described below. Figures 13 to 16 show the configuration of a neural network when the present embodiment is realized by a recurrent neural network (RNN).

[0065] Figure 13 shows an RNN that identifies whether the speaker is speaking to a group or an individual. Generic features 1310 are composed of information that is useful for identifying the speaker's speaking destination. For example, information indicating who the speaker was speaking to in the previous utterance, information indicating whether the spoken text began with a conjunction, etc. can be used. This information is organized in the form of a vector and used as a generic feature.

[0066] The RNN 1320 receives the speech features of the speaker obtained from the speech feature extraction unit 405. Here, a Long short-term memory (LSTM) is used as the RNN cell.

[0067] The details of the processing in the LSTM 1320 are shown in Fig. 14A. Acoustic features 1410 of the speaker's voice at each time are input to cell 1430 of the LSTM's forward network and cell 1440 of the backward network. The vectors of the hidden layer values ​​of the final cell of the forward calculation and the final cell of the backward calculation are concatenated to obtain the final LSTM output vector 1450.

[0068] 13, a layer 1330 that performs a logistic regression calculation reduces the dimensionality of the output vector of the LSTM 1320. This logistic regression 1330 may be omitted.

[0069] The word embedding layer 1340 converts words contained in the features expressed in natural language into vectors. The output of the word embedding layer 1340 is input to the LSTM 1350. The features expressed in natural language include, first, the text spoken by the speaker. In addition, there are also the speaker's soft skills, hard skills, role, and name. These are in other words, the speaker's unique features. Note that features other than these may also be input as long as they are expressed in natural language.

[0070] The details of the part where words are input to the word embedding layer 1340 and the LSTM 1350 will be described with reference to Fig. 14B. Each word 1461 of the features expressed in natural language is input to the word embedding layer 1462 and converted into a fixed-length vector. The output of the word embedding layer 1462 is input to a cell 1463 of the forward network of the LSTM and a cell 1464 of the backward network. The vectors of the values ​​of the hidden layers of the final cell of the forward calculation and the final cell of the backward calculation are concatenated to obtain the final LSTM output vector 1465.

[0071] 13, layer 1360 constructs one multidimensional vector by concatenating the input vectors. The concatenated multidimensional vector is input to multilayer perceptron 1370, which outputs probability 1380 that the utterance destination is a group.

[0072] FIG. 15 shows an RNN that determines which listener is the destination when the destination is an individual. In this embodiment, a neural network is configured to calculate the probability that each listener is the destination. Neural networks 1570 and 1571 calculate the probability that the first and i-th listeners are the destination, respectively. Since the configuration and parameters of the networks inside neural networks 1570 and 1571 are the same, only neural network 1570 will be described here.

[0073] The word embedding layer 1510 converts each word contained in the features expressed in natural language into a finite-length vector. This word embedding layer receives as input the text spoken by the speaker, the text previously spoken by the listener, and the text representing the listener's unique features. By inputting the words contained in each text into the word embedding layer, a fixed-length vector is obtained for each word.

[0074] MaLSTM1520 is a neural network that uses LSTM to calculate the magnitude of the relationship between two texts based on the Manhattan distance. Each MaLSTM takes as input the text spoken by the speaker and one of the features that can be expressed in the text related to the listener.

[0075] The configuration of MaLSTM 1520 is shown in Fig. 16. MaLSTM accepts information 1610 expressed in text about the speaker and information 1620 expressed in text about the listener. The information expressed in text here may be the text of a speech or the text of the unique features of the listener and the speaker.

[0076] Speaker information is represented by a word string 1611. This is converted into a vector 1612 for each word by inputting it into a word embedding layer. The word vectors are then input into a forward network 1613 of an LSTM and a backward network 1614. Then, by concatenating the vectors of the hidden layer values ​​of the final cells of the forward network and the backward network, a vector 1615 of speaker information is obtained.

[0077] The listener information is processed in the same way as the speaker information. That is, a listener information vector 1625 is obtained by concatenating a word string 1621, a word vector 1622 which is a word embedding representation, a forward LSTM network 1623, a backward LSTM network 1624, and the value of the hidden layer of the final cell.

[0078] Using information about the speaker and information about the listener, the relationship between the two is calculated. Here, in layer 1630, the magnitude of the relationship is calculated based on the Manhattan distance using the following formula: Ds-l=exp(-||hs-hl||) Ds-l represents the magnitude of the relationship between speaker s's information 1615 and listener l's information 1625. The result 1640 is output.

[0079] Returning to Figure 15, there are multiple MaLSTMs 1520. All of them accept the speaker's utterance text, but the other input is different for each MaLSTM. This other input could be, for example, the text of the corresponding listener's past utterances or the text of the listener's unique features. Layer 1540 concatenates the outputs of these MaLSTMs 1520 to form a single vector.

[0080] Other features may be used, such as focus information 1530, which indicates whether the speaker was looking at the corresponding listener. This focus information may be detected from the speaker's face or eye images. The vector output by layer 1540 is input to multi-layer perceptron 1550, which outputs probability 1560 that the corresponding listener is the intended listener.

[0081] By the above neural network processing, it is possible to calculate the probability that the destination of an utterance is a group or not, and the probability that the destination of an utterance is each listener.

[0082] This makes it possible to more accurately estimate the listener to whom the utterance is directed. Note that some of the listener information shown in Fig. 9 may be omitted, for example, the listener's unique feature, and the speaker's voice feature and video feature may be omitted.

[0083] The input feature of the speaker may include, for example, a text feature, a voice feature, and a video feature currently uttered by the speaker. The input feature of the speaker may include a feature specific to the speaker. The text feature, the voice feature, and the video feature can be obtained from the utterance text field 473, the voice feature field 475, and the video feature field 476, respectively, in the record of the current utterance in the utterance history management information 423.

[0084] The text feature 512 of the past utterance of the listener can be acquired from the utterance text field 473 of the utterance history management information 423. A preset number of text feature values ​​of the past utterances of the listener, for example k, are selected. For example, text feature values ​​of a plurality of past utterances are selected. When the k past utterance texts are used, a special word ( <sep>The concatenation of word strings, including the word sequence, can be used as input. This allows for more accurate calculation of destination probability.

[0085] The listener specific feature 513 is obtained from the listener record of the participant specific feature management information 421. In one embodiment of the present specification, the listener specific feature 513 referenced for calculating the listener's utterance destination probability includes βsoft skill, βhard skill, βrole, and βname.

[0086] In one embodiment of this specification, βsoft skill, βhard skill, βrole, and βname are each terms of speaker features input to the neural networks 501, 1340, and 1510 for each item of listener-specific features, for example, input speech text features or speaker-specific features.

[0087] Furthermore, instead of using the word as it is, it is also possible to use the word that has the closest relationship to the word. Specifically, βsoft skill, βhard skill, βrole, and βname are the words that have the closest relationship to the input word in the listener's "soft skill," "hard skill," "role," and "name," respectively.

[0088] In one embodiment of the present specification, an eigenfeature that has a high semantic relationship with an input word for a speaker is selected. The semantic relationship can be expressed as a distance in a semantic space (semantic distance). Several methods for calculating the semantic distance are considered.

[0089] For example, to calculate semantic relationships using Manhattan distance, the formula is: Ds-t=exp(-||hs-ht||) Ds-t is the semantic distance (distance score) between speaker s (e.g., the word "machine" in the speaker's text feature) and listener (destination) t (e.g., the word "machine learning" in the listener's hard skill feature). The speaker and listener feature vectors are passed to, for example, an LSTM network to update the hidden states (context features) hs and ht, respectively. The output of the LSTM network indicates the value of the semantic relationship between the input features.

[0090] The output relevance can be shown diagrammatically as in Figure 10. For example, the word "machine" is <name>It is related to "AI" in <hard skills>, "machine learning" in <hard skills>, "teamwork" in <soft skills>, and "designer" in <roles>. Among the types of unique features of each participant, the words closest to the word "machine" are βsoft skill, βhard skill, βrole, and βname.

[0091] As described above, the destination determination unit 411 calculates the destination probability of each listener. In this way, the probability of each listener is calculated independently of each other, so that the probability of any number of listeners can be calculated and the destination can be determined. The destination determination unit 411 can determine the listener with the highest probability as the destination.

[0092] The destination determination unit 411 compares the probabilities of all listeners, and if a predetermined condition is satisfied, determines that the destination is a group (all listeners). For example, the destination determination unit 411 may determine that the destination is a group if the difference between the minimum and maximum destination probabilities is smaller than a preset threshold, or may determine that the destination is a group if the variance is smaller than a threshold.

[0093] A semantic relationship score between the features of an utterance by a speaker and the features of a listener can be calculated, and the destination of the utterance can be determined based on the score. The semantic relationship score can be expressed, for example, as a Manhattan distance. As mentioned above, in a conversation with three or more participants, an utterance may be directed to the group.

[0094] The destination determination unit 411 can use a single-task or multi-task neural network to estimate whether an utterance is “group-directed” or “individual-directed.” Figure 11 shows an example of a destination determination process using multi-tasks.

[0095] In Figure 11, we first calculate the speaker's input features (speech text and unique features), then calculate the features for each listener (past speech text and unique features). After that, we determine whether the speaker's utterance was directed to a group or an individual (first task), and if it was directed to an individual, we determine which listener the utterance was directed to (second task).

[0096] In step 601, features of the current speaker are calculated based on the spoken text and the unique features. In step 605, the features of the speaker are input into a neural network to calculate the probability that the speaker is speaking to a group or an individual. In step 606, if the probability that the speaker is speaking to a group is greater than 0.5, the process proceeds to step 610, where information that the speaker is speaking to a group is saved and the process ends.

[0097] If it is not a group, the process proceeds to step 611 and a loop is performed. Here, the number of the target listener is represented as i. In the process of step 604, the feature amount of the target i-th listener is calculated. Here, for example, the text previously spoken by listener i and the unique feature amount are used.

[0098] In the process of step 607, the destination determination unit 411 inputs the features of the speaker and the features of the listener i to the neural network trained for the second task, and outputs the probability that the speaker spoke to the listener i. This value is usually in the range of 0 to 1.

[0099] Steps 612 and 613 are processes that repeat a loop. That is, the probability that the speaker spoke to all listeners is calculated. Finally, in the process of step 608, the listener with the maximum probability output in step 607 is found, and that listener is saved as the speaker, and the process ends.

[0100] The destination determination unit 411 starts the determination of "group" or "individual" as the first task in the multitask neural network (605). The neural network estimates whether the destination is a group or an individual from the input feature amount of the vectorized speaker. For example, if the probability (group probability) output by the neural network is higher than 0.5 (606: YES), the destination belongs to a group.

[0101] If the group probability is less than or equal to 0.5 (606: NO), the destination is determined to be an individual. The destination determination unit 411 performs additional estimation using a neural network to obtain a probability score for each listener (607, 608). Figure 11 shows an example of the calculation of scores for only two listeners.

[0102] The vectorized speaker input features obtained for group / individual are used. The vectorized speaker input features are shared between the group / individual judgment task and the listener destination probability calculation task.

[0103] The destination determination unit 411 can calculate the probability based on the semantic relationship between the vectorized features of the speaker and the features of the listener. The semantic relationship can be calculated using, for example, the Manhattan distance, and is output as a value between 0 and 1.

[0104] The destination decision unit 411 utilizes the "shared information" of the two tasks. Unlike a single task, a multitask neural network can simultaneously optimize both a first task (a task for determining whether a response is group-oriented or individual-oriented) and one or more second tasks (calculating the probability score for each listener) that may be related to the first task.

[0105] Here, we will explain how to "share" two tasks using a concrete example. In one embodiment of this specification, a word embedding layer that converts words into word vectors is used in multiple places (1340 in FIG. 13, 1510 in FIG. 15). Although each of these is used for a different task, they can all be trained as a word embedding layer with common parameters. This is the first example of sharing.

[0106] The second shared example is about how to define a loss function used in training a neural network. In one embodiment of the present specification, the loss of a neural network is calculated by the following formula: Llot=λ1*LG+λ2*LID LG is the loss function for the group / individual judgment task, LID is the loss function for the listener probability score calculation task, and Llot is the overall loss function. That is, the neural network is trained while simultaneously considering the properties of both the group judgment and the individual judgment. λ1 and λ2 are weights selected to control the influence of each loss function in the neural network.

[0107] The destination determination unit 411 compares the "probability of being a destination" of each listener and regards the candidate with the highest probability as the true destination. The destination determination unit 411 determines the participant ID corresponding to that listener as the destination ID.

[0108] An example of a conversation managed by the conversation management system 101 will be described below. Figures 12A to 12H show an example of a series of utterances that constitute a conversation. The conversation management system 101 executes an agent 413 and controls the agent 413 according to the determined utterance destination. In the example of Figures 12A to 12H, three users 105A to 105C participate in the conversation together with the agent 413.

[0109] 12A, user 105A makes an utterance 651. Destination determination unit 411 determines the destination to be "group" and returns it to agent 413. This means that anyone, including agent 413, can respond. Agent 413 may wait a predetermined amount of time for the user to speak before starting to respond.

[0110] 12B, user 105B issues a response 652 to an utterance 651 made immediately before by user 105A. Utterance destination determination unit 411 determines that user 105A is the utterance destination, and returns the determination result to agent 413.

[0111] 12C, user 105A issues a response 653 to an utterance 652 from previous user 105B. Utterance destination determination section 411 determines the utterance destination to be a “group”, and returns this to agent 413.

[0112] 12D, user 105B responds to previous utterance 653 of user 105A with response 654. Destination decision unit 411 decides that the destination is an "agent" and returns this to agent 413. Agent 413 generates a response based on the content of the utterance and its own knowledge.

[0113] 12E, agent 413 issues response 655 to previous utterance 654 of user 105B. Utterance destination determination section 411 determines the utterance destination to be “group”, and returns this to agent 413.

[0114] 12F, user 105C utters response 656 in response to previous utterance 655 of agent 413. Utterance destination determination section 411 determines the utterance destination to be "group" and returns this to agent 413.

[0115] 12G, user 105A utters response 657 in response to utterance 656 from previous user 105C. Utterance destination determination section 411 determines the utterance destination to be “group”, and returns this to agent 413.

[0116] 12H, agent 413 issues response 658 to previous utterance 657 of user 105A. Since the previous utterance was to a group, agent 413 can respond. Utterance destination determination unit 411 determines that the utterance destination is the "group" and returns this to agent 413.

[0117] As described above, in a multi-user conversation, by estimating the destination of a conversation and controlling an agent based on the estimation result, it is possible to more appropriately support the progress of the multi-user conversation. Note that the determination result of the destination of a conversation according to the present disclosure can be applied to a purpose other than controlling an agent. For example, the destination of a conversation may be included in the conversation history, or information indicating the destination may be transmitted to the user terminal 104 of the destination of a conversation.

[0118] The present invention is not limited to the above-mentioned embodiment, but includes various modified examples. For example, the above-mentioned embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0119] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0120] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0121] 101 Conversation Management System 104 User terminal 105 users 401 Voice Recognition Unit 403 Participant Identification Department 405 Speech Feature Extraction Unit 407 Video Feature Extraction Unit 411 Speech destination determination unit 413 Agent 421 Participant-specific feature management information 423 Speech History Management Information< / name> < / sep>

Claims

1. 1. A conversation management system, comprising: one or more storage devices; one or more computing devices; The one or more storage devices store past speech history information of each of a plurality of participants in the conversation; The one or more computing devices, Obtaining text features of a first current utterance of a speaker; acquiring text features of past utterances of each of a plurality of listeners from the past utterance history information; Calculating a destination probability of each listener based on the text feature of the first current utterance and the text feature of the past utterance of each listener; determining a destination of the first current utterance based on destination probabilities of the plurality of listeners; and determining whether the destination of the utterance is the entire listener based on the text feature of the first current utterance.

2. A conversation management system, comprising: one or more storage devices; one or more computing devices; The one or more storage devices store past speech history information of each of a plurality of participants in the conversation; The one or more computing devices, Obtaining text features of a first current utterance of a speaker; acquiring text features of past utterances of each of a plurality of listeners from the past utterance history information; Calculating a destination probability of each listener based on the text feature of the first current utterance and the text feature of the past utterance of each listener; determining a destination of the first current utterance based on destination probabilities of the plurality of listeners; Calculating a group probability indicating a probability that the second current utterance is addressed to all listeners based on the text feature of the second current utterance; If the group probability exceeds a threshold, the conversation management system determines that the destination of the second current utterance is the entire listeners.

3. A conversation management system according to claim 1 or 2, the one or more storage devices store features associated with each of the plurality of participants; The one or more computing devices, obtaining, from the one or more storage devices, features associated with each of the plurality of listeners; The conversation management system calculates the destination probability for each listener based on the feature associated with each listener.

4. A conversation management system according to claim 1 or 2, The one or more computing devices calculate the destination probability of each listener based on text features of multiple past utterances of each listener.

5. A conversation management system as claimed in claim 2, The one or more computing devices, performing a first task and a second task subsequent to the first task with a neural network; a loss function of the neural network includes a loss in the first task and a loss in the second task; The first task calculates the group probability based on text features of the second current utterance; The second task calculates a destination probability for each listener based on the text features of the second current utterance used in the first task and the text features of the past utterances of each listener.

6. A conversation management system according to claim 1 or 2, The one or more computing devices, Extracting speech features of the first current utterance; The conversation management system calculates the destination probability of each listener based on the speech features.

7. A conversation management system according to claim 1 or 2, The one or more computing devices, Execute the programs of the agents who are participants in the conversation, A conversation management system that controls the agent based on the determined conversation destination.

8. A conversation management system as claimed in claim 1, The one or more computing devices, Execute the programs of the agents who are participants in the conversation, a conversation management system that permits the agent to respond to the first current utterance when it is determined that the destination of the utterance is all of the plurality of listeners or the agent.

9. A conversation management system according to claim 1 or 2, The one or more computing devices, A conversation management system that determines the speaker based on an utterance position determined based on audio of the first current utterance.

10. A method for a conversation management system to manage a conversation having multiple participants, comprising: The conversation management system stores past speech history information of each of a plurality of participants in a conversation; The method comprises: The conversation management system acquires text features of a first current utterance of a speaker; the conversation management system acquires text features of past utterances of each of a plurality of listeners from the past utterance history information; the conversation management system calculates a destination probability of each listener based on the text feature of the first current utterance and the text feature of the past utterance of each listener; The conversation management system determines a destination of the first current utterance based on destination probabilities of the plurality of listeners; The method includes the conversation management system determining whether the target of the utterance is an entire listener based on text features of the first current utterance.

11. A method for a conversation management system for managing a conversation having multiple participants, comprising: The conversation management system stores past speech history information of each of a plurality of participants in a conversation; The method comprises: The conversation management system acquires text features of a first current utterance of a speaker; the conversation management system acquires text features of past utterances of each of a plurality of listeners from the past utterance history information; the conversation management system calculates a destination probability of each listener based on the text feature of the first current utterance and the text feature of the past utterance of each listener; The conversation management system determines a destination of the first current utterance based on destination probabilities of the plurality of listeners; the conversation management system calculates a group probability indicating a probability that the second current utterance is addressed to all listeners based on the text feature of the second current utterance; the conversation management system determining that the second current utterance is addressed to an entire listener if the group probability exceeds a threshold.

Images