Estimation device, estimation method, and estimation program

Abstract

To enable a plurality of attributes to be estimated concurrently with a single estimator for utterances issued between speakers.SOLUTION: An estimation device 10 includes: a first conversion unit 11 that, using a CNN 111, converts mel spectrograms corresponding to respective utterances of a plurality of speakers into a voice feature vector indicating a co-occurrence relation between voice features of the respective speakers; a second conversion unit 12 that, using a BERT model 121, converts data on written text of contents of utterances of the speakers into a distributed expression vector; a connection unit 13 that connects the voice feature vector to the distributed expression vector and merges the connected vectors; a discourse action estimation unit 14 that, based on an output from the connection unit 13, estimates a class of a discourse action of the utterances of the speakers, and outputs an estimation result; and an important statement estimation unit 15 that, based on the output from the connection unit 13, estimates whether or not the utterances of the speakers are important statements, and outputs an estimation result.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an estimation device, an estimation method, and an estimation program.

Background Art

[0002] Conventionally, there is a technique for estimating whether a statement made in a group meeting is important in the meeting with respect to the attributes of the statement (Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technique described in Non-Patent Document 1, the attributes that can be estimated are limited to only one type of whether the statement is important in the meeting. In order to estimate a plurality of attributes, it is necessary to create an estimator for each attribute for estimating each attribute. That is, in order to estimate a plurality of attributes, it is necessary to create a plurality of estimators for each attribute.

[0005] ​The present invention has been made in consideration of the above, and aims to provide an estimation device, an estimation method, and an estimation program that can simultaneously estimate multiple attributes using a single estimator for utterances generated between multiple speakers. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the estimation device of the present invention is characterized by having: a first conversion unit that uses a convolutional neural network to convert mel spectrograms corresponding to each utterance of a plurality of speakers into speech feature vectors that indicate co-occurrence relationships of speech features of each speaker; a second conversion unit that uses a language representation model to convert data obtained by transcribing the content of the speakers' utterances into text into distributed representation vectors; a concatenation unit that concatenates the speech feature vectors and the distributed representation vectors and fuses the concatenated vectors; a first estimation unit that estimates the discourse act class of the speaker's utterance based on the output from the concatenation unit and outputs the estimation result; and a second estimation unit that estimates whether the speaker's utterance is an important statement based on the output from the concatenation unit and outputs the estimation result. [Effects of the Invention]

[0007] According to the present invention, a single estimator can simultaneously estimate a plurality of attributes for an utterance generated between a plurality of speakers. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram schematically illustrating an example of the configuration of an estimation device according to an embodiment. [Figure 2] Figure 2 shows a diagram of discourse act classes. [Figure 3] FIG. 3 is a flowchart illustrating a processing procedure of the estimation process according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a computer that implements the estimation device by executing a program. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.

[0010] [Embodiment Mode] In this embodiment, we will describe an estimation device that uses a single estimator to simultaneously estimate multiple attributes of utterances made between multiple speakers. The estimation device according to the embodiment estimates multiple types of attributes of utterances made in a group conference, and independently estimates two attributes: the discourse act class of each speaker's utterance, and whether the speaker's utterance is an important utterance.

[0011] [Estimation device] Next, an estimation device according to an embodiment will be described. Fig. 1 is a diagram schematically illustrating an example of the configuration of an estimation device according to an embodiment.

[0012] The estimation device 10 according to the embodiment is realized by loading a predetermined program into a computer or the like including a ROM (Read Only Memory), a RAM (Random Access Memory), a CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The estimation device 10 also has a communication interface for transmitting and receiving various information to and from other devices connected via a network, etc.

[0013] The estimation device 10 shown in FIG. 1 receives as input mel spectrograms corresponding to each utterance of multiple speakers (all conversation participants) and text transcription data of the utterances of the conversation participants. The estimation device 10 uses deep learning technology as an estimator and simultaneously outputs two attributes: the discourse act class of the utterance of the utterance and whether the utterance of the utterance is important or not. The mel spectrogram indicates the intensity of each frequency component with respect to the time component, for example, by color. As shown in FIG. 1, the estimation device 10 includes a first conversion unit 11, a second conversion unit 12, a connection unit 13, a discourse act estimation unit 14 (first estimation unit), and an important utterance estimation unit 15 (second estimation unit).

[0014] The first conversion unit 11 has a convolutional neural network (CNN) 111. In the embodiment, the CNN 111 is employed for the purpose of capturing co-occurrence relationships between speech features of the speech of multiple speakers by using information on temporal and frequency neighbors of the speech information of multiple speakers.

[0015] The first conversion unit 11 receives input of mel spectrograms corresponding to each of the utterances of multiple speakers. The first conversion unit 11 receives input of each mel spectrogram of one voiced section of one speaker. The first conversion unit 11 then uses a CNN 111 to convert the mel spectrograms corresponding to each of the utterances of the multiple speakers into speech feature vectors indicating the co-occurrence relationships of the speech features of each speaker. In the example of FIG. 1, the mel spectrograms of four speakers are received as input, and a 200-dimensional vector is obtained using a CNN 122. The mel spectrograms of the four speakers are created by performing speech analysis on the speech information of the four speakers.

[0016] The second conversion unit 12 has a language representation model. For example, the second conversion unit 12 has, as the language representation model, a Bidirectional Encoder Representations from Transformers (BERT) model 121. The BERT model 121 receives an utterance transcription as input and outputs a distributed representation vector.

[0017] The second conversion unit 12 receives input of data obtained by transcribing the content of a speaker's utterance into text. The second conversion unit 12 receives input of data obtained by transcribing the content of a single voiced section of a single speaker into text. The second conversion unit 12 converts the data obtained by transcribing the content of a speaker's utterance into text into a distributed representation vector using a BERT model 121. The second conversion unit 12 applies the transcription of the utterance to the BERT model 121 at the same time as vectorizing the speech information in the first conversion unit 11, and obtains a 768-dimensional vector.

[0018] The connection unit 13 includes a fully connected block 131. The fully connected block 131 is a fully connected neural network. The fully connected block 131 uses, for example, ReLU as an activation function.

[0019] The concatenation unit 13 uses a fully connected block 131 to concatenate the speech feature vector obtained from the first conversion unit 11 and the distributed representation vector output from the second conversion unit 12, and fuse the two vectors. The fully connected block 131 is used to optimize the class score of the discourse act estimation block (described later) and the class score of the important utterance estimation block (described later). The fully connected block 131 concatenates the speech feature vector and the distributed representation vector to create a 968-dimensional vector, which is then passed through a fully connected neural network FC to create a 100-dimensional vector (fusion vector).

[0020] The discourse act estimation unit 14 has a discourse act estimation block 141. The discourse act estimation block 141 is a fully connected neural network, and uses, for example, softmax as an activation function. Based on the fusion vector output from the connection unit 13, the discourse act estimation block 141 estimates the discourse act to which the target utterance belongs as the likelihood of n types (e.g., 11 types) of class labels.

[0021] Fig. 2 is a diagram showing classes of discourse acts. As shown in Fig. 2, discourse act classes include, for example, information provision, information request, suggestion, and offer. Information provision indicates that the utterance to be processed is an utterance in which the speaker intends to provide information. Information request indicates that the utterance to be processed is an utterance in which the speaker intends to request information from the interlocutor.

[0022] The discourse act estimation unit 14 estimates the discourse act class of the speaker's utterance using the discourse act estimation block 141 based on the output from the connection unit 13, and outputs the estimation result. The first estimation unit estimates the discourse act class to which the utterance of one voiced section of one speaker belongs.

[0023] The important utterance estimation unit 15 has an important utterance estimation block 151. The important utterance estimation block 151 is a fully connected neural network, and uses, for example, softmax as an activation function. Based on the fusion vector output from the connection unit 13, the important utterance estimation block 151 estimates whether the utterance to be processed is an important utterance as the likelihood of two types of class labels.

[0024] The important utterance estimation unit 15 estimates whether the speaker's utterance is an important utterance or not, based on the output from the connection unit 13, using the important utterance estimation block 151, and outputs the estimation result. The important utterance estimation unit 15 estimates whether the utterance in one voiced section is an important utterance or not.

[0025] In this way, the estimation device 10 receives as input, as audio information, the mel spectrograms of each of the four conversation participants in the utterance section, and as linguistic information, the transcription of the utterance by the generator (speaker) of the utterance in the utterance section, and outputs likelihoods of n types of class labels of discourse acts and likelihoods of two types of class labels of whether the utterance is important or not.

[0026] Note that the important utterance estimation unit 15 is not limited to outputting two types of class labels, i.e., whether a utterance is important or not, but may output likelihoods of three or more types of class labels. Furthermore, the estimation device 10 trains the CNN 111, the fully connected block 131, the discourse act estimation block 141, and the important utterance estimation block 151 end-to-end using, for example, a publicly available dataset as training and learning. Therefore, the estimation device 10 is multitasking and does not require training for each task. Therefore, compared to training for each task, it does not require a large training cost and the process up to application to actual estimation is simple.

[0027] [Estimation processing procedure] Next, the estimation process according to the embodiment will be described with reference to a flowchart of FIG.

[0028] As shown in FIG. 3, the estimation device 10 receives, as input, mel spectrograms corresponding to the utterances of a plurality of speakers, and data obtained by transcribing the contents of the utterances of the speakers into text (steps S1 and S3).

[0029] In the estimation device 10, the first conversion unit 11 performs a first conversion process to convert mel spectrograms corresponding to the respective utterances of multiple speakers into speech feature vectors indicating co-occurrence relationships of speech features of each speaker using the CNN 111 (step S2). Meanwhile, in the estimation device 10, the second conversion unit 12 converts data obtained by transcribing the content of the utterances of the speakers into text into distributed representation vectors using the BERT model 121 (step S4).

[0030] The concatenation unit 13 uses the fully connected block 131 to concatenate the obtained speech feature vector output from the first conversion unit 11 and the distributed representation vector output from the second conversion unit 12, and performs a concatenation process to fuse the two vectors (step S5).

[0031] Then, the discourse act estimation unit 14 performs discourse act estimation processing to estimate the discourse act class of the speaker's utterance using the discourse act estimation block 141 based on the output from the connection unit 13 (step S6), and outputs the discourse act estimation result (step S7).

[0032] On the other hand, based on the output from the connection unit 13, the important utterance estimation unit 15 performs an important utterance estimation process using the important utterance estimation block 151 to estimate whether the speaker's utterance is an important utterance or not (step S8), and outputs the important utterance estimation result (step S9).

[0033] [Effects of the embodiment] As described above, the estimation device 10 has a configuration in which, before the discourse act estimation unit 14 and the important utterance estimation unit 15, there are provided a CNN 111 that converts speech information, a BERT model 121 that processes transcription data of utterances, and a concatenation unit 13 that concatenates the speech feature vector output from the CNN 111 and the distributed representation vector output from the BERT model 121 and fuses the two vectors. This allows the estimation device 10 to simultaneously estimate multiple attributes, such as the discourse act class of an utterance and whether or not the utterance is an important utterance, for utterances generated between multiple speakers, using only one estimation device 10.

[0034] Furthermore, in the embodiment, multiple estimators for estimating multiple attributes are not created, but multiple attributes are simultaneously estimated using only one estimation device 10. Therefore, according to the embodiment, it can be expected that the different characteristics between the attributes captured by the estimation device 10 will complementarily improve the estimation performance of each attribute, rather than estimating each attribute independently.

[0035] The input to the estimation device 10 is not limited to speech information and linguistic information. The estimation device can also perform more detailed estimation using facial expression information related to the facial expressions of interlocutors (for example, an image capturing the speaker's facial features) and gaze information indicating the direction of the speaker's gaze, in addition to speech information and linguistic information. In this case, the estimation device arranges a neural network (NN) that processes facial expression information and a neural network (NN) that processes gaze information in parallel with the CNN 111 and the BERT model 121. The estimation device connects and fuses the conversion information of facial expression information, the conversion information of gaze information, the speech feature vector obtained by the CNN 111, and the distributed representation vector obtained by the BERT model 121, and inputs the fusion vector to the discourse act estimation block and the important utterance estimation block to perform each estimation.

[0036] [System configuration of the embodiment] Each component of the estimation device 10 is a functional concept and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of the functions of the estimation device 10 is not limited to that shown in the figure, and all or part of the functions can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0037] Furthermore, all or any part of the processes performed in the estimation device 10 may be realized by a CPU, a GPU (Graphics Processing Unit), and a program analyzed and executed by the CPU and the GPU. Furthermore, each process performed in the estimation device 10 may be realized as hardware using wired logic.

[0038] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated can be changed as appropriate unless otherwise specified.

[0039] [program] 4 is a diagram showing an example of a computer in which the estimation device 10 is realized by executing a program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0040] The memory 1010 includes a ROM 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0041] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the estimating device 10 is implemented as a program module 1093 in which code executable by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the estimating device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0042] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary and executes them.

[0043] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0044] Although the present invention has been described above as an embodiment, the present invention is not limited to the descriptions and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention. [Explanation of symbols]

[0045] 10 Estimation device 11 First conversion section 12 Second conversion section 13 Connecting part 14 Discourse act estimation part 15 Important Statement Estimation Department 111 CNN 121 BERT model 131 fully connected blocks 141 Discourse Act Inference Block 151 Important statement estimation block

Claims

1. a first conversion unit that converts, using a convolutional neural network, mel spectrograms corresponding to each of the utterances of a plurality of speakers into speech feature vectors that indicate co-occurrence relationships of speech features of each speaker; a second conversion unit that converts data obtained by transcribing the content of the speaker's utterance into text using a language representation model into a distributed representation vector; a concatenation unit that concatenates the speech feature vector and the distributed representation vector and fuses the concatenated vectors; a first estimation unit that estimates a discourse act class of the utterance of the speaker based on an output from the concatenation unit and outputs an estimation result; a second estimation unit that estimates whether the utterance of the speaker is an important utterance based on an output from the connection unit, in parallel with the estimation process in the first estimation unit, and outputs an estimation result; and An estimation device characterized by simultaneously estimating and outputting a plurality of attributes, including a discourse act class of the speaker's utterance and an estimation result of whether the speaker's utterance is an important utterance.

2. the second conversion unit receives as input data obtained by transcribing the content of an utterance in one voiced section of one speaker into text; the first transform unit receives as input each mel spectrogram of the one voiced section, 2. The estimation device according to claim 1, wherein the first estimation unit estimates a discourse act class to which the utterance in the one voiced section belongs, and the second estimation unit estimates whether the utterance in the one voiced section is an important utterance.

3. 3. The estimation device according to claim 1, wherein the language representation model is a Bidirectional Encoder Representations from Transformers (BERT) model.

4. An estimation method executed by an estimation device, comprising: a first conversion step of converting, using a convolutional neural network, mel spectrograms corresponding to each of the utterances of a plurality of speakers into speech feature vectors indicating co-occurrence relationships of speech features of each speaker; a second conversion step of converting data obtained by transcribing the content of the speaker's utterance into text using a language representation model into a distributed representation vector; a concatenation step of concatenating the speech feature vector and the distributed representation vector and fusing the concatenated vectors; a first estimation step of estimating a discourse act class of the utterance of the speaker based on the output of the concatenation step and outputting the estimation result; a second estimation step of estimating whether the utterance of the speaker is an important utterance or not based on the output of the connection step, in parallel with the estimation in the first estimation step, and outputting an estimation result; Including, The estimation method is characterized by simultaneously estimating and outputting a plurality of attributes, namely, the discourse act class of the speaker's utterance and an estimation result of whether the speaker's utterance is an important utterance or not.

5. An estimation program for causing a computer to function as the estimation device according to any one of claims 1 to 3.

Images