Estimation device, estimation method, and estimation program

Abstract

An estimation device according to the present invention divides input sound to be subjected to diarization into voice segments for each prescribed time period. Thereafter, the estimation device inputs the voice of each voice segment into a neural network (NN) and obtains: diarization information indicating the presence or absence of speech from speakers during each time period of each voice segment; and speaker embedding information for identifying to which speaker the speaker indicated by the diarization information corresponds. Subsequently, the estimation device clusters the speaker embedding information of each voice segment. In addition, the estimation device uses the clustering results to estimate that speakers for whom the speaker embedding information is closely related, from among speakers indicated by the diarization information of each voice segment, are the same speaker.

Description

[Technical field]

[0001] The present invention relates to an estimation device, an estimation method, and an estimation program. [Background technology]

[0002] Conventionally, there is a dialerization technique that estimates who spoke when from speech data of multiple people conversing. This dialerization technique includes a clustering-based technique that divides the input speech data into small segments, calculates the feature values ​​of each segment, and estimates that segments with similar features are from the same speaker.

[0003] There is also an NN-based method that trains a neural network (NN) that outputs whether or not each speaker spoke at each time for the entire input voice data (input sound), and then uses the trained NN to estimate who spoke and when. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] G. Sell, et al. “Diarization is Hard: Some Experiences and Lessons Learned for the JHU Team in the Inaugural DIHARD Challenge”, Proc. Interspeech, pp. 2808-2812, 2018. [Non-Patent Document 2] Y. Fujita, et al. “END-TO-END NEURAL SPEAKER DIARIZATION WITH SELF-ATTENTION”, arXiv, 1909.06247v1, 2019. Summary of the Invention [Problem to be solved by the invention]

[0005] The above clustering-based method has a problem that it cannot be applied to input sounds with overlapping speech. In addition, when the input sound is divided into segments, the NN-based method cannot match the speakers between the divided segments. Therefore, due to memory limitations, the NN-based method has a problem that it cannot be applied to long input sounds or input sounds from multiple speakers.

[0006] Therefore, an object of the present invention is to solve the above-mentioned problems and perform dialerization even for input sounds of a long duration and from multiple speakers, in which speech is overlapped. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the present invention is characterized in that it comprises a division unit that divides an input sound to be diarized into speech intervals of a predetermined time interval, an output processing unit that inputs the speech divided into the predetermined time intervals into a model that receives as input the speech divided into the divided speech intervals, and outputs diarization information indicating the presence or absence of speech by each speaker speaking within the divided speech intervals at each time within the speech interval, and speaker features which are features for identifying which speaker corresponds to which speaker indicated in the diarization information, and inputs the speech of each speech interval of the input sound to be diarized, and obtains output results of the diarization information and the speaker features for each speech interval from the model, and an estimation unit that estimates that speakers indicated in the diarization information for each speech interval who have similar values ​​of the speaker features are the same speaker. Effect of the Invention

[0008] According to the present invention, dialerization can be performed even for input sounds of a long duration and from many speakers, in which utterances are overlapped. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining an overview of diarization. [Diagram 2] FIG. 2 is a diagram for explaining an outline of diarization by the estimation device of this embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of an estimation device according to the present embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of a processing procedure of the estimation device of this embodiment. [Diagram 5] FIG. 5 is a diagram showing experimental results of diarization performed by the estimation device of this embodiment. [Figure 6] FIG. 6 is a diagram showing the experimental results of diarization performed by the estimation device of this embodiment. [Figure 7] FIG. 7 is a diagram showing experimental results of diarization performed by the estimation device of this embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of the configuration of a computer that executes an estimation program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, a mode (embodiment) for carrying out the present invention will be described with reference to the drawings. The present invention is not limited to the embodiment described below.

[0011] [overview] The estimation device of this embodiment performs diarization on input speech. For example, as shown in FIG. 1, the estimation device estimates when and which speaker spoke based on conversation data by multiple speakers (e.g., speaker 1, 2, 3, . . . ).

[0012] Here, the estimation device of this embodiment performs diarization using both End-to-End Neural Diarization (EEND) and clustering.

[0013] For example, the estimation device divides the input speech into multiple speech segments (e.g., X1, X2, ...) as shown in Fig. 2. Then, the estimation device inputs the speech of each speech segment to the NN of EEND, and obtains the diarization result of each speech segment and speaker embedding information corresponding to the speaker.

[0014] Then, the estimation device clusters the speaker embedding information of each of the above speech segments, and matches the speaker indicated in the diarization result of each speech segment based on the clustering result.

[0015] For example, the estimation device clusters the speaker embedding information of each speech section as shown by reference numeral 201. Then, based on the result of this clustering, the estimation device estimates that a speaker corresponding to speaker embedding information e1,1 and a speaker corresponding to speaker embedding information e2,2, which belong to the same cluster, are the same speaker (e.g., Speaker A in FIG. 2). In other words, the estimation device estimates that speaker 1,1 in speech section X1 and speaker 2,2 in speech section X2 are the same speaker (Speaker A).

[0016] In this way, the estimation device matches the speaker of each speech segment using the clustering results of the speaker embedding information of each speech segment, so that it is possible to divide the input speech into multiple speech segments and perform diarization by applying EEND to each speech segment.

[0017] As a result, the estimation device does not require a large memory even if the input speech is a long speech. Also, there is no need to design a special model for long speech. Furthermore, the estimation device can perform diarization even in a situation where multiple speakers speak simultaneously in each speech section. Furthermore, the estimation device does not need to construct a model corresponding to the speech of multiple speakers even if the entire input speech is the speech of multiple speakers.

[0018] [Configuration example] Next, a configuration example of the estimation device will be described with reference to Fig. 3. The estimation device 10 includes, for example, an input / output unit 11, a storage unit 12, and a control unit 13, as shown in Fig. 3.

[0019] The input / output unit 11 inputs and outputs various data. For example, the input / output unit 11 receives input of speech to be diarized and outputs the results of diarization of the input speech.

[0020] The storage unit 12 stores data referred to when the control unit 13 executes various processes and results of processes performed by the control unit 13. For example, the storage unit 12 stores a model (model parameters) learned by the control unit 13.

[0021] This model takes speech divided into predetermined time segments as input, and outputs diarization information indicating whether or not each speaker speaks at each time point in the divided speech, and speaker features, which are features for identifying which speaker corresponds to which speaker is indicated in the diarization information.

[0022] The control unit 13 is responsible for controlling the entire estimation device 10, and includes a learning unit 131, a division unit 132, an output processing unit 133, a clustering unit 136, an estimation unit 137, and an estimation result output unit 138.

[0023] The learning unit 131 receives a speech segmented at a predetermined time interval as an input, and learns a model that outputs diarization information indicating an estimation result of whether each speaker who is speaking in the segmented speech interval speaks at each time in the segmented speech interval, and a speaker feature (e.g., speaker embedding information) for identifying which speaker corresponds to which speaker indicated in the diarization information. This model is realized, for example, by a neural network.

[0024] For example, the learning unit 131 uses the correct answer data of the diarization to learn the NN so as to minimize the loss shown in the following formula (1).

[0025]

number

[0026] L in Equation (1) diarization is the loss associated with the estimated diarization result, and L speaker is the loss related to the estimation result of the speaker embedding information.

[0027] For example, L diarization uses the value of the binary cross entropy between the diarization estimation result and the correct answer data. In this case, the learning unit 131 trains the NN so that the value calculated by the formula (2) is as small as possible. Note that S Local is the maximum number of active speakers in each speech interval.

[0028]

number

[0029] Also, L speaker For example, the distance between the estimated speaker embedding information of a certain speaker and an element corresponding to the certain speaker in the dictionary of speaker embedding information optimized by learning, and the distance between the estimated speaker embedding information of the certain speaker and an element corresponding to a speaker other than the certain speaker in the dictionary of speaker embedding information are used. In this case, the learning unit 131 calculates L based on the formula (3). speaker The NN is trained to make as small as possible.

[0030]

number

[0031] That is, the training unit 131 receives, for example, estimated speaker embedding information for a certain speaker.

number

number

number

[0032] The division unit 132 divides the input sound to be diarized into voice segments each having a predetermined time (for example, T).

[0033] The output processing unit 133 inputs the speech for each speech section of the input sound to be diarized to the model trained by the training unit 131, and obtains from the model the diarization information for each speech section and speaker features for identifying which speaker corresponds to which speaker is indicated in the diarization information.

[0034] The output processing unit 133 includes a diarization processing unit 134 and a feature output unit 135. The diarization processing unit 134 obtains diarization information for each voice section of the input sound to be diarized from the above model. The feature output unit 135 obtains speaker features (e.g., speaker embedding information) from the above model for identifying which speaker corresponds to which speaker is indicated in the diarization information.

[0035] The clustering unit 136 clusters the speaker features of each speech section obtained by the feature output unit 135 based on the values ​​of the speaker features. For example, when the total number of speakers of the input sound is known in advance, the clustering unit 136 clusters the speaker embedding information of each speech section obtained by the feature output unit 135 based on the total number of speakers of the input sound (see reference numeral 201 in FIG. 2).

[0036] Here, the clustering unit 136 may impose a constraint that speaker features obtained from the same speech section are not clustered into the same cluster among the speaker features of each speech section. By imposing such a constraint, the clustering unit 136 can cluster the speaker features of each speech section for each speaker with high accuracy.

[0037] The estimation unit 137 aligns the speakers indicated in the diarization information of each speech section by using the result of clustering of the speaker features by the clustering unit 136. That is, the estimation unit 137 associates the speakers indicated in the diarization information of each speech section with each other.

[0038] Here, for example, the estimation unit 137 estimates that, among the speakers indicated in the diarization information of each speech section, speakers who belong to the same cluster and have the same speaker feature are the same speaker. In other words, the estimation unit 137 estimates that, among the speakers indicated in the diarization information of each speech section, speakers who have similar values ​​of the speaker feature are the same speaker.

[0039] The estimation result output unit 138 outputs the result of diarization of the entire input sound based on the processing result by the estimation unit 137.

[0040] [Example of processing procedure] Next, an example of the processing procedure of the estimation device 10 will be described with reference to FIG. 4. It is assumed that the estimation device 10 has already completed model (e.g., NN) learning. In addition, each voice section of the input sound to be diarized includes a maximum of S Local Assume there are active speakers.

[0041] First, the division unit 132 of the estimation device 10 divides the input sound into speech segments of length T (S1). Then, the output processing unit 133 divides each speech segment into S LocalThe diarization result (diarization information) for the speakers and the embedded information (speaker embedded information) corresponding to the speakers are output (S2). After that, the clustering unit 136 clusters the embedded information corresponding to the speakers (S3). For example, when the total number of speakers of the input sound is known in advance, the clustering unit 136 clusters the embedded information corresponding to the speakers into clusters of the total number of speakers.

[0042] Then, the estimation unit 137 aligns the diarization results of each speech section based on the clustering result of S3 (S4). For example, the estimation unit 137 estimates that speakers who belong to the same cluster in the clustering of S3 among the speakers indicated in the diarization information of each speech section are the same speaker. Then, the estimation result output unit 138 outputs the diarization result of the entire input sound based on the processing result of S4 (S5).

[0043] According to the estimation device 10, it is possible to perform dialerization even for input sounds having a long duration and from multiple speakers, in which utterances are overlapped.

[0044] [Experimental Results] Next, experimental results of the estimation device 10 will be described with reference to Fig. 5 to Fig. 7. Note that the EEND-vector clust. in each figure is the diarization by the estimation device 10.

[0045] FIG. 5 shows a comparison result between the diarization by the estimation device 10 and the diarization by the conventional EEND (1.EEND and 2.EEND). The difference between 1.EEND and 2.EEND is whether chunking (dividing the test data at a predetermined time interval) was performed or not. The test data was conversation data of two speakers. The duration of the test data was 3 minutes, 5 minutes, 10 minutes, and 20 minutes.

[0046] As shown in FIG. 5, it was confirmed that the diarization by the estimation device 10 resulted in a lower DER than the conventional EEND for all of the test data for 3 minutes, 5 minutes, 10 minutes, and 20 minutes.

[0047] FIG. 6 shows the results of a comparison between the diarization by the estimation device 10 and the conventional diarization by EEND (x-vector clustering and EDA-EEND) for test data with a known number of speakers.

[0048] The test data includes conversation data of 2, 3, 4, 5, and 6 people, and all conversation data of people 2 to 6. The division times T of the test data by the estimation device 10 were 15 seconds, 20 seconds, 30 seconds, and 50 seconds.

[0049] As shown in FIG. 6, it was confirmed that the diarization by the estimation device 10 achieved a lower DER than the conventional x-vector clustering and EDA-EEND for all test data with two, three, four, five, and six speakers.

[0050] FIG. 7 shows a comparison result between the diarization by the estimation device 10 and the conventional diarization by EEND (x-vector clustering and EDA-EEND) for test data with an unknown number of speakers.

[0051] 6, the test data includes conversation data of 2, 3, 4, 5, and 6 people, and all conversation data of people 2 to 6. The division times T of the test data by the estimation device 10 were 15 seconds, 20 seconds, 30 seconds, and 50 seconds.

[0052] As shown in Fig. 7, it was confirmed that the diarization by the estimation device 10 resulted in a lower DER than the conventional x-vector clustering and EDA-EEND for all test data with 2, 3, 4, 5, and 6 speakers. In other words, it was confirmed that the estimation device 10 can perform diarization with high accuracy even for test data with an unknown number of speakers.

[0053] [System configuration, etc.] In addition, each component of each unit shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in part by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0054] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0055] [program] The above-mentioned estimation device 10 can be implemented by installing a program (estimation program) as package software or online software on a desired computer. For example, by executing the above-mentioned program on an information processing device, the information processing device can function as the estimation device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), as well as terminals such as PDAs (Personal Digital Assistants).

[0056] 8 is a diagram showing an example of a computer that executes an estimation program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0057] The memory 1010 includes a read only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0058] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process executed by the above-mentioned estimation device 10 is implemented as a program module 1093 in which a computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing a process similar to the functional configuration of the estimation device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0059] Data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0060] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0061] 10 Estimation device 11 Input / output section 12 Storage section 13 Control section 131 Learning Department 132 Division 133 Output Processing Section 134 Dialysis Processing Unit 135 Feature Output Unit 136 Clustering Department 137 Estimation Department 138 Estimation result output part

Claims

1. A division unit that divides an input sound to be diarized into speech segments of a predetermined time interval; an output processing unit that inputs a voice divided at predetermined time intervals into a model that outputs diarization information indicating whether or not each speaker who is speaking within the divided voice intervals speaks at each time interval of the voice interval, and speaker features that are features for identifying which speaker corresponds to which speaker indicated in the diarization information, and inputs the voice of each voice interval of the input sound to be diarized, and obtains an output result of the diarization information and the speaker features for each voice interval from the model; an estimation unit that estimates, among the speakers indicated in the diarization information for each speech section, speakers having similar values ​​of the speaker feature amount as being the same speaker; An estimation device comprising:

2. A clustering unit that clusters speaker features of each speech segment based on values ​​of the speaker features, The estimation unit is As a result of clustering the speaker features of each speech section, speakers whose speaker features belong to the same cluster are estimated to be the same speaker. The estimation device according to claim 1 .

3. The clustering unit includes: The clustering is performed under a constraint that speaker features obtained from the same speech section are not clustered in the same cluster among the speaker features of each speech section.

3. The estimation device according to claim 2 .

4. A learning unit that learns the model based on the correct answer data of the diarization. The estimation device according to claim 1 , further comprising:

5. An estimation method performed by an estimation device, comprising: A step of dividing an input sound to be subjected to diarization into speech segments each having a predetermined time period; a step of inputting a voice divided at predetermined time intervals into a model that outputs diarization information indicating whether or not each speaker who is speaking within the divided voice intervals speaks at each time interval of the voice interval, and speaker features that are features for identifying which speaker corresponds to which speaker indicated in the diarization information, inputting the voice of each voice interval of the input sound to be diarized, and obtaining an output result of the diarization information and the speaker features for each voice interval from the model; a step of estimating that speakers having similar values ​​of the speaker feature among the speakers indicated in the diarization information for each speech section are the same speaker; The estimation method according to claim 1,

6. An estimation program for causing a computer to function as the estimation device according to any one of claims 1 to 4.

Images