Speech classification device and method

Abstract

To provide a speech classification device capable of accurately classifying the number of speakers participating in a meeting regardless of the presence / absence of pre-input information about the number of participants in the meeting, and speech classification method.SOLUTION: A speaker classification system 50 includes: a voice input unit 251 that inputs a voice file containing speeches made by multiple speakers; a feature quantity extraction unit 252 that extracts the feature quantity of respective speeches within multiple speech segments in the voice file; and a voice clustering unit 253 that divides respective speeches into groups per speaker based on the feature quantity of the respective speeches. The voice clustering unit 253 is configured so as to, when predefined conditions are met, perform group composition to combine the multiple groups as a group of one speaker.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an utterance classification device and a method for utterance classification. [Background technology]

[0002] Patent Document 1 discloses a minutes output device that outputs minutes in which speakers in a meeting are identified with high accuracy. This minutes output device acquires information on the number of participants in a meeting and data on the voice in the meeting, recognizes voice based on the data on the voice, and converts it into text as the speaker's remarks. The minutes output device identifies the speaker based on the information on the number of participants and the data on the voice, and outputs minutes in which a label indicating the speaker is associated with the content of the converted remarks. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-95210 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the configuration of Patent Document 1, information on the number of participants in a meeting is given as prerequisite information for identifying speakers. For this reason, if information on the number of speakers in a meeting is not given, it is assumed that the minutes output device of Patent Document 1 has difficulty in estimating how many people are speaking during the meeting. For example, when there is audio data for which minutes of a meeting are to be created, the longer the time has passed since the audio data was recorded, the more fuzzy the memory of the number of speakers becomes, and moreover, it is often difficult to accurately grasp the number of speakers among the participants who actually spoke. For this reason, it can be said that there is room for improvement in automatic classification without inputting the number of participants in a meeting in advance.

[0005] The present disclosure has been devised in consideration of the above-described conventional situation, and aims to provide an utterance classification device and an utterance classification method that classify utterances with high accuracy while estimating the number of speakers participating in a conference, regardless of whether information regarding the number of speakers in the conference is input in advance. [Means for solving the problem]

[0006] The present disclosure provides a method and apparatus for processing speech data including a plurality of speeches, the method comprising: an input unit for inputting a speech file including speeches of a plurality of speakers; a feature extraction unit for extracting features of each speech within a plurality of speech sections included in the speech file; and a classification unit for classifying the speeches into groups for each speaker based on the features of each speech, the classification unit performing group synthesis for synthesizing the plurality of groups into a group for one speaker when a predetermined condition is satisfied. the classification unit derives a similarity between a plurality of groups having voice data of one or more utterances for each speaker and the voice data of the plurality of speech sections included in the voice file, derives a correlation coefficient between the groups using the derivation result of the similarity, and determines that the predetermined condition is met when the correlation coefficient is equal to or greater than a predetermined threshold, and synthesizes two or more groups whose correlation coefficient is equal to or greater than the predetermined threshold into a group of one speaker. , an utterance classification device is provided.

[0007] The present disclosure also provides 1. A method for speech classification executed by a speech classification device that is a computer, comprising: The method includes the steps of: inputting a voice file including speech from a plurality of speakers; extracting features of each speech in a plurality of speech sections included in the voice file; classifying the speech into groups by speaker based on the features of each speech; and performing group synthesis for synthesizing the grouped groups into a single speaker group when a predetermined condition is satisfied. and, as the group synthesis, deriving a similarity between a plurality of groups having voice data of one or more utterances for each of the speakers and the voice data of the plurality of speech sections included in the voice file, deriving a correlation coefficient between the groups using the derived result of the similarity, and synthesizing two or more groups whose correlation coefficient is equal to or greater than the predetermined threshold as a group of one speaker, assuming that the predetermined condition is met when the correlation coefficient is equal to or greater than a predetermined threshold. The present invention provides a method for classifying utterances. Effect of the Invention

[0008] According to the present disclosure, utterances can be classified with high accuracy while estimating the number of speakers participating in a conference, regardless of whether information regarding the number of speakers in the conference is input in advance. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration of a speaker classification system according to a first embodiment. [Diagram 2] Block diagram showing an example of a PC hardware configuration [Diagram 3]FIG. 1 is a diagram showing an example of a procedure of an overall operation of a minutes generating device; [Figure 4] A schematic diagram showing an example of the output of minutes [Diagram 5] FIG. 13 is a diagram showing an example of parameters determined according to conditions; [Figure 6] A flowchart showing an example of an operation procedure of the minutes generating device according to the first embodiment. [Figure 7] A diagram showing an example of a speech waveform based on continuous speech by one speaker. [Figure 8] FIG. 2 is a diagram showing a first example of a segmentation of an utterance section used for feature extraction; [Figure 9] FIG. 13 is a diagram showing a second example of a segmentation of an utterance section used for feature extraction; [Figure 10] FIG. 8 shows an example of a speech waveform based on the continuous speech of one speaker shown in FIG. 7 followed by speech of another speaker. [Figure 11] A flowchart showing an example of an operation procedure of the minutes generating device according to the second embodiment. [Figure 12] A diagram showing an example of a feature file used for feature extraction. [Figure 13] A diagram showing an example of the procedure for clustering operations to estimate the number of speakers included in a feature file [Figure 14] Graph showing an example of clustering results without data division [Figure 15] A table showing an example of calculation of the similarity between each utterance and each cluster obtained by clustering. [Figure 16] A graph showing the behavior of the similarity between each utterance and each cluster based on the calculation results in Figure 15. [Figure 17] 11 is a flowchart showing an example of a basic operation procedure of the minutes generating device according to the third embodiment in the case where data division processing is not performed. [Figure 18] A flowchart showing an example of the operation procedure of a subroutine of the cluster synthesis process shown in FIG. 17. [Figure 19] Graph showing examples of clustering results when data division processing is performed [Figure 20]A graph showing an example of a combination of the clustering process results of FIG. [Figure 21] A table showing an example of calculating the similarity between clusters [Figure 22] A graph showing the behavior of the similarity between each utterance and each cluster obtained by clustering [Figure 23] A table showing an example of calculation of the similarity between each utterance and each cluster obtained by clustering. [Figure 24] A flowchart showing an example of a basic operation procedure of the minutes generating device according to the third embodiment in the case where data division processing is performed DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, with reference to the drawings as appropriate, an embodiment specifically disclosing an utterance classification device and an utterance classification method according to the present disclosure will be described in detail. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters and duplicate explanation of substantially the same configuration may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the attached drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0011] In each of the following embodiments, a conference is taken as an example of an event in which speech from multiple speakers is expected, and the speech classification device of the present disclosure uses audio files in which each utterance by multiple speakers during the conference is recorded, groups each utterance by speaker (hereinafter referred to as "clustering"), and outputs minutes obtained by performing voice recognition on the utterances of each grouped speaker.

[0012] (Embodiment 1) FIG. 1 is a diagram showing an example of a system configuration of a speaker classification system 50 according to the first embodiment. The speaker classification system 50 is used, for example, during a conference in which a plurality of speakers participate, and includes a sound collection device MIC, a PC (Personal Computer) 10, and a minutes-of-meeting creating device 20. The sound collection device MIC and the PC 10 are both placed in a conference room and are connected, for example, by wire, so that data can be input and output between them. The PC 10 and the minutes-of-meeting creating device 20 are connected via a network NW1 so that data can be sent and received. Note that the network NW1 may be omitted, in which case the PC 10 and the minutes-of-meeting creating device 20 are directly connected.

[0013] The network NW1 may be a wired network or a wireless network. The wired network may be, for example, a wired LAN (Local Area Network) such as Ethernet (registered trademark) or a Universal Serial Bus (USB) standard. On the other hand, the wireless network may be, for example, a wireless LAN such as Wi-fi (registered trademark), Bluetooth (registered trademark), or WiGig using a high frequency such as the 60 GHz band, and the type is not particularly limited.

[0014] There are two possible use cases for the speaker classification system, for example. However, the use cases are not limited to these two.

[0015] In the first use case U1 shown in FIG. 1, for example, four speakers A, B, C, and D participate in a conference, and each speaker speaks toward the sound collection device MIC. The sound collection device MIC includes one or more sound collection devices such as microphones, collects the voices uttered by each speaker during the conference, and sends the collected voice data to the PC10. The PC10 uses the voice data sent from the sound collection device MIC to generate a voice file including the voice data of the contents of the speeches of the speakers during the conference, and sends the voice file to the minutes generation device 20 via the network NW1. The minutes generation device 20 receives the voice file from the PC10, analyzes the voice file (see below for details), and clusters (groups) each utterance included in the voice file by speaker. The minutes generation device 20 recognizes the voice data of each clustered utterance, converts it into text, and outputs the text (see FIG. 4). A detailed configuration example of the PC10 will be described with reference to FIG. 2, which will be described later.

[0016] In the second use case U2 shown in FIG. 1, for example, two speakers E and F each participate in a conference, and speaker E speaks toward the PC 30, and speaker F speaks toward the PC 40. As will be described later with reference to FIG. 2, the PCs 30 and 40 are equipped with one or more microphones or other sound collection devices 17, and collect the voices uttered by the speakers E and F during the conference. Using the collected voice data, the PCs 30 and 40 generate voice files each including the voice data of the contents of the speeches by the speakers E and F during the conference, and send the voice files to the minutes generation device 20 via the network NW2. The minutes generation device 20 receives the voice files from the PCs 30 and 40, synthesizes both voice files, analyzes the synthesized voice file (see later for details), and clusters (groups) each utterance included in the voice file by speaker. The minutes generation device 20 recognizes the voice data of each clustered utterance, converts it into text, and outputs the text (see FIG. 4). The network NW2 may be a wired network (see above) or a wireless network (see above) like the network NW1. The network NW2 may be omitted, in which case the PCs 30 and 40 are directly connected to the minutes generating device 20. A detailed configuration example of the PCs 30 and 40 will be described later with reference to FIG. 2.

[0017] The minutes creating device 20 is connected to the PC 10 via a network NW1 so that data can be sent and received, and is further connected to the PCs 30 and 40 via a network NW2 so that data can be sent and received, and is configured using a server computer such as a cloud server having high-performance specifications. The minutes creating device 20 may be configured using a PC, may be realized by one computer, or may be realized by multiple different computers. The minutes creating device 20 may be realized by each or any one of the PCs 10, 30, and 40.

[0018] A minutes generating device 20 as an example of an utterance classification device according to the present disclosure includes a communication unit 21, a read only memory (ROM) 22, a random access memory (RAM) 23, a storage 24, and a processor 25.

[0019] The communication unit 21 is configured using a communication circuit capable of transmitting and receiving data to and from an external device. The communication unit 21 is connected, for example, via a network NW1 to enable data transmission and reception to and from the PC 10, and is further connected via a network NW2 to enable data transmission and reception to and from the PCs 30 and 40. The communication unit 21 receives audio file data from the PC 10 via the network NW1 and sends it to the processor 25. Similarly, the communication unit 21 receives audio file data from the PCs 30 and 40 via the network NW2 and sends it to the processor 25.

[0020] The ROM 22 stores programs and data necessary for the processor 25 to execute operations (processing). In other words, the processor 25 executes processes defined in the programs using the programs and data stored in the ROM 22, thereby enabling the minutes generating device 20, which is a computer, to execute various processes related to the utterance classification method according to the present disclosure.

[0021] The RAM 23 temporarily stores data or information generated or acquired during the operation (processing) of the processor 25 .

[0022] The storage 24 is configured using a storage medium such as a flash memory, a hard disk drive (HDD), or a solid state drive (SSD), and stores data or information generated by the processor 25.

[0023] The processor 25 is configured using, for example, a central processing unit (CPU), a digital signal processor (DSP), a graphical processing unit (GPU) or a field programmable gate array (FPGA). The processor 25 functions as a controller that manages the overall operation of the minutes generating device 20, and performs control processing for managing the operation of each part of the minutes generating device 20, input / output processing of data between each part of the minutes generating device 20, data arithmetic processing and data storage processing. The processor 25 can operate functionally as a voice input unit 251, a feature extraction unit 252, a voice clustering unit 253 and a minutes generating output unit 254 according to the programs and data stored in the ROM 22. The processor 25 uses the RAM 23 during operation, and temporarily stores data or information generated or acquired by the processor 25 in the RAM 23.

[0024] The audio input unit 251, which is an example of an input unit, acquires audio file data from the PC 10 and audio file data from the PCs 30 and 40 from the communication unit 21 and inputs them.

[0025] The feature extraction unit 252, which is an example of a speech extraction unit, detects speech sections by analyzing a voice file (for example, a data file composed of voice data of speech contents by multiple speakers during a conference) input by the voice input unit 251, and extracts the total number of utterances and the total speech time of the voice data included in the voice file. For example, a sound pressure threshold (for example, a sound pressure value) used to detect a speech section may be stored in the ROM 22 or the RAM 23 in advance, and the feature extraction unit 252 may detect a section in which a sound pressure equal to or higher than the sound pressure threshold is detected as a speech section. Note that the method of detecting a speech section by the feature extraction unit 252 is not limited to the above-mentioned method, and may be realized by, for example, the feature extraction unit 252 reading out from the ROM 22 or the storage 24 a trained model of AI (artificial intelligence) (for example, a trained model that performs VAD (Voice Active Detector) processing) that has already been generated based on the execution of a learning process and executing it.

[0026] In addition, the feature extraction unit 252, which is an example of a division unit, determines the number of divisions (an example of a parameter, see Figure 5) of the audio data contained in the audio file based on the extraction results of the total number of utterances and the total total speech time, and divides the audio data contained in the audio file according to the determined number of divisions.

[0027] Furthermore, the feature extraction unit 252 extracts features of the voice data from the speech of the speakers participating in the conference for each speech section of the voice data detected for the entire voice file or for each speech section of the divided voice data. Here, the features of the voice data are indicated by a string of numbers, such as MFCC (Mel-Frequency Cepstrum Coefficient), formant frequency, voice frequency spectrum (e.g., amplitude spectrum or phase spectrum), and voiceprint values ​​indicated in a spectrogram. Furthermore, the features of the voice data need not be limited to those described above.

[0028] Furthermore, the feature extraction unit 252 as an example of a parameter determination unit determines parameters (see Fig. 5) for clustering performed by the voice clustering unit 253, using at least one extraction result of the total number of utterances and the total total speech time of the voice data detected for the entire voice file, or at least one extraction result of the total number of utterances and the total total speech time of the divided voice data, and the table of Fig. 5. Note that, when the feature extraction unit 252 is unable to extract (including not extracting) at least one of the total number of utterances and the total total speech time of the voice data detected for the entire voice file, it may use a parameter of a fixed value corresponding to at least one of the total number of utterances and the total total speech time that cannot be extracted.

[0029] Here, the parameters will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of parameters determined according to conditions. The table shown in FIG. 5 is stored in, for example, the storage 24, and specifies the number of data divisions, the maximum number of classifications, and the number of data N additions that are determined for each condition (specifically, the total time of the total speech length and the number of speech sections). The number of data divisions, the maximum number of classifications, and the number of data N additions are each an example of a parameter. Note that the parameters are not limited to these, and may also include, for example, a lower limit value of the time of data (voice data) required for the learning process of the learned model used in clustering, an initial value of classification for clustering, and the like.

[0030] The total speech length is the total time value of the speaking parts included in the audio file, and is different from the duration of the audio data included in the audio file. The number of speech sections is the number of speaking parts (speech sections) included in the audio file.

[0031] The number of data divisions is determined based on the total length of the total speech length of the audio files input to the audio input unit 251, with the aim of improving the accuracy of the clustering processing by the audio clustering unit 253, and indicates a value for dividing the audio data contained in the audio files into time lengths suitable for clustering.

[0032] The maximum number of categories indicates the maximum number of categories resulting from clustering performed by the voice clustering unit 253, for the purpose of suppressing an increase in the processing load of clustering performed by the voice clustering unit 253.

[0033] The data N addition number indicates the value of N when the number of data required to perform the learning process of the trained model of AI (artificial intelligence) performed before clustering by the voice clustering unit 253 can be prepared by multiplying the number of voice data in the speech section by N.

[0034] For example, if the extraction results of the total speech time and the number of speech sections included in the audio file input to the audio input unit 251 are "1 minute, 10", the feature extraction unit 252 determines the parameters as "none" for the number of data divisions, "2" for the maximum number of classifications, and "50" for the number of data N additions. In other words, when a short audio file with a total time of 1 minute is input, the feature extraction unit 252 calculates the features of the speech sections of the audio data included in the audio file without dividing the data, and performs a learning process by increasing the number of audio data included in the audio file by 50 times before clustering, and then performs clustering using the trained model obtained by the learning process so that each utterance included in the audio file corresponds to at most two speakers.

[0035] Furthermore, for example, if the extraction results of the total speech time and the number of speech sections included in the audio file input to the audio input unit 251 are "10 minutes, 100", the feature extraction unit 252 determines the parameters as the number of data divisions to be "none", the maximum number of classifications to be "10", and the number of data N increases to be "10". In other words, if an audio file with a total time of 10 minutes is input, the feature extraction unit 252 calculates the features of the speech sections of the audio data included in the audio file without dividing the data, and performs a learning process by increasing the number of audio data included in the audio file by 10 times before clustering, and then performs clustering using the trained model obtained by the learning process so that each utterance included in the audio file corresponds to at most 10 speakers.

[0036] Furthermore, for example, when the extraction results of the total speech time and the number of speech sections included in the audio file input to the audio input unit 251 are "100 minutes, 1000", the feature extraction unit 252 determines the number of data divisions to be "10", the maximum number of classifications to be "2", and the number of data N increments to be "50" as parameters. In other words, when a long audio file with a total time of 100 minutes is input, the feature extraction unit 252 divides the audio data included in the audio file into 10 parts, calculates the features of the speech sections of the audio data divided into 10 parts, and performs a learning process without increasing the number of audio data as learning data included in the audio file before clustering, and then performs clustering using the trained model obtained by the learning process so that each utterance included in the audio file corresponds to at most 20 speakers.

[0037] The voice clustering unit 253, which is an example of a classification unit, uses the output of the feature extraction unit 252 (specifically, the feature of the utterance in the utterance section, the parameters described with reference to FIG. 5) to group (cluster) each utterance of the detected voice data for the entire voice file or each utterance of the divided voice data by speaker. For example, the voice clustering unit 253 classifies utterances with similar feature amounts (in other words, utterances whose difference between the feature amounts is less than a predetermined value registered in the ROM 22) into the same group (hereinafter also referred to as a "cluster"). In addition, the voice clustering unit 253 may calculate the similarity between the clusters, and combine both clusters when the calculation results are similar (in other words, when the difference between the similarity amounts is less than a predetermined value registered in the ROM 22).

[0038] The minutes generation output unit 254, which is an example of an output processing unit, performs speech recognition on the voice data of each utterance by each speaker grouped (clustered) by the voice clustering unit 253, converts it into text, and generates the textual sentences (e.g., words uttered during the conference) as minutes (see FIG. 4). The minutes generation output unit 254 displays the generated data of the minutes on PC 10 via network NW1, or on PCs 30 and 40 via network NW2.

[0039] FIG. 4 is a diagram showing a schematic example of an output of minutes. As shown in FIG. 4, the display 18 (see FIG. 2) of the PC 10, 30, 40 displays the data of the minutes sent from the minutes generating device 20 in a manner that allows the speaker to be identified. For example, utterance RT1 of "Let's start the meeting. First, please tell us about the progress of project Pa," utterance RT2 of "Yes, I understand. This week, the progress of Pa is...," utterance RT3 of "Thank you. Next, please tell us about the progress of project Pb," and utterance RT4 of "Yes, I understand. This week, the progress of Pb is..." are displayed. In particular, since the voice clustering unit 253 has identified that the speakers of utterances RT1 and RT3 are the same, utterances RT1 and RT3 are displayed in a manner that allows the speaker to be identified as being the same (for example, in the same color).

[0040] In the minutes taking device 20 according to the first embodiment, it is not necessary that information regarding the number of speakers is registered in advance as in Patent Document 1. Information regarding the number of speakers may be registered, but the minutes taking device 20 according to the first embodiment can perform clustering (in other words, classifying the voice data uttered by each speaker) using the voice data of each utterance by each speaker by analyzing the input voice file, regardless of whether information regarding the number of speakers is registered.

[0041] Fig. 2 is a block diagram showing an example of the hardware configuration of the PCs 10, 30, and 40. The PCs 10, 30, and 40 each include an interface 11, an imaging unit 12, a processor 13, a storage 14, a communication device 15, an operation device 16, a sound collection device 17, a display 18, and an external interface 19. Since the PCs 10, 30, and 40 have the same configuration, the following description of Fig. 2 will be given using the PC 10 as an example, but the same description can also be applied to the PCs 30 and 40.

[0042] The interface 11 connects the various components of the PC 10 so as to enable input and output of data. In Fig. 2, the interface is abbreviated as IF.

[0043] The imaging unit 12 includes at least a lens and an image sensor. The lens includes, for example, a focus lens and a zoom lens, and forms an optical image on the light receiving surface (in other words, the imaging surface) of the image sensor by receiving light from at least one speaker in front of the PC 10. The image sensor performs photoelectric conversion to convert the light received on the light receiving surface (in other words, the imaging surface) into an electrical signal. The image sensor is configured using, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The image sensor sends an electrical signal (analog signal) corresponding to the light received on the light receiving surface (in other words, the imaging surface) to the processor 13 at predetermined time intervals. This analog signal is converted by the processor 13 into captured image data in a digital format. As a result, the captured image data is generated by the processor 13. The conversion from this analog signal to a digital format may be performed by the image sensor. In this way, the PC 10 can acquire captured image data in a digital format at predetermined time intervals. The imaging unit 12 may be omitted from the configuration of the PC 10.

[0044] The processor 13 is configured using, for example, a CPU, a DSP, a GPU, or an FPGA. The processor 13 functions as a controller that manages the overall operation of the PC 10, and performs control processing for managing the operation of each part of the PC 10, input / output processing of data between each part of the PC 10, calculation processing of data, and storage processing of data. The processor 13 operates according to the programs and data stored in the storage 14. The processor 13 uses the storage 14 during operation, and temporarily stores data or information generated or acquired by the processor 13 in the storage 14.

[0045] The storage 14 includes, for example, a ROM, and stores in the ROM programs and data required for the processor 13 to execute operations (processing). In other words, the processor 13 can execute various processes related to the speech classification method according to the present disclosure on the computer PC 10 by executing processes defined in the programs using the programs and data stored in the storage 14. The storage 14 also includes, for example, a RAM, and temporarily holds in the RAM data or information generated or acquired during the operation (processing) of the processor 13. The storage 14 also includes a storage medium, such as a flash memory, HDD, or SSD, and stores in the storage medium data or information generated by the processor 13.

[0046] The communication device 15 is configured using a communication circuit capable of transmitting and receiving data to and from an external device. The communication device 15 is connected to the minutes generating device 20 via a network NW1, for example, so as to be capable of transmitting and receiving data therebetween. The communication device 15 receives minutes data from the minutes generating device 20 via the network NW1 and sends it to the processor 13.

[0047] The operation device 16 is a device used by the speaker during operation, and may be, for example, a mouse, a keyboard, a touch pad, a touch panel, or a combination of a plurality of these. A signal from the operation device 16 based on the speaker's operation is input to the processor 13 via the interface 11.

[0048] The sound collection device 17 is composed of, for example, one or more microphones, and collects the voices spoken by each speaker in front of or around the PC 10 during the conference, and sends the collected voice data to the processor 13.

[0049] The display 18 is constructed, for example, using an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) device, and outputs (e.g., displays), for example, data of the speaker's facial image captured by the imaging unit 12 and data of the minutes of the meeting from the minutes of the meeting generation device 20.

[0050] The external interface 19 inputs and outputs data to and from an external medium (for example, an external storage medium such as an SD card) via a cable conforming to the USB standard. The external interface 19 may be connected to the minutes generating device 20.

[0051] 3 is a diagram showing an example of an overall operation procedure of the minutes generating device 20. Each process shown in FIG. 3 is a schematic and chronological diagram of the operation procedure of the minutes generating device 20, and is mainly executed by the processor 25 of the minutes generating device 20.

[0052] 3, when processor 25 receives a voice file from PC 10, for example, it analyzes (e.g., VAD processing) the voice file (e.g., a data file composed of voice data of speech contents of multiple speakers in a conference) in feature extraction unit 252 to detect speech sections (Step 1). Processor 25 extracts the total number of utterances and the total total speech time of the voice data included in the voice file based on the speech sections detected in Step 1. Processor 25 determines the division number K (an example of a parameter) of the voice data included in the voice file in feature extraction unit 252 based on the extraction results of the total number of utterances and the total total speech time, and divides the voice data included in the voice file in feature extraction unit 252 according to the determined division number K.

[0053] Processor 25 generates an audio file (e.g., a wav file) consisting of audio data of an utterance section for each of the K divided pieces of audio data (Step 2), and extracts features of the audio data included in the audio file in feature extraction unit 252 (Step 3). Processor 25 determines parameters (see FIG. 5) for clustering performed by audio clustering unit 253 for each of the K divided pieces of audio data, using the extraction results of the total number of utterances and the total total speaking time of the audio data and the table of FIG. 5.

[0054] Processor 25 uses the output of feature extraction unit 252 (specifically, the features of the utterance in the speech section, the parameters described with reference to FIG. 5) to group (cluster) each utterance of the divided speech data by speaker in speech clustering unit 253 (Step 4). Processor 25 uses the results of the clustering process in Step 4 to perform a cluster synthesis process as necessary in speech clustering unit 253 (Step 5). Processor 25 outputs the results of the synthesis process in Step 5 (Step 6, see FIG. 4).

[0055] Next, an operation procedure of the minutes generating device 20 according to the first embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of an operation procedure of the minutes generating device 20 according to the first embodiment. Each process shown in Fig. 6 is mainly executed by the processor 25 of the minutes generating device 20.

[0056] 6, when processor 25 receives data of an audio file from communication unit 21, for example, processor 25 analyzes (for example, VAD processing) the audio file (for example, a data file composed of audio data of utterances by multiple speakers during a conference) in feature extraction unit 252 (St1). This allows processor 25 to detect sections of audio data (speech sections) of each utterance by multiple speakers included in the entire audio file input from communication unit 21 in feature extraction unit 252. Processor 25 extracts the total number of utterances and the total total speaking time of the audio data included in the entire audio file input from communication unit 21 based on the detected speech sections in feature extraction unit 252.

[0057] Processor 25 determines the number of divisions (one example of a parameter) of the voice data included in the entire voice file in feature extraction unit 252 based on the extraction results of the total number of utterances and the total total speech time, and divides the voice data included in the entire voice file in feature extraction unit 252 according to the determined number of divisions (St2). For example, K is an integer equal to or greater than 2 and indicates the number of divisions. Processor 25 acquires one piece of voice data divided in step St2 in feature extraction unit 252 (St3), and cuts out a speech section of the voice data (St4). In step St4, processor 25 detects, as a speech section, a section in which a sound pressure equal to or greater than a sound pressure threshold (see above) stored in, for example, ROM 22 or RAM 23 is detected.

[0058] The processor 25 calculates and extracts features of the voice data of each utterance section cut out in step St4 in the feature extraction unit 252 (St5). The processor 25 determines parameters (see FIG. 5) for clustering performed in the voice clustering unit 253 in the feature extraction unit 252 using the extraction results of the total number of utterances and the total speech time of the voice data of the utterance section cut out in step St4 and the table of FIG. 5 (St6).

[0059] The processor 25 performs grouping (clustering) of each utterance of one piece of voice data acquired in step St3 by speaker in the voice clustering unit 253 using the parameters determined in step St6 and the feature amount of the voice data of each utterance section extracted in step St5 (St7). At this time, the processor 25 performs synthesis processing (e.g., addition processing) of the feature amount of the voice data of each utterance included in each cluster indicating the grouping result of each utterance, and stores it in the RAM 23.

[0060] The processor 25 determines whether or not to perform a process of synthesizing clusters that indicate the grouping results of each utterance (St8). For example, when the number of speakers information from the PCs 10, 30, and 40 is specified in advance (that is, when the number of speakers is known), the clustering in step St7 is performed so as to obtain a number of clusters that matches the specified number of speakers, and therefore the process of synthesizing clusters is not performed. When the process of synthesizing clusters is not performed (St8, NO), the process of the processor 25 proceeds to step St11.

[0061] On the other hand, when the processor 25 determines to perform the synthesis process between the clusters (St8, YES), the processor 25 performs a similarity calculation between the clusters obtained by the clustering in step St7 in the voice clustering unit 253 (St9). This similarity calculation between the clusters may be realized by an existing technique (see FIG. 13), such as cosine distance. For example, a similarity threshold (similarity value) used in the synthesis process between the clusters is stored in the ROM 22 or the RAM 23 in advance, and the processor 25 performs the synthesis process between the clusters in the voice clustering unit 253 only when the similarity between the clusters calculated in step St9 is equal to or greater than the similarity threshold (St10). On the other hand, when the similarity between the clusters calculated in step St9 is less than the similarity threshold, the processor 25 does not perform the synthesis process between the clusters.

[0062] The processor 25 determines whether or not the series of processes from step St3 to step St10 has been performed on each of all the voice data obtained by the division process in step St2 (St11). The processor 25 repeats the series of processes from step St3 to step St10 on one piece of voice data obtained in step St3 until it determines that the series of processes from step St3 to step St10 has been performed on each of all the voice data (St11, NO).

[0063] On the other hand, when the processor 25 judges that the series of processes from step St3 to step St10 have been performed on each of all the voice data obtained by the division process in step St2 (St11, YES), it judges whether or not the division process in step St2 was effective in order to determine whether or not a synthesis process is required for clusters obtained from each of the K voice data divided in step St2 (St12). If the division process has not been performed in step St2 (St12, NO), the process of the processor 25 in FIG. 6 ends.

[0064] On the other hand, when the processor 25 determines that the division process in step St2 was effective (i.e., the division process was performed in step St2) (St12, YES), it performs a similarity calculation across the clusters obtained from each of the voice data divided in step St2 (St13). The similarity calculation in step St13 may be the same as or different from the method in step St9. The processor 25 performs a synthesis process on the clusters in the voice clustering unit 253 only when the similarity between the clusters calculated in step St13 is equal to or greater than the similarity threshold (see above) (St14). Furthermore, the processor 25 performs a synthesis process (e.g., an addition process) on the features of the voice data of each utterance that constitutes the synthesized cluster, and stores the result in the RAM 23.

[0065] As described above, the minutes creation device 20 of embodiment 1 includes an input unit (e.g., the audio input unit 251) that inputs an audio file containing utterances from multiple speakers, an utterance extraction unit (e.g., the feature extraction unit 252) that analyzes the audio file and extracts the number of utterances and the utterance length of each utterance contained in the audio file, a parameter determination unit (e.g., the feature extraction unit 252) that determines multiple parameters for grouping multiple speakers based on at least one of the number of utterances and the utterance length of each utterance, the feature extraction unit 252 that extracts features of each utterance in multiple speech sections contained in the audio file based on the parameters, and a classification unit (e.g., the audio clustering unit 253) that groups each utterance by speaker based on the features of each utterance.

[0066] As a result, the minutes generation device 20 can determine clustering parameters from the number of utterances and the length of each utterance contained in the input audio file, regardless of whether information regarding the number of speakers at the meeting has been input in advance, and can perform clustering using these parameters, thereby enabling the minutes generation device 20 to classify the utterances with high accuracy while estimating the number of speakers participating in the meeting.

[0067] The minutes generating device 20 further includes a division unit (e.g., feature extraction unit 252) that divides the audio file into a plurality of speech sections based on at least one of the number of utterances and the length of each utterance. The feature extraction unit 252 extracts features of the utterances in each speech section divided by the division unit. This allows the minutes generating device 20 to divide the audio data for obtaining parameters appropriate for clustering when the total utterance length included in the input audio file is long.

[0068] The multiple parameters include a first parameter determined based on the number of utterances (e.g., how many seconds or more of data to use) and a second parameter determined based on the length of each utterance (e.g., the maximum number of classifications). This allows the minutes generating device 20 to quickly and efficiently perform the learning process of the trained model used for clustering by considering the number of utterances of the audio data included in the input audio file and the length of each utterance, and as a result, can quickly and efficiently perform clustering using the trained model obtained by the learning process.

[0069] Furthermore, the parameter determination unit determines the first parameter (e.g., the number of data N additions) by inputting the number of utterances into a predetermined calculation formula (e.g., if the number of utterances in the voice data is "100" and the number of data required for the learning process of the trained model is M (an integer), then "M / 100" becomes N) or a predetermined table (see FIG. 5). Similarly, if the number of utterances in the voice data is "1000" and the number of data required for the learning process of the trained model is M (an integer), then "M / 1000" becomes N. This enables the minutes generation device 20 to easily determine the first parameter for classifying (clustering) speakers with a certain degree of accuracy.

[0070] The parameter determination unit also determines the second parameter (e.g., the maximum number of classifications) by inputting the utterance length of each utterance into a predetermined calculation formula (e.g., if data of 60 seconds or more is required to classify utterances with a certain degree of accuracy while estimating the number of speakers, and the total utterance length of the voice data is L (an integer), then "L / 60" is used) or a predetermined table (see FIG. 5). This enables the minutes generating device 20 to easily determine the second parameter for classifying (clustering) speakers with a certain degree of accuracy.

[0071] The minutes generating device 20 further includes an output processing unit (e.g., the minutes generating output unit 254) that recognizes the voice data of each utterance grouped by speaker and outputs the voice recognition result in a manner that allows each speaker to be identified. This allows the minutes generating device 20 to output the contents of utterances made by multiple speakers in a meeting so that it is intuitively clear whether the speakers are the same or different.

[0072] (Embodiment 2) In the second embodiment, an example of a minutes generating device will be described as an example of an utterance feature extraction device that improves the accuracy of feature extraction processing compared to the first embodiment. Since the configuration of the speaker classification system according to the second embodiment is the same as the configuration of the speaker classification system 50 according to the first embodiment, the same reference numerals are used to simplify or omit the description of the same contents, and the different contents will be described.

[0073] When extracting features, it is considered desirable to extract features by utterance, not by word, spoken by the speaker. For example, assume that speaker A (see Figure 1) utters "Good morning." In this case, in extracting features of the voice data of speaker A's "Good morning," the first method is to extract four features for each of "o," "ha," "yo," and "u," and the second method is to extract a feature for "Good morning." As a result, the second method is considered to be superior. This is because, for example, even if a single sound is heard by the human ear, it is difficult to tell who is speaking, and a certain length of sound such as "Good morning" is more likely to reveal the atmosphere of the speaker's utterance (in other words, the feature of the voice data).

[0074] FIG. 7 is a diagram showing an example of a speech waveform based on continuous speech by one speaker A. The speech waveform RWD1 shown in FIG. 7 is the speech waveform of speaker A's speech, "Good morning, how are you?" If features can be extracted on an utterance-by-utterance basis, such as "Good morning, how are you?", it is preferable because it is easier to detect the atmosphere of the speech that is characteristic of that speaker (see above). However, with conventional general sound pressure analysis, it has been difficult to detect when the speaker switches midway. The longer the detected speech section is, the higher the possibility that multiple speakers are speaking, so it is preferable to divide it.

[0075] For example, the audio waveform RWD1 shows a section P1 including an utterance section P2 of "Good morning," a silent section N1, and an utterance section P3 of "How are you?" The minutes generating device 20 according to the second embodiment adopts the utterance sections P2 and P3 as valid utterance sections for extracting features, rather than the entire section P1. In other words, the minutes generating device 20 does not extract features of the speech part of the audio waveform RWD1 for the entire section P1 in which the utterance sections P2 and P3 continue almost continuously, but performs feature extraction processing for each of the utterance sections P2 and P3.

[0076] FIG. 8 is a diagram showing a first division example of the speech section used for feature extraction. FIG. 9 is a diagram showing a second division example of the speech section used for feature extraction. Here, it is assumed that, for example, voice data (voice waveform) of a 10-second speech section is obtained. As a first division example (division example), the minutes generating device 20 according to the second embodiment may divide the entire 10-second speech waveform RWD2 into a first 5-second section P11 and a last 5-second section P12 in the middle in time. Also, as a second division example (division example), the minutes generating device 20 according to the second embodiment may divide the entire 10-second speech waveform RWD3 into a first 6-second section P13 and a last 4-second section P14 so as not to include the silent section N2 (i.e., the section where the sound pressure is considerably reduced) in the middle in the feature extraction section.

[0077] Fig. 10 is a diagram showing an example of a speech waveform based on the continuous speech by one speaker A shown in Fig. 7 and the subsequent speech by another speaker B. When cutting out a speech section to extract features of speech data, it is ideal that the features of one speaker are included from one speech section, but if the end of the speech (i.e., the end of speech) is not detected easily and the speech section becomes too long, there is a possibility that another speaker will be mixed in.

[0078] For example, the audio waveform RWD4 shows a section P21 including an utterance section P22 of "Good morning," an utterance section P23 of "How are you?", and an utterance section P24 of "How are you?". The minutes generating device 20 according to the second embodiment adopts, as a valid utterance section for extracting a feature, not the entire section P21, but the utterance sections P22, P23, and P24 obtained by dividing the section P21 by a predetermined number of seconds (for example, 5 seconds). In other words, the minutes generating device 20 does not extract a feature of the utterance part of the audio waveform RWD4 for the entire section P21 in which the utterance sections P22, P23, and P24 continue almost continuously, but performs a feature extraction process for each of the utterance sections P22, P23, and P24 in the feature extraction unit 252.

[0079] Next, the operation procedure of the minutes generating device 20 according to the second embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the operation procedure of the minutes generating device 20 according to the second embodiment. Each process shown in Fig. 11 is mainly executed by the feature extraction unit 252 of the processor 25 of the minutes generating device 20. Moreover, the minutes generating device 20 may execute each process shown in Fig. 11 as a subroutine of the feature extraction of step St5 shown in Fig. 6.

[0080] In Fig. 11, when the processor 25 inputs the voice data to be subjected to the feature extraction process in the feature extraction unit 252, the processor 25 detects the speech section of the voice data (St21) and analyzes the speech time in the feature extraction unit 252 (St21). This analysis can be realized, for example, by using a trained model that performs VAD processing and that can be executed by the feature extraction unit 252. The processor 25 determines in the feature extraction unit 252 whether the length of the speech time (i.e., the speech length) extracted by the analysis in step St21 is equal to or greater than the time length threshold (e.g., 5 seconds, see Fig. 10) stored in the ROM 22 or RAM 23 (St22). If it is determined that the length of the speech time (i.e., the speech length) is equal to or less than the time length threshold (St22, NO), the process of the processor 25 proceeds to step St24.

[0081] On the other hand, when the processor 25 determines that the length of the utterance time (i.e., the utterance length) is equal to or greater than the time length threshold (St22, YES), the feature extraction unit 252 divides the voice data input in step St21 into a plurality of pieces (St23). The processor 25 extracts and detects the voice data divided in step St23 or the voice data input in step St21 as a feature extraction section in the feature extraction unit 252 (St24). The processor 25 calculates and extracts the feature of the voice data of the feature extraction section cut out in step St24 in the feature extraction unit 252 (St25).

[0082] After step St25, the processor 25 determines whether or not the division process has been performed in step St23 (St26). If it is determined that the division process has not been performed in step St23 (St26, NO), the process of the processor 25 in FIG. 11 ends.

[0083] On the other hand, when the processor 25 determines that the division process has been performed in step St23 (St26, YES), the feature extraction unit 252 calculates the similarity between the divided voice data (for example, the difference value of the feature of the voice data) (St27). When the division process has been performed in step St23, there is a high probability that the same speaker originally spoke in one speech section. For this reason, the processor 25 calculates the similarity between the voice data before and after the division process (i.e., before and after the break), and if this similarity is equal to or greater than a fourth threshold (see below), combines the features of each voice data.

[0084] When the processor 25 determines that the similarity between the voice data calculated in step St27 is equal to or greater than the fourth threshold (similarity value) stored in the ROM 22 or the RAM 23 (St28, YES), the processor 25 synthesizes the features of the voice data (St29). On the other hand, when the processor 25 determines that the similarity between the voice data is equal to or less than the fourth threshold (St28, NO), the processor 25 ends the process of FIG.

[0085] As described above, the minutes creation device 20 of embodiment 2 comprises an input unit (e.g., the audio input unit 251) that inputs an audio file containing speeches from multiple speakers, a speech interval detection unit (e.g., the feature extraction unit 252) that analyzes the audio file using a first algorithm for detecting speech intervals and detects multiple speech intervals contained in the audio file, a feature extraction interval detection unit (e.g., the feature extraction unit 252) that extracts feature extraction intervals in each of the multiple speech intervals detected using a second algorithm for detecting speech intervals, and the feature extraction unit 252 that extracts features of the extracted extraction intervals.

[0086] As a result, the minutes generating device 20 can use a second algorithm different from the first algorithm for detecting speech sections included in an audio file to appropriately extract sections (extraction sections) for extracting features from each of the multiple speech sections detected using the first algorithm. Therefore, the minutes generating device 20 can appropriately extract features of the audio data of each utterance of multiple speakers, which can contribute to improving the accuracy of the clustering process.

[0087] The minutes generating device 20 further includes a classification unit (e.g., the voice clustering unit 253) that groups each utterance in a plurality of speech sections included in the audio file by speaker based on the feature amount of the extracted extraction section. This allows the minutes generating device 20 to group (cluster) each utterance by speaker with high accuracy while estimating the number of multiple speakers.

[0088] In addition, in the minutes generating device 20, the speech section detection unit divides a detected speech section into multiple speech sections when the detected speech section is longer than a predetermined time (for example, the above-mentioned time length threshold). This allows the minutes generating device 20 to reduce the possibility of another speaker being mixed in due to the speech section becoming too long, and appropriately extract the feature amount of the voice data of an utterance by one speaker.

[0089] The first algorithm detects a detection section of a sound pressure equal to or greater than a first sound pressure threshold (an example of a sound pressure threshold) as a speech section. The second algorithm has stricter detection conditions than the first algorithm for detecting a speech section included in an audio file in order to detect an extraction section of a feature, and detects a detection section of a sound pressure equal to or greater than a second sound pressure threshold that is greater than the first sound pressure threshold as an extraction section of a feature. For example, if the first sound pressure threshold is a preset reference sound pressure (e.g., 20 μPa), the first sound pressure threshold is 10 dB, and the second sound pressure threshold is 15 dB. This allows the minutes generating device 20 to further carefully select the voice data of the speech section detected by the first algorithm and detect the extraction section of the feature with high accuracy.

[0090] The first algorithm detects a detection section of a sound pressure equal to or greater than a first sound pressure threshold (an example of a sound pressure threshold) as a speech section. The second algorithm detects a detection section of a sound signal in a frequency band (e.g., 100 to 1000 Hz) included in human voice as a feature extraction section. This enables the minutes generating device 20 to carefully select sound data having a frequency band of human voice from the sound data of the speech section detected by the first algorithm, and detect the feature extraction section with high accuracy.

[0091] Furthermore, the second algorithm detects a detection section where the sound pressure is equal to or greater than a second sound pressure threshold (see above) that is greater than the first sound pressure threshold as an extraction section of a feature. As a result, the minutes generating device 20 not only carefully selects sound data having a frequency band of human voice from the sound data of the speech section detected by the first algorithm, but also carefully selects a second sound pressure threshold that is greater than the first sound pressure threshold from the sound data of the speech section detected by the first algorithm, so that the extraction section of a feature can be detected with higher accuracy.

[0092] Furthermore, in the minutes generating device 20, when one speech section detected using the first algorithm contains multiple (e.g., two) extraction sections detected using the second algorithm, the feature extractor 252 extracts the averaged result of the features corresponding to the multiple extraction sections as the feature corresponding to the multiple extraction sections. This allows the minutes generating device 20 to obtain a feature of the voice data that is more likely to represent the speaker by averaging (smoothing) the features in the multiple extraction sections.

[0093] (Embodiment 3) In the third embodiment, a further improvement example of the clustering according to the first embodiment will be described. Since the configuration of the speaker classification system according to the third embodiment is the same as the configuration of the speaker classification system 50 according to the first embodiment, the same reference numerals are used to simplify or omit the description of the same contents, and the differences will be described.

[0094] Unlike Patent Document 1, the minutes generating device 20 according to each embodiment groups (clusters) each utterance by multiple speakers participating in a conference by speaker even when information on the number of speakers is not available in advance. In the third embodiment, in order to perform such clustering (i.e., to appropriately narrow down the speakers), the minutes generating device 20 narrows down the speakers by synthesizing clusters as necessary based on not only the similarity between clusters obtained by clustering (see FIG. 13 described later) but also a correlation coefficient (see later) based on the similarity between the voice data of the utterances and the clusters obtained by clustering.

[0095] Fig. 12 is a diagram showing an example of a feature file used for feature extraction. For example, the speech waveform RWD5 shows a section 31 including a feature extraction section P32, a feature extraction section P33, and a feature extraction section P34. Each of the extraction sections P32, P33, and P34 is obtained by the detection method described with reference to the second embodiment, for example, but may be obtained by a detection method other than the detection method. Features are calculated and extracted for each of the extraction sections P32, P33, and P34, and in Fig. 12, for example, a feature file storing three feature values ​​is obtained.

[0096] Here, if information regarding the number of speakers (e.g., X people) is given in advance as in Patent Document 1, the minutes generating device 20 may cluster the feature files into X pieces as a result of the clustering process so as to match the given number of speakers (X people). However, if information regarding the number of speakers (e.g., X people) is not given in advance (so-called unsupervised), the minutes generating device 20 must perform clustering while estimating how many people are speaking.

[0097] For this purpose, a method (comparison example) can be considered in which clustering is first performed using appropriate initial values ​​(classification initial values), and then clusters with high similarity are synthesized using the similarity between each cluster obtained as a result of the processing (see Fig. 13). Fig. 13 is a diagram showing an example of the procedure of the clustering operation overview for estimating the number of speakers included in a feature file.

[0098] In Fig. 13, each of a plurality of (e.g., 36) feature files (see Fig. 12) extracted from an audio file is plotted in a two-dimensional coordinate system (Step 1). This plot can be realized by, for example, arranging the values ​​of two types of feature calculated based on the feature files at positions Q1, Q2, Q3, ..., Q36 specified as coordinate values. After Step 1, clustering is performed with an appropriate initial value (classification initial value, for example, 5), resulting in classification into five clusters CL1, CL2, CL3, CL4, and CL5 (Step 2). Note that this initial value (classification initial value) is an example of a parameter determined by the feature extraction unit 252 described with reference to the first embodiment.

[0099] After Step 2, the similarities between the clusters CL1 to CL5 obtained in Step 2 are calculated (Step 3). In Step 3, for example, the distance between the center positions (coordinate values) of the clusters CL1 to CL5 is calculated as the similarity. Note that the similarity does not have to be calculated based on the distance between the center positions of the clusters. If it is determined that the similarity calculated in Step 3 is equal to or greater than a predetermined threshold (for example, the similarity threshold described with reference to the first embodiment), the corresponding clusters are combined (Step 4). For example, in FIG. 13, the clusters CL2 and CL3 are combined to generate the cluster CL6, and the clusters CL4 and CL5 are combined to generate the cluster CL7.

[0100] However, there was a high possibility that the similarity between clusters alone would not be enough to properly synthesize the clusters. This was because, for example, when the similarity between clusters was close to the above-mentioned predetermined threshold, there was a possibility that speech from different speakers would be erroneously classified as being in the same cluster.

[0101] Therefore, in the third embodiment, the search results are narrowed down by combining clusters as necessary based not only on the similarity between clusters obtained by clustering (see FIG. 13 described later) but also on correlation coefficients (see later) based on the similarity between the speech data and the clusters obtained by clustering (see FIG. 14 to FIG. 24).

[0102] First, an overview of the processing of the minutes generating device 20 according to embodiment 3 will be described by exemplifying a case where the speech section included in the audio file input to the processor 25 is not very long (i.e., a case where data division processing is not required). Fig. 14 is a graph showing an example of a clustering processing result when data division processing is not performed. Fig. 15 is a table showing an example of calculation of the similarity between each utterance and each cluster obtained by clustering. Fig. 16 is a graph showing the behavior of the similarity between each utterance and each cluster based on the calculation result of Fig. 15.

[0103] For example, as shown in Fig. 14, assume that utterances corresponding to feature files (see Fig. 12) are plotted two-dimensionally and classified into three clusters CLx, CLy, and CLz by clustering by the voice clustering unit 253 according to the third embodiment. The numbers in circles in Fig. 14 correspond to the utterance numbers, and it is shown here that a total of 10 utterances (feature files) have been classified. Cluster CLx has three utterances: utterances (1), (2), and (3). Cluster CLy has three utterances: utterances (4), (5), and (6). Cluster CLz has four utterances: utterances (7), (8), (9), and (10).

[0104] The voice clustering unit 253 calculates the similarity between each cluster CLx, CLy, CLz and all the utterances (1) to (10) (see FIG. 15). Here, the similarity is calculated to have a value of 0 to 100, and so on. As shown in FIG. 14, the cluster CLx has three utterances, utterances (1), (2), and (3), so the similarity between the cluster CLx and each of the utterances (1) to (3) is very high (see the dotted line frame). Similarly, the cluster CLy has three utterances, utterances (4), (5), and (6), so the similarity between the cluster CLy and each of the utterances (4) to (6) is very high (see the dotted line frame). Similarly, the cluster CLz has four utterances, utterances (7), (8), (9), and (10), so the similarity between the cluster CLz and each of the utterances (7) to (10) is very high (see the dotted line frame).

[0105] Furthermore, the voice clustering unit 253 calculates a correlation coefficient G indicating the degree of correlation between the clusters, using the calculation result of the similarity between each cluster CLx, CLy, CLz and all utterances (1) to (10) and a predetermined evaluation function ES (cluster, utterance number, similarity). The correlation coefficient G is obtained from the output value of the predetermined evaluation function ES (cluster, utterance number, similarity), and takes a value of, for example, 0 to 1. In other words, if the correlation coefficient G is equal to or greater than a predetermined correlation threshold value stored in advance in the ROM 22 or RAM 23, the voice clustering unit 253 performs synthesis processing on each cluster corresponding to the correlation coefficient G. On the other hand, if the correlation coefficient G is less than the above-mentioned predetermined correlation threshold value, the voice clustering unit 253 does not perform synthesis processing on each cluster corresponding to the correlation coefficient G.

[0106] Fig. 16 shows a graph illustrating the characteristics of the vertical axis (similarity) against the horizontal axis (utterance number) using the similarity described with reference to Fig. 15. As can be seen from the behavior of the similarity (score) between the clusters CLx, CLy, and CLz and each utterance, there was no correlation coefficient G between the clusters that exceeded a predetermined correlation threshold. In other words, the voice clustering unit 253 does not perform synthesis processing on each cluster corresponding to the correlation coefficient G (specifically, the pair of clusters CLx and CLy, the pair of clusters CLy and CLz, and the pair of clusters CLz and CLx).

[0107] Next, an example of the operation procedure of the minutes generating device 20 according to the third embodiment in the case where data division processing is not required will be described with reference to Figs. 17 and 18. Fig. 17 is a flowchart showing an example of the basic operation procedure of the minutes generating device according to the third embodiment in the case where data division processing is not performed. Fig. 18 is a flowchart showing an example of the operation procedure of the subroutine of the cluster synthesis processing shown in Fig. 17. Each process shown in Figs. 17 and 18 is mainly executed by the voice clustering unit 253 of the processor 25 of the minutes generating device 20. The minutes generating device 20 may execute each process shown in Fig. 17 as a subroutine of the clustering of step St7 shown in Fig. 6. In addition, the minutes generating device 20 executes each process shown in Fig. 18 as a subroutine of the cluster synthesis processing of step St33 shown in Fig. 17.

[0108] In FIG. 17, the processor 25 acquires data to be input for clustering (e.g., voice data of all utterances (in other words, data of feature files), parameters described with reference to embodiment 1, and a trained model for clustering) in the voice clustering unit 253 (St31). The processor 25 performs clustering in the voice clustering unit 253 using the data acquired in step St31 (St32). After step St32, the processor 25 performs cluster synthesis processing in the voice clustering unit 253 (St33).

[0109] 18, the processor 25 calculates the similarity between each cluster classified by the clustering in step St32 in the voice clustering unit 253 (St41). If it is determined that the similarity between each cluster calculated in step St41 is less than the similarity threshold (see embodiment 1) (St42, NO), the process of the processor 25 proceeds to step St44. On the other hand, if the processor 25 determines that the similarity between each cluster calculated in step St41 is equal to or greater than the similarity threshold (see embodiment 1) (St42, YES), the processor 25 performs synthesis processing of the corresponding clusters in the voice clustering unit 253 (St43).

[0110] Furthermore, the processor 25 calculates the similarity between each cluster classified by the clustering in step St32 and the voice data of all utterances acquired in step St31 in the voice clustering unit 253 (St44, see FIG. 15). The processor 25 calculates the correlation coefficient G indicating the degree of correlation between the clusters in the voice clustering unit 253 using the calculation result of the similarity between each cluster and all utterances and a predetermined evaluation function ES (cluster, utterance number, similarity) (St45). If the processor 25 determines that the correlation coefficient G calculated in step St44 is equal to or greater than a predetermined correlation threshold previously stored in the ROM 22 or RAM 23 (St46, YES), the voice clustering unit 253 performs a synthesis process on each cluster corresponding to the correlation coefficient G (St47). On the other hand, if it is determined that the correlation coefficient G is less than the above-mentioned predetermined correlation threshold (St46, NO), the process of the processor 25 in FIG. 18 ends.

[0111] Next, an overview of the processing of the minutes generating device 20 according to embodiment 3 will be described by taking as an example a case where the speech section included in the audio file input to the processor 25 is long (i.e., when data division processing is required). FIG. 19 is a graph showing an example of each clustering processing result when data division processing is performed. FIG. 20 is a graph showing an example of synthesis of each clustering processing result of FIG. 19. FIG. 21 is a table showing an example of calculation of the similarity between each cluster. FIG. 22 is a graph showing the behavior of the similarity between each utterance and each cluster obtained by clustering. FIG. 23 is a table showing an example of calculation of the similarity between each utterance and each cluster obtained by clustering.

[0112] For example, as shown in FIG. 19, when the speech section included in the audio file is long (for example, when there is audio data for 20 utterances), the data is divided by the audio clustering unit 253 (see FIG. 24), and two classification results corresponding to the clustering process results are shown, showing that the audio data is divided into two pieces of audio data. Specifically, the cluster CLx1 in FIG. 19 has three utterances, utterances (1), (2), and (3). The cluster CLy1 has three utterances, utterances (4), (5), and (6). The cluster CLz1 has four utterances, utterances (7), (8), (9), and (10). Similarly, the cluster CLx2 in FIG. 19 has three utterances, utterances (11), (12), and (13). The cluster CLy2 has three utterances, utterances (14), (15), and (16). Cluster CLz2 has four utterances: utterances (17), (18), (19), and (20). Figure 20 shows how the clusters shown in Figure 19 are superimposed.

[0113] The voice clustering unit 253 calculates the similarity between each cluster (specifically, clusters CLx1, CLx2, CLy1, CLy2, CLz1, and CLz2). The results of this similarity calculation are shown in FIG. 21. According to FIG. 21, the similarity (98) between clusters CLx1 and CLx2 and the similarity (91) between clusters CLy1 and CLy2 are very high, so they may be subject to synthesis processing (in other words, clusters CLx1 and CLx2, and clusters CLy1 and CLy2 may be identified as the same speaker) (see FIG. 20). However, although the similarity (62) between clusters CLy1 and CLz2 and the similarity (58) between clusters CLy2 and CLz2 seem high, it is difficult to know whether they may be subject to synthesis processing.

[0114] Therefore, the voice clustering unit 253 calculates the similarity between each of the clusters CLx1, CLx2, CLy1, CLy2, CLz1, and CLz2 and all of the utterances (1) to (20) (see FIG. 23). As shown in FIG. 23, since the cluster CLx1 has three utterances, utterances (1), (2), and (3), the similarity between the cluster CLx1 and each of the utterances (1) to (3) is very high (see the dotted line frame). Similarly, the similarity between the cluster CLx2 and each of the utterances (11) to (13) is very high (see the dotted line frame). Similarly, since the cluster CLy1 has three utterances, utterances (4), (5), and (6), the similarity between the cluster CLy1 and each of the utterances (4) to (6) is very high (see the dotted line frame). Similarly, cluster CLy2 has three utterances, utterances (14), (15), and (16), so the similarity between cluster CLy2 and each of utterances (14) to (16) is very high (see dotted line frame). Similarly, cluster CLz1 has four utterances, utterances (7), (8), (9), and (10), so the similarity between cluster CLz1 and each of utterances (7) to (10) is very high (see dotted line frame). Similarly, cluster CLz2 has four utterances, utterances (17), (18), (19), and (20), so the similarity between cluster CLz2 and each of utterances (17) to (20) is very high (see dotted line frame).

[0115] Furthermore, the voice clustering unit 253 calculates a correlation coefficient G indicating the degree of correlation between the clusters using the calculation results of the similarities between each cluster CLx1, CLx2, CLy1, CLy2, CLz1, and CLz2 and all utterances (1) to (20) and a predetermined evaluation function ES (cluster, utterance number, similarity). The correlation coefficient G is calculated from the output value of the predetermined evaluation function ES (cluster, utterance number, similarity), and takes a value of, for example, 0 to 1. In other words, if the correlation coefficient G is equal to or greater than a predetermined correlation threshold value stored in advance in the ROM 22 or RAM 23, the voice clustering unit 253 performs synthesis processing on each cluster corresponding to the correlation coefficient G. On the other hand, if the correlation coefficient G is less than the above-mentioned predetermined correlation threshold value, the voice clustering unit 253 does not perform synthesis processing on each cluster corresponding to the correlation coefficient G.

[0116] Fig. 22 shows a graph illustrating the characteristics of the vertical axis (similarity) against the horizontal axis (utterance number) using the similarity explained with reference to Fig. 23. As can be seen from the behavior of the similarity (score) between the clusters CLx1, CLx2, CLy1, CLy2, CLz1, and CLz2 and each utterance, the correlation coefficient G between the clusters that exceeds a predetermined correlation threshold is the correlation coefficient between the clusters CLx1 and CLx2 and the correlation coefficient between the clusters CLy1 and CLy2. In other words, the voice clustering unit 253 performs cluster synthesis processing only for the pair of clusters CLx1 and CLx2 and the pair of clusters CLy1 and CLy2.

[0117] Next, an example of an operation procedure of the minutes generating device 20 according to the third embodiment when data division processing is required will be described with reference to Fig. 24. Fig. 24 is a flowchart showing an example of a basic operation procedure of the minutes generating device 20 according to the third embodiment when data division processing is required. Each process shown in Fig. 24 is mainly executed by the voice clustering unit 253 of the processor 25 of the minutes generating device 20. The minutes generating device 20 executes each process shown in Fig. 18 as a subroutine of the cluster synthesis process of steps St33 and St53 shown in Fig. 24. In the description of Fig. 24, the same processes as those in Fig. 18 are given the same step numbers, and the description is simplified or omitted, and different contents are described.

[0118] In Fig. 24, the processor 25 divides the audio data included in the entire input audio file in the audio clustering unit 253 (St51). The processor 25 executes the series of processes from step St31 to step St33 described with reference to Fig. 18 in the audio clustering unit 253 for each piece of audio data divided in step St51. The processor 25 determines whether or not the series of processes from step St31 to step St33 has been executed for all the audio data divided in step St51 (St52). The processor 25 repeats the execution of the series of processes from step St31 to step St33 for the remaining audio data divided in step St51 until the series of processes from step St31 to step St33 has been executed for all the audio data divided in step St51.

[0119] On the other hand, if the processor 25 determines that the series of processes from step St31 to step St33 have been performed on all of the voice data divided in step St51 (St52, YES), it performs a cluster synthesis process (see Figure 18) between each of the voice data that constitutes all of the voice data divided in step St51 (St53, see Figures 19, 20, 22 and 23).

[0120] As described above, the minutes generating device 20 according to the third embodiment includes an input unit (e.g., a voice input unit 251) for inputting a voice file including utterances of multiple speakers, a feature extraction unit 252 for extracting features of each utterance in multiple speech sections included in the voice file, and a classification unit (e.g., a voice clustering unit 253) for grouping each utterance by speaker based on the feature of each utterance. When a predetermined condition is met, the classification unit performs group synthesis to synthesize the grouped groups into a group of one speaker.

[0121] As a result, the minutes generation device 20 can perform clustering using the features of the audio data of each utterance contained in the input audio file, regardless of whether information regarding the number of speakers at a meeting has been input in advance, and can perform cluster synthesis processing when certain conditions are met, thereby performing appropriate clustering (speaker classification) to narrow down and estimate the number of speakers with greater accuracy.

[0122] In addition, in the minutes creation device 20, the classification unit derives the similarity (for example, brute-force calculation of similarity, see FIG. 23) between multiple groups having one or more utterances of audio data for each speaker and the audio data of multiple utterance sections included in the audio file as group synthesis, and derives a correlation coefficient between the groups using the derived similarity. In addition, the classification unit determines that a predetermined condition is met when the correlation coefficient is equal to or greater than a predetermined threshold, and synthesizes two or more groups whose correlation coefficient is equal to or greater than a predetermined threshold into a group of one speaker (i.e., cluster synthesis process). This allows the minutes creation device 20 to perform cluster synthesis process with high accuracy using not only the similarity between clusters, but also the correlation coefficient based on the similarity between the clusters and the utterance audio data.

[0123] The minutes generating device 20 further includes an utterance extraction unit (e.g., feature extraction unit 252) that analyzes the audio file and extracts the number of utterances and the utterance length of each utterance included in the audio file, and a division unit (e.g., feature extraction unit 252) that divides the audio file into a plurality of utterance sections based on the number of utterances and the utterance length of each utterance. As a result, even if the utterance section included in the audio file is long, the minutes generating device 20 can adaptively divide the audio data of the input audio file based on the number of utterances and the utterance length of each utterance, and therefore can perform highly accurate clustering.

[0124] In addition, in the minutes generating device 20, the classification unit groups each utterance in each divided speech section by speaker. When a predetermined condition is satisfied, the classification unit uses the grouping process results for each divided speech section to perform group synthesis between the grouping process results for different speech sections. As a result, even when the same speaker speaks in a time-discontinuous manner, the minutes generating device 20 can appropriately perform synthesis process using the clustering process results (i.e., clusters) between different divided voice data, thereby accurately improving the estimation accuracy of the number of speakers (in other words, the accuracy of narrowing down the number of speakers).

[0125] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, corrections, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure. In addition, the components in the various embodiments described above may be arbitrarily combined within the scope of the invention. [Industrial Applicability]

[0126] INDUSTRIAL APPLICABILITY The present disclosure is useful as an utterance classification device and an utterance classification method that classify utterances with high accuracy while estimating the number of speakers participating in a conference, regardless of whether information regarding the number of speakers in the conference is input in advance. [Explanation of symbols]

[0127] 10, 30, 40 PCS 11 Interface 12 Imaging unit 13, 25 processor 14, 24 Storage 15. Communication Devices 16 Operating Devices 17 Sound Recording Device 18 Display 19 External Interface 21 Communications Department 22 ROM 23 RAM 50 Speaker Classification System 251 Audio input section 252 Feature Extraction Unit 253 Voice Clustering Department 254 Minutes Generation Output Unit MIC sound pickup device

Claims

1. an input unit for inputting an audio file including speech from multiple speakers; a feature extraction unit that extracts features of each utterance in a plurality of speech sections included in the audio file; A classification unit that classifies each of the utterances into groups for each speaker based on a feature amount of each of the utterances, The classification unit performs group synthesis to synthesize the plurality of groups into one speaker group when a predetermined condition is satisfied, The classification unit performs the group synthesis as follows: deriving a similarity between a plurality of groups each having voice data of one or more utterances for each speaker and the voice data of the plurality of speech sections included in the voice file; deriving a correlation coefficient between the groups using the derived result of the similarity; When the correlation coefficient is equal to or greater than a predetermined threshold, it is determined that the predetermined condition is satisfied, and two or more groups for which the correlation coefficient is equal to or greater than the predetermined threshold are synthesized as one speaker group. Speech classifier.

2. an utterance extraction unit that analyzes the audio file and extracts the number of utterances and the length of each utterance included in the audio file; a division unit that divides the audio file into a plurality of speech sections based on the number of utterances and the length of each utterance. The speech classification device of claim 1 .

3. The classification unit includes: For each of the divided speech sections, the utterances in the speech section are grouped by speaker; performing the group synthesis between the processing results of the grouping for the divided utterance sections when the predetermined condition is satisfied, using the processing results of the grouping for the divided utterance sections; The speech classification device of claim 2 .

4. A method for speech classification executed by a speech classification device that is a computer, comprising: A step of inputting an audio file including utterances from multiple speakers; extracting features of each utterance in a plurality of speech sections included in the audio file; classifying the utterances into groups for each speaker based on the feature amount of each utterance; and performing group synthesis of synthesizing the plurality of groups into one speaker group when a predetermined condition is satisfied, As the group synthesis, deriving a similarity between a plurality of groups each having voice data of one or more utterances for each speaker and the voice data of the plurality of speech sections included in the voice file; deriving a correlation coefficient between the groups using the derived result of the similarity; When the correlation coefficient is equal to or greater than a predetermined threshold, it is determined that the predetermined condition is satisfied, and two or more groups for which the correlation coefficient is equal to or greater than the predetermined threshold are synthesized as one speaker group. Speech classification methods.

Images