Classification device, classification method, recording medium, and information display device

Abstract

This classification device comprises: a first classification means that classifies a plurality of section voices generated by dividing an input voice into a plurality of sections into a plurality of first section voices having a first sound quality and a plurality of second section voices having a second sound quality different from the first sound quality; and a second classification means that classifies the plurality of second section voices as any of a plurality of groups based on the plurality of first section voices or a new group different from the plurality of groups.

Description

Classification device, classification method, recording medium, and information display device

[0001] The present disclosure relates to the technical fields of a classification device, a classification method, a recording medium, and an information display device.

[0002] As an example of this type of device, a device has been proposed that identifies a portion of voice data corresponding to a desired keyword from voice data generated by recording a telephone call without requiring voice recognition processing (see Patent Document 1). Other technologies related to this disclosure include Patent Documents 2 to 4.

[0003] JP 2011-053563 A JP 2022-144927 A JP 2015-040931 A JP 2013-228472 A

[0004] An object of this disclosure is to provide a voice processing device, a voice processing method, a recording medium, and an information display device that aim to improve upon the techniques described in prior art documents.

[0005] One aspect of the classification device includes a first classification means that classifies a plurality of section sounds generated by dividing an input voice into a plurality of sections into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality, and a second classification means that classifies the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups.

[0006] In one aspect of the classification method, a computer classifies a plurality of section sounds generated by dividing an input voice into a plurality of sections into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality, and classifies the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups.

[0007] One aspect of the recording medium has recorded thereon a computer program that causes a computer to execute a classification method that classifies a plurality of section sounds generated by dividing an input voice into a plurality of sections into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality, and classifies the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups.

[0008] One aspect of the information display device comprises a first classification means for classifying a plurality of section sounds generated by dividing a first input voice into a plurality of sections into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality; a second classification means for classifying the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups; a first generation means for generating first speaker information indicating a speaker of each of the plurality of section sounds based on the plurality of groups and the new group; a second generation means for generating character information indicating a character string corresponding to the at least one section sound based on at least one section sound of the plurality of section sounds; and a display means for displaying the speaker indicated by the generated first speaker information and the character string indicated by the generated character information.

[0009] FIG. 1 is a block diagram showing an example of the configuration of a classification device; FIG. 2 is a block diagram showing another example of the configuration of a classification device; FIG. 3 is a flowchart showing the operation of a classification device according to the present disclosure; FIG. 4 is a diagram showing a concept of speech detection processing; FIG. 5 is a diagram showing an example of similarity; FIG. 6 is a diagram showing an example of a display screen; FIG. 7 is a diagram showing an example of a speaker input screen; and FIG. 8 is a diagram showing another example of a display screen.

[0010] First Embodiment An embodiment relating to a classification device, a classification method, and a recording medium will be described with reference to Fig. 1. In the following, the classification device, the classification method, and the recording medium will be described using a classification device 10.

[0011] 1 , the classification device 10 includes a classification unit 11 and a clustering unit 12. The classification unit 11 classifies a plurality of section sounds into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality. Note that the first sound quality may be higher in quality than the second sound quality.

[0012] The plurality of section sounds may be generated by dividing the input sound into a plurality of sections. Existing technology can be applied to the method of dividing the input sound into a plurality of section sounds. One example is a method of dividing the input sound into a plurality of section sounds in predetermined time units (for example, in units of 2 seconds). Another example is a method of dividing the input sound into a plurality of section sounds in predetermined time units with overlap. The input sound may be acquired by a sound collection device such as a microphone.

[0013] The clustering unit 12 classifies each of the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups. Here, "classifying" is not limited to dividing according to a predetermined standard (for example, according to a predetermined correct answer), but also includes dividing based on the similarity between data (for example, between one section sound and another section sound). In other words, "classification" according to this embodiment may include clustering.

[0014] The plurality of groups based on the plurality of first section sounds may be generated (or set) by, for example, the following method. The classifier 11 may extract features of each of the plurality of first section sounds. An example of the section sound features (i.e., sound features) is an i-vector calculated using Mel-Frequency Cepstrum Coefficients (MFCC), which are acoustic features. Another example of the section sound features is an x-vector calculated using a DNN (Deep Neural Network).

[0015] The classification unit 11 may calculate the similarity between one first section speech and another first section speech based on the features of a plurality of first section speeches. An example of the similarity between two speeches is the similarity of i-vectors or x-vectors calculated by PLDA (Probabilistic Linear Discriminant Analysis). The similarity may also be referred to as an index value indicating the degree of similarity. The extraction of the features of the section speeches and the calculation of the similarity may be performed by the clustering unit 12 instead of the classification unit 11.

[0016] The clustering unit 12 may perform clustering processing on the plurality of first section sounds based on similarities between the plurality of first section sounds. An example of the clustering processing is processing using hierarchical agglomerative clustering (AHC). The plurality of first section sounds may be grouped by the clustering processing, thereby generating (or setting) the plurality of groups (i.e., the plurality of groups based on the plurality of first section sounds).

[0017] The clustering unit 12 may classify each of the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups, for example, by the following method. The clustering unit 12 may extract features of each of the plurality of second section sounds. The features of each of the plurality of second section sounds may be extracted by the same method as the features of the first section sound.

[0018] The clustering unit 12 may calculate a similarity between the one second section sound and the one or more first section sounds based on the feature of the one second section sound and the feature of one or more first section sounds included in one of the multiple groups based on the multiple first section sounds. The clustering unit 12 may further calculate a similarity between the one second section sound and the one or more first section sounds based on the feature of the one second section sound and the feature of one or more first section sounds included in another of the multiple groups based on the multiple first section sounds.

[0019] The clustering unit 12 may identify the maximum similarity among the similarities calculated as described above. The clustering unit 12 may identify a combination of one second section speech and one first section speech that results in the identified maximum similarity, thereby identifying a group including the one first section speech (i.e., a group based on multiple first section speeches). If the identified maximum similarity is equal to or greater than a predetermined threshold, the clustering unit 12 may classify the one second section speech into the identified group. On the other hand, if the identified maximum similarity is less than the predetermined threshold, the clustering unit 12 may classify the one second section speech into a new group different from the multiple groups based on the multiple first section speeches.

[0020] In this way, the classification device 10 performs a classification method in which a plurality of section sounds generated by dividing the input sound into a plurality of sections are classified into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality, and each of the plurality of second section sounds is classified into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups.

[0021] The classification device 10 may be realized by a computer reading a computer program recorded on a recording medium. In this case, it can be said that the recording medium has recorded thereon a computer program that causes the computer to execute a classification method that classifies a plurality of section sounds generated by dividing an input speech into a plurality of sections into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality, and classifies each of the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups.

[0022] (Technical Effect) Research by the inventors of the present application has revealed the following: A first feature of a first speech associated with one speaker and a second feature of a second speech associated with the same speaker are similar to each other because they are spoken by the same speaker. A first feature of a first speech associated with one speaker and a third feature of a third speech associated with another speaker are dissimilar to each other because they are spoken by different speakers. Therefore, the similarity between the first speech and the second speech based on the first and second features is higher than the similarity between the first speech and the third speech based on the first and third features. In other words, the similarity between the first speech and the third speech based on the first and third features is lower than the similarity between the first speech and the second speech based on the first and second features. When the speech has high sound quality, the similarity when the two speeches are spoken by the same speaker is clearly different from the similarity when the two speeches are spoken by different speakers.

[0023] For example, if the sound quality of the first voice is high and the sound quality of the second voice is low, the similarity between the first voice and the second voice will be lower than when the sound quality of the first voice and the second voice are high. On the other hand, if the sound quality of the first voice is high and the sound quality of the third voice is low, the similarity between the first voice and the third voice will be higher than when the sound quality of the first voice and the third voice are high. As a result, the second feature of the second voice may be neither similar nor dissimilar to the first feature of the first voice. Similarly, the third feature of the third voice may be neither similar nor dissimilar to the first feature of the first voice. For this reason, when clustering processing is performed on multiple voices with different sound qualities, there is a possibility that the multiple voices will not be properly separated by speaker.

[0024] In contrast, in the classification device 10, the classification unit 11 classifies multiple section speeches into multiple first section speeches having a first sound quality and multiple second section speeches having a second sound quality different from the first sound quality. Then, the clustering unit 12 classifies each of the multiple second section speeches into one of multiple groups based on the multiple first section speeches or into a new group different from the multiple groups. In other words, in the classification device 10, after each of the multiple section speeches is classified by sound quality, the multiple section speeches are grouped by sound quality. Therefore, the classification device 10 can appropriately classify multiple speeches by speaker.

[0025] Second Embodiment An embodiment relating to a classification device, a classification method, a recording medium, and an information display device will be described with reference to Fig. 2 to Fig. 7. In the following, the classification device, the classification method, the recording medium, and the information display device will be described using a classification device 20.

[0026] 2 , the classification device 20 includes a calculation device 21, a storage device 22, and a communication device 23. The classification device 20 may further include an input device 24 and an output device 25. Note that the classification device 20 does not necessarily include at least one of the input device 24 and the output device 25. In the classification device 20, the calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0027] The arithmetic device 21 may include, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a TPU (Tensor Processing Unit), and a quantum processor.

[0028] The storage device 22 may include, for example, at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and an optical disk array. In other words, the storage device 22 may include a non-transitory recording medium. The storage device 22 is capable of storing desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic device 21. The storage device 22 may temporarily store data that is temporarily used by the arithmetic device 21 when the arithmetic device 21 is executing a computer program.

[0029] The communication device 23 may be capable of communicating with devices external to the classification device 20. The communication device 23 may perform wired communication or wireless communication.

[0030] The input device 24 is a device capable of accepting information input to the classification device 20 from outside. The input device 24 may include an operation device (e.g., a keyboard, a mouse, a touch panel, etc.) that can be operated by an operator of the classification device 20. The input device 24 may include a recording medium reading device that can read information recorded on a recording medium that is detachable from the classification device 20, such as a USB (Universal Serial Bus) memory. Note that when information is input to the classification device 20 via the communication device 23 (in other words, when the classification device 20 obtains information via the communication device 23), the communication device 23 may function as an input device.

[0031] The output device 25 is a device capable of outputting information to the outside of the classification device 20. The output device 25 may output visual information such as text or images, auditory information such as sound, or tactile information such as vibration, as the information. The output device 25 may include, for example, at least one of a display, a speaker, a printer, and a vibration motor. The output device 25 may be capable of outputting information to a recording medium that is detachable from the classification device 20, such as a USB memory. Note that when the classification device 20 outputs information via the communication device 23, the communication device 23 may function as the output device.

[0032] The calculation device 21 may have a classification unit 211, a clustering unit 212, a speaker information generation unit 213, a character information generation unit 214, and a display control unit 215 as logically realized functional blocks or as physically realized processing circuits.

[0033] At least one of the classification unit 211, the clustering unit 212, the speaker information generation unit 213, the character information generation unit 214, and the display control unit 215 may be realized in a form in which a logical functional block and a physical processing circuit (i.e., hardware) are mixed. When at least some of the classification unit 211, the clustering unit 212, the speaker information generation unit 213, the character information generation unit 214, and the display control unit 215 are functional blocks, at least some of the classification unit 211, the clustering unit 212, the speaker information generation unit 213, the character information generation unit 214, and the display control unit 215 may be realized by the calculation device 21 executing a predetermined computer program.

[0034] The arithmetic device 21 may acquire (in other words, read) the predetermined computer program from the storage device 22. The arithmetic device 21 may read the predetermined computer program stored in a computer-readable, non-transitory recording medium using a recording medium reading device (not shown) included in the classification device 20. The arithmetic device 21 may acquire (in other words, download or read) the predetermined computer program from a device (not shown) external to the classification device 20 via the communication device 23. Note that the recording medium for recording the predetermined computer program executed by the arithmetic device 21 may be at least one of an optical disk, a magnetic medium, a magneto-optical disk, a semiconductor memory, and any other medium capable of storing a program.

[0035] The "classification unit 211" and the "clustering unit 212" are components corresponding to the "classification unit 11" and the "clustering unit 12" in the first embodiment, respectively.

[0036] The operations of the classification unit 211 and the clustering unit 212 will be described with reference to Figures 3 to 5. Note that, for the classification unit 211 and the clustering unit 212, descriptions that overlap with those in the first embodiment will be omitted as appropriate.

[0037] 3 , the classification unit 211 detects a section in which speech is present from the input speech (in other words, a section including an acoustic signal corresponding to speech) (step S101). A section in which speech is present will hereinafter be referred to as a "speech section" as appropriate. In the processing of step S101, the classification unit 211 may detect the speech section by performing speech detection processing on the input speech. Note that various existing aspects can be applied to the speech detection processing, and therefore detailed description thereof will be omitted. Note that when detecting speech related to human speech, the speech detection processing may be referred to as speech detection processing.

[0038] The input audio may include audio and non-audio sounds (for example, at least one of sounds, music, and animal cries). The input audio may also include silent intervals. As a result of the processing in step S101, audio intervals VS1, VS2, VS3, VS4, and VS5 may be detected from the input audio, as shown in FIG. 4, for example. The intervals BS1, BS2, BS3, and BS4 in FIG. 4 are examples of intervals other than audio intervals. The non-audio intervals may include, for example, intervals where sounds other than audio exist and silent intervals.

[0039] The classification unit 211 may divide one input speech into a plurality of section speeches by extracting detected speech segments (e.g., speech segments VS1, VS2, VS3, VS4, and VS5) from the input speech. In this case, one speech segment may correspond to one section speech. Alternatively, the classification unit 211 may divide one input speech to generate a plurality of section speeches, and then perform speech detection processing on each of the plurality of section speeches as the input speech. In this case, the classification unit 211 may extract a speech segment from each of the plurality of section speeches. In other words, the classification unit 211 may further divide one section speech by extracting a speech segment from the one section speech.

[0040] After the process of step S101, the classification unit 211 extracts speaker features from the detected speech segments (for example, speech segments VS1, VS2, VS3, VS4, and VS5) (step S102). The speaker features may be at least one of an i-vector and an x-vector. The classification unit 211 may extract speaker features using a learning model that, when a speech segment is input, outputs speaker features from the input speech segment. Such a learning model may be constructed by machine learning using speech data corresponding to the speech segment.

[0041] Next, the classification unit 211 calculates the similarity between each of the multiple speech segments based on the speaker features extracted in the processing of step S102 (step S103). Note that a speech segment can be considered to be one segment of one input speech. Therefore, a speech segment may also be referred to as segment speech. Therefore, calculating the similarity between each of the multiple speech segments can be considered to be calculating the similarity between each of the multiple segment speeches. Note that the similarity may be the similarity of an i-vector or x-vector calculated by PLDA. Note that the classification unit 211 may calculate the similarity using a learning model that, when two speaker features are input, outputs the similarity between two speech segments corresponding to the two input speaker features. Such a learning model may be constructed by machine learning using training data corresponding to the speaker features.

[0042] For example, the speaker features of the speech segments VS1, VS2, VS3, VS4, and VS5 are defined as speaker features SF1, SF2, SF3, SF4, and SF5, respectively. The calculation results of the similarity between each of the speech segments VS1, VS2, VS3, VS4, and VS5 can be represented in the table shown in Fig. 5. In Fig. 5, for example, the value at the position where the speaker feature SF1 on the vertical axis intersects with the speaker feature SF2 on the horizontal axis (specifically, "0.1") indicates the similarity between the speech segments VS1 and VS2. In this embodiment, the more similar the speaker feature of one speech segment is to the speaker feature of another speech segment, the larger the similarity value.

[0043] The classification unit 211 may calculate distance instead of similarity. That is, the classification unit 211 may calculate distance instead of similarity as an index indicating the degree of similarity between a speaker feature of one speech section and a speaker feature of another speech section. In this case, the value indicating the distance may be smaller as the speaker feature of one speech section and a speaker feature of another speech section are more similar.

[0044] After the processing of step S103, the classification unit 211 classifies the multiple voice segments into voice segments having a first sound quality and voice segments having a second sound quality different from the first sound quality (step S104). In the processing of step S104, the classification unit 211 may classify the multiple voice segments into voice segments having a first sound quality and voice segments having a second sound quality based on the sound quality of each of the multiple voice segments and the reference sound quality. For example, if the sound quality of a voice segment is higher than the reference sound quality, the classification unit 211 may classify the voice segment into a voice segment having the first sound quality. On the other hand, if the sound quality of a voice segment is lower than the reference sound quality, the classification unit 211 may classify the voice segment into a voice segment having the second sound quality. Note that if the sound quality of a voice segment and the reference sound quality are "equal," the voice segment may be treated as including either case. With this configuration, multiple voice segments can be easily classified into voice segments having the first sound quality and voice segments having the second sound quality. The reference sound quality may be, for example, an average value of the sound quality of the speech data used to construct a learning model related to speaker features. The first sound quality may be higher than the second sound quality.

[0045] For example, among the voice sections VS1, VS2, VS3, VS4, and VS5, the voice sections VS1, VS2, and VS3 are voice sections with high sound quality (in other words, voice sections with a first sound quality), and the voice sections VS4 and VS5 are voice sections with low sound quality (in other words, voice sections with a second sound quality). Furthermore, the speaker involved in the voice sections VS1 and VS3 is a first speaker, and the speaker involved in the voice sections VS2, VS4, and VS5 is a second speaker. In this case, in the processing of step S104, the classification unit 211 may classify the voice sections VS1, VS2, and VS3 as voice sections with the first sound quality, and may classify the voice sections VS4 and VS5 as voice sections with the second sound quality.

[0046] The clustering unit 212 performs a first clustering process using the plurality of voice segments classified as having the first sound quality in the process of step S104 (step S105). An example of the first clustering process is a process using hierarchical agglomerative clustering (AHC).

[0047] The first clustering process will be specifically described using the similarities shown in Fig. 5. The clustering unit 212 may identify a combination of speech segments that has the greatest similarity among the respective speech segments VS1, VS2, and VS3. In the example shown in Fig. 5, the value (i.e., "0.9") at the position where the speaker feature SF1 of the speech segment VS1 intersects with the speaker feature SF3 of the speech segment VS3 is the greatest. Therefore, the clustering unit 212 may identify the combination of the speech segment VS1 and the speech segment VS3 as the combination of speech segments that has the greatest similarity.

[0048] Next, the clustering unit 212 may determine whether the similarity between the voice section VS1 and the voice section VS3 is equal to or greater than a first threshold. Here, the first threshold is assumed to be "0.5." The similarity between the voice section VS1 and the voice section VS3 is "0.9," which is greater than the first threshold. Therefore, the clustering unit 212 may classify the voice section VS1 and the voice section VS3 into the same group (which may be referred to as a "cluster"). Here, the group including the voice sections VS1 and VS3 is referred to as group C1.

[0049] Next, the clustering unit 212 may calculate the similarity between the voice sections VS1 and VS3 and the voice section VS2. In this case, the clustering unit 212 may determine the similarity between the voice sections VS1 and VS3 and the voice section VS2 as the similarity between the voice sections VS1 and VS3 and the voice section VS2 by dividing the sum of the similarity between the voice sections VS1 and VS2 and the similarity between the voice sections VS2 and VS3 by 2.

[0050] 5, the value at the position where the speaker feature SF1 of the speech section VS1 intersects with the speaker feature SF2 of the speech section VS2 is "0.1". The value at the position where the speaker feature SF2 of the speech section VS2 intersects with the speaker feature SF3 of the speech section VS3 is "0.1". Therefore, the similarity between the speech sections VS1 and VS3 and the speech section VS2 is "(0.1 + 0.1) / 2 = 0.1".

[0051] The similarity between the voice segments VS1 and VS3 and the voice segment VS2 (i.e., 0.1) is smaller than the first threshold (i.e., 0.5). Therefore, the clustering unit 212 may classify the voice segment VS2 into a group different from the voice segments VS1 and VS3. Here, the group including the voice segment VS2 is referred to as group C2.

[0052] The groups C1 and C2 may be generated (or set) by performing a first clustering process using the voice segments VS1, VS2, and VS3 as voice segments having the first sound quality. Therefore, it can be said that the groups C1 and C2 are multiple groups based on the voice segments having the first sound quality.

[0053] After the process of step S105, the clustering unit 212 performs a second clustering process (step S106). In the process of step S106, the clustering unit 212 performs the second clustering process to classify the voice segments having the second sound quality into one of a plurality of groups (e.g., groups C1 and C2) based on the voice segments having the first sound quality, or into a new group different from the plurality of groups.

[0054] The second clustering process will be specifically described using the similarity shown in Fig. 5. As described above, the voice segments VS4 and VS5 correspond to examples of voice segments having the second sound quality. Therefore, by performing the second clustering process, the clustering unit 212 classifies each of the voice segments VS4 and VS5 into one of the above-described groups C1 and C2, or into a new group different from groups C1 and C2.

[0055] In the second clustering process, the clustering unit 212 may calculate the similarity between the voice section VS4 and each of the groups C1 and C2, and the similarity between the voice section VS5 and each of the groups C1 and C2. Here, the similarity between the voice section and a group may be calculated based on the similarity between the voice section and one or more voice sections included in the group.

[0056] For example, the similarity between the voice section VS4 and the group C1 may be calculated based on the similarity between the voice section VS4 and each of the voice sections VS1 and VS3 included in the group C1. In this case, the clustering unit 212 may determine the similarity between the voice section VS4 and the group C1 as the similarity between the voice section VS4 and the group C1 by dividing the sum of the similarity between the voice section VS1 and the voice section VS4 and the similarity between the voice section VS3 and the voice section VS4 by 2.

[0057] In FIG. 5, the value at the position where the speaker feature SF1 of the speech section VS1 intersects with the speaker feature SF4 of the speech section VS4 is "0.5". The value at the position where the speaker feature SF3 of the speech section VS3 intersects with the speaker feature SF4 of the speech section VS4 is "0.4". Therefore, the similarity between the speech section VS4 and the group C1 is "(0.5 + 0.4) / 2 = 0.45". By a similar calculation, the similarity between the speech section VS4 and the group C2 is "0.4". The similarity between the speech section VS5 and the group C1 is "0.4". The similarity between the speech section VS5 and the group C2 is "0.5".

[0058] The clustering unit 212 may identify the combination with the highest similarity among the calculated similarities. Here, the combination of the voice section VS5 and the group C2 may be identified as the combination with the highest similarity. Next, the clustering unit 212 may determine whether the similarity between the voice section VS5 and the group C2 is equal to or greater than a second threshold. The second threshold may be the same as or different from the first threshold (i.e., the threshold related to the first clustering process). Here, the second threshold is assumed to be "0.5."

[0059] The similarity between the voice section VS5 and group C2 is "0.5," which is equal to the second threshold. Therefore, the clustering unit 212 may classify the voice section VS5 into group C2. If the similarity between the voice section VS5 and group C2 is smaller than the second threshold, the clustering unit 212 may classify the voice section VS5 into a new group C3 that is different from groups C1 and C2.

[0060] Next, the clustering unit 212 may recalculate the similarity between the voice section VS4 and each of the groups C1 and C2. Because group C2 includes voice sections VS2 and VS5, the similarity between the voice section VS4 and group C2 is "0.55." The clustering unit 212 may determine whether the similarity between the voice section VS4 and group C2 is equal to or greater than a second threshold. The similarity between the voice section VS4 and group C2 is "0.55," which is greater than the second threshold. Therefore, the clustering unit 212 may classify the voice section VS4 into group C2.

[0061] As a result of the first clustering process and the second clustering process, the voice sections VS1 and VS3 are classified into group C1, and the voice sections VS2, VS4, and VS5 are classified into group C2. As described above, the speaker involved in the voice sections VS1 and VS3 is the first speaker, and the speaker involved in the voice sections VS2, VS4, and VS5 is the second speaker. Therefore, it can be seen that the voice sections VS1, VS2, VS3, VS4, and VS5 have been appropriately divided by speaker by the first clustering process and the second clustering process.

[0062] The speaker information generation unit 213 generates speaker information indicating speakers of a speech section (e.g., speech sections VS1, VS2, VS3, and VS4) based on the results of the clustering process (specifically, the first clustering process and the second clustering process) performed by the clustering unit 212. The speaker information generation unit 213 may generate speaker information indicating speakers in a manner that allows one speaker to be distinguished (or identified) from other speakers. The speaker information generation unit 213 may represent the speakers as, for example, "speaker 1" and "speaker 2," etc.

[0063] The speaker information generation unit 213 may generate speaker information in which the display mode of a speaker of one speech section (i.e., a speech section having a first sound quality) classified into one group (e.g., one of groups C1 and C2) by the first clustering process is different from that of a speaker of another speech section (i.e., a speech section having a second sound quality) classified into the same group by the second clustering process. In this case, the speaker information may include characters indicating a speaker who is the speaker of one speech section and characters suggesting the speaker as the speaker of another speech section. For example, if the characters indicating a speaker are "Speaker 1," the "characters suggesting a speaker" may be either "Maybe Speaker 1" or "Speaker 1 (guessed)." This configuration allows the likelihood of clustering to be presented to the user of the classification device 20. If a speech section is classified into a new group (e.g., a group different from groups C1 and C2) by the second clustering process, the speaker information generation unit 213 may not generate speaker information indicating the speaker of the speech section classified into the new group. Alternatively, if a speech section is classified into a new group (e.g., a group different from groups C1 and C2) by the second clustering process, the speaker information generation unit 213 may represent the speaker of the speech section classified into the new group as, for example, "Unknown." Note that "Unknown" is just an example and is not limited to this.

[0064] The speaker information generation unit 213 may identify a specific person corresponding to a single speaker by performing at least one of speaker recognition processing and voice authentication processing on one or more speech segments classified as speech related to a single speaker based on the result of the clustering processing by the clustering unit 212. In this case, the speaker information generation unit 213 may represent the speaker by a specific person's name.

[0065] 3, the text information generation unit 214 generates text information indicating a character string corresponding to the speech, based on the speech related to the speech section detected in the processing of step S101. Note that various existing methods can be applied to the method of generating text information indicating a character string from speech (so-called transcription), and therefore detailed description thereof will be omitted.

[0066] The display control unit 215 controls the output device 25, which can function as a display device, to display the speakers indicated by the speaker information generated by the speaker information generation unit 213 and the character strings indicated by the character information generated by the character information generation unit 214 in association with each other. As a result, the output device 25 may display, for example, the image shown in Fig. 6. The display control unit 215 may control the output device 25 to display an image (e.g., an icon) instead of or in addition to the character strings indicating the speakers. Since the classification device 20 displays the speakers indicated by the speaker information and the character strings indicated by the character information, it may also be referred to as an information display device.

[0067] In addition, when the speaker information generation unit 213 does not perform speaker recognition processing and voice authentication processing, the display control unit 215 may control the output device 25 to display, for example, a speaker input screen shown in FIG. 7 . A user of the classification device 20 may input, via the input device 24, the name of at least one speaker among the speakers displayed on the speaker input screen. The speaker information generation unit 213 may change the speaker indicated by the speaker information based on the input name. The display control unit 215 may control the output device 25 to display the speaker indicated by the speaker information changed by the speaker information generation unit 213 and the character string indicated by the character information in association with each other.

[0068] (Technical Effect) When a clustering process (corresponding to the first clustering process described above) is performed on multiple voices with different sound qualities, there is a possibility that the multiple voices will not be properly separated by speaker. For example, when the first clustering process described above is performed on voice segments VS1, VS2, VS3, VS4, and VS5, the voice segments VS1 and VS2 are classified into group C1, the voice segment VS2 is classified into group C2, and the voice segments VS4 and VS5 are classified into group C3, which is different from groups C1 and C2. Note that a description of the calculation process will be omitted.

[0069] As described above, the speaker involved in the voice sections VS1 and VS3 is Speaker 1, and the speaker involved in the voice sections VS2, VS4, and VS5 is Speaker 2. Therefore, when the first clustering process described above is performed on the voice sections VS1, VS2, VS3, VS4, and VS5, it is found that the voice sections VS1, VS2, VS3, VS4, and VS5 are not divided by speaker.

[0070] In contrast, in the classification device 20, the classification unit 211 classifies multiple voice segments into voice segments having a first sound quality and voice segments having a second sound quality. The clustering unit 212 performs a first clustering process on the multiple voice segments classified as the voice segments having the first sound quality. The clustering unit 212 then classifies the voice segments classified as the voice segments having the second sound quality into one of multiple groups based on the voice segments having the first sound quality, or into a new group different from the multiple groups. In this way, by classifying multiple voice segments by sound quality and then performing a clustering process for each sound quality, multiple voice segments with different sound qualities can be appropriately separated for each speaker.

[0071] Furthermore, by performing the first clustering process on a plurality of speech segments having the first sound quality, which is higher than the second sound quality, it is possible to perform grouping more appropriately than when the first clustering process is performed on a plurality of speech segments having the second sound quality. Character strings corresponding to speech are displayed in association with speakers, which is useful in practice.

[0072] Third Embodiment An embodiment relating to a classification device, a classification method, a recording medium, and an information display device will be described with reference to FIG. 8 . Hereinafter, the classification device, the classification method, the recording medium, and the information display device will be described using the classification device 20. In the third embodiment, the operation of the display control unit 215 will be mainly described. The other configurations may be the same as those in the second embodiment described above. Descriptions of the third embodiment that overlap with those of the second embodiment will be omitted as appropriate.

[0073] The input device 24 may include a microphone. The classification unit 211 may detect a speech segment by performing a speech detection process on sounds sequentially acquired by the microphone serving as the input device 24. The character information generation unit 214 may generate first character information indicating a character string corresponding to the speech, based on the speech related to the detected speech segment. The display control unit 215 may control the output device 25 to display the character string indicated by the first character information generated by the character information generation unit 214. As a result, the output device 25 may display, for example, an image shown in FIG. 8 . In this case, since the speaker information generation unit 213 has not generated speaker information, information indicating the speaker (e.g., a character or an image) is not displayed.

[0074] In parallel with the above process, the classification unit 211 may extract speaker features for each of the multiple voice segments. The classification unit 211 may calculate a similarity between each of the multiple voice segments based on the extracted speaker features. The classification unit 211 may then classify the multiple voice segments into voice segments having a first sound quality and voice segments having a second sound quality. The clustering unit 212 may perform a first clustering process on the multiple voice segments classified as the voice segments having the first sound quality. The clustering unit 212 may perform a second clustering process to classify the voice segments classified as the voice segments having the second sound quality into one of multiple groups based on the voice segments having the first sound quality, or into a new group different from the multiple groups.

[0075] The speaker information generation unit 213 may generate first speaker information indicating a speaker of each of the multiple speech segments based on the results of the clustering process (specifically, the first clustering process and the second clustering process). The display control unit 215 may control the output device 25 to display the speaker indicated by the first speaker information generated by the speaker information generation unit 213 and the character string indicated by the first character information generated by the character information generation unit 214. As a result, the output device 25 may display, for example, an image shown in FIG. 6 .

[0076] The classification device 20 may treat a sound acquired by a microphone in a first period as one input sound, and a sound acquired by a microphone in a second period following the first period as another input sound. Alternatively, the classification device 20 may treat a sound acquired by a microphone in the first period and a sound acquired by a microphone in a second period following the first period together as one input sound. In this case, the one input sound is updated when a new sound is acquired by the microphone.

[0077] The operation of the classification device 20 when it combines a sound acquired by a microphone in a first period and a sound acquired by a microphone in a second period following the first period and treats them as a single input sound will be described.

[0078] The classification unit 211 may detect a voice segment by performing a voice detection process on one input voice, which is sound acquired by a microphone during a first time period. The classification unit 211 may extract speaker features for each of a plurality of voice segments. The classification unit 211 may calculate a similarity between each of the plurality of voice segments based on the extracted speaker features. The classification unit 211 may then classify the plurality of voice segments into voice segments having a first sound quality and voice segments having a second sound quality. The clustering unit 212 may perform a first clustering process on the plurality of voice segments classified as the voice segment having the first sound quality. The clustering unit 212 may perform a second clustering process to classify the voice segment classified as the voice segment having the second sound quality into one of a plurality of groups based on the voice segment having the first sound quality, or into a new group different from the plurality of groups. The speaker information generation unit 213 may generate first speaker information indicating a speaker of each of the plurality of voice segments based on the result of the clustering process. The character information generating unit 214 may generate first character information indicating a character string corresponding to the voice, based on the voice related to the detected voice section. The display control unit 215 may control the output device 25 to display the speaker indicated by the first speaker information generated by the speaker information generating unit 213 and the character string indicated by the first character information generated by the character information generating unit 214.

[0079] When a sound captured by the microphone in a second period following the first period is input, the classification device 20 may update the input audio by combining the sound captured by the microphone in the first period with the sound captured by the microphone in the second period.

[0080] The classification unit 211 may detect a voice segment by performing a voice detection process on the updated input voice. In this case, the classification unit 211 may perform the voice detection process again on sounds acquired by the microphone during a first period included in the updated input voice. Note that the classification unit 211 does not need to perform the voice detection process on sounds acquired by the microphone during the first period included in the updated input voice. The classification unit 211 may extract speaker features for each of the multiple voice segments. The classification unit 211 may calculate a similarity between each of the multiple voice segments based on the extracted speaker features. Thereafter, the classification unit 211 may classify the multiple voice segments into voice segments having a first sound quality and voice segments having a second sound quality. The clustering unit 212 may perform a first clustering process on the multiple voice segments classified as voice segments having the first sound quality. The clustering unit 212 may perform a second clustering process to classify the voice segments classified as having the second sound quality into one of a plurality of groups based on the voice segments having the first sound quality, or into a new group different from the plurality of groups. The speaker information generation unit 213 may generate second speaker information indicating a speaker of each of the plurality of voice segments based on the result of the clustering process. The character information generation unit 214 may generate second character information indicating a character string corresponding to the voice, based on the voice related to the detected voice segment. The display control unit 215 may control the output device 25 to display the speaker indicated by the second speaker information generated by the speaker information generation unit 213 and the character string indicated by the second character information generated by the character information generation unit 214.

[0081] When the speaker of a character string included in both the first character information and the second character information (e.g., a character string corresponding to a sound included in a sound acquired by a microphone during a first period) differs between the speaker indicated by the first speaker information and the speaker indicated by the second speaker information, the display control unit 215 may control the output device 25, when the second speaker information is generated, to change the speaker displayed in association with the character string from the speaker indicated by the second speaker information to the speaker indicated by the second speaker information. As a result, the output device 25 may change the speaker displayed in association with the character string, for example, from "Speaker 1" to "Speaker 2."

[0082] (Technical Effect) According to the classification device 20 of this embodiment, it is possible to display a character string corresponding to an utterance in real time while the utterance is being made, which makes it possible to generate minutes of a meeting in real time, for example.

[0083] <Supplementary Notes> The following supplementary notes are further disclosed regarding the above-described embodiment.

[0084] (Supplementary Note 1) A classification device comprising: a first classification means for classifying a plurality of section sounds generated by dividing an input speech into a plurality of sections into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality; and a second classification means for classifying the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups.

[0085] (Supplementary Note 2) The second classifying means calculates a second index value indicating a degree of similarity between one second section sound and one group based on one or more first index values ​​indicating a degree of similarity between one second section sound among the plurality of second section sounds and one or more first section sounds of the plurality of first section sounds included in one group among the plurality of groups, among a plurality of first index values ​​each indicating a degree of similarity between the one section sound among the plurality of section sounds and each of the plurality of section sounds; and the second classifying means classifies the one second section sound into a group corresponding to a maximum second index value among the plurality of second index values ​​calculated for each of the plurality of groups, the maximum second index value being equal to or greater than a predetermined threshold. This is the classification device described in Supplementary Note 1.

[0086] (Supplementary Note 3) The second classifying means calculates a second index value indicating a degree of similarity between one second section sound and one group based on one or more first index values ​​indicating a degree of similarity between one second section sound among the plurality of second section sounds and one or more first section sounds of the plurality of first section sounds included in one group among the plurality of groups, among a plurality of first index values ​​each indicating a degree of similarity between the one section sound among the plurality of section sounds and each of the plurality of section sounds; and the second classifying means classifies the one second section sound into the new group when all of the plurality of second index values ​​calculated for each of the plurality of groups are smaller than a predetermined threshold. This is the classification device described in Supplementary Note 1 or 2.

[0087] (Supplementary Note 4) The classification device according to any one of Supplementary Notes 1 to 3, wherein the classification means classifies the plurality of section sounds into the plurality of first section sounds and the plurality of second section sounds based on a sound quality of each of the plurality of section sounds and a reference sound quality.

[0088] (Supplementary Note 5) The classification device according to any one of Supplementary Notes 1 to 4, wherein the second sound quality is lower than the first sound quality.

[0089] (Supplementary Note 6) A classification method in which a computer classifies a plurality of section sounds generated by dividing an input sound into a plurality of sections into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality, and classifies the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups.

[0090] (Supplementary Note 7) A recording medium having recorded thereon a computer program for causing a computer to execute a classification method of classifying a plurality of section sounds generated by dividing an input voice into a plurality of sections into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality, and classifying the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups.

[0091] (Supplementary Note 8) An information display device comprising: a first classification means for classifying a plurality of section sounds generated by dividing a first input speech into a plurality of sections, into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality; a second classification means for classifying the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups; a first generation means for generating first speaker information indicating a speaker of each of the plurality of section sounds based on the plurality of groups and the new group; a second generation means for generating character information indicating a character string corresponding to the at least one section sound, based on at least one section sound of the plurality of section sounds; and a display means for displaying the speaker indicated by the generated first speaker information and the character string indicated by the generated character information.

[0092] (Supplementary Note 9) The information display device described in Supplementary Note 8, wherein the display means displays a character string corresponding to a section sound that corresponds to a first section sound classified into a first group among the plurality of groups and the new group, among the at least one section sound, in association with a speaker indicated by the generated first speaker information.

[0093] (Appendix 10) The information display device described in claim 8 or 9, wherein the first speaker information indicates a first speaker who is the speaker of a first section audio classified into a second group among the plurality of groups, and indicates characters suggesting the first speaker as the speaker of a second section audio classified into the second group.

[0094] (Supplementary Note 11) When a plurality of other section sounds generated by dividing a second input sound into a plurality of sections are added to a plurality of section sounds generated by dividing the first input sound into a plurality of sections, the first classifying means classifies the other plurality of section sounds into a plurality of third section sounds having the first sound quality and a plurality of fourth section sounds having the second sound quality, the second classifying means classifies the second section sounds and the fourth section sounds into one of a plurality of groups based on the plurality of first section sounds and the plurality of third section sounds or into a new group different from the plurality of groups, and the first generating means generates second speaker information indicating speakers of the plurality of section sounds and the other plurality of section sounds based on the plurality of groups and the new group, An information display device as described in any one of Appendices 8 to 10, wherein if the speaker of a section audio corresponding to a character string indicated by the generated character information is different between the speaker indicated by the first speaker information and the speaker indicated by the second speaker information, when the second speaker information is generated, the display means changes the speaker associated with the character string from the speaker indicated by the first speaker information to the speaker indicated by the second speaker information.

[0095] This disclosure is not limited to the above-described embodiments, but may be modified as appropriate within the scope of the claims and the gist or idea of ​​the invention as can be read from the entire specification, and classification devices, classification methods, recording media, and information display devices that involve such modifications are also included in the technical scope of this disclosure.

[0096] 10, 20 Classification device 11, 211 Classification unit 12, 212 Clustering unit 213 Speaker information generation unit 214 Character information generation unit 215 Display control unit

Claims

1. a first classification means for classifying a plurality of section sounds generated by dividing an input voice into a plurality of sections into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality; a second classification means for classifying the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups; A classification device comprising:

2. the second classification means calculates a second index value indicating a degree of similarity between one second section sound and one of the groups based on one or more first index values ​​indicating a degree of similarity between one second section sound of the plurality of second section sounds and one or more first section sounds of the plurality of first section sounds included in one of the plurality of groups, among a plurality of first index values ​​each indicating a degree of similarity between the one second section sound of the plurality of second section sounds and each of the plurality of section sounds; The second classification means classifies the one second section speech into a group corresponding to a second index value that is the largest of the plurality of second index values ​​calculated for each of the plurality of groups and is equal to or larger than a predetermined threshold value. The classification device of claim 1 .

3. the second classification means calculates a second index value indicating a degree of similarity between one second section sound and one of the groups based on one or more first index values ​​indicating a degree of similarity between one second section sound of the plurality of second section sounds and one or more first section sounds of the plurality of first section sounds included in one of the plurality of groups, among a plurality of first index values ​​each indicating a degree of similarity between the one second section sound of the plurality of second section sounds and each of the plurality of section sounds; The second classification means classifies the one second section speech into the new group when all of the plurality of second index values ​​calculated for each of the plurality of groups are smaller than a predetermined threshold. The classification device according to claim 1 or 2.

4. The classification means classifies the plurality of section sounds into the plurality of first section sounds and the plurality of second section sounds based on a sound quality of each of the plurality of section sounds and a reference sound quality. The classification device according to claim 1 or 2.

5. The second sound quality is lower than the first sound quality. The classification device according to claim 1 or 2.

6. The computer classifying a plurality of section sounds generated by dividing the input voice into a plurality of sections into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality; Classifying the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups. Classification method.

7. On the computer, classifying a plurality of section sounds generated by dividing the input voice into a plurality of sections into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality; Classifying the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups. A recording medium on which a computer program for executing the classification method is recorded.

8. a first classification means for classifying a plurality of section sounds generated by dividing a first input voice into a plurality of sections into a plurality of first section sounds having a first sound quality and a plurality of second section sounds having a second sound quality different from the first sound quality; a second classification means for classifying each of the plurality of second section sounds into one of a plurality of groups based on the plurality of first section sounds or into a new group different from the plurality of groups; a first generating means for generating first speaker information indicating speakers of the plurality of section sounds based on the plurality of groups and the new group; a second generating means for generating character information indicating a character string corresponding to at least one section sound based on at least one section sound of the plurality of section sounds; a display means for displaying a speaker indicated by the generated first speaker information and a character string indicated by the generated character information; An information display device comprising:

9. The display means displays a character string corresponding to a section voice that corresponds to a first section voice classified into a first group among the plurality of groups and the new group, among the at least one section voice, in association with a speaker indicated by the generated first speaker information. The information display device according to claim 8.

10. The first speaker information indicates a first speaker who is a speaker of a first section voice classified into a second group among the plurality of groups, and indicates characters suggesting the first speaker as a speaker of a second section voice classified into the second group.

10. The information display device according to claim 8 or 9.