Audio processing device, audio processing method, recording medium, and information display device

Abstract

This audio processing device comprises: a classification means for classifying a plurality of audio sections, which were generated by dividing input audio into a plurality of sections, into an audio section having a feature of a speaker and an audio section not having said feature of the speaker, said classification being made in accordance with a reference for determining the presence / absence of the feature of the speaker; and an extraction means for extracting, from among the plurality of audio sections, the audio section classified into the audio section having the feature of the speaker.

Description

Audio processing device, audio processing method, recording medium, and information display device

[0001] The present disclosure relates to the technical fields of an audio processing device, an audio processing method, a recording medium, and an information display device.

[0002] One such device proposed is one that divides a first voice into a plurality of first voice segments and determines whether or not a voice corresponding to the first voice is included in one or more clusters based on a second voice, based on the similarity between the plurality of first voice segments and one or more clusters (see Patent Document 1).Other prior art documents related to this disclosure include Patent Documents 2 and 3.

[0003] International Publication No. 2020 / 246041 Japanese Patent Application Laid-Open No. 2020-067566 Japanese Patent Application Laid-Open No. 11-175090

[0004] An object of this disclosure is to provide a voice processing device, a voice processing method, a recording medium, and an information display device that aim to improve upon the techniques described in prior art documents.

[0005] One aspect of the speech processing device includes a classification means for classifying a plurality of section speeches generated by dividing an input speech into a plurality of sections into section speeches having the characteristics of the speaker and section speeches not having the characteristics of the speaker according to a criterion for determining whether or not the speaker has the characteristics, and an extraction means for extracting section speeches classified as section speeches having the characteristics of the speaker from the plurality of section speeches.

[0006] In one aspect of the speech processing method, a computer classifies a plurality of section speeches generated by dividing an input speech into a plurality of sections into section speeches having the characteristics of the speaker and section speeches not having the characteristics of the speaker according to a criterion for determining whether or not the section speeches have the characteristics of the speaker, and extracts section speeches classified as section speeches having the characteristics of the speaker from among the plurality of section speeches.

[0007] One aspect of the recording medium has recorded thereon a computer program that causes a computer to execute a speech processing method that classifies a plurality of section speeches generated by dividing input speech into a plurality of sections into section speeches that have the characteristics of the speaker and section speeches that do not have the characteristics of the speaker according to criteria for determining whether or not the speaker has the characteristics, and extracts section speeches classified as section speeches that have the characteristics of the speaker from the plurality of section speeches.

[0008] One aspect of an information display device comprises a classification means for classifying a plurality of section sounds generated by dividing an input voice into a plurality of sections into section sounds having the characteristics of the speaker and section sounds not having the characteristics of the speaker according to a criterion for determining whether or not the section sounds have the characteristics of the speaker; an extraction means for extracting one or more section sounds from the plurality of section sounds that have been classified as section sounds having the characteristics of the speaker; a first generation means for generating first speaker information indicating a speaker of each of the extracted one or more section sounds; a second generation means for generating character information indicating a character string corresponding to the at least one section sound based on at least one section sound of the plurality of section sounds; and a display means for displaying the speaker indicated by the generated first speaker information and the character string indicated by the generated character information.

[0009] 1 is a block diagram showing an example of a speech processing device; FIG. 2 is a block diagram showing another example of a speech processing device; FIG. 3 is a flowchart showing the operation of a speech processing device according to the present disclosure; FIG. 4 is a diagram showing a concept of speech detection processing; FIG. 5 is a diagram showing an example of similarity; FIG. 6 is a diagram showing an example of frequency distribution of similarity; FIG. 7 is a diagram showing another example of frequency distribution of similarity; FIG. 8 is a diagram showing an example of similarity that is a target of clustering processing; FIG. 9 is a diagram showing an example of a display screen; FIG. 10 is a diagram showing an example of a speaker input screen; and FIG. 11 is a diagram showing another example of a display screen.

[0010] First Embodiment An embodiment relating to a voice processing device, a voice processing method, and a recording medium will be described with reference to Fig. 1. In the following, the voice processing device 10 will be used to describe the embodiment relating to the voice processing device, the voice processing method, and the recording medium.

[0011] 1, a speech processing device 10 includes a classification unit 11 and an extraction unit 12. The classification unit 11 classifies each of a plurality of section sounds into section sounds having speaker characteristics and section sounds not having speaker characteristics, according to a criterion for determining whether or not a speaker characteristic is present.

[0012] The plurality of section sounds may be generated by dividing one input sound into a plurality of sections. Existing technology can be applied to the method of dividing one input sound into a plurality of section sounds. One example is a method of dividing one input sound into a plurality of section sounds in predetermined time units (for example, in units of two seconds). Another example is a method of dividing one input sound into a plurality of section sounds in predetermined time units with overlap. Note that the input sound may be acquired by a sound collection device such as a microphone.

[0013] The concept of "classifying" is not limited to separating according to a predetermined standard (e.g., according to a predetermined correct answer), but also includes separating based on the similarity between data (e.g., between one section audio and another section audio). "Speaker characteristics" may refer to the characteristics of a person's speech. Examples of audio that does not have speaker characteristics include audio that is different from speech, such as sounds, music, and animal sounds, as well as audio that includes the speech of multiple people. In other words, audio that does not have speaker characteristics may refer to at least one of audio that does not include the characteristics of a person's speech and audio that has relatively few characteristics of a person's speech (e.g., the ratio of the characteristics of a person's speech to the characteristics related to the section audio is less than a predetermined ratio). Audio that has speaker characteristics may refer to audio that has relatively many characteristics of a person's speech (e.g., the ratio of the characteristics of a person's speech to the characteristics related to the section audio is more than a predetermined ratio).

[0014] An example of a section speech feature (i.e., a speech feature) is an i-vector calculated using Mel-Frequency Cepstrum Coefficients (MFCC), which are acoustic features. Another example of a section speech feature is an x-vector calculated using a DNN (Deep Neural Network).

[0015] The extraction unit 12 extracts section sounds classified as section sounds having speaker characteristics from among the plurality of section sounds.

[0016] In other words, the speech processing device 10 performs a speech processing method in which each of a plurality of section speeches generated by dividing the input speech into a plurality of sections is classified into section speeches having the speaker's characteristics and section speeches not having the speaker's characteristics according to a criterion for determining whether or not the speaker's characteristics are present, and the section speeches classified as section speeches having the speaker's characteristics are extracted from the plurality of section speeches.

[0017] The speech processing device 10 may be realized by a computer reading a computer program recorded on a recording medium. In this case, it can be said that the recording medium records a computer program that causes the computer to execute a speech processing method that classifies each of a plurality of section speeches generated by dividing an input speech into a plurality of sections into section speeches that have speaker characteristics and section speeches that do not have speaker characteristics according to criteria for determining whether or not the section speeches have speaker characteristics, and extracts section speeches classified as section speeches that have speaker characteristics from the plurality of section speeches.

[0018] (Technical Effect) A technology (so-called speaker clustering) has been proposed that classifies multiple voices (e.g., multiple section voices) by speaker. With this technology, for example, multiple voices are classified by speaker based on the similarity between two voices. Note that an example of the similarity between two voices is similarity based on voice features. Specifically, for example, the similarity of i-vectors or x-vectors calculated by PLDA (Probabilistic Linear Discriminant Analysis) is exemplified.

[0019] For example, when multiple voices are separated into speakers based on similarities based on voice characteristics, the following technical problem may occur: For example, voices generated by multiple people speaking simultaneously and / or voices related to singing sounds as environmental sounds have voice characteristics. Due to the similarities based on such voice characteristics, the multiple voices may not be properly separated into speakers.

[0020] In contrast, in the speech processing device 10, the classification unit 11 classifies each of the multiple section sounds into section sounds having speaker characteristics and section sounds not having speaker characteristics according to criteria for determining whether or not the section sounds have speaker characteristics. Then, the extraction unit 12 extracts section sounds classified as section sounds having speaker characteristics from the multiple section sounds. As described above, "speaker characteristics" may mean characteristics of a person's speech. In other words, section sounds having speaker characteristics are section sounds having characteristics of a person's speech. Therefore, based on the similarity of the section sounds extracted by the extraction unit 12 (i.e., section sounds having speaker characteristics), the multiple section sounds can be appropriately divided by speaker.

[0021] Second Embodiment An embodiment relating to an audio processing device, an audio processing method, a recording medium, and an information display device will be described with reference to Fig. 2 to Fig. 10. In the following, an audio processing device 20 will be used to describe the embodiment relating to an audio processing device, an audio processing method, a recording medium, and an information display device.

[0022] 2 , the audio processing device 20 includes a calculation device 21, a storage device 22, and a communication device 23. The audio processing device 20 may further include an input device 24 and an output device 25. Note that the audio processing device 20 does not necessarily include at least one of the input device 24 and the output device 25. In the audio processing device 20, the calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0023] The arithmetic device 21 may include, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a TPU (Tensor Processing Unit), and a quantum processor.

[0024] The storage device 22 may include, for example, at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and an optical disk array. In other words, the storage device 22 may include a non-transitory recording medium. The storage device 22 is capable of storing desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic device 21. The storage device 22 may temporarily store data that is temporarily used by the arithmetic device 21 when the arithmetic device 21 is executing a computer program.

[0025] The communication device 23 may be capable of communicating with devices external to the audio processing device 20. The communication device 23 may perform wired communication or wireless communication.

[0026] The input device 24 is a device capable of accepting information input to the voice processing device 20 from the outside. The input device 24 may include an operation device (e.g., a keyboard, a mouse, a touch panel, etc.) that can be operated by an operator of the voice processing device 20. The input device 24 may include a recording medium reading device that can read information recorded on a recording medium that is detachable from the voice processing device 20, such as a USB (Universal Serial Bus) memory. Note that when information is input to the voice processing device 20 via the communication device 23 (in other words, when the voice processing device 20 acquires information via the communication device 23), the communication device 23 may function as an input device.

[0027] The output device 25 is a device capable of outputting information to the outside of the voice processing device 20. The output device 25 may output visual information such as text or images, auditory information such as sound, or tactile information such as vibration, as the information. The output device 25 may include, for example, at least one of a display, a speaker, a printer, and a vibration motor. The output device 25 may be capable of outputting information to a recording medium detachable from the voice processing device 20, such as a USB memory. Note that when the voice processing device 20 outputs information via the communication device 23, the communication device 23 may function as the output device.

[0028] The calculation device 21 may have, as logically realized functional blocks or as physically realized processing circuits, a classification unit 211, an extraction unit 212, a clustering unit 213, a speaker information generation unit 214, a character information generation unit 215, and a display control unit 216.

[0029] At least one of the classification unit 211, extraction unit 212, clustering unit 213, speaker information generation unit 214, character information generation unit 215, and display control unit 216 may be realized in a format in which a logical functional block and a physical processing circuit (i.e., hardware) are mixed. When at least some of the classification unit 211, extraction unit 212, clustering unit 213, speaker information generation unit 214, character information generation unit 215, and display control unit 216 are functional blocks, at least some of the classification unit 211, extraction unit 212, clustering unit 213, speaker information generation unit 214, character information generation unit 215, and display control unit 216 may be realized by the calculation device 21 executing a predetermined computer program.

[0030] The arithmetic device 21 may acquire (in other words, read) the predetermined computer program from the storage device 22. The arithmetic device 21 may read the predetermined computer program stored in a computer-readable, non-transitory recording medium using a recording medium reading device (not shown) included in the audio processing device 20. The arithmetic device 21 may acquire (in other words, download or read) the predetermined computer program from a device (not shown) external to the audio processing device 20 via the communication device 23. Note that the recording medium for recording the predetermined computer program executed by the arithmetic device 21 may be at least one of an optical disk, a magnetic medium, a magneto-optical disk, a semiconductor memory, and any other medium capable of storing a program.

[0031] The "classifying unit 211" and the "extracting unit 212" are components corresponding to the "classifying unit 11" and the "extracting unit 12" in the first embodiment, respectively.

[0032] The operations of the classification unit 211, extraction unit 212, and clustering unit 213 will be described with reference to Figures 3 to 8. Note that, for the classification unit 211 and extraction unit 212, descriptions that overlap with those in the first embodiment will be omitted as appropriate.

[0033] 3 , the classification unit 211 performs a predetermined voice detection process on the input voice (step S101). Here, the input voice may include voice and sounds other than voice. The input voice may also include silent intervals. By performing the predetermined voice detection process on the input voice, the classification unit 211 may distinguish between voice intervals in which voice is present (in other words, intervals including an acoustic signal corresponding to voice) and intervals other than voice intervals. The intervals other than voice intervals may include, for example, intervals in which sounds other than voice (e.g., sounds, music, animal calls, etc.) are present and silent intervals. Since various existing aspects can be applied to the voice detection process, detailed description thereof will be omitted. When detecting voice related to human speech, the voice detection process may be referred to as speech detection process.

[0034] As a result of the processing in step S101, for example, speech segments VS1, VS2, VS3, VS4, and VS5 may be detected from the input speech as shown in Fig. 4. Note that segments BS1, BS2, BS3, and BS4 in Fig. 4 are examples of segments other than speech segments.

[0035] The classification unit 211 may divide one input speech into a plurality of section speeches by extracting detected speech segments (e.g., speech segments VS1, VS2, VS3, VS4, and VS5) from the input speech. In this case, one speech segment may correspond to one section speech. Alternatively, the classification unit 211 may divide one input speech to generate a plurality of section speeches, and then perform speech detection processing on each of the plurality of section speeches as the input speech. In this case, the classification unit 211 may extract a speech segment from each of the plurality of section speeches. In other words, the classification unit 211 may further divide one section speech by extracting a speech segment from the one section speech.

[0036] After the process of step S101, the classification unit 211 extracts speaker features from the detected speech segments (for example, speech segments VS1, VS2, VS3, VS4, and VS5) (step S102). As described above, the speaker features may be at least one of an i-vector and an x-vector.

[0037] Next, the classification unit 211 calculates the similarity between each of the multiple speech segments based on the speaker features extracted in the processing of step S102 (step S103). Note that a speech segment can be considered to be one segment of one input speech. Therefore, a speech segment may also be referred to as segment speech. Therefore, calculating the similarity between each of the multiple speech segments can be considered to be calculating the similarity between each of the multiple segment speeches.

[0038] For example, the speaker features of the speech segments VS1, VS2, VS3, VS4, and VS5 are designated as speaker features SF1, SF2, SF3, SF4, and SF5, respectively. The calculation results of the similarity between each of the speech segments VS1, VS2, VS3, VS4, and VS5 can be represented as the table shown in Fig. 5. In Fig. 5, for example, the value at the position where the speaker feature SF1 on the vertical axis intersects with the speaker feature SF2 on the horizontal axis (specifically, "0.1") indicates the similarity between the speech segments VS1 and VS2. In this embodiment, the more similar the speaker feature of one speech segment is to the speaker feature of another speech segment, the larger the value indicating the similarity.

[0039] The classification unit 211 may calculate distance instead of similarity. That is, the classification unit 211 may calculate distance instead of similarity as an index indicating the degree of similarity between a speaker feature of one speech section and a speaker feature of another speech section. In this case, the value indicating the distance may be smaller as the speaker feature of one speech section and a speaker feature of another speech section are more similar.

[0040] After the process of step S103, the classification unit 211 performs score cleansing (step S104). Score cleansing refers to a process of classifying multiple speech segments into speech segments that have speaker characteristics and speech segments that do not have speaker characteristics, according to a criterion for determining whether or not a speaker characteristic is present.

[0041] The "criteria for determining the presence or absence of speaker features" will now be described. The first speaker feature of a first speech associated with one speaker and the second speaker feature of a second speech associated with the same speaker are similar to each other because they are spoken by the same speaker. The first speaker feature of a first speech associated with one speaker and the third speaker feature of a third speech associated with another speaker are dissimilar to each other because they are spoken by different speakers. Therefore, the similarity between the first speech and the second speech based on the first and second speaker features is higher than the similarity between the first speech and the third speech based on the first and third speaker features. In other words, the similarity between the first speech and the third speech based on the first and third speaker features is lower than the similarity between the first speech and the second speech based on the first and second speaker features.

[0042] For example, when similarity is expressed as a numerical value ranging from 0 to 1, the similarity value between the first voice and the second voice is often close to 1. On the other hand, the similarity value between the first voice and the third voice is often close to 0. In this way, the similarity when the two voices are spoken by the same speaker is clearly different from the similarity when the two voices are spoken by different speakers.

[0043] The similarity between the fourth speech, which is miscellaneous speech, and the first speech, based on the fourth speaker feature of the fourth speech and the first speaker feature of the first speech associated with one speaker, is often moderate. In other words, the fourth speech and the first speech can be said to be similar, or the fourth speech and the first speech can be said to be dissimilar. Similarly, the similarity between the fourth speech and the third speech, based on the fourth feature of the fourth speech, which is miscellaneous speech, and the third feature of the third speech associated with another speaker, is often moderate. In other words, the fourth speech and the third speech can be said to be similar, or the fourth speech and the third speech can be said to be dissimilar. For example, when similarity is expressed as a numerical value ranging from 0 to 1, a moderate similarity value is often close to 0.5. Note that "0.5" is an example and is not limited to this. Examples of miscellaneous speech include speech containing speech from multiple people, speech containing laughter, and speech containing singing as environmental sound.

[0044] When one voice is voice related to one person (for example, one speaker and one of the other speakers), the frequency distribution of the similarity between the one voice and each of the multiple other voices is, for example, the distribution shown in Figure 6. In the distribution shown in Figure 6, one peak is formed in an area of ​​high similarity, and another peak is formed in an area of ​​low similarity. Because the distribution shown in Figure 6 has two peaks, it can be said to be a multi-peak distribution.

[0045] On the other hand, when the first voice is a miscellaneous voice, the frequency distribution of the similarity between the first voice and each of the plurality of other voices will be, for example, the distribution shown in Fig. 7. In the distribution shown in Fig. 7, one peak is formed in the region where the similarity is medium.

[0046] For example, the similarity when the two voices are spoken by the same person is called the first similarity. The similarity when the two voices are spoken by different people is called the second similarity. The similarity when at least one of the two voices is a mixed voice is called the third similarity. The third similarity is different from the first similarity and the second similarity. Note that the similarity can be considered an index value indicating the degree of similarity. Therefore, the first similarity, the second similarity, and the third similarity may be referred to as the first index value, the second index value, and the third index value, respectively.

[0047] Using the first, second, and third similarities, the distribution shown in Fig. 6 can be expressed as the proportion of similarities corresponding to the first similarity and the proportion of similarities corresponding to the second similarity being higher than the proportion of similarities corresponding to the third similarity. Similarly, the distribution shown in Fig. 7 can be expressed as the proportion of similarities corresponding to the third similarity being higher than the proportion of similarities corresponding to the first similarity and the proportion of similarities corresponding to the second similarity.

[0048] 6 and 7, the frequency distribution of similarity when a given voice is a voice related to one person (e.g., one speaker and one of the other speakers) is clearly different from the frequency distribution of similarity when the given voice is a miscellaneous voice. This means that it is possible to determine whether a given voice has the characteristics of a speaker based on the similarity between the given voice and each of the other voices.

[0049] Therefore, the "criteria for determining the presence or absence of speaker characteristics" may include that the percentage of similarity corresponding to the first similarity when the two voices are spoken by the same speaker and the percentage of similarity corresponding to the second similarity when the two voices are spoken by different speakers are higher than the percentage of similarity corresponding to the third similarity different from the first similarity and the second similarity. Alternatively, the "criteria for determining the presence or absence of speaker characteristics" may include that the percentage of similarity corresponding to the third similarity different from the first similarity and the second similarity is lower than the percentage of similarity corresponding to the first similarity when the two voices are spoken by the same speaker and the percentage of similarity corresponding to the second similarity when the two voices are spoken by different speakers.

[0050] The "criteria for determining the presence or absence of speaker characteristics" may include a condition in which the proportion of similarities corresponding to a third similarity level, which is different from the first similarity level and the second similarity level, is higher than a proportion of similarities corresponding to the first similarity level when the speakers of the two voices are the same and a proportion of similarities corresponding to the second similarity level when the speakers of the two voices are different. Alternatively, the "criteria for determining the presence or absence of speaker characteristics" may include a condition in which the proportion of similarities corresponding to the first similarity level when the speakers of the two voices are the same and a proportion of similarities corresponding to the second similarity level when the speakers of the two voices are different are lower than a proportion of similarities corresponding to a third similarity level, which is different from the first similarity level and the second similarity level.

[0051] The "criteria for determining the presence or absence of a speaker's characteristics" may include that the frequency distribution of the similarity between one voice and each of the other multiple voices is a multi-modal distribution. Alternatively, the "criteria for determining the presence or absence of a speaker's characteristics" may include that the frequency distribution of the similarity between one voice and each of the other multiple voices is not a multi-modal distribution.

[0052] For example, by defining "criteria for determining whether or not a speaker has characteristics" as described above, it is possible to appropriately distinguish the frequency distribution of similarities between one voice related to one person and each of multiple other voices, and the frequency distribution of similarities between one voice as a miscellaneous voice and each of multiple other voices.

[0053] As described above, the similarity when two voices are spoken by the same speaker is clearly different from the similarity when two voices are spoken by different speakers. Therefore, the first range, which is the range of values ​​that the similarity can take when two voices are spoken by the same speaker, is different from the second range, which is the range of values ​​that the similarity can take when two voices are spoken by different speakers. For example, if a value between the first range and the second range is set as the predetermined value PV, in the distribution shown in FIG. 6 , the similarity value corresponding to one peak is greater than the predetermined value PV, and the similarity value corresponding to another peak is smaller than the predetermined value PV. The predetermined value PV may be the same as a threshold value used in the clustering process described below (e.g., a threshold value used to determine whether two voices are spoken by the same speaker).

[0054] Therefore, the "criteria for determining whether or not a speaker has characteristics" may include that in the frequency distribution of similarities between one voice and each of a plurality of other voices, the similarity value corresponding to one peak is greater than a predetermined value PV, and the similarity values ​​corresponding to other peaks are smaller than the predetermined value PV.

[0055] Research by the present inventors has revealed the following: As the number of voice segments detected in the process of step S101 in FIG. 3 increases, the detected voice segments are more likely to include voice segments related to multiple miscellaneous voices. The similarity between one miscellaneous voice and other miscellaneous voices may differ from the similarity between the one miscellaneous voice and other voices related to one person. As a result, the frequency distribution of similarities between the one miscellaneous voice and each of the other voices may be multi-peaked. Even in such cases, if the "criteria for determining the presence or absence of speaker characteristics" include a requirement that, in the frequency distribution of similarities between the one voice and each of the other voices, a similarity value corresponding to one peak is greater than a predetermined value PV and a similarity value corresponding to another peak is smaller than a predetermined value PV, it is possible to appropriately distinguish the frequency distribution of similarities between the one voice related to one person and each of the other voices from the frequency distribution of similarities between the one miscellaneous voice and each of the other voices.

[0056] In the processing of step S104 in Fig. 3, the classification unit 211 may perform score cleansing in accordance with the above-mentioned "criteria for determining the presence or absence of speaker features." For example, in Fig. 5, the values ​​at the positions where the speaker feature SF5 on the vertical axis intersects with each of the speaker features SF1, SF2, SF3, and SF4 on the horizontal axis are "0.5", "0.5", "0.5", and "0.5". In the example shown in Fig. 5, the number of similarities (i.e., the number of samples) is small, but it is highly likely that the frequency distribution of the similarities between the speech section VS5 and each of the speech sections VS1, VS2, VS3, and VS4 will not be a multi-peak distribution.

[0057] Therefore, the speech in the speech section VS5 is likely to be miscellaneous speech. Therefore, the classification unit 211 may determine that the speech section VS5 is a speech section that does not have speaker characteristics. Also, the classification unit 211 may determine that the speech sections VS1, VS2, VS3, and VS4 are speech sections that have speaker characteristics.

[0058] After step S104, the extraction unit 212 may extract voice segments VS1, VS2, VS3, and VS4 determined to be voice segments having speaker characteristics. The clustering unit 213 may perform clustering processing based on the similarity between each of the voice segments VS1, VS2, VS3, and VS4 extracted by the extraction unit 212. Note that, instead of extracting voice segments having speaker characteristics, the extraction unit 212 may extract voice segments not having speaker characteristics. For example, the extraction unit 212 may extract voice segment VS5 determined to be a voice segment not having speaker characteristics. The clustering unit 213 may perform clustering processing based on the similarity between each of the voice segments VS1, VS2, VS3, and VS4 not extracted by the extraction unit 212.

[0059] That is, the clustering unit 213 may perform clustering processing based on the similarity of the unshaded parts in Fig. 8. Note that various existing aspects can be applied to the clustering processing. One example of the clustering processing is processing using hierarchical agglomerative clustering (AHC).

[0060] The speaker information generating unit 214 generates speaker information indicating the speakers of the speech sections (e.g., speech sections VS1, VS2, VS3, and VS4) determined to be speech sections having speaker characteristics based on the results of the clustering process by the clustering unit 213. The speaker information generating unit 214 may generate speaker information indicating the speakers in a manner that allows one speaker to be distinguished (or identified) from other speakers. The speaker information generating unit 214 may represent the speakers as, for example, "speaker 1" and "speaker 2," etc.

[0061] The speaker information generation unit 214 may identify a specific person corresponding to a single speaker by performing at least one of speaker recognition processing and voice authentication processing on one or more pieces of speech classified as speech related to a single speaker based on the result of the clustering processing by the clustering unit 213. In this case, the speaker information generation unit 214 may represent the speaker by a specific person's name.

[0062] The speaker information generating unit 214 may include speaker information indicating the speaker of a voice section determined by the classification unit 211 to be a voice section not having a speaker characteristic (e.g., voice section VS5) in speaker information indicating the speaker of a voice section determined to be a voice section having a speaker characteristic. In this case, the speaker information generating unit 214 may represent the speaker of the voice section not having a speaker characteristic as, for example, "Unknown." Note that "Unknown" is an example and is not limited to this. As long as it is clear that the speaker of the voice section not having a speaker characteristic is different from the speaker of the voice section having a speaker characteristic, the speaker of the voice section not having a speaker characteristic may be represented by any expression.

[0063] 3, the text information generation unit 215 generates text information indicating a character string corresponding to the speech, based on the speech related to the speech section detected in the processing of step S101. Note that various existing methods can be applied to the method of generating text information indicating a character string from speech (so-called transcription), and therefore detailed description thereof will be omitted.

[0064] The display control unit 216 controls the output device 25, which can function as a display device, to display the speakers indicated by the speaker information generated by the speaker information generation unit 214 and the character strings indicated by the character information generated by the character information generation unit 215 in association with each other. As a result, the output device 25 may display, for example, the image shown in FIG. 9 . The display control unit 216 may control the output device 25 to display an image (e.g., an icon) instead of or in addition to the character strings indicating the speakers. For example, the display control unit 216 may control the output device 25 to display the speaker of a voice section determined to be a voice section having a speaker characteristic as a human-shaped icon, and to display the speaker of a voice section determined to be a voice section not having a speaker characteristic as an icon of an object other than a human. The display control unit 216 may control the output device 25 to display the character string corresponding to the voice related to the voice section not having the speaker characteristic and the speaker (e.g., "Unknown") of the voice section not having the speaker characteristic at positions different from the character string and the speaker related to the voice section having the speaker characteristic. The speech processing device 20 may be called an information display device because it displays the speaker indicated by the speaker information and the character string indicated by the character information.

[0065] In addition, when the speaker information generation unit 214 does not perform speaker recognition processing and voice authentication processing, the display control unit 216 may control the output device 25 to display, for example, a speaker input screen shown in FIG. 10 . A user of the speech processing device 20 may input, via the input device 24, the name of at least one speaker among the speakers displayed on the speaker input screen. The speaker information generation unit 214 may change the speaker indicated by the speaker information based on the input name. The display control unit 216 may control the output device 25 to display the speaker indicated by the speaker information changed by the speaker information generation unit 214 and the character string indicated by the character information in association with each other.

[0066] (Technical Effect) As described above, it can be said that miscellaneous speech and speech related to one speaker are similar, or that miscellaneous speech and speech related to one speaker are dissimilar. It can also be said that miscellaneous speech and speech related to another speaker are similar, or that miscellaneous speech and speech related to another speaker are dissimilar.

[0067] If no measures are taken, the clustering process may determine that one miscellaneous voice and a voice associated with one speaker are the voices of the same speaker. In addition, the clustering process may determine that the one miscellaneous voice and a voice associated with another speaker are the voices of the same speaker. As a result, the clustering process may determine that a voice associated with one speaker and a voice associated with another speaker are the voices of the same speaker. In other words, if miscellaneous voices are included in the voices to be subjected to the clustering process, there is a possibility that each of the multiple voices will not be properly separated by speaker. Note that miscellaneous voices often have voice characteristics, and therefore it is difficult to exclude them, for example, in the voice detection process of step S101.

[0068] In contrast, in the speech processing device 20, the classification unit 211 classifies each of multiple speech segments (e.g., speech segments VS1, VS2, VS3, VS4, and VS5) into speech segments having speaker characteristics and speech segments not having speaker characteristics according to criteria for determining the presence or absence of speaker characteristics. The clustering unit 213 then performs clustering processing based on the similarity between each of the speech segments determined to have speaker characteristics. This configuration makes it possible to prevent miscellaneous speech from being included in the speech to be subjected to the clustering processing. Therefore, the speech processing device 20 can appropriately classify each of multiple speeches by speaker. In addition, character strings corresponding to the speech are displayed in association with the speakers, which is practically useful. As described above, miscellaneous speech may adversely affect the results of the clustering processing. Therefore, the "criteria for determining the presence or absence of speaker characteristics" used to distinguish (or identify) whether a speech has the characteristics of a single person's speech or is miscellaneous speech may also be referred to as the "criteria for determining the presence or absence of a negative impact on the clustering processing" or the "criteria for determining miscellaneous speech."

[0069] Third Embodiment An embodiment relating to an audio processing device, an audio processing method, a recording medium, and an information display device will be described with reference to FIG. 11 . Hereinafter, the embodiment relating to an audio processing device, an audio processing method, a recording medium, and an information display device will be described using an audio processing device 20. In the third embodiment, the operation of the display control unit 216 will be mainly described. The other configurations may be the same as those in the second embodiment described above. Descriptions of the third embodiment that overlap with those of the second embodiment will be omitted as appropriate.

[0070] The input device 24 may include a microphone. The classification unit 211 may perform a voice detection process on sounds sequentially acquired by the microphone serving as the input device 24. The character information generation unit 215 may generate first character information indicating a character string corresponding to the voice, based on the voice related to the detected voice section. The display control unit 216 may control the output device 25 to display the character string indicated by the first character information generated by the character information generation unit 215. As a result, the output device 25 may display, for example, an image shown in FIG. 11 . In this case, since the speaker information generation unit 214 has not generated speaker information, information indicating the speaker (e.g., a character or an image) is not displayed.

[0071] In parallel with the above process, the classification unit 211 may extract speaker features from each of the detected speech segments. The classification unit 211 may calculate the similarity between each of the multiple speech segments based on the extracted speaker features. Thereafter, the classification unit 211 may perform score cleansing. That is, the classification unit 211 may classify speech segments into speech segments having speaker features and speech segments not having speaker features according to a criterion for determining the presence or absence of speaker features. The clustering unit 213 may perform a clustering process based on the similarity between each of the speech segments determined to be speech segments having speaker features. The speaker information generation unit 214 may generate first speaker information indicating the speaker of a speech segment determined to be a speech segment having speaker features based on the result of the clustering process. The speaker information control unit 214 may include speaker information indicating the speaker of a speech segment determined to be a speech segment not having speaker features in the first speaker information. The display control unit 216 may control the output device 25 to display the speaker indicated by the first speaker information generated by the speaker information generation unit 214 and the character string indicated by the first character information generated by the character information generation unit 215. As a result, the output device 25 may display, for example, the image shown in FIG.

[0072] The audio processing device 20 may treat a sound acquired by a microphone during a first period as one input audio, and a sound acquired by a microphone during a second period following the first period as another input audio. Alternatively, the audio processing device 20 may treat a sound acquired by a microphone during the first period and a sound acquired by a microphone during a second period following the first period as one input audio. In this case, the one input audio is updated when a new sound is acquired by the microphone.

[0073] The operation of the audio processing device 20 when it combines a sound acquired by a microphone during a first period and a sound acquired by a microphone during a second period following the first period and treats them as a single input audio will be described.

[0074] The classification unit 211 may perform a speech detection process on one input speech, which is a sound acquired by a microphone during a first period. The classification unit 211 may extract speaker features for each of the detected speech segments. The classification unit 211 may calculate a similarity between each of the detected speech segments based on the extracted speaker features. Thereafter, the classification unit 211 may perform score cleansing. That is, the classification unit 211 may classify speech segments into speech segments having speaker features and speech segments not having speaker features according to a criterion for determining whether or not a speaker feature is present. The clustering unit 213 may perform a clustering process based on the similarity between each of the speech segments determined to be speech segments having speaker features. The speaker information generation unit 214 may generate first speaker information indicating a speaker of a speech segment determined to be a speech segment having speaker features based on the result of the clustering process. The speaker information control unit 214 may include speaker information indicating a speaker of a speech segment determined to be a speech segment not having speaker features in the first speaker information. The character information generating unit 215 may generate first character information indicating a character string corresponding to the voice, based on the voice related to the detected voice section. The display control unit 216 may control the output device 25 to display the speaker indicated by the first speaker information generated by the speaker information generating unit 214 and the character string indicated by the first character information generated by the character information generating unit 215.

[0075] When a sound captured by a microphone is input during a second period following a first period, the audio processing device 20 may update one input audio by combining the sound captured by the microphone during the first period with the sound captured by the microphone during the second period.

[0076] The classification unit 211 may perform a speech detection process on the updated input speech. In this case, the classification unit 211 may perform a speech detection process again on sounds acquired by the microphone during a first period included in the updated input speech. Note that the classification unit 211 does not need to perform a speech detection process on sounds acquired by the microphone during a first period included in the updated input speech. The classification unit 211 may extract speaker features from each of the detected speech segments. The classification unit 211 may calculate a similarity between each of the multiple speech segments based on the extracted speaker features. Thereafter, the classification unit 211 may perform score cleansing. That is, the classification unit 211 may classify the speech segments into speech segments having speaker features and speech segments not having speaker features according to a criterion for determining the presence or absence of speaker features. The clustering unit 213 may perform a clustering process based on the similarity between each of the speech segments determined to be speech segments having speaker features. The speaker information generation unit 214 may generate second speaker information indicating a speaker of a voice section determined to be a voice section having speaker characteristics based on the result of the clustering process. The speaker information control unit 214 may include speaker information indicating a speaker of a voice section determined to be a voice section not having speaker characteristics in the second speaker information. The character information generation unit 215 may generate second character information indicating a character string corresponding to the voice based on the voice related to the detected voice section. The display control unit 216 may control the output device 25 to display the speaker indicated by the second speaker information generated by the speaker information generation unit 214 and the character string indicated by the second character information generated by the character information generation unit 215.

[0077] When the speaker of a character string included in both the first character information and the second character information (e.g., a character string corresponding to a sound included in a sound acquired by a microphone during a first period) differs between the speaker indicated by the first speaker information and the speaker indicated by the second speaker information, the display control unit 216 may control the output device 25, when the second speaker information is generated, to change the speaker displayed in association with the character string from the speaker indicated by the second speaker information to the speaker indicated by the second speaker information. As a result, the output device 25 may change the speaker displayed in association with the character string, for example, from "Speaker 1" to "Speaker 2."

[0078] (Technical Effect) According to the voice processing device 20 of this embodiment, it is possible to display a character string corresponding to an utterance in real time while the utterance is being made, which makes it possible to generate minutes of a meeting in real time, for example.

[0079] <Supplementary Notes> The following supplementary notes are further disclosed regarding the above-described embodiment.

[0080] (Supplementary Note 1) A speech processing device comprising: a classification means for classifying a plurality of section speeches generated by dividing an input speech into a plurality of sections into section speeches having the characteristics of the speaker and section speeches not having the characteristics of the speaker according to a criterion for determining whether or not the characteristics of the speaker are present; and an extraction means for extracting section speeches classified as section speeches having the characteristics of the speaker from the plurality of section speeches.

[0081] (Supplementary Note 2) The speech processing device according to Supplementary Note 1, wherein the classification means calculates a plurality of index values ​​each indicating a degree of similarity between one section voice of the plurality of section voices and each of the plurality of section voices, and the classification means classifies the plurality of section voices into section voices having the speaker's characteristics and section voices not having the speaker's characteristics based on the calculated plurality of index values ​​and a criterion for determining whether or not the speaker's characteristics are present.

[0082] (Supplementary Note 3) The speech processing device according to Supplementary Note 2, wherein the plurality of index values ​​include a first index value when a speaker associated with the one section sound is the same as a speaker associated with another section sound different from the one section sound, a second index value when a speaker associated with the one section sound is different from a speaker associated with the other section sound, and a third index value different from the first index value and the second index value, and wherein criteria for determining whether or not the speaker has a characteristic include that a proportion of index values ​​corresponding to the third index value among the plurality of index values ​​is higher than a proportion of index values ​​corresponding to the first index value and a proportion of index values ​​corresponding to the second index value.

[0083] (Supplementary Note 4) The speech processing device according to Supplementary Note 2 or 3, wherein the plurality of index values ​​include a first index value when a speaker associated with the one section audio is the same as a speaker associated with another section audio different from the one section audio, a second index value when a speaker associated with the one section audio is different from a speaker associated with the other section audio, and a third index value different from the first index value and the second index value, and criteria for determining whether or not a feature of the speaker is present include a ratio of index values ​​corresponding to the first index value and a ratio of index values ​​corresponding to the second index value being higher than a ratio of index values ​​corresponding to the third index value among the plurality of index values.

[0084] (Supplementary Note 5) The speech processing device according to any one of Supplementary Notes 2 to 4, wherein the criteria for determining whether or not the speaker has a characteristic include whether the frequency distribution of the plurality of index values ​​is a multi-modal distribution.

[0085] (Supplementary Note 6) The speech processing device according to claim 5, wherein the criteria for determining whether or not the speaker has a characteristic include that an index value corresponding to one peak of the multimodal distribution is greater than a predetermined value, and that index values ​​corresponding to other peaks of the multimodal distribution are smaller than the predetermined value.

[0086] (Supplementary Note 7) A speech processing method in which a computer classifies a plurality of section speeches generated by dividing an input speech into a plurality of sections into section speeches having the characteristics of the speaker and section speeches not having the characteristics of the speaker according to a criterion for determining whether or not the speaker has the characteristics, and extracts section speeches classified as section speeches having the characteristics of the speaker from the plurality of section speeches.

[0087] (Supplementary Note 8) A recording medium having recorded thereon a computer program for causing a computer to execute a speech processing method, the method comprising: classifying a plurality of section speeches generated by dividing input speech into a plurality of sections into section speeches having the characteristics of the speaker and section speeches not having the characteristics of the speaker according to a criterion for determining whether or not the speaker has the characteristics; and extracting section speeches classified as section speeches having the characteristics of the speaker from the plurality of section speeches.

[0088] (Supplementary Note 9) An information display device comprising: a classification means for classifying a plurality of section sounds generated by dividing an input speech into a plurality of sections into section sounds having the characteristics of the speaker and section sounds not having the characteristics of the speaker according to a criterion for determining whether or not a speaker has a characteristic; an extraction means for extracting one or more section sounds classified as section sounds having the characteristics of the speaker from the plurality of section sounds; a first generation means for generating first speaker information indicating a speaker of each of the extracted one or more section sounds; a second generation means for generating character information indicating a character string corresponding to the at least one section sound, based on at least one section sound of the plurality of section sounds; and a display means for displaying the speaker indicated by the generated first speaker information and the character string indicated by the generated character information.

[0089] (Supplementary Note 10) The information display device according to Supplementary Note 9, wherein the display means displays a character string corresponding to the section sound corresponding to the one or more extracted section sounds from the at least one section sound in association with a speaker indicated by the generated first speaker information.

[0090] (Supplementary Note 11) The information display device described in Supplementary Note 9 or 10, wherein the display means displays a character string corresponding to a section sound that does not correspond to one or more of the extracted section sounds among the at least one section sound, in association with at least one of a character and an image indicating a speaker different from the speaker indicated by the generated first speaker information.

[0091] (Supplementary Note 12) When new section audio is extracted by the extraction means, the first generation means generates second speaker information indicating the speaker of each of multiple section audios including the extracted new section audio, and the display means displays the speaker indicated by the generated second speaker information and a character string indicated by the generated character information. An information display device as described in any one of Supplementary Notes 9 to 11.

[0092] (Appendix 13) When the speaker of a section audio corresponding to a character string indicated by the generated character information is different between the speaker indicated by the first speaker information and the speaker indicated by the second speaker information, the display means changes the speaker associated with the character string from the speaker indicated by the first speaker information to the speaker indicated by the second speaker information when the second speaker information is generated. This is an information display device as described in Appendix 12.

[0093] This disclosure is not limited to the above-described embodiments, but may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and audio processing devices, audio processing methods, recording media, and information display devices that involve such modifications are also included in the technical scope of this disclosure.

[0094] 10, 20 Speech processing device 11, 211 Classification unit 12, 212 Extraction unit 213 Clustering unit 214 Speaker information generation unit 215 Character information generation unit 216 Display control unit

Claims

1. a classification means for classifying a plurality of section sounds generated by dividing an input speech into a plurality of sections into section sounds having the characteristics of the speaker and section sounds not having the characteristics of the speaker, according to a criterion for determining whether or not the characteristics of the speaker exist; an extraction means for extracting section sounds classified as section sounds having the characteristics of the speaker from the plurality of section sounds; An audio processing device comprising:

2. the classification means calculates a plurality of index values ​​each indicating a degree of similarity between one of the plurality of section sounds and each of the plurality of section sounds; The classification means classifies the plurality of section sounds into section sounds having the characteristics of the speaker and section sounds not having the characteristics of the speaker based on the calculated plurality of index values ​​and a criterion for determining whether or not the characteristics of the speaker are present. The audio processing device according to claim 1 .

3. the plurality of index values ​​include a first index value when a speaker associated with the one section sound and a speaker associated with another section sound different from the one section sound are the same, a second index value when a speaker associated with the one section sound and a speaker associated with the other section sound are different, and a third index value different from the first index value and the second index value, The criteria for determining whether or not the speaker has a characteristic include that, among the plurality of index values, a proportion of index values ​​corresponding to the third index value is higher than a proportion of index values ​​corresponding to the first index value and a proportion of index values ​​corresponding to the second index value. The audio processing device according to claim 2 .

4. the plurality of index values ​​include a first index value when a speaker associated with the one section sound and a speaker associated with another section sound different from the one section sound are the same, a second index value when a speaker associated with the one section sound and a speaker associated with the other section sound are different, and a third index value different from the first index value and the second index value, The criteria for determining whether or not the speaker has a characteristic include that, among the plurality of index values, a proportion of index values ​​corresponding to the first index value and a proportion of index values ​​corresponding to the second index value are higher than a proportion of index values ​​corresponding to the third index value.

4. The audio processing device according to claim 2 or 3.

5. The criteria for determining whether or not a speaker has a characteristic include whether the frequency distribution of the plurality of index values ​​is a multi-modal distribution.

4. The audio processing device according to claim 2 or 3.

6. The criteria for determining whether or not a speaker characteristic exists include that an index value corresponding to one peak of the multimodal distribution is greater than a predetermined value, and an index value corresponding to another peak of the multimodal distribution is smaller than the predetermined value. The audio processing device according to claim 5 .

7. The computer A plurality of section sounds generated by dividing an input speech into a plurality of sections are classified into section sounds having the characteristics of the speaker and section sounds not having the characteristics of the speaker according to a criterion for determining whether or not the characteristics of the speaker are present; extracting section sounds classified as section sounds having the characteristics of the speaker from the plurality of section sounds; Audio processing methods.

8. On the computer, A plurality of section sounds generated by dividing an input speech into a plurality of sections are classified into section sounds having the characteristics of the speaker and section sounds not having the characteristics of the speaker according to a criterion for determining whether or not the characteristics of the speaker are present; extracting section sounds classified as section sounds having the characteristics of the speaker from the plurality of section sounds; A recording medium on which a computer program for executing an audio processing method is recorded.

9. a classification means for classifying a plurality of section sounds generated by dividing an input speech into a plurality of sections into section sounds having the characteristics of the speaker and section sounds not having the characteristics of the speaker, according to a criterion for determining whether or not the characteristics of the speaker exist; an extraction means for extracting one or more section sounds classified as section sounds having the characteristics of the speaker from the plurality of section sounds; a first generating means for generating first speaker information indicating a speaker of each of the extracted one or more section sounds; a second generating means for generating character information indicating a character string corresponding to at least one section sound based on at least one section sound of the plurality of section sounds; a display means for displaying a speaker indicated by the generated first speaker information and a character string indicated by the generated character information; An information display device comprising:

10. The display means displays a character string corresponding to the extracted one or more section sounds from the at least one section sound in association with a speaker indicated by the generated first speaker information. The information display device according to claim 9.