Voiceprint matching support system, voiceprint matching support method, and program

The voiceprint matching support system automates the extraction of time intervals spoken by the target speaker, improving accuracy and efficiency by excluding non-target speaker utterances, addressing the inefficiencies and inaccuracies of manual processes.

JP7948336B2Active Publication Date: 2026-11-11NEC CORP
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2024562653
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2022-12-05
Filing Date
2023-11-16
Publication Date
2026-11-11
Estimated Expiration
2043-11-16

AI Technical Summary

Technical Problem

Existing voiceprint matching systems require manual and time-consuming processes, leading to potential fatigue and variations in accuracy, and fail to exclude overlapping utterances by speakers other than the target speaker.

Method used

A voiceprint matching support system that includes an input unit for sample voice data, an extraction unit to identify time intervals of the target speaker, and a display unit to present these intervals to the user, reducing manual effort and excluding non-target speaker utterances.

Benefits of technology

The system enhances accuracy and efficiency by automating the extraction of time intervals spoken by the target speaker, minimizing overlap with other speakers, thus reducing the burden on operators and stabilizing accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007948336000001
    Figure 0007948336000001
  • Figure 0007948336000002
    Figure 0007948336000002
  • Figure 0007948336000003
    Figure 0007948336000003
Patent Text Reader

Abstract

Provided is a voiceprint collation assistance system that can suppress inclusion of a portion of voice data to be voiceprint-collated that is uttered by someone other than a subject of voiceprint collation when a portion uttered by the subject is presented to a user. The voiceprint collation assistance system pertaining to the present disclosure comprises an input unit, an extraction unit, and a display unit. The input unit inputs sample voice data of a subject for voiceprint collation. The extraction unit extracts a time segment in which the subject is speaking from voice data to be voiceprint-collated, on the basis of the sample voice data. The display unit presents the time segment extracted by the extraction unit to the user.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a voiceprint verification support system, a voiceprint verification support method, and a program.

Background Art

[0002] In voiceprint verification for voiceprint appraisal and the like, an operator manually extracts segments by listening to the entire audio or the entire video containing audio, in order to find segments where the speaker to be verified is presumed to be speaking.

[0003] Further, Patent Document 1 discloses a speaker search system for solving the problem that, in a system for searching for a speaker, when the speakers of detected audio are similar, it is difficult to determine whether the detection result is actually uttered by the person that the user intends to find. The speaker search system described in Patent Document 1 includes an audio database that accumulates audio data, an optimal listening segment detection unit, a speaker search unit, and a search result presentation unit. The optimal listening segment detection unit detects an optimal listening segment with high speaker uniqueness from the accumulated audio data. The speaker search unit searches the accumulated audio data for audio data uttered by the same speaker as the audio or speaker according to the audio or speaker name input by the user. The search result presentation unit presents information related to the audio data obtained by the speaker search unit together with information related to the optimal listening segment with high speaker uniqueness of the audio data detected by the optimal listening segment detection unit.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] As mentioned above, voiceprint matching for voiceprint analysis and other purposes requires manual work by operators. This process requires repeatedly listening to the entire audio, which is extremely time-consuming. It is easy to imagine that fatigue could lead to a decrease in the accuracy of voiceprint matching and variations in accuracy.

[0006] Furthermore, although Patent Document 1 describes detecting utterance segments of the same speaker by clustering, the technology described in Patent Document 1 does not take into account the possibility that utterances by speakers other than the specific speaker may overlap within the utterance segment. Therefore, the technology described in Patent Document 1 cannot present the user with the utterance segment spoken by the specific speaker while excluding the overlap of utterances by speakers other than the specific speaker, and there is room for improvement in the information presented.

[0007] This disclosure is made to solve the above-mentioned problems and aims to provide the following voiceprint matching support system, voiceprint matching support method, and program. Specifically, this disclosure aims to provide a voiceprint matching support system, etc., that can suppress the inclusion of parts spoken by persons other than the target person when presenting to the user the parts spoken by the person to be voiceprint matching from voice data to be voiceprint matching. [Means for solving the problem]

[0008] The voiceprint matching support system relating to this disclosure comprises an input unit for inputting sample voice data of a person to be voiceprint matched, an extraction unit for extracting time intervals in which the person is speaking from the voiceprint matching target voice data based on the sample voice data, and a display unit for presenting the time intervals extracted by the extraction unit to the user.

[0009] Furthermore, the voiceprint matching support method relating to this disclosure comprises an input process for inputting sample voice data of a person to be voiceprint matched, an extraction process for extracting time intervals in which the person is speaking from the voiceprint matching target voice data based on the sample voice data, and a display process for displaying the extracted time intervals on a display device for presentation to the user.

[0010] Furthermore, the program relating to this disclosure is a program that causes a computer to execute a voiceprint matching support process which includes input processing to input sample voice data of a person to be matched; extraction processing to extract time intervals in which the person is speaking from the voice data to be matched based on the sample voice data; and display processing to display the extracted time intervals on a display device for presentation to the user. [Effects of the Invention]

[0011] According to this disclosure, it is possible to provide a voiceprint matching support system, etc., that can suppress the inclusion of parts spoken by persons other than the target person when presenting the portion of the voiceprint matching target audio data to the user. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing an example configuration of the voiceprint matching support system related to this disclosure. [Figure 2] Figure 1 is a flowchart illustrating an example of a voiceprint matching support method in the voiceprint matching support system. [Figure 3] This block diagram shows another example configuration of the voiceprint matching support system related to this disclosure. [Figure 4] This figure shows an example of the waveform of voice data to be processed by the voiceprint matching support system shown in Figure 3. [Figure 5] Figure 3 is a conceptual diagram showing an example of voiceprint matching target audio data processed by the voiceprint matching support system, and Figure 4 is a schematic diagram showing the waveform of the voiceprint matching target audio data separated by speaker. [Figure 6] Figure 3 shows an example of the waveform of sample voice data used in the voiceprint matching support system. [Figure 7] Figure 3 is a conceptual diagram illustrating an example of sample voice data used in the voiceprint matching support system, and Figure 6 is a schematic diagram showing the waveform of the sample voice data. [Figure 8]This is a diagram showing a display example of extracted time intervals for the waveform of voiceprint verification target audio data in FIG. 4. [Figure 9] This is a diagram showing another display example of extracted time intervals for the waveform of voiceprint verification target audio data in FIG. 4. [Figure 10] This is a diagram showing another display example of extracted time intervals for the waveform of voiceprint verification target audio data in FIG. 4. [Figure 11] This is a diagram showing another display example of extracted time intervals for the waveform of voiceprint verification target audio data in FIG. 4. [Figure 12] This is a flowchart for explaining the creation operation of a voiceprint verification target file according to Comparative Example 1. [Figure 13] This is a flowchart for explaining the creation operation of a voiceprint verification target file according to Comparative Example 2. [Figure 14] This is a diagram showing an example of the hardware configuration included in the apparatus. [Mode for Carrying Out the Invention]

[0013] Hereinafter, the embodiments will be described with reference to the drawings. For clarity of explanation, the following description and drawings are appropriately omitted and simplified. In addition, in each drawing, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted as necessary.

[0014] <Embodiment 1> FIG. 1 is a block diagram showing a configuration example of a voiceprint verification support system according to the present disclosure. The voiceprint verification support system 1 according to the present disclosure shown in FIG. 1 can be a system that supports the user's voiceprint verification work by presenting the user with audio data that is a candidate target for voiceprint verification. The voiceprint verification support system 1 can be used, for example, in voiceprint identification work in police forensics and various other operations that require voiceprint verification.

[0015] For such support, the voiceprint verification support system 1 shown in Figure 1 may include an input unit 1a, an extraction unit 1b, and a display unit 1c.

[0016] The input unit 1a inputs sample voice data of a subject for voiceprint verification. Here, the number of subjects for voiceprint verification is one. The sample voice data is voiceprint Verification 1-channel or multi-channel voice data recorded to include only the utterance of the relevant subject. Since the input of the sample voice data is referred to when the extraction unit 1b executes extraction, it only needs to be stored inside the voiceprint verification support system 1 so as to be referable before the extraction is executed. Examples of the input source include a voice acquisition device such as a microphone that acquires voice data, or a computer such as a server computer that stores pre-acquired sample voice data. These voice acquisition devices or computers may also be provided in the voiceprint verification support system 1. The input unit 1a is a component that inputs sample voice data and passes it to the extraction unit 1b, and may be provided with an input interface such as a communication interface for that purpose.

[0017] The extraction unit 1b extracts, based on the sample voice data, the time interval in which the subject is speaking from the voice data targeted for voiceprint verification. Since this extraction is based on the sample voice data, the extraction unit 1b analyzes both pieces of data and extracts time intervals that are highly relevant to the sample voice data. The extraction unit 1b may, for example, divide the sample voice data at predetermined time intervals, divide the voice data targeted for voiceprint verification also at the aforementioned predetermined time intervals, and extract the time intervals in which the subject is speaking for each of the predetermined time intervals. Note that processing for recognizing the voice content of the voice data targeted for voiceprint verification is not required here.

[0018] Furthermore, the extraction unit 1b may also perform time interval extraction using a learning model that has been trained to receive, for example, the sample voice data and the voice data targeted for voiceprint verification as inputs, and output the time intervals in which the subject is speaking. The algorithm or the like of this learning model is not limited, and the input target data may be, for example, data divided at the aforementioned predetermined time intervals.

[0019] The voiceprint matching target audio data can be recorded audio data from one channel or multiple channels. In particular, in this embodiment, since the voiceprint matching target audio data is extracted based on the subject's sample audio data, it is possible to extract the time interval in which the subject is speaking, even if the voiceprint matching target audio data is monaural audio data. Furthermore, the voiceprint matching target audio data can be data associated with video data, in other words, video data with audio.

[0020] Furthermore, the voice data to be matched can be stored in advance on a server computer or other computer and retrieved from that computer, or it can be stored on a memory device provided in the voiceprint matching support system 1 and read from that memory device. This computer can also be provided in the voiceprint matching support system 1.

[0021] Alternatively, the voice data to be matched can be data acquired from a real-time conversation using a voice acquisition device such as a microphone, or data acquired from a real-time telephone conversation. In cases where the voice data to be matched is acquired in real time, as in these examples, the extraction and presentation processes by the display unit 1c can also be performed in real time.

[0022] The display unit 1c presents the time interval extracted by the extraction unit 1b to the user. The display unit 1c may include a display device and a display control unit that controls the display on the display device. The display device only needs to be able to display an image and can be a display such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or a projector. The display device may be, for example, a display included in a smartphone or tablet device.

[0023] Furthermore, the voiceprint matching support system 1 can be, for example, a server computer, a personal computer, or a smartphone. Specifically, the voiceprint matching support system 1 may be configured to include a computer device that includes hardware such as one or more processors and one or more memories. At least some of the functions of each part within the voiceprint matching support system 1 can be realized by one or more processors operating according to a program read from one or more memories.

[0024] In other words, the voiceprint matching support system 1 may include a control unit (not shown) that controls the entire system. This control unit can be implemented, for example, by a processor, working memory, and a non-volatile storage device that stores a program. This processor can be, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). This program can be a program that causes the processor to execute the input processing of the input unit 1a, the extraction processing of the extraction unit 1b, and the display control processing of the display unit 1c. The voiceprint matching support system 1 may also include a storage device that stores input sample voice data, voiceprint matching target voice data, and information indicating the extracted time interval. This storage device can be, for example, a storage device provided in the control unit described above. The voiceprint matching support system 1 may also be configured as a device equipped with dedicated hardware.

[0025] Furthermore, the voiceprint matching support system 1 is not limited to being configured as a single device; it can also be constructed as multiple devices with distributed functions, and the method of distribution is not limited. One of the distributed devices can, of course, include a display device. When constructing a voiceprint matching support system with functions distributed among multiple devices, each device is equipped with a control unit, a communication unit, and, if necessary, a storage unit, etc. In this case, the multiple devices can be connected as needed via wireless or wired communication and cooperate to realize the functions described in the voiceprint matching support system 1.

[0026] Next, we will explain an example of the processing of the voiceprint matching support system 1 with reference to Figure 2. Figure 2 is a flowchart illustrating an example of the voiceprint matching support method in the voiceprint matching support system 1 shown in Figure 1.

[0027] This voiceprint matching support method performs the following voiceprint matching support process. The voiceprint matching support process performs an input process (step S1) to input sample voice data of the person to be voiceprinted. Next, the voiceprint matching support process performs an extraction process (step S2) to extract the time interval in which the person is speaking from the voice data to be voiceprinted, based on the sample voice data. Then, the voiceprint matching support process performs a display process (step S3) to display the extracted time interval on a display device for the user to see, and then terminates the process. The above program can be a program that causes a computer to perform the voiceprint matching support process, including the input process, extraction process, and display process.

[0028] In this way, the voiceprint matching support system 1 receives sample voice data of the person (speaker) to be matched, analyzes and extracts time intervals from the voiceprint matching target voice data that are highly related to the sample voice data, and returns time information indicating these time intervals to the user.

[0029] After this voiceprint matching support processing, the user who receives the presentation can select the audio data to be voiceprint matched from the voiceprint matching target audio data, taking the presentation content into consideration, and then perform the listening work and voiceprint matching processing by the voiceprint matching device. In this embodiment, with this voiceprint matching support, the user can perform voiceprint matching processing on the selected audio data that places a greater burden on the user's listening work and on the processing load of the voiceprint matching device. This audio data is the data that will actually be voiceprint matched, while the voiceprint matching target audio data that is processed by the extraction unit 1b is audio data for extracting the target of voiceprint matching. Therefore, the voiceprint matching target audio data can also be called the extraction target data or the selection target data.

[0030] As described above, according to this embodiment, it is possible to suppress the inclusion of parts spoken by persons other than the target person and present to the user only the parts of the voice data subject to voiceprint matching that are spoken by the person to be matched.

[0031] Let me elaborate on this effect. In order to perform voiceprint matching, that is, speaker matching based on speech, with high accuracy, it is necessary to extract the sections in which only a specific speaker is speaking. Currently, in voiceprint matching work such as voiceprint analysis, long periods of audio data are listened to and time sections suitable for voiceprint matching are extracted. This places a heavy burden on the worker due to the long hours of work, and there are concerns that the accuracy will not be stable due to the worker's subjectivity. In contrast, in this embodiment, with the aim of extracting time sections in which only the target person for voiceprint matching is speaking, especially time sections that do not overlap with the speech of others, the extraction is performed based on the target person's sample audio data, and the time sections are presented to the user. Since the extraction is based on the target person's sample audio data, the extraction results will have little overlap with the speech of others. The user can then check the results of the voiceprint matching support system 1's automatic extraction of time sections in which the target person is speaking, and narrow down the time sections for voiceprint matching by listening, etc.

[0032] Thus, in this embodiment, it is possible to present to the user a section in which a specific speaker is speaking while suppressing the overlap of utterances by speakers other than the specific speaker within the utterance section. Therefore, according to this embodiment, in tasks requiring voiceprint matching, such as voiceprint analysis, the user acting as the operator can streamline the process by focusing on listening to the proposed section, eliminating the need to listen to the entire speech. Consequently, according to this embodiment, it is possible to reduce the manual effort involved in user extraction, prevent a decrease in accuracy, and prevent variations in accuracy.

[0033] Furthermore, although this embodiment was described assuming that there is one person to be subjected to voiceprint matching, there may be multiple people. In that case, based on the sample voice data of each of those multiple people, the time intervals in which those multiple people are speaking simultaneously can be extracted from the voiceprint matching target voice data and presented to the user.

[0034] <Embodiment 2> Embodiment 2 will be described with reference to Figures 3 to 13, focusing on the differences from Embodiment 1, although various examples described in Embodiment 1 can be applied. First, other configuration examples of the voiceprint matching support system according to this disclosure will be described with reference to Figure 3. Figure 3 is a block diagram showing other configuration examples of the voiceprint matching support system according to this disclosure.

[0035] As shown in Figure 3, the voiceprint matching support system 100 according to this disclosure may include a server computer (hereinafter referred to as server) 10, a user terminal 20, a voice acquisition device 30 for processing, and a sample voice acquisition device 40.

[0036] The server 10 may include a control unit 11, a storage unit 12, and a communication unit 13. The control unit 11 is the part that controls the entire server 10 and can be realized by, for example, a processor such as a CPU or GPU, working memory, and a non-volatile storage device that stores the program. This program can be a program that causes the processor to execute processing that includes part of the input processing of the input unit 1a, the extraction processing of the extraction unit 1b, and the display control processing of the display unit 1c. Here, part of the display control processing refers to the processing that instructs the user terminal 20 via the communication unit 13 to display the extracted time interval on the user terminal 20. When the control unit 11 reads and executes the above program, the server 10 can realize functions that include part of the input function of the input unit 1a, the extraction function of the extraction unit 1b, and the display control function of the display unit 1c.

[0037] The storage unit 12 can be a storage device that stores sample audio data input from the sample audio acquisition device 40, voiceprint matching target audio data (hereinafter referred to as "processing target audio data") input from the processing target audio acquisition device 30, and information indicating the extracted time interval. The storage unit 12 can also store thresholds and time intervals used in the extraction process.

[0038] The communication unit 13 can serve as a communication interface that communicates with the user terminal 20, the voice acquisition device 30, and the sample voice acquisition device 40 via a wired or wireless network.

[0039] The user terminal 20 can be an information processing device such as a personal computer or a smartphone, and may include a control unit 21, a storage unit 22, a communication unit 23, and a display unit 24.

[0040] The control unit 21 is the part that controls the entire user terminal 20 and can be realized by, for example, a processor such as a CPU or GPU, working memory, and a non-volatile storage device that stores the program. This program can be a program that causes the processor to execute processing that includes processing to access the server 10 via the communication unit 23 and part of the display control processing of the display unit 1c. When the control unit 21 reads and executes the above program, the user terminal 20 can realize the functions to perform these processing. Here, part of the display control function can refer to the processing that, when an instruction to display the extracted time interval is received from the server 10 via the communication unit 23, displays it on the display unit 24 according to that instruction.

[0041] The storage unit 22 can be a storage device that stores various settings related to the display of time intervals. These settings may include, for example, display settings for the user interface for displaying time intervals. The communication unit 23 can be a communication interface that communicates with the server 10 via a wired or wireless network.

[0042] The voice acquisition device 30 is a device that acquires voice data to be processed by the server 10 and transmits it to the server 10. The voice acquisition device 30 can be a telephone system, a network system capable of voice calls, a recording device equipped with a microphone, etc.

[0043] The sample audio acquisition device 40 is a device that acquires sample audio data to be used for extraction processing on the server 10 and transmits it to the server 10, and can be a recording device equipped with a microphone or the like.

[0044] With the configuration described above, the voiceprint matching support system 100 can store the target voice data acquired by the target voice acquisition device 30 in the server 10, and also store the sample voice data acquired by the sample voice acquisition device 40 in the server 10. The order of storage does not matter.

[0045] Then, as described in the function of the extraction unit 1b, the control unit 11 of the server 10 performs an extraction process to extract the time interval in which the subject is speaking from the stored audio data to be processed, based on the stored sample audio data.

[0046] In Embodiment 1, the extraction process is performed based on the subject's sample voice data. Therefore, the extracted voice data, which is extracted from the processed voice data, includes the voice data of the section in which the subject is speaking, and that section is presented to the user. However, in the extraction process in Embodiment 1, voice data of the section in which the subject is conversing with someone else may also be extracted.

[0047] Therefore, the extraction process in this embodiment extracts time intervals in which the subject is speaking alone, that is, it extracts time intervals while excluding intervals in which others are speaking.

[0048] Furthermore, the control unit 11 sends an instruction to the user terminal 20 via the communication unit 13 to display the extracted time interval on the user terminal 20 for the user to see. The user terminal 20 receives this instruction from the server 10 via the communication unit 23, and the control unit 21 controls the display unit 24 to display the time interval, resulting in the display of the time interval on the display unit 24.

[0049] This allows users to focus their listening on the time intervals suggested as the display, streamlining the process of listening (viewing) the entire content. It should be noted that the information indicating the time intervals to be displayed is merely a suggestion, and voiceprint matching for these time intervals is not performed automatically. Of course, for example, the user can select the time intervals requiring voiceprint matching from the user terminal 20, and have the voiceprint matching performed by a voiceprint matching device that does not display the voiceprint matching process. This voiceprint matching device can also be installed on the server 10.

[0050] In this embodiment, by employing the extraction process described above, it is possible to exclude audio data from sections in which the subject is conversing with others and where the other person's speech is mixed in, thus eliminating the overlap of other people's voices compared to Embodiment 1. In other words, according to this embodiment, it is possible to prevent other people's voices from being mixed in during voiceprint matching.

[0051] Thus, according to this embodiment, when presenting the user with the portion of the audio data to be processed that is spoken by the person to be matched with a voiceprint, it is possible to exclude portions spoken by persons other than the target person. In other words, in this embodiment, it is possible to present the user with the portion of the speech spoken by a specific speaker while excluding any overlapping speech by persons other than the specific speaker within that speech segment.

[0052] Therefore, according to this embodiment, the user, who is the operator, can further improve the efficiency of the work, which previously involved listening to (watching) the entire recording, by focusing their listening on the proposed section, compared to Embodiment 1.

[0053] Furthermore, the control unit 11 can extract time intervals based on the similarity score between the feature quantities of the sample audio data and the feature quantities of the audio data to be processed, as part of the extraction process. Existing methods can be used for the type of feature quantities, the calculation of feature quantities, and the calculation method of the similarity score. For example, feature quantities can be calculated at predetermined time intervals, similarity scores can be calculated at those predetermined time intervals, and time intervals can be extracted using those predetermined time intervals as the unit. In this method, within the predetermined time interval intervals of the audio data to be processed, the similarity score will decrease in intervals where the voices of people other than the target person indicated by the feature quantities of the sample audio data are mixed in.

[0054] In particular, the control unit 11 may, in the extraction process, extract time intervals in the audio data to be processed in which the similarity score is equal to or greater than a predetermined threshold. By employing such an extraction process, time intervals can be presented to the user while excluding intervals in which other people's voices are mixed. This makes it possible to present to the user only the information that is highly relevant to the target person (high similarity score), that is, only the information indicating the intervals necessary for listening, in an easy-to-understand manner.

[0055] Specific examples of this processing will be explained with reference to Figures 4 to 11. Figure 4 shows an example of the waveform of the audio data to be processed by the voiceprint matching support system 100. Figure 5 is a conceptual diagram that conceptually shows an example of the audio data to be processed by the voiceprint matching support system 100, and schematically shows the waveform of the audio data to be processed in Figure 4, separated by speaker. Figure 6 shows an example of the waveform of the sample audio data used in the voiceprint matching support system 100. Figure 7 is a conceptual diagram that conceptually shows an example of the sample audio data used in the voiceprint matching support system 100, and schematically shows the waveform of the sample audio data in Figure 6.

[0056] The audio data to be processed, acquired by the audio acquisition device 30 and processed by the server 10, will have a waveform like the one shown in Figure 4. The audio data to be processed, as illustrated in Figure 4, can be divided into one or more speakers or one or more noises depending on the features, for example, by frequency analysis or clustering. Figure 5 shows a schematic example of the features for speaker A, speaker B, speaker C, and noise. The features are values ​​that represent various types of features, including volume and sound pressure.

[0057] Furthermore, the sample speech data acquired by the sample speech acquisition device 40 and used for extraction processing on the server 10 will have a waveform like the one shown in Figure 6. Figure 6 shows an example of sample speech data from speaker C. For comparison with Figure 5, an example of the feature quantities of the sample speech data with the waveform exemplified in Figure 6 is schematically shown in Figure 7. Also, Figure 5 is merely an illustration to explain an example of the overlap of multiple speakers and noise (background noise), and is only included to explain that the feature quantities of speaker C shown in Figure 7 can be extracted.

[0058] Examples of displaying time intervals will be explained with reference to Figures 8 to 11. Figure 8 shows an example of displaying the extracted time intervals for the waveform of the audio data to be processed in Figure 4. Figures 9 to 11 show other examples of displaying the extracted time intervals for the waveform of the audio data to be processed in Figure 4.

[0059] The server 10 can instruct the display unit 24 to display the extracted time intervals in a different display format from other intervals, while displaying the waveform of the audio data to be processed, including the time axis corresponding to the audio data to be processed, as illustrated in Figure 8. The display unit 24 can then perform such a display. In the example in Figure 8, rectangular regions c1 and cs2 are shown on the waveform, indicating the calculation target time for which the similarity score is above a predetermined threshold. Here, an example is given in which the similarity score is in the range of 0 to 100 and the predetermined threshold is set to 70. In Figure 8, the rectangular regions c1 and cs2 indicating the calculation target time for the score value correspond to the feature quantities c1 and c2 of the speaker C feature quantities in Figure 5, respectively.

[0060] The display format is not limited to this; extracted time intervals can also be highlighted, or higher similarity scores can be displayed with darker gradations or color differences. An example of the latter is to divide the similarity score into multiple threshold ranges and display each range in a different format. This allows for the display of highly relevant time intervals in a ranked format based on the similarity score, making it easier for users to visually identify the degree of relevance. The ranked number can also be displayed. Another example of a display format is to display the feature quantities exemplified in Figure 5 instead of the waveform in Figure 4, as shown in the example in Figure 8.

[0061] Furthermore, while an example showing a waveform including a time axis was given, it is also possible to simply display only the time axis without displaying the waveform. For example, as illustrated in Figure 9, rectangular regions c1 and cs2 can be displayed on the time axis indicating the calculation target time for which the similarity score is above a predetermined threshold. In Figure 9, the rectangular regions c1 and cs2, which indicate the magnitude of the similarity score (height showing the value) and the calculation target time for the similarity score, correspond to the regions of feature quantities c1 and c2 among the speaker C features in Figure 5, respectively.

[0062] The display example in Figure 8 also shows the similarity score value and corresponds to the following process. That is, the server 10 can instruct the display unit 24 to display the waveform of the audio data to be processed, including the time axis corresponding to the audio data to be processed, and to visualize and display the similarity score for at least the extracted time interval. The display unit 24 can then perform such a display. Here, visualization of the similarity score may mean visualizing the similarity score value itself, as in the example in Figure 8, or visualizing it in a way that shows the magnitude indicated by the similarity score value, as in the example in Figure 9.

[0063] The server 10 can also instruct the display unit 24 to display only the waveform or time axis of the audio data to be processed, including the time axis corresponding to the audio data to be processed, as illustrated in Figure 10, and to display the density of similarity scores for at least the extracted time intervals. In other words, the server 10 can instruct the display unit 24 to visualize and display the density of intervals with high similarity scores. Intervals with high similarity scores can refer to intervals that are higher than, for example, other predetermined thresholds. The above predetermined thresholds can be set lower than the above predetermined thresholds. The display unit 24 can then perform such a display.

[0064] In the example in Figure 10, the density is displayed with no hatching for sparse areas and hatching for dense areas, with the hatching density increasing as the density increases. However, in the example in Figure 10, the predetermined time interval that serves as the extraction unit is shorter than in the example in Figure 8. The display of density is not limited to this example; any method that makes it possible to visually identify areas with high similarity scores is acceptable, and the density can be represented using a so-called one-dimensional heatmap or a similar display format.

[0065] Furthermore, as can be seen from the difference between Figure 8 and Figure 10, the server 10 may also be equipped with an interval setting unit that sets the time interval (the predetermined time interval) which will be the extraction unit in the extraction process. This interval setting unit can be implemented, for example, by storing an interval setting program that performs such setting in the storage unit 12 in a state that can be executed by the control unit 11. This allows the user to select the section to listen to while changing the suggested content by changing the time interval.

[0066] Such time interval settings can be accepted through the user interface (UI). For example, the server 10 can, either spontaneously or upon request from the user terminal 20, present a UI 50, as illustrated in Figure 11, to the user terminal 20 and display it on the display unit 24. When the user enters an interval in the input field 55 for the extracted time interval in the UI 50 displayed on the display unit 24, the graph 58, which includes information indicating the time interval displayed in the area below, will be changed and displayed based on the entered time interval.

[0067] In this example, Graph 58 shows an example where the example in Figure 8 is displayed. However, it is also possible to display other graphs, such as the graphs exemplified in Figure 9 or Figure 10, and it is also possible to allow selection of the graph type.

[0068] UI50 includes an input area 51 in addition to the graph 58, and this input area 51 can include an input field 55. The input area 51 can also include other elements, such as a button 52 to delete low score areas, a button 53 to delete while keeping selected areas, a button 54 to delete selected areas, an input field 56 for the extraction threshold, and a button 57 to create a voiceprint matching target file.

[0069] Furthermore, the server 10 may also include a threshold setting unit for setting the predetermined thresholds described above. This threshold setting unit can be implemented, for example, by storing a threshold setting program for performing such settings in the storage unit 12 in a state that can be executed by the control unit 11. Such predetermined threshold settings can also be accepted via the UI. For example, the server 10 can, either spontaneously or upon request from the user terminal 20, present the user terminal 20 with a UI 50 including an extraction threshold input field 56 as illustrated in Figure 11, and display it on the display unit 24. When the user enters an interval in the extraction threshold input field 56 in the UI 50, the graph 58 displayed in the area below will be changed and displayed based on the entered extraction threshold.

[0070] Furthermore, the server 10 may include a deletion unit that deletes data from time intervals other than those extracted from the audio data to be processed. This deletion unit can be implemented as a function of the control unit 11, for example, by storing a deletion program that performs such deletions in an executable state in the control unit 11. Such deletions can also be accepted via the UI.

[0071] For example, the server 10 can, either spontaneously or upon request from the user terminal 20, present a UI 50 to the user terminal 20 that includes at least one of the buttons 52-54 as illustrated in Figure 11, and display it on the display unit 24. The user can delete the area indicated by the button from the audio data to be processed by selecting one of the buttons 52-54 on the UI 50 displayed on the display unit 24. Here, if button 52 is selected, data corresponding to a similarity score below a predetermined threshold is deleted from the audio data to be processed. Buttons 53 and 54 become selectable by selecting a time area on the graph 58 in advance. It is also desirable to configure the system so that time intervals such as areas cs1 and cs2 can be selected at this time.

[0072] Furthermore, when ultimately generating a voiceprint matching target file, selecting the voiceprint matching target file creation button 57 allows the data resulting from the removal of such areas from the processed audio data to be saved as a voiceprint matching target file. The save location can be at least one of the storage unit 12 of the server 10 and the storage unit 22 of the user terminal 20, but other save locations are also possible, and the save location may also be selectable via UI 50.

[0073] Next, in relation to the effects of this embodiment, we will describe examples of work in which only recorded audio data and video data are used as the audio data to be processed, with reference to Figures 12 and 13, respectively, as comparative examples. Figure 12 is a flowchart illustrating the process of creating a voiceprint matching target file according to Comparative Example 1, and Figure 13 is a flowchart illustrating the process of creating a voiceprint matching target file according to Comparative Example 2.

[0074] The example shown in Figure 12 illustrates the procedure for creating a voiceprint matching file from the audio data to be processed.

[0075] First, the user plays the recording data to be processed (step S11) and listens to it. In Comparative Example 1, the recording data to be processed is partially or entirely saved with the generated text transcript. The user determines whether or not transcript data with a clear subject can be referenced (exists) in the recording data to be processed (step S12). If it cannot be referenced, the recording data is discarded (step S18), and the process ends. If the answer in step S12 is YES, the user listens only to the section in which the subject is speaking and performs the task of segmenting the waveform (step S13).

[0076] Next, the user determines whether the subject's voice overlaps with other voices or noise in the recording data processed in step S13 (step S14), and if it does, discards the overlapping section in the recording data (step S15). This partially deletes the recording data processed in step S13. Next, the user determines whether the duration of the subject's voice is longer than a specified value, such as 5 seconds (step S16). If the answer in step S14 is NO, the process proceeds to step S16 without going through step S15. If the answer in step S16 is YES, the remaining recording data up to that point is registered as a voiceprint matching target file (step S17), and the process ends. On the other hand, if the answer in step S16 is NO, the remaining recording data up to that point is registered as a component file so that it is longer than or equal to the specified value (step S19), and the process ends.

[0077] Comparative Example 2, shown in Figure 13, illustrates the procedure for creating a voiceprint matching target file from video data that includes audio recordings.

[0078] First, the user plays the video data to be processed (step S21) and watches it. In Comparative Example 2, since video exists, the user checks the video and determines whether or not the subject is clearly visible in the video data to be processed (step S22). If it is not clear, the video data is excluded from processing (step S28), and the process ends. If the answer in step S22 is YES, the user listens only to the section in which the subject is speaking and performs the task of segmenting the waveform (step S23).

[0079] Next, the user determines whether the subject's voice overlaps with other voices or noise in the video data processed in step S23 (step S24), and if it does, discards the overlapping section in the video data (step S25). This partially deletes the video data processed in step S23. Next, the user determines whether the duration of the subject's voice is greater than or equal to a specified value, such as 5 seconds (step S26). If the answer in step S24 is NO, the process proceeds to step S26 without going through step S25. If the answer in step S26 is YES, the audio data of the remaining video data up to that point is registered as a voiceprint matching target file (step S27), and the process ends. On the other hand, if the answer in step S26 is NO, the remaining video data up to that point is treated as data not to be processed (step S28), and the process ends. If the answer in step S26 is NO, the remaining video data up to that point may be registered as a component file to ensure it is greater than or equal to the specified value, similar to step S19, before the process ends.

[0080] In both Comparative Examples 1 and 2, it is clear that creating the files to be used for voiceprint matching requires considerable effort and time. In contrast, in this embodiment, by employing the extraction process described above, audio data in sections where the subject is conversing with others and where the speech of others is mixed in can be excluded from the extraction. Therefore, according to this embodiment, the user performing the task can improve the efficiency of the work, which previously involved listening to (viewing) the entire recording, by focusing on the suggested sections, compared to Comparative Examples 1 and 2. In fact, since video and audio data under investigation often contain voices other than those of the subject, this embodiment is extremely beneficial.

[0081] <Variation> Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0082] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments rather than with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps shown in any of the drawings may be changed as appropriate.

[0083] Furthermore, one or more devices constituting the voiceprint matching support system relating to this disclosure may each have the following hardware configuration. Figure 14 shows an example of the hardware configuration included in the device.

[0084] The device 1000 shown in Figure 14 comprises a processor 1001, a memory 1002, and a communication interface 1003. The functions of each device can be realized by the processor 1001 reading a program stored in the memory 1002 and executing it in cooperation with the communication interface 1003.

[0085] The program described above includes, when loaded into a computer, a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disk (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals.

[0086] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0087] (Note 1) An input section for inputting sample voice data of the person to be matched for voiceprint matching, Based on the aforementioned sample audio data, an extraction unit extracts the time interval in which the subject is speaking from the voice data to be matched with the voiceprint, A display unit that presents the time interval extracted by the extraction unit to the user, A voiceprint matching support system equipped with [unspecified features]. (Note 2) The extraction unit extracts a time interval in which the subject is speaking alone. The voiceprint matching support system described in Appendix 1. (Note 3) The system includes a deletion unit that deletes data from the voiceprint matching target audio data for segments other than the time segment extracted by the extraction unit. The voiceprint matching support system described in Appendix 1 or 2. (Note 4) The extraction unit extracts the time interval based on the similarity score between the feature quantities of the sample audio data and the feature quantities of the audio data to be matched with the voiceprint. A voiceprint matching support system as described in any one of the appendices 1 to 3. (Note 5) The extraction unit extracts, as the time interval, the interval in which the similarity score in the voice data subject to voiceprint matching is equal to or greater than a predetermined threshold. The voiceprint matching support system described in Appendix 4. (Note 6) The system includes a threshold setting unit for setting the predetermined threshold, The voiceprint matching support system described in Appendix 5. (Note 7) The display unit displays a time axis corresponding to the voiceprint matching target audio data or a waveform of the voiceprint matching target audio data including the time axis, and visualizes and displays the similarity score for at least the time interval extracted by the extraction unit. A voiceprint matching support system as described in any one of the items 4 to 6 of the appendix. (Note 8) The display unit displays a time axis corresponding to the voice data to be matched, or the waveform of the voice data to be matched including the time axis, and visualizes and displays the density of the intervals with high similarity scores for at least the time intervals extracted by the extraction unit. A voiceprint matching support system as described in any one of the items 4 to 7 of the appendix. (Note 9) The display unit displays the time interval extracted by the extraction unit in a different display format from other intervals, while displaying the time axis corresponding to the voice data to be matched or the waveform of the voice data to be matched including the time axis. A voiceprint matching support system as described in any one of the appendices 1 to 8. (Note 10) The extraction unit includes an interval setting unit for setting the time interval that will be the extraction unit. A voiceprint matching support system as described in any one of the appendices 1 to 9. (Note 11) Input process involves inputting sample voice data of the person to be matched for voiceprint matching, Based on the aforementioned sample audio data, an extraction process is performed to extract the time interval in which the subject is speaking from the voiceprint matching target audio data. A display process that causes the extracted time interval to be displayed on a display device for the user to see, A voiceprint matching support method equipped with [a specific feature]. (Note 12) The extraction process extracts time intervals in which the subject is speaking alone, as the time intervals. The voiceprint matching support method described in Appendix 11. (Note 13) This process includes deleting data from the voiceprint matching target audio data for time intervals other than those extracted in the extraction process, The voiceprint matching support method described in Appendix 11 or 12. (Note 14) The extraction process is performed based on the similarity score between the feature quantities of the sample audio data and the feature quantities of the audio data to be matched with the voiceprint. A voiceprint matching support method described in any one of the appendices 11 to 13. (Note 15) The extraction process extracts, as the time interval, the interval in which the similarity score in the voice data subject to voiceprint matching is equal to or greater than a predetermined threshold. The voiceprint matching support method described in Appendix 14. (Note 16) The process includes setting the predetermined threshold, The voiceprint matching support method described in Appendix 15. (Note 17) The display process includes, in a display device, a process that visualizes and displays the similarity score for at least the time interval extracted in the extraction process, while displaying a time axis corresponding to the voice data to be matched or the waveform of the voice data to be matched including the time axis. A voiceprint matching support method described in any one of the appendices 14 to 16. (Note 18) The display process includes, in a state in which the display device displays a time axis corresponding to the voiceprint matching target audio data or the waveform of the voiceprint matching target audio data including the time axis, a process that visualizes and displays the density of the intervals with high similarity scores for at least the time intervals extracted in the extraction process. A voiceprint matching support method described in any one of the appendices 14 to 17. (Note 19) The display process includes, in a state where the display device displays a time axis corresponding to the voiceprint matching target audio data or the waveform of the voiceprint matching target audio data including the time axis, a process that displays the time interval extracted in the extraction process in a different display format from other intervals. A voiceprint matching support method described in any one of the appendices 11 to 18. (Note 20) This includes a process for setting the time interval that will be the extraction unit in the extraction process, A voiceprint matching support method described in any one of the appendices 11 to 19. (Note 21) On the computer, Input process involves inputting sample voice data of the person to be matched for voiceprint matching, Based on the aforementioned sample audio data, an extraction process is performed to extract the time interval in which the subject is speaking from the voiceprint matching target audio data. A display process that causes the extracted time interval to be displayed on a display device for the user to see, A program that performs voiceprint matching support processing, including the necessary steps. (Note 22) The extraction process extracts time intervals in which the subject is speaking alone, as the time intervals. The program described in Appendix 21. (Note 23) The voiceprint matching support process includes a process to delete data from the voiceprint matching target audio data other than the time interval extracted in the extraction process. The program described in Appendix 21 or 22. (Note 24) The extraction process is performed based on the similarity score between the feature quantities of the sample audio data and the feature quantities of the audio data to be matched with the voiceprint. The program described in any one of the items in Appendix 21 to 23. (Note 25) The extraction process extracts, as the time interval, the interval in which the similarity score in the voice data subject to voiceprint matching is equal to or greater than a predetermined threshold. The program described in Appendix 24. (Note 26) The voiceprint matching support process includes a process for setting the predetermined threshold, The program described in Appendix 25. (Note 27) The display process includes, in a display device, a process that visualizes and displays the similarity score for at least the time interval extracted in the extraction process, while displaying a time axis corresponding to the voice data to be matched or the waveform of the voice data to be matched including the time axis. The program described in any one of the items in Appendix 24-26. (Note 28) The display process includes, in a state in which the display device displays a time axis corresponding to the voiceprint matching target audio data or the waveform of the voiceprint matching target audio data including the time axis, a process that visualizes and displays the density of the intervals with high similarity scores for at least the time intervals extracted in the extraction process. The program described in any one of the items in Appendix 24-27. (Note 29) The display process includes, in a state where the display device displays a time axis corresponding to the voiceprint matching target audio data or the waveform of the voiceprint matching target audio data including the time axis, a process that displays the time interval extracted in the extraction process in a different display format from other intervals. The program described in any one of the items in Appendix 21 to 28. (Note 30) The voiceprint matching support process includes a process for setting the time interval that will be the extraction unit in the extraction process. The program described in any one of the items in Appendix 21 to 29.

[0088] This application claims priority based on Japanese Patent Application No. 2022-194509, filed on 5 December 2022, and incorporates all of its disclosures herein. [Explanation of symbols]

[0089] 1. 100 Voiceprint Matching Support System 1a Input section 1b Extraction part 1c Display section 10 servers 11 Control Unit 12 Storage section 13 Communications Department 20 User Terminals 21 Control Unit 22 Memory section 23 Communications Department 24 Display 1000 devices 1001 Processor 1002 memory 1003 Communication Interface

Claims

1. An input method for inputting sample voice data of the person to be matched for voiceprint matching, Based on the aforementioned sample audio data, an extraction means for extracting time intervals in which the subject is speaking alone, by excluding sections containing utterances by persons other than the subject from the audio data to be matched with a voiceprint, A display means for presenting the time interval extracted by the extraction means to the user, Equipped with, The extraction means extracts the time interval based on the similarity score between the feature quantities of the sample audio data and the feature quantities of the audio data to be matched with the voiceprint. The display means displays a time axis corresponding to the voice data to be matched, or the waveform of the voice data to be matched including the time axis, and visualizes and displays the density of the intervals with high similarity scores for at least the time intervals extracted by the extraction means. Voiceprint matching support system.

2. The system includes a deletion means for deleting data from the voiceprint matching target audio data other than the time interval extracted by the extraction means, The voiceprint matching support system according to claim 1.

3. The extraction means extracts, as the time interval, the interval in which the similarity score in the voice data subject to voiceprint matching is equal to or greater than a predetermined threshold. The voiceprint matching support system according to claim 1 or 2.

4. The system includes threshold setting means for setting the predetermined threshold, The voiceprint matching support system according to claim 3.

5. The extraction means includes an interval setting means for setting the time interval that will be the extraction unit, The voiceprint matching support system according to claim 1 or 2.

6. Input process involves inputting sample voice data of the person to be matched for voiceprint matching, Based on the aforementioned sample audio data, an extraction process is performed to extract time intervals in which the subject is speaking alone, by excluding sections containing utterances by persons other than the subject from the audio data to be matched with voiceprints. A display process that causes the extracted time interval to be displayed on a display device for the user to see, Equipped with, The extraction process extracts the time interval based on the similarity score between the feature quantities of the sample audio data and the feature quantities of the audio data to be matched with the voiceprint. The display process displays the time axis corresponding to the voiceprint matching target audio data or the waveform of the voiceprint matching target audio data including the time axis, and visualizes and displays the density of the intervals with high similarity scores for at least the time intervals extracted in the extraction process. Voiceprint matching support method.

7. On the computer, Input process involves inputting sample voice data of the person to be matched for voiceprint matching, Based on the aforementioned sample audio data, an extraction process is performed to extract time intervals in which the subject is speaking alone, by excluding sections containing utterances by persons other than the subject from the audio data to be matched with voiceprints. A display process that causes the extracted time interval to be displayed on a display device for the user to see, Includes, The extraction process extracts the time interval based on the similarity score between the feature quantities of the sample audio data and the feature quantities of the audio data to be matched with the voiceprint. The display process displays the time axis corresponding to the voiceprint matching target audio data or the waveform of the voiceprint matching target audio data including the time axis, and visualizes and displays the density of the intervals with high similarity scores for at least the time intervals extracted in the extraction process. A program that performs voiceprint matching support processing.

Citation Information

Patent Citations

  • Utterance identification system and method therefor

    JP2005321530A

  • Voice processing device

    JP2005326576A

  • Speaker display system, speaker display method and speaker display program

    JP2008032825A

  • Harmful customer detection system, its method and harmful customer detection program

    JP2010113167A

  • Passive and continuous multi-speaker voice biometrics

    US20210326421A1