Extraction system and extraction method

Abstract

Provided is an extraction system for extracting a speaker feature amount from input audio including voices of a plurality of speakers. To enable extraction of a speaker feature amount with a single extraction model, without relying on separation of sound sources, the extraction system comprises: an extraction model for outputting a speaker feature amount of one or more speakers on the basis of input audio including voices of the speakers; a training unit for training the extraction model by using training data that includes mixed sound of a plurality of speaker voices and identification information for identifying the speaker voices; and an estimation unit for inputting input audio including voices of one or more speakers to the trained extraction model and estimating the speaker feature amounts of the speakers.

Description

Extraction system and extraction method

[0001] The present invention relates to an extraction system and an extraction method.

[0002] There are techniques for extracting speaker features from speech from multiple speakers. For example, a technique is known in which speech is separated into individual speakers in advance and speaker features are extracted from each separated speech (see, for example, Non-Patent Document 1). Also, a technique is known in which speaker features for two speakers are extracted assuming that the input speech is a mixture of speech from two speakers (see, for example, Non-Patent Documents 2 and 3).

[0003] Xiong Xiao et al., "Microsoft Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2020", in Proc. ICASSP 2021.Tobias Cord-Landwehr et al., "A Teacher-Student approach for extracting informative speaker embeddings from speech mixtures", in Proc. Interspeech 2023.Hyewon Han, Soo-Whan Chung, Hong-Goo Kang, "MIRNet: Learning multiple identities representations in overlapped speech", in Proc. Interspeech 2020.

[0004] The technology disclosed in Non-Patent Document 1 enables feature extraction for each speaker by applying sound source separation prior to the extraction of speaker features. However, it is empirically known that sound source separation causes distortion in the speech, which adversely affects subsequent speech processing. Furthermore, there is a risk that the speaker characteristics of the speaker features may differ from the original due to distortion caused by the separation.

[0005] Furthermore, the techniques disclosed in Non-Patent Documents 2 and 3 are capable of outputting speaker features corresponding to each speaker even when the number of speakers included in the speech is multiple (these documents particularly mention the case where there are two speakers). However, since it is necessary to use different models depending on whether the number of speakers included in the input speech is one or two, it is necessary to prepare different models according to the number of speakers.

[0006] The embodiment of the present invention has been made in consideration of the above-mentioned problems, and in an extraction system that extracts speaker features from input speech containing speech from multiple speakers, it is possible to extract speaker features using a single extraction model without separating sound sources.

[0007] In order to solve the above-described problems, an extraction system according to an embodiment of the present invention includes an extraction model that outputs speaker features of one or more speakers based on input speech containing the speech of the speakers; a training unit that trains the extraction model using training data that includes a mixture of speeches of a plurality of speakers and identification information that identifies the speeches of the speakers; and an estimation unit that inputs input speech containing the speeches of one or more speakers to the trained extraction model and estimates speaker features of the speakers.

[0008] According to an embodiment of the present invention, in an extraction system that extracts speaker features from input speech including speech from multiple speakers, it becomes possible to extract speaker features using a single extraction model without separating sound sources.

[0009] FIG. 1 is a diagram illustrating an example of the configuration of an extraction system according to the present embodiment; FIG. 2 is a diagram illustrating an example of the configuration of a conventional speaker feature extractor; FIG. 3 is a diagram illustrating an example of a conventional pooling unit; FIG. 4 is a diagram illustrating the pooling unit according to the present embodiment; FIG. 4 is a diagram illustrating an example of the configuration of an extraction system according to Example 1; FIG. 5 is a flowchart (1) illustrating an example of the processing of the extraction system according to Example 1; FIG. 6 is a diagram illustrating the pooling unit according to Example 1; FIG. 7 is a diagram illustrating the estimation processing according to Example 1; FIG. 8 is a flowchart (2) illustrating an example of the processing of the extraction system according to Example 1; FIG. 9 is a diagram illustrating an example of the configuration of an extraction system according to Example 2; FIG. 10 is a diagram illustrating the pooling unit according to Example 2; and FIG. 11 is a flowchart illustrating an example of the processing of the extraction system according to Example 2.

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] <Overview> (Background) Spoken dialogue understanding is a technology for recognizing and analyzing conversational speech, and plays an important role in communication involving many people. For example, it is expected to be applied to accurately recording the content of meetings or business negotiations involving many people, supporting consensus building, and improving the quality of discussions. To achieve spoken dialogue understanding, it is necessary not only to perform speech recognition to transcribe spoken speech, but also to determine who spoke each utterance. A simple implementation is to transcribe each utterance, then extract speaker features from the speech waveform of each utterance section to determine the speaker.

[0012] Here, the speaker feature is a vector representation of the identity of a speaker, and by evaluating the similarity, it is possible to determine, for example, which person from among those registered in the system spoke a certain utterance (speaker recognition).In addition, by evaluating the similarity, it is also possible to determine whether a certain utterance and another utterance in a conversation were spoken by the same speaker (speaker diarization).

[0013] Speaker features can also be used for target speaker speech recognition. Target speaker speech recognition is a technology that transcribes only the speech of a speaker of interest. This is made possible by inputting the speaker features of the speaker to be recognized in addition to the speech waveform into the model. Target speaker speech recognition makes it possible to transcribe only the speech of the speaker of interest, for example, at a party attended by an unspecified number of people.

[0014] (Problem) Conventional speaker feature extractors assume that the input is the voice of a single speaker, and are unable to extract speaker features from voices that may be spoken by multiple speakers.

[0015] To solve this problem, efforts have been made to extract speaker features from the speech of multiple speakers. For example, the technology disclosed in Non-Patent Document 1 applies sound source separation prior to the extraction of speaker features, making it possible to extract features for each speaker. However, it is empirically known that sound source separation causes distortion in the speech, which adversely affects subsequent speech processing. Furthermore, there is a risk that the speaker characteristics of the speaker features may differ from their original characteristics due to distortion caused by separation.

[0016] Furthermore, the techniques disclosed in Non-Patent Documents 2 and 3 are capable of outputting speaker features corresponding to each speaker even when the number of speakers included in the speech is multiple (these documents particularly mention the case where there are two speakers). However, since it is necessary to use different models depending on whether the number of speakers included in the input speech is one or two, it is necessary to prepare different models according to the number of speakers.

[0017] In order to solve the above problems, the extraction system according to this embodiment has a configuration as shown in FIG. 1, for example.

[0018] <Configuration of Extraction System> Fig. 1 is a diagram showing an example of the configuration of an extraction system according to this embodiment. The extraction system 1 is a system that extracts speaker features of a speaker based on input speech containing speech from one or more speakers. As mentioned above, the techniques disclosed in Non-Patent Documents 2 and 3 require different models to be used depending on whether the input speech contains one or two speakers, and therefore require the preparation of separate models according to the number of speakers. In contrast, the extraction system 1 according to this embodiment can extract speaker features of a speaker using a single extraction model, regardless of the number of speakers contained in the input speech.

[0019] The extraction system 1 is configured, for example, by one extraction device 100, as shown in FIG. 1 . However, this is just one example, and the extraction system 1 may be configured by multiple information processing devices. For example, the extraction system 1 may be divided into an extraction device used during learning and an extraction device used during inference. Here, as an example, the following description will be given assuming that the extraction system 1 is configured by one extraction device 100.

[0020] The extraction device 100 is, for example, an information processing device having a computer configuration, or a system including multiple computers. The extraction device 100 realizes each functional configuration shown in Fig. 1 by, for example, executing a predetermined program on one or more computers included in the extraction device 100. In the example of Fig. 1, the extraction device 100 includes a mixed sound generation unit 101, a learning unit 102, an estimation unit 103, and an extraction model 110 (extraction models 110-1 and 110-2).

[0021] In the following description, the extraction model 110 before learning will be referred to as an extraction model 110-1, and the learned extraction model 110 will be referred to as an extraction model 110-2. Furthermore, when there is no particular distinction between the extraction model 110 before learning and the learned extraction model 110, it will simply be referred to as the extraction model 110.

[0022] The mixed sound generation unit 101 executes mixed sound generation processing to generate a mixed sound of multiple speaker voices from original training data 10. The original training data 10 includes a large amount of speaker voices 11 with speaker IDs. The speaker ID is identification information for identifying a speaker or a speaker voice. The speaker voice is audio data obtained by recording the speaker's speech.

[0023] For example, the mixed sound generation unit 101 selects speaker voices of n speakers (n is an integer of 2 or more) from the original training data 10 and adds them together at an arbitrary signal ratio to generate a mixed sound. The generated mixed sound, together with a speaker ID that identifies the speaker voice (or speaker), is used as training data for training the extraction model 110.

[0024] In addition, when a mixed sound of multiple speaker voices and identification information for identifying the speaker voices are obtained (when data exists in which the mixed sound and the speakers contained therein are known), the data can be used as training data for training the extraction model 110, and in this case, the extraction system 1 does not need to have the mixed sound generation unit 101.

[0025] The training unit 102 executes a training process to train the extraction model 110 using training data including a mixed sound of multiple speaker voices and identification information (e.g., speaker ID) for identifying the speaker voices. For example, the training unit 102 trains the pre-training extraction model 110-1 using training data including the mixed sound generated by the mixed sound generation unit 101 and the speaker ID included in the mixed sound, thereby creating a trained extraction model 110-2. Note that a specific example of the processing by the training unit 102 will be described later.

[0026] The estimation unit 103 inputs input speech including speech of one or more speakers to the trained extraction model 110-2, and executes estimation processing to estimate speaker features of the speakers.

[0027] As an example, the extraction model 110 receives input speech and the number of speakers as input data and outputs speaker features for the number of speakers. In this case, the estimation unit 103 inputs the input speech and the number of speakers into the trained extraction model 110-2 and estimates speaker features for the number of speakers. For example, if the number of speakers is one, the estimation unit 103 estimates speaker features for one speaker, and if the number of speakers is two, the estimation unit 103 estimates and outputs speaker features for two speakers.

[0028] As another example, the extraction model 110 uses input speech as input data, estimates the number of speakers (one or more), and outputs speaker features equal to the estimated number of speakers. In this case, the estimation unit 103 inputs the input speech to the trained extraction model 110-2 and estimates speaker features equal to the number of speakers.

[0029] As another example, the extraction model 110 receives input data including an input speech and information indicating a speech section of an extraction target speaker from among one or more speakers from which speaker features are to be extracted and / or a speaker other than the extraction target speaker, and outputs the speaker features of the extraction target speaker. The learning unit 102 also uses information on the speech sections of the speeches of multiple speakers to train the extraction model 110.

[0030] In this case, the estimation unit 103 inputs the input speech and information indicating the speech sections of the extraction target person and / or speakers other than the extraction target person among one or more speakers into the trained 110-2 extraction model, and estimates the speaker features of the extraction target person.

[0031] The extraction model 110 is a speaker feature extractor realized by a neural network or the like that outputs speaker features of a speaker based on input speech including speech of one or more speakers.

[0032] Fig. 2 is a diagram showing an example of the configuration of a conventional speaker feature extractor. As shown in Fig. 2, a conventional speaker feature extractor 200 based on a general neural network includes an encoder unit 201, a pooling unit 202, a conversion unit 203, etc.

[0033] The encoder 201 converts a variable-length acoustic feature sequence extracted from input speech into an embedding vector of the same sequence length. The pooling unit 202 compresses the variable-length embedding vector sequence in the time direction to generate a single embedding vector. The converter 203 converts the embedding vector into speaker features.

[0034] Fig. 3 is a diagram illustrating an example of a conventional pooling unit. The conventional pooling unit 202 calculates attention weight A using equation (2) in Fig. 3 for the output E from the encoder unit 201 as shown in equation (1) in Fig. 3. The sum of each row of attention weight A is 1, and a weighted average is taken by attention through utterance-level embedding, and compressed in the time direction to generate a single embedding vector.

[0035] Here, since the pooling unit 202 generally outputs one vector, that is, the input to the conventional speaker feature extractor 200 is assumed to be the voice of one speaker. However, with this method, for example, when transcribing a dialogue containing multiple speakers, speaker features of the multiple speakers are extracted sequentially and speech recognition of the target speaker is performed for each of them. Therefore, with this method, since speaker features must be extracted from speech that may be spoken by multiple speakers, it is inconvenient to assume the voice of one speaker.

[0036] To solve this problem, efforts have been made to extract speaker features from the speech of multiple speakers. For example, Non-Patent Document 1 discloses a method of separating speech into individual speakers in advance and extracting speaker features from each separated speech. Furthermore, Non-Patent Documents 2 and 3 disclose models that assume that the input is a mixture of speech from two speakers and extract speaker features for two speakers.

[0037] The technology disclosed in Non-Patent Document 1 enables feature extraction for each speaker by applying sound source separation prior to the extraction of speaker features. However, it is empirically known that sound source separation causes distortion in the speech, which adversely affects subsequent speech processing. Furthermore, there is a risk that the speaker characteristics of the speaker features may differ from the original due to distortion caused by the separation.

[0038] The techniques disclosed in Non-Patent Documents 2 and 3 are capable of outputting speaker features corresponding to each speaker even when the number of speakers included in the speech is multiple. However, since it is necessary to use different models depending on whether the number of speakers included in the input speech is one or two, not only is it necessary to prepare different models depending on the number of speakers, but it is also necessary to know the number of speakers included in the input speech.

[0039] Therefore, in this embodiment, a vector c that conditions the speaker is introduced into the calculation of the attention weight used in the pooling unit, and only the features of the speaker that meet the conditions are extracted from the speech that may include multiple speakers.

[0040] 4 is a diagram for explaining the pooling unit according to this embodiment. This embodiment particularly targets a speaker feature extractor (extraction model 110) that uses an attention function (attention mechanism) 401 in a pooling unit 400. The attention function 401 determines which time and which dimension of the embedding vector sequence extracted from speech should be focused on to extract speaker features.

[0041] A typical speaker feature extractor 200 targets speaker feature extraction from the speech of a single speaker, so attention is focused on sections containing speaker information. In this embodiment, this is extended to input speech from multiple speakers. Specifically, it is assumed that each time and each dimension of the embedding vector sequence contains information from at most one speaker, and by using a conditioning vector c in attention calculations, attention is focused only on sections containing speaker information that meets the conditions. This allows the speaker feature extractor (extraction model 110) according to this embodiment to extract speaker features corresponding to the speaker.

[0042] Specifically, the speaker feature extractor (extraction model 110) according to this embodiment calculates the attention weight A by equation (3) in Fig. 4. In equation (3), W ｃ A term c is added, where c is the conditioning vector and W ｃis a parameter. Note that the conditioning vector will be explained by taking an example as an example.

[0043] [Example 1] Fig. 5 is a diagram illustrating an example of the configuration of an extraction system according to Example 1. In Example 1, as shown in Fig. 5, an example will be described in which the estimation unit 103 estimates speaker features for the number of speakers using input speech including speech from one or more speakers, the number of speakers, and a trained extraction model 110-2. Note that the number of speakers may be estimated by the trained extraction model 110-2 based on the input speech.

[0044] As shown in Fig. 3, the conventional pooling unit 202 uses only the time-series embedding sequence derived from the encoder to calculate attention weights used for extracting speaker features. On the other hand, the pooling unit 400 of the extraction model 110 according to the first embodiment also uses the attention weights used for extracting speaker features up to the n-1th speaker when extracting the nth speaker feature (n is an integer equal to or greater than 2). This enables the extraction model 110 according to the first embodiment to obtain the nth speaker feature by focusing on areas that have not previously been attended to (areas that did not contain information about the 1st to n-1th speakers).

[0045] <Processing Flow> Next, an example of a method for extracting speaker features according to the first embodiment will be described.

[0046] (Processing 1 of the extraction system) Fig. 6 is a flowchart (1) illustrating an example of processing of the extraction system according to Example 1. This processing illustrates an example of processing executed by the extraction system 1 when the input data input to the estimation unit 103 in Fig. 5 is input speech including speech of one or more speakers and the number of speakers.

[0047] In step S601, the mixed sound generation unit 101 generates a mixed sound of multiple speaker voices from the original training data 10. For example, the mixed sound generation unit 101 selects speaker voices of any number of speakers from the original training data 10 and adds them at any signal ratio to generate a mixed sound. The mixed sound generation unit 101 may also perform data extension, as necessary, by superimposing noise on the mixed sound or by convolving an impulse response to add reverberation. For efficient training, the length of the mixed sound is generally fixed (e.g., 3 seconds). However, each speaker does not necessarily have to speak for 3 seconds. For example, a mixed sound in which speaker 1 speaks between 0 and 2 seconds and speaker 2 speaks between 1 and 3 seconds may be used. The mixed sound generated by the mixed sound generation unit 101 and speaker IDs identifying the speaker voices (or speakers) included in the mixed sound are used as training data in subsequent training.

[0048] In step S602, the training unit 102 trains the extraction model 110 using training data including the mixed sound of multiple speaker voices generated by the mixed sound generation unit 101 and a speaker ID for identifying the speaker voice (or speaker).

[0049] Here, the extraction model 110 is assumed to be a model based on a neural network. Such a model based on a neural network can generally be optimized by mini-batch learning based on the stochastic gradient descent method. In this case, the number of speakers may differ for each mini-batch, or different values ​​may be used for each sample in the mini-batch. The speaker features thus obtained, corresponding to the number of speakers, are optimized using a speaker identification loss based on, for example, cross entropy. Here, it is not obvious which speaker ID each of the speaker features, corresponding to the number of speakers, is associated with. However, as disclosed in Non-Patent Document 3, for example, the model can be trained using a permutation-free loss.

[0050] 7 is a diagram illustrating the pooling unit according to the first embodiment. As described above, in this embodiment, it is assumed that each time and each dimension of the embedding vector sequence contains information about at most one speaker, and a conditioning vector c is used for attention calculation. This allows attention to be applied only to parts containing speaker information that meets the conditions, thereby realizing the extraction of speaker features corresponding to that speaker.

[0051] Furthermore, in the first embodiment, a method is realized for extracting speaker features corresponding to K speakers (K is an integer equal to or greater than 1) from mixed speech of K speakers. Specifically, in order to calculate attention weights used for extracting the n-th speaker feature in the pooling unit 800, instead of using only the time-series embedding sequence derived from the encoder as in the conventional configuration described in Fig. 3, attention weights used for extracting the n-1th speaker features are also used as shown in Fig. 7.

[0052] For example, the pooling unit 700 according to the first embodiment calculates the weight A of the attention of the n-th person using the following equation (4): （ｎ） Calculate.

[0053] Here, the conditioning vector c according to Example 1 ｔ （ｎ） is the sum of the attention weights up to the n-1th person, and is expressed by the following equation (5).

[0054] However, a ｔ （ｍ） is a D-dimensional zero vector.

[0055] This allows the speaker feature (extraction model 110) to obtain the nth speaker feature by focusing on areas that have not previously received attention (areas that did not contain information about the 1st to n-1th speakers).

[0056] Through the processing of steps S601 and S602 (learning processing), the extraction system 1 can obtain a trained extraction model 110-2. Using this trained extraction model 110-2, the extraction system 1 executes the processing of steps S603 and S604 (inference processing). However, the inference processing can be executed at any timing after the execution of the learning processing.

[0057] Furthermore, when executing the processes of steps S603 and S604, the extraction device 100 does not need to include the mixed sound generation unit 101 and the learning unit 102. For example, the processes of steps S603 and S604 may be executed by an information processing device other than the extraction device 100 that executed the processes of steps S601 and S602.

[0058] In step S604, the estimation unit 103 inputs input speech including speech of one or more speakers from which speaker features are to be extracted and the number of speakers, which is the number of speakers included in the input speech, into the trained extraction model 110-2, and estimates speaker features for the number of speakers. For example, the estimation unit 103 acquires speaker features for the number of speakers output by the trained extraction model 110-2.

[0059] 8 is a diagram for explaining the estimation process according to Example 1. The estimation unit 103 is capable of extracting a variable number of speaker features according to the number of speakers, without the speakers being fixed by the network structure as in Non-Patent Documents 2 and 3.

[0060] As mentioned above, when a 3-second speech is used during model training, attention weights calculated for the 3-second speech are used. However, when focusing on attention weights in a certain dimension, the sum of attentions is always 1. Therefore, as shown in Figure 8, the longer the sequence length, the lower the overall attention weight value.

[0061] Therefore, for example, if the length of the input speech at the time of inference is longer than 3 seconds, the attention weight value will be low overall. As a result, for example, there is a risk that attention will be again focused on the part that was already focused on when extracting speaker features of the first speaker for the second speaker and onwards. Therefore, in this embodiment, when the length of the speech used at the time of learning is T and the length of the speech input at the time of inference is T', as shown in the following equation (6), W ｃ c ｔ （ｎ） By multiplying by T' / T, the distribution of attention weight values ​​is corrected and the influence of the input sequence length is removed.

[0062]

[0063] In step S604, the estimation unit 103 outputs the speaker features for the estimated number of speakers to a predetermined output destination.

[0064] According to the first embodiment, the processing in FIG. 6 enables the extraction device 100 (or the extraction system 1) that extracts speaker features from input speech including speech from multiple speakers to extract speaker features using a single extraction model without separating sound sources.

[0065] The speaker features output by the estimation unit 103 can be used, for example, in a speaker recognition system for determining which person among those registered in advance in the system made a certain utterance. The speaker features output by the estimation unit 103 can also be used, for example, in a speaker diarization system for evaluating similarity to determine whether a certain utterance and another utterance in a conversation were made by the same speaker. Furthermore, the speaker features output by the estimation unit 103 can also be used in a target speaker speech recognition system for transcribing only the utterance of a speaker of interest.

[0066] (Processing 2 of the extraction system) Fig. 9 is a flowchart (2) showing an example of processing of the extraction system according to the first embodiment. This processing shows an example of processing executed by the extraction system 1 when the input data input to the estimation unit 103 in Fig. 5 is input speech including speech from one or more speakers and the number of speakers is not specified. Note that the basic processing content is the same as Processing 1 of the extraction system described in Fig. 6, and therefore detailed description of the processing similar to the processing in Fig. 6 will be omitted here.

[0067] In step S901, the mixed sound generation unit 101 generates a mixed sound of the voices of a plurality of speakers from the original training data 10. This process may be similar to the process in step S601 in FIG.

[0068] In step S902, the training unit 102 trains the extraction model 110 using training data including the mixed sound of multiple speaker voices generated by the mixed sound generation unit 101 and a speaker ID for identifying the speaker voice (or speaker).

[0069] Here, by training the extraction model 110 including speaker number estimation, for example, as shown in FIG. 7, by also executing process 701 in the pooling unit 700 for speaker number estimation, inference becomes possible even when the number of speakers is unknown.

[0070] Through the processing of steps S901 and S902 (learning processing), the extraction system 1 can obtain a trained extraction model 110-2. Using this trained extraction model 110-2, the extraction system 1 executes the processing of steps S903 and S904 (inference processing). However, the inference processing can be executed at any timing after the execution of the learning processing.

[0071] Furthermore, when executing the processes of steps S903 and S904, the extraction device 100 does not need to include the mixed sound generation unit 101 and the learning unit 102. For example, the processes of steps S903 and S904 may be executed by an information processing device other than the extraction device 100 that executed the processes of steps S901 and S902.

[0072] In step S903, the estimation unit 103 inputs input speech including speech of one or more speakers from which speaker features are to be extracted to the trained extraction model 110-2, and estimates speaker features for the number of speakers. For example, the estimation unit 103 acquires speaker features for the number of speakers output by the trained extraction model 110-2.

[0073] At this time, the pooling unit 700 of the trained extraction model 110-2 executes process 701 in FIG. 7 for estimating the number of speakers. For example, when the pooling unit 700 generates the embedding vector for the nth person, it identifies whether the nth person is a speaker or a non-speaker. If the nth person is a speaker, it executes estimation processing for the (n+1)th person, and if the nth person is a non-speaker, it terminates the estimation processing. In this way, the extraction device 100 can extract speaker features for the number of speakers from input speech containing speech from one or more speakers, even when the number of speakers is unknown.

[0074] In step S904, the estimation unit 103 outputs the speaker features for the estimated number of speakers to a predetermined output destination.

[0075] By the process of FIG. 9, the extraction device 100 can extract speaker features for the number of speakers from input speech containing speech from one or more speakers, even when the number of speakers is unknown.

[0076] 10 is a diagram illustrating an example of the configuration of an extraction system according to Example 2. In Example 2, an example will be described in which the estimation unit 103 extracts speaker features of an extraction target person using input speech including speech of one or more speakers and information indicating speech periods of an extraction target person and / or a speaker other than the extraction target person as input data.

[0077] The information of the speaker to be extracted is given as, for example, the speech section of the speaker. This makes it possible to identify the speaker of each utterance for a speech recognition system that can obtain speech recognition results and their start and end times, as shown in Reference 1, for example.

[0078] As a conditioning vector obtained from speech section information, for example, as shown in Figure 11, the pooling unit 1100 conditions the extraction target person using information on the speech section of the extraction target person and / or the speech section of non-target speakers in each frame.

[0079] For example, the pooling unit 1200 according to the second embodiment calculates the weight A of the attention of the n-th person using the following equation (7): （ｎ） Calculate.

[0080] Here, c ｔ is the conditioning vector.

[0081] Conditioning vector c ｔ is determined using, for example, one of the following two methods.

[0082] First method: If the speaker to be extracted is speaking at time t, a D-dimensional all-zero vector is used as the conditioning vector c. ｔ This makes it possible to extremely reduce the attention weight in sections where the speaker to be extracted does not exist, and to prevent information about speakers other than the speaker to be extracted from being included in the speaker features.

[0083] Second method: The arithmetic mean or weighted mean of embeddings in frames where the target speaker is not speaking is used as the conditioning vector c ｔ In this case, the conditioning vector c ｔ is a vector that does not depend on t. This makes it possible to avoid outputting speaker vectors other than those of the speaker to be extracted.

[0084] The extraction system 1 uses the conditioning vector c ｔ and the conditioning vector c ｔ In addition, the extraction system 1 may use the conditioning vector c ｔ A neural network that outputs the above may be simultaneously trained.

[0085] Furthermore, information on the speech periods of the target speaker and non-target speakers at each time t may be input to the encoder together with the acoustic features at 201 in Fig. 2. This allows the encoder itself to have the effect of highlighting the information of the target speaker and suppressing the information of the non-target speakers.

[0086] <Processing Flow> Next, a processing flow of the speaker feature extraction method according to the second embodiment will be described.

[0087] (Processing of Extraction System) Fig. 12 is a flowchart showing an example of processing of the extraction system according to Example 2. This processing shows an example of processing executed by the extraction system 1 shown in Fig. 10 .

[0088] In step S1201, the mixed sound generation unit 101 generates a mixed sound of multiple speaker voices from the original training data 10. In Example 2, speakers who have exactly the same speech interval cannot be distinguished from each other, and therefore, unlike Example 1, only a mixed sound in which the voices of multiple speakers partially overlap is generated. Here, the mixed sound generated by the mixed sound generation unit 101, a speaker ID that identifies the speaker voice (or speaker) included in the mixed sound, and information indicating the speech interval of each speaker are used as training data in subsequent training.

[0089] In step S1202, the training unit 102 trains the extraction model 110, which outputs speaker features of the speaker to be extracted, using training data including the generated mixed sounds, speaker IDs, and information indicating the speech periods of each speaker. The extraction model 110 is optimized using a speaker identification loss based on cross entropy, as in the first embodiment.

[0090] Through the processing of steps S1201 and S1202 (learning processing), the extraction system 1 can obtain a trained extraction model 110-2. Using this trained extraction model 110-2, the extraction system 1 executes the processing of steps S1203 and S1204 (inference processing). However, the inference processing can be executed at any timing after the training processing is executed.

[0091] Furthermore, when executing the processes of steps S1203 and S1204, the extraction device 100 does not need to include the mixed sound generation unit 101 and the learning unit 102. For example, the processes of steps S1203 and S1204 may be executed by an information processing device other than the extraction device 100 that executed the processes of steps S1201 and S1202.

[0092] In step S1203, the estimation unit 103 inputs input speech including speech of one or more speakers from which speaker features are to be extracted and information indicating speech periods of the extraction targets and / or speakers other than the extraction targets to the trained extraction model 110-2, and estimates speaker features for the number of the extraction targets. For example, the estimation unit 103 acquires speaker features of the extraction targets output by the trained extraction model 110-2.

[0093] The information indicating the speech intervals of the extraction target person and / or speakers other than the extraction target person is, for example, a vector extracted using the speech intervals of the extraction target person, who is the speaker from whom speaker features are to be extracted, and other speakers.

[0094] In step S1204, the estimation unit 103 outputs the estimated speaker features of the extraction target person to a predetermined output destination.

[0095] According to the second embodiment, the processing of FIG. 6 enables the extraction device 100 (or the extraction system 1) that extracts speaker features from input speech including speech of multiple speakers to extract speaker features of an extraction target person using a single extraction model without separating sound sources.

[0096] The speaker features of the extraction target person output by the estimation unit 103 can be suitably used for, for example, the above-mentioned target speaker speech recognition that transcribes only the speech of a speaker of interest.

[0097] <Hardware Configuration> The extraction device 100 according to this embodiment has, for example, the hardware configuration of a computer 1300 as shown in Fig. 13. Alternatively, the extraction device 100 is realized by a plurality of computers 1300. Note that the computers are not limited to physical machines and may be, for example, virtual machines on the cloud.

[0098] Fig. 13 is a diagram showing an example of the hardware configuration of a computer. In the example of Fig. 13, a computer 1300 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, and an output device 1008, all of which are interconnected by a bus B. The computer 1300 may further include another processor such as a GPU (Graphics Processing Unit).

[0099] A program for implementing processing on the computer 1300 is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0100] The memory device 1003 reads and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the extraction device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a communication network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, and / or a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0101] The CPU 1004 may be another processor such as a DSP (Digital Signal Processor), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0102] According to the present embodiment, the extraction device 100 (or the extraction system 1) extracts speaker features from input speech containing speech from multiple speakers, and can extract speaker features using a single extraction model without relying on sound source separation. Furthermore, because sound source separation is not used, changes in speaker characteristics due to speech distortion do not occur.

[0103] Furthermore, according to the first embodiment, the extraction device 100 (or the extraction system 1) can extract speaker features for the number of speakers from input speech including a plurality of speakers.

[0104] Furthermore, according to the second embodiment, the extraction device 100 (or the extraction system 1) can extract speaker features of an extraction target person from input speech including speeches of multiple speakers.

[0105] Summary of Embodiments This specification discloses at least the extraction systems, extraction methods, and extraction devices described in the following paragraphs. (Item 1) An extraction system comprising: an extraction model that, based on input speech including speech of one or more speakers, outputs speaker features of the speakers; a training unit that trains the extraction model using training data including a mixture of speech of multiple speakers and identification information that identifies the speaker speech; and an estimation unit that inputs the input speech including speech of one or more speakers to the trained extraction model and estimates speaker features of the speakers. (Item 2) The extraction system described in item 1, wherein the extraction model takes the input speech and the number of speakers of the one or more speakers as input data and outputs the speaker features for the number of speakers, and the estimation unit inputs the input speech and the number of speakers to the trained extraction model and estimates the speaker features for the number of speakers. 10. The extraction system according to claim 1, wherein: (Clause 3) the extraction model uses the input speech as input data to estimate the number of speakers among the one or more speakers and output the speaker features for the number of speakers, and the estimation unit inputs the input speech to the trained extraction model to estimate the speaker features for the number of speakers. 11. The extraction system according to claim 1, wherein: (Clause 4) the extraction model uses the input speech and information indicating a speech period of an extraction target person among the one or more speakers from which the speaker features are to be extracted and / or a speaker other than the extraction target person as input data, and outputs the speaker features of the extraction target person, and the training unit trains the extraction model further using information on the speech periods of the speeches of the multiple speakers, and the estimation unit inputs the input speech and information indicating the speech period of the extraction target person and / or a speaker other than the extraction target person among the one or more speakers to the trained extraction model to estimate the speaker features of the extraction target person. (5) An extraction system described in any one of paragraphs 1 to 3, wherein the extraction model has an attention function that determines attention, indicating which part of the embedding vector sequence extracted from the input speech to focus on when extracting the speaker features, and the attention function uses the sum of attention weights calculated for speaker features that have already been extracted to calculate the attention weight.(6) The extraction system according to paragraphs 1 or 4, wherein the extraction model has an attention function that determines attention indicating which part of the embedding vector sequence extracted from the input speech to focus on when extracting the speaker features, and the attention function uses a conditioning vector calculated using information indicating the speech section of each speaker to calculate the attention weights. (7) The extraction system according to any of paragraphs 1 to 4, wherein the extraction model corrects the distribution of the attention weights at the time of inference based on the input length of the mixed sound at the time of training and the input length of the input sound at the time of inference. (8) An extraction method, comprising an extraction system having an extraction model that extracts speaker features of a speaker based on input speech including speech of one or more speakers, the extraction system performing the following steps: a training process that trains the extraction model using training data including the mixed sound of multiple speaker speeches and identification information that identifies the speaker speeches; and an estimation process that inputs speech including the speech of one or more speakers into the trained extraction model to estimate speaker features of the speaker. (Clause 9) An extraction device comprising: an extraction model that outputs speaker features of one or more speakers based on input speech including the speech of the speakers; a learning unit that trains the extraction model using training data including a mixed sound of multiple speaker speeches and identification information that identifies the speaker speeches; and an estimation unit that inputs input speech including the speech of one or more speakers to the trained extraction model and estimates speaker features of the speakers.

[0106] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0107] (References) Reference 1: Naoyuki Kanda et al., "Simultaneous speech recognition and speaker diarization for monaural dialogue recordings with target-speaker acoustic models," in Proc. ASRU 2019.

[0108] 1 Extraction system 10 Original learning data 100 Extraction device 101 Mixed sound generation unit 102 Learning unit 103 Estimation unit 110 Extraction model 110-1 Extraction model (before learning) 110-2 Extraction model (trained) (trained extraction model) 1300 Computer

Claims

1. An extraction system comprising: an extraction model that outputs speaker features of one or more speakers based on input speech containing the speech of the speakers; a training unit that trains the extraction model using training data that includes a mixture of speech from multiple speakers and identification information that identifies the speech of the speakers; and an estimation unit that inputs input speech containing the speech of one or more speakers to the trained extraction model and estimates speaker features of the speakers.

2. The extraction system according to claim 1, wherein the extraction model receives the input speech and the number of speakers, and outputs the speaker features for the number of speakers, and the estimation unit inputs the input speech and the number of speakers to the trained extraction model to estimate the speaker features for the number of speakers.

3. The extraction system according to claim 1, wherein the extraction model uses the input speech as input data, estimates the number of speakers of the one or more speakers, and outputs the speaker features for the number of speakers, and the estimation unit inputs the input speech into the trained extraction model and estimates the speaker features for the number of speakers.

4. The extraction system described in claim 1, wherein the extraction model receives as input data the input speech and information indicating the speech periods of an extraction target person among the one or more speakers from which the speaker features are to be extracted and / or speakers other than the extraction target person, and outputs the speaker features of the extraction target person; the learning unit learns the extraction model further using information on the speech periods of the speeches of the multiple speakers; and the estimation unit inputs the input speech and information indicating the speech periods of the extraction target person among the one or more speakers and / or speakers other than the extraction target person into the trained extraction model, and estimates the speaker features of the extraction target person.

5. An extraction system as claimed in any one of claims 1 to 3, wherein the extraction model has an attention function that determines attention indicating which part of the embedding vector sequence extracted from the input speech to focus on when extracting the speaker features, and the attention function uses the sum of attention weights calculated for speaker features that have already been extracted to calculate the attention weight.

6. The extraction system described in claim 1 or 4, wherein the extraction model has an attention function that determines attention indicating which part of the embedding vector sequence extracted from the input speech to focus on to extract the speaker features, and the attention function uses a conditioning vector calculated using information indicating the speech section for each speaker to calculate the attention weight.

7. An extraction system according to any one of claims 1 to 4, wherein the extraction model has an attention function that determines attention indicating which part of the embedding vector sequence extracted from the input speech to focus on when extracting the speaker features, and during inference, corrects the distribution of attention weights based on the input length of the mixed sound during training and the input length of the input sound during inference.

8. An extraction method in which an extraction system having an extraction model that extracts speaker features of a speaker based on input speech including the speech of one or more speakers executes the following steps: a training process that trains the extraction model using training data that includes a mixture of speech from multiple speakers and identification information that identifies the speaker speech; and an estimation process that inputs input speech including the speech of one or more speakers to the trained extraction model and estimates speaker features of the speakers.

Images