SPEAKER EMBEDDING APPARATUS, SPEAKER EMBEDDING METHOD, AND SPEAKER EMBEDDING PROGRAM

Abstract

Provided is a system that generates utterance unit segmentation information indicated by the duration length of each utterance of a speaker on the basis of speech data of the speaker. The system learns a speaker identification model using the duration length of each utterance indicated in the utterance unit segmentation information. When thereafter receiving an input of speech data to be converted into a speaker vector, the system generates utterance unit segmentation information of the speech data. The system inputs, to the learned speaker identification model, the duration length of each utterance indicated in the generated utterance unit segmentation information, and outputs, as the speaker vector, an output in an intermediate layer of the speaker identification model.

Description

[Technical field]

[0001] The present invention relates to a speaker embedding device, a speaker embedding method, and a speaker embedding program. [Background technology]

[0002] Conventionally, in the field of speech processing, a technology for vectorizing speaker information (speaker embedding technology) has been proposed (for example, Non-Patent Document 1, etc.). By using the above technology to vectorize speaker information and represent it in a space of continuous values, it is possible to perform various speech processing tasks such as speaker identification, speaker authentication, speech recognition, speech synthesis, and speech conversion.

[0003] In conventional speaker embedding technology, a neural network is first trained to identify speakers from a large amount of speech data of the speakers. The trained neural network is then input with the speech of the speaker to be converted into a speaker vector, and the information of the intermediate layer of the neural network is output as a speaker vector.

[0004] When training the above-mentioned neural network, it is common to use acoustic features extracted from speech (e.g., speech spectrum, Mel Frequency Cepstrum (MFCC), Mel Spectrogram, etc.) as input features. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] David Snyder et al.:X-VECTORS: ROBUST DNN EMBEDDINGS FOR SPEAKER RECOGNITION,2018 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP),2018.4 Summary of the Invention [Problem to be solved by the invention]

[0006] According to the above technology, it is possible to capture the characteristics of a speaker's voiceprint, but it is difficult to capture the characteristics of the speaker's speaking style, such as the rhythm of speech, etc. Therefore, for example, even if the speaker vector obtained by the above technology is used to perform the above voice processing task, there is a problem that the performance may not be improved.

[0007] Therefore, an object of the present invention is to solve the above-mentioned problems and to extract a speaker vector that captures the speech rhythm of a speaker. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the present invention is characterized by comprising an input unit that accepts input of voice data, an information generation unit that generates utterance unit segmentation information indicating the duration of each utterance of a speaker in the input voice data, and a learning unit that uses the duration of each utterance indicated in the generated utterance unit segmentation information as learning data and trains a speaker identification model that outputs a speaker identification result when the duration of each utterance of a speaker is input. Effect of the Invention

[0009] According to the present invention, it is possible to extract a speaker vector that captures the speech rhythm of a speaker. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram for explaining an overview of a speaker embedding system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a speaker embedding system according to the first embodiment. [Diagram 3] FIG. 3 is a diagram for explaining an overview of the learning device of FIG. [Figure 4] FIG. 4 is a diagram for explaining an overview of the vector conversion device of FIG. [Diagram 5] FIG. 5 is a flowchart illustrating an example of a processing procedure of the learning device of FIG. [Figure 6] FIG. 6 is a flowchart illustrating an example of a processing procedure of the vector conversion device of FIG. [Figure 7] FIG. 7 is a diagram showing an example of the configuration of a computer that executes a speaker embedding program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, a mode (embodiment) for carrying out the present invention will be described with reference to the drawings, with a first embodiment and a second embodiment being described separately. The present invention is not limited to the embodiments described below.

[0012] [First embodiment] First, an overview of a speaker embedding device (speaker embedding system) according to a first embodiment will be described with reference to FIG.

[0013] The speaker embedding system (hereinafter, abbreviated as the system) is characterized by using the speaker's utterance unit segmentation information as a feature to be input to the speaker identification model. This utterance unit segmentation information is, for example, information indicating the duration of the utterance unit (e.g., phoneme, mora, syllable, phrase, etc.) for the speaker's voice data (see reference numeral 101 in FIG. 1).

[0014] For example, as shown at 102, the system uses a large amount of speech unit segmentation information of a speaker to train a speaker identification model, and obtains a speaker vector using the trained speaker identification model.

[0015] This speaker identification model is a model that outputs a speaker identification result when the duration of each utterance is input. For example, the speaker identification model is a model that outputs a speaker posterior probability (e.g., p(s(1)), p(s(2)), ..., p(s(N-1)), p(s(N))) when a sequence of durations of each utterance in the utterance unit segmentation information (d(1), d(2), ..., d(Nf-1), d(N)) is input. This speaker identification model is realized, for example, by a neural network.

[0016] The system trains a speaker identification model using the duration of each utterance indicated in a large amount of utterance unit segmentation information. The system then inputs the duration of each utterance of the speech data to be converted into a speaker vector to the trained speaker identification model, and outputs the output of the intermediate layer in the speaker identification model as the speaker vector of the speaker.

[0017] In this way, the system uses the duration of each speaker's utterance as a feature to be input into the speaker identification model, thereby enabling extraction of a speaker vector that captures the speaker's speech rhythm.

[0018] [Configuration example] Next, an example of the system configuration will be described with reference to Fig. 2. As shown in Fig. 2, the system 1 includes a learning device 10 and a vector conversion device 20. The learning device 10 learns a speaker identification model using utterance unit segmentation information. The vector conversion device 20 converts the inputted speaker's voice data into a speaker vector using the learned speaker identification model.

[0019] [Learning device] The learning device 10 performs, for example, representation conversion of the utterance unit segmentation information for training by the representation conversion unit 132. Then, the learning device 10 uses the representation-converted utterance unit segmentation information to train a speaker identification model (see FIG. 3).

[0020] The learning device 10 includes an input / output unit 11, a storage unit 12, and a control unit 13. The input / output unit 11 controls input and output of various data. The input / output unit 11 receives, for example, input of speech data for learning.

[0021] The storage unit 12 stores data that the control unit 13 refers to when performing various processes and data generated by the control unit 13. For example, the storage unit 12 stores utterance unit segmentation information of training voice data generated by the control unit 13, a speaker identification model trained by the control unit 13, and the like.

[0022] The control unit 13 is responsible for controlling the entire learning device 10, and includes, for example, an information generation unit 131, an expression conversion unit 132, and a learning unit 133.

[0023] The information generating unit 131 generates utterance unit segmentation information indicating the duration of each utterance (e.g., phoneme) of the speaker based on the voice data of the speaker. This utterance unit segmentation information is stored in the storage unit 12, for example.

[0024] For example, the information generating unit 131 uses a voice recognition device to generate utterance unit segmentation information in which the duration of each phoneme is assigned to the speech data of N speakers, which is the training speech data. This utterance unit segmentation information indicates the duration of each phoneme included in the speech data, for example, as indicated by reference numeral 101 in Fig. 1. This utterance unit segmentation information may include the start time and end time of an utterance (e.g., a phoneme), as indicated by reference numeral 101.

[0025] The number of speakers of the speech data used for training the speaker identification model is, for example, several hundred or more, and the number of sentences uttered by each speaker is, for example, several tens or more.

[0026] The representation converting unit 132 converts the utterance unit segmentation information into a representation that can be used by the learning unit 133. For example, the representation converting unit 132 converts the utterance unit segmentation information generated by the information generating unit 131 into a one-dimensional numerical representation.

[0027] For example, the number of phonemes in the nth sentence of speaker s is T sn In the case of a one-dimensional numerical representation, the duration in the utterance unit segmentation information is expressed as a one-dimensional vector d sn Treat as (t).

[0028] Furthermore, the representation converting unit 132 can also convert the utterance unit segmentation information into a one-hot representation shown in the following formula (1).

[0029]

number

[0030] In the case of the one-hot representation shown in the above formula (1), d sn (t)(v) is d sn It represents the v-th dimension of (t). Also, d sn Each dimension in (t) corresponds to V clusters obtained by clustering the duration of each utterance in the utterance unit segmentation information, for example, by the k-means method. For example, the representation conversion unit 132 may convert the dimension (d sn Convert the vector into one with (t)(v) set to 1 and all other dimensions set to 0.

[0031] The learning unit 133 uses the speech unit segmentation information after the expression conversion by the expression conversion unit 132 to train a speaker identification model. The speaker identification model is realized, for example, by a neural network that converts the speech unit segmentation information after the expression conversion (one-dimensional numerical expression or V-dimensional one-hot expression) into a one-hot expression (N-dimensional vector) of the speaker. This neural network is realized, for example, by using f d→p It can be expressed as follows.

[0032]

number

[0033] The neural network used in the speaker identification model may be a normal multilayer perceptron (MLP) or other neural networks. For example, the neural network used in the speaker identification model may be a neural network that can take into account surrounding words, such as a recurrent neural network (RNN) or an RNN-LSTM (RNN-Long Short Term Memory). The neural network used in the speaker identification model may be a combination of the above neural networks.

[0034] [Vector conversion device] Next, a description will be given of the vector transformation device 20. Note that, in the following, a case will be described in which the speaker identification model used by the vector transformation device 20 is a speaker identification model using a neural network.

[0035] The vector conversion device 20 performs, for example, representation conversion of the utterance unit segmentation information of the speech data to be converted into a speaker vector, by the representation conversion unit 132. Next, the vector conversion device 20 inputs the representation-converted utterance unit segmentation information to the trained speaker identification model. Then, the vector conversion device 20 performs forward propagation processing of the neural network of the speaker identification model, and outputs any bottleneck feature of the neural network as the speaker vector of the speaker (see FIG. 4).

[0036] The vector conversion device 20 includes an input / output unit 21, a storage unit 22, and a control unit 23. The input / output unit 21 controls input and output of various data. The input / output unit 21 receives, for example, input of speech data of a speaker to be converted into a speaker vector.

[0037] The storage unit 22 stores data that the control unit 23 refers to when performing various processes and data generated by the control unit 23. For example, the storage unit 22 stores utterance unit segmentation information to be converted into a speaker vector, etc., generated by the control unit 23.

[0038] The control unit 23 is responsible for controlling the entire vector conversion device 20, and includes, for example, an information generation unit 231, an expression conversion unit 232, and a speaker vector output unit 233.

[0039] The information generating unit 231, like the information generating unit 131 of the learning device 10, generates utterance unit segmentation information indicating the duration of each utterance of a speaker, based on the speech data of the speaker.

[0040] For example, the information generating unit 231 uses a voice recognition device to generate utterance unit segmentation information in which the duration of each phoneme is added to the voice data of a speaker to be converted into a speaker vector. The utterance unit segmentation information generated by the information generating unit 231 is stored in, for example, the storage unit 22.

[0041] The representation conversion unit 232 converts the utterance unit segmentation information into a representation that can be processed by the speaker identification model. For example, like the representation conversion unit 132 of the learning device 10, the representation conversion unit 232 converts the utterance unit segmentation information generated by the information generation unit 231 into a one-dimensional numerical representation.

[0042] The speaker vector output unit 233 converts the utterance unit segmentation information generated by the information generation unit 231 into a speaker vector using the trained speaker identification model. For example, the speaker vector output unit 233 inputs the utterance unit segmentation information after the expression conversion by the expression conversion unit 232 to the trained speaker identification model. Next, the speaker vector output unit 233 performs a forward propagation process of the neural network in the speaker identification model. Then, the speaker vector output unit 233 outputs the output of an arbitrary intermediate layer (bottleneck feature) of the neural network as a speaker vector of the voice data input from the input / output unit 21.

[0043] In this way, system 1 uses speech unit segmentation information indicating the duration of a speaker's speech units as a feature to be input into the speaker identification model, thereby enabling extraction of a speaker vector that captures the speaker's speech rhythm.

[0044] [Example of processing procedure] Next, an example of a processing procedure of the system 1 will be described with reference to Fig. 5 and Fig. 6. First, the learning device 10 of the system 1 accepts input of speech data of a speaker for learning (S1 in Fig. 5). Next, the information generating unit 131 of the learning device 10 generates utterance unit segmentation information of the speech data input in S1 (S2). Then, the representation converting unit 132 converts the utterance unit segmentation information generated in S2 into a one-dimensional numerical expression that can be processed by a speaker identification model (S3). Thereafter, the learning unit 133 uses the one-dimensional numerical expression converted in S3 to train the speaker identification model (S4).

[0045] After S4 in Fig. 5, the vector conversion device 20 accepts input of speech data of a speaker to be converted into a speaker vector (S11 in Fig. 6). Next, the information generation unit 231 of the vector conversion device 20 generates utterance unit segmentation information of the speech data input in S11 (S12). Then, the representation conversion unit 232 converts the utterance unit segmentation information generated in S12 into a one-dimensional numerical representation that can be processed by a speaker identification model (S13).

[0046] Thereafter, the speaker vector output unit 233 inputs the one-dimensional numerical representation converted in S13 to the trained speaker identification model (S14). Then, the speaker vector output unit 233 outputs the output of the intermediate layer of the speaker identification model as a speaker vector (S15).

[0047] In this way, the system 1 can extract a speaker vector that captures the speech rhythm of the speaker.

[0048] [Second embodiment] Next, a second embodiment of the present invention will be described. The same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0049] The system 1 of the second embodiment is characterized in that not only the duration of an utterance included in the utterance unit segmentation information, but also utterance information (for example, phoneme information) is used to train a speaker identification model.

[0050] In other words, when the representation conversion unit 132 of the learning device 10 and the representation conversion unit 223 of the vector conversion device 20 in the system 1 of the second embodiment convert the utterance unit segmentation information into a one-dimensional notification representation that can be processed by a speaker identification model, they convert not only the duration of the utterance contained in the utterance unit segmentation information, but also the information of the utterance.

[0051] For example, the representation conversion unit 132 of the learning device 10 converts the duration in the utterance unit segmentation information into a one-dimensional numerical representation or a one-hot representation, as in the first embodiment. Next, the representation conversion unit 132 converts the information on the utterance (e.g., phonemes) in the utterance unit segmentation information into a one-hot representation.

[0052] For example, if the number of phonemes in the training speech data is I, then Ph sn (t) is converted to the one-hot representation shown in the following equation (3).

[0053]

number

[0054] In the above formula (3), Ph sn (t)(i) is Ph sn For example, the representation conversion unit 132 converts the i-th dimension (Ph sn (t)(i)) is set to 1, and the other dimensions are set to 0. Then, the representation conversion unit 132 converts the vector d sn (t) and the vector Ph for phoneme names sn Output the combined vector of (t).

[0055] The learning unit 133 then converts the duration-related vector d sn (t) and the vector Ph for phoneme names sn The combined vector (t) is used to train a speaker identification model.

[0056] In addition, the representation conversion unit 223 of the vector conversion device 20 converts the utterance unit segmentation information of the speech data to be converted into a speaker vector into a vector d sn(t) and the vector Ph for phoneme names sn Output the combined vector of (t).

[0057] Then, the speaker vector output unit 233 inputs the vector output by the representation conversion unit 232 to the trained speaker identification model, and outputs a speaker vector.

[0058] In this way, the system 1 can extract a speaker vector that captures the speech rhythm with greater accuracy.

[0059] Although the system 1 in each of the above-described embodiments is configured to include the learning device 10 and the vector conversion device 20, this is not limiting. For example, the system 1 may be configured to include only the learning device 10 or the vector conversion device 20. Furthermore, although the utterance unit segmentation information is generated by the system 1, this is not limiting. For example, the system 1 may execute various processes using the utterance unit segmentation information generated by an external device.

[0060] [System configuration, etc.] In addition, each component of each unit shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in part by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0061] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0062] [program] The above-mentioned system 1 can be implemented by installing a program as package software or online software on a desired computer. For example, by executing the above-mentioned program on an information processing device, the information processing device can function as the system 1 of each embodiment. The information processing device here includes desktop or notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), and further terminals such as PDA (Personal Digital Assistant), etc.

[0063] The system 1 can also be implemented as a server device that provides services related to the above-mentioned processes to a client, with the terminal device used by the user being the client. In this case, the server device may be implemented as a Web server, or may be implemented as a cloud that provides services related to the above-mentioned processes by outsourcing.

[0064] 7 is a diagram showing an example of a computer that executes a speaker embedding program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. Each of these components is connected by a bus 1080.

[0065] The memory 1010 includes a read only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0066] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process executed by the above-mentioned system 1 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration in the system 1 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD.

[0067] Data used in the processing of each of the above-described embodiments is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes them.

[0068] The program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0069] 1 System 10 Learning Device 11,21 Input / output section 12,22 Storage section 13,23 Control section 20 Vector conversion device 131,231 Information generation section 132,232 Expression conversion unit 133 Learning Department 233 Speaker Vector Output Unit

Claims

1. an input unit that accepts input of voice data; an information generating unit that generates utterance unit segmentation information indicating a duration of each utterance of a speaker in the input speech data; a learning unit that uses the duration of each utterance indicated in the generated utterance unit segmentation information as learning data, and learns a speaker identification model that outputs a speaker identification result when the duration of each utterance of a speaker is input; A speaker vector output unit, After training the speaker identification model, The input unit includes: Accepting input of speech data to be converted into a speaker vector; The information generating unit generating the utterance unit segmentation information of the input speech data to be converted into the speaker vector; The speaker vector output unit The duration of each utterance indicated in the generated utterance unit segmentation information is input to the speaker identification model after training, and an output in an intermediate layer of the speaker identification model is output as a speaker vector of the speech data. A speaker embedding device comprising:

2. The learning unit is training a speaker identification model that uses the utterances and durations of the utterances indicated in the utterance unit segmentation information as learning data, and outputs a speaker identification result when a speaker's utterance and the durations of the utterances are input; The speaker vector output unit The utterances and the durations of the utterances indicated in the generated utterance unit segmentation information are input to the speaker identification model after training, and an output in an intermediate layer of the speaker identification model is output as a speaker vector of the speech data. The speaker embedding device according to claim 1 .

3. An expression conversion unit that converts the duration of each utterance indicated in the generated utterance unit segmentation information into a one-dimensional numerical expression, The learning unit is The one-dimensional numerical representation of the duration of each utterance thus converted is used as training data to train the speaker identification model. The speaker embedding device according to claim 1 .

4. The representation conversion unit is converting the utterances and durations of the utterances indicated in the generated utterance unit segmentation information into one-dimensional numerical expressions; The learning unit is The converted utterances and the one-dimensional numerical representation of the duration of each utterance are used as training data to train the speaker identification model. The speaker embedding device according to claim 3 .

5. A speaker embedding method performed by a speaker embedding device, comprising: accepting input of voice data; generating utterance unit segmentation information indicating the duration of each utterance of a speaker in the input speech data; a step of training a speaker identification model which uses the duration of each utterance indicated in the generated utterance unit segmentation information as training data, and outputs a speaker identification result when the duration of each utterance of a speaker is input; receiving input of speech data to be converted into a speaker vector; generating the utterance unit segmentation information of the input speech data to be converted into the speaker vector; inputting the duration of each utterance indicated in the generated utterance unit segmentation information into the trained speaker identification model, and outputting an output of an intermediate layer of the speaker identification model as a speaker vector of the speech data; A speaker embedding method comprising:

6. accepting input of voice data; generating utterance unit segmentation information indicating the duration of each utterance of a speaker in the input speech data; a step of training a speaker identification model which uses the duration of each utterance indicated in the generated utterance unit segmentation information as training data, and outputs a speaker identification result when the duration of each utterance of a speaker is input; receiving input of speech data to be converted into a speaker vector; generating the utterance unit segmentation information of the input speech data to be converted into the speaker vector; inputting the duration of each utterance indicated in the generated utterance unit segmentation information into the trained speaker identification model, and outputting an output of an intermediate layer of the speaker identification model as a speaker vector of the speech data; A speaker embedding program that causes a computer to execute the above steps.

Images