Training device, inference device, training method, inference method, and program

Abstract

This training device for training a target speaker task model comprises: a random projection unit for generating a target speaker expression by performing random projection on an acoustic feature amount of the speech of a target speaker; and a training unit for training the target speaker task model by using the target speaker expression and acoustic feature amounts of speech including speech of the target speaker and speech of another speaker.

Description

Learning device, inference device, learning method, inference method, and program

[0001] The present invention relates to a technology for recognizing the speech of a desired person (target speaker) from among speeches of multiple people.

[0002] In a real speech listening environment, it is rare for only a single voice to be input. Rather, it is common for the input to contain not only the voice of interest (target speaker) but also noise and the voices of other speakers.

[0003] In such a situation, in a conventional speech recognition / speech enhancement system, it is not obvious whose speech should be recognized / enhanced, and therefore the task accuracy deteriorates significantly. Therefore, by pre-registering the target speaker's speech utterance, extracting target speaker expressions from the target speaker's speech, and using them for conditioning, it is possible to realize speech recognition / speech enhancement that focuses only on the target speaker. For example, the technology disclosed in Non-Patent Document 1 realizes target speaker speech recognition. Non-Patent Document 2 provides a general overview of target speaker speech enhancement.

[0004] In target speaker tasks, a speaker model is often used to extract target speaker expressions. The speaker model used here is generally a model pre-trained for a speaker recognition task (a task of classifying input speech into classes), such as d-vector or x-vector, or a model that is simultaneously optimized during training for the target speaker task (during training for a speech recognition task in the case of target speaker speech recognition).

[0005] Moriya, T., Sato, H., Ochiai, T., Delcroix, M., & Shinozaki, T. (2022). Streaming target-speaker ASR with neural transducer. arXiv preprint arXiv:2209.04175.Zmolikova, K., Delcroix, M., Ochiai, T., Kinoshita, K., Cernocky, J., & Yu, D. (2023). Neural target speech extraction: An overview. IEEE Signal Processing Magazine, 40(3), 8-29. Chiu, CC, Qin, J., Zhang, Y., Yu, J., & Wu, Y. (2022, June). Self-supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning (pp. 3915-3924). PMLR.

[0006] Speaker models used for pre-training or joint optimization generally consist of multiple layers of LSTMs or CNNs, and the training costs are not negligible. In addition, labeled data is generally required for supervised learning. For example, pre-training requires speaker labels, and joint optimization requires speaker labels and task-specific label transcriptions. The labeling costs limit the number of speakers that can be covered. Therefore, the estimation accuracy for speakers not included in the training may be degraded.

[0007] The present invention has been made in consideration of the above points, and aims to provide a technology for reducing training costs in target speaker task technology by omitting the pre-training or simultaneous optimization of speaker models.

[0008] According to the disclosed technology, there is provided a learning device that trains a target speaker task model, comprising: a random projection unit that generates a target speaker expression by performing random projection on acoustic features of a target speaker's speech; and a learning unit that trains the target speaker task model using the target speaker expression and acoustic features of speech that includes the target speaker's speech and speech of another speaker.

[0009] According to the disclosed technology, in the target speaker task technology, it is possible to omit pre-training or simultaneous optimization of speaker model training itself, thereby reducing training costs.

[0010] FIG. 1 is a diagram showing an example of the configuration of a learning device 100. FIG. 2 is a diagram showing an example of the configuration of a learning device 200. FIG. 3 is a diagram showing an example of the configuration of a random projection device 300. FIG. 4 is a flowchart showing the operation of the random projection device 300. FIG. 5 is a flowchart showing the operation of the learning device 200. FIG. 6 is a diagram showing an example of the configuration of an inference device 400. FIG. 7 is a diagram showing an example of the configuration of an inference device 500. FIG. 8 is a diagram showing an example of the configuration of a random projection device 600. FIG. 9 is a diagram showing an example of the hardware configuration of the device.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0012] (Outline of the embodiment) As described above, in the conventional technology, training a speaker model to extract target speaker expressions to be used in a target speaker task is costly, and estimation accuracy for speakers who do not appear in the training of the speaker model may be degraded.

[0013] Therefore, in this embodiment, random projection (random matrix or random quantization) is used as the speaker model in the target speaker task, thereby omitting the pre-training or training of a speaker model that is simultaneously optimized. This reduces the training cost while improving robustness against speakers that do not appear in the training data. The key points of the technology according to this embodiment are as follows:

[0014] As described above, this embodiment uses random projection (random matrix or random quantization) as the speaker model. For the target speaker task model, it is sufficient to be able to identify the target speaker (the target of speech recognition or speech enhancement) from speech containing other speakers using a separately input pre-registered utterance as a clue. Therefore, pre-trained speaker embeddings based on criteria (such as d-vectors or x-vectors) that make it easier to identify speakers (such as for speaker recognition tasks) or sophisticated speaker embeddings that are simultaneously optimized are not necessarily required.

[0015] Therefore, in this embodiment, random projection is used. In random projection, input speaker speech is converted into a single embedding or speaker code by simply using a random projection matrix or quantization module, and this is then input as a clue to the speech recognition model or speech enhancement model. In this case, since the parameters of the random projection remain random values, training is not required, and the training cost can be saved. In addition, since training is not performed using explicit speaker data, it is possible to operate robustly even for speakers that do not appear in training, and it is possible to achieve high task accuracy. The configuration and operation of the device according to this embodiment will be described in detail below.

[0016] (Device Configuration Example) Fig. 1 shows an example configuration of a learning device 100 in this embodiment. As shown in Fig. 1, the learning device 100 has a speech signal acquisition unit 101, a pre-registered speech speech signal acquisition unit 102, a speech digital signal storage unit 103, a pre-registered speech speech digital signal storage unit 104, a feature analysis unit 105, a pre-registered utterance feature analysis unit 106, a feature storage unit 107, a pre-registered utterance feature storage unit 108, a random projection unit 109, and a target speaker task learning unit 110. The functions of each unit are as follows. In the explanation of the functions, "input," "output," and "processing" are clearly stated.

[0017] <Audio signal acquisition unit 101> The audio signal acquisition unit 101 acquires an audio signal and outputs an audio digital signal. More specifically, the audio signal acquisition unit 101 converts an analog audio signal into a digital audio digital signal. The input, output, and processing are as follows. Note that the "audio signal" may also be simply called "audio."

[0018] Input: Audio signal Output: Audio digital signal Processing: AD conversion <Pre-registered utterance audio signal acquisition unit 102> The pre-registered utterance audio signal acquisition unit 102 acquires a pre-registered utterance audio signal and outputs a pre-registered utterance audio digital signal. More specifically, the pre-registered utterance audio signal acquisition unit 102 converts an analog audio signal into a digital audio digital signal. The input, output, and processing are as follows:

[0019] Input: Pre-registered speech voice signal Output: Pre-registered speech voice digital signal Processing: AD conversion <Speech digital signal storage unit 103> The speech digital signal storage unit 103 stores the speech digital signal. The input and processing are as follows.

[0020] Input: Audio digital signal Processing: Storage of audio digital signal <Pre-registered speech audio digital signal storage unit 104> The pre-registered speech audio digital signal storage unit 104 stores the pre-registered speech audio digital signal. The input and processing are as follows.

[0021] Input: Pre-registered speech digital signal Processing: Accumulation of pre-registered speech digital signal <Feature analysis unit 105> The feature analysis unit 105 performs feature analysis on the speech digital signal and outputs a feature sequence. The input, output, and processing are as follows. Note that the "feature sequence" may also be called "feature."

[0022] Input: Audio digital signal Output: Feature sequence Processing: Feature analysis More specifically, the feature analysis unit 105 extracts acoustic features from the audio digital signal read from the audio digital signal storage unit 103 through signal processing, and acquires an (acoustic) feature sequence for each utterance.

[0023] The extracted features include, for example, dimensions 1 to 12 of MFCC (Mel-Frequency Cepstrum Coefficient) based on short-time frame analysis of the speech signal, and its dynamic features such as dynamic parameters ΔMFCC, ΔΔMFCC, power, Δpower, and ΔΔpower. Furthermore, CMN (Cepstrum Mean Normalization) processing may be performed on the MFCC. A log-log-mel filter bank (FBANK) may also be used.

[0024] The feature amount is not limited to MFCC or power, and parameters used for identifying special utterances (for example, autocorrelation peak value, group delay, etc.) may also be used as the feature amount.

[0025] <Pre-registered speech feature analysis unit 106> The pre-registered speech feature analysis unit 106 performs feature analysis on the pre-registered speech digital signal and outputs a pre-registered speech feature sequence. The input, output, and processing are as follows.

[0026] Input: Pre-registered speech voice digital signal Output: Pre-registered speech feature sequence Processing: Feature analysis The processing content of the pre-registered speech feature analysis unit 106 is the same as the processing content of the feature analysis unit 105 .

[0027] <Feature Amount Accumulation Unit 107> The feature amount accumulation unit 107 accumulates the feature amount series analyzed by the feature amount analysis unit 105. The input and processing are as follows.

[0028] Input: Feature sequence Processing: Accumulation of feature sequence <Pre-registered speech feature storage unit 108> The pre-registered speech feature storage unit 108 accumulates the pre-registered speech feature sequence analyzed by the pre-registered speech feature analysis unit 106. The input and processing are as follows.

[0029] Input: Pre-registered speech feature sequence Processing: Accumulation of pre-registered speech feature sequence <Random projection unit 109> The random projection unit 109 extracts a target speaker expression using the pre-registered speech feature sequence read from the pre-registered speech feature accumulation unit 108. The input, output, and processing are as follows:

[0030] Input: Pre-registered utterance feature sequence Output: Target speaker expression Processing: Generation of target speaker expression More specifically, the random projection unit 109 performs random projection processing on the pre-registered utterance feature sequence read from the pre-registered utterance feature storage unit 108 as input, to generate a target speaker expression. When training a model to be used in a target speaker task, the parameters of the random projection unit 109 are not updated, and the random weights are used as they are.

[0031] For random projection, for example, a random matrix or random quantization is used. In the case of a random matrix, a linear transformation layer or an affine transformation layer that has undergone certain weight initialization (for example, Xavier initialization and He initialization) is used with its initial values ​​unchanged. In the case of random quantization, a quantization module is used. For example, a Random-Projection Quantizer as used in Non-Patent Document 3 can be used as the quantization module.

[0032] Note that while both the random matrix and random quantization are output in frame units, a single vector (not a frame-by-frame vector, but a single vector pooled in the time direction) is generally used as the target speaker representation. Therefore, when using the output of the random matrix and random quantization, the information is degenerated in the time direction through average pooling or max pooling. In other words, the random projection information output from the above-mentioned linear transformation layer / affine transformation layer or quantization module is input to an average pooling or max pooling layer, and the obtained information (degenerated in the time direction) is used as the output of the random projection unit 109.

[0033] <Target speaker task training unit 110> The target speaker task training unit 110 trains a target speaker task model using the acoustic features stored in the feature storage unit 107 and the target speaker expressions generated by the random projection unit 109. The input, output, and processing are as follows.

[0034] Input: feature sequence, target speaker expression Output: target speaker task model Processing: training of target speaker task model More specifically, the target speaker task training unit 110 trains a target speaker task model using the feature sequence and the target speaker expression generated using the pre-registered utterance feature sequence.

[0035] Examples of target speaker tasks include target speaker speech recognition and target speaker speech enhancement. For example, for target speaker speech recognition, a Transformer or Conformer can be used as a model, and Connectionist Temporal Classification (CTC) Loss or Transducer Loss can be used as a training criterion. For target speaker speech enhancement, a Bidirectional Long Short Term Memory (BLSTM) can be used as a model, and Mean Square Loss or Scale-invariant Signal-to-distortion Ratio Loss can be used as a training criterion.

[0036] (Other Configuration Examples) The configuration shown in Fig. 1 may be a configuration of a single device or a configuration consisting of multiple devices. For example, it may include the learning device 200 shown in Fig. 2 and the random projection device 300 shown in Fig. 3.

[0037] The learning device 200 shown in FIG. 2 includes a speech signal acquisition unit 101, a speech digital signal storage unit 103, a feature analysis unit 105, a feature storage unit 107, a pre-registered speech feature storage unit 108, and a target speaker task learning unit 110.

[0038] The random projection device 300 shown in FIG. 3 includes a pre-registered speech sound signal acquisition unit 102, a pre-registered speech sound digital signal storage unit 104, a pre-registered speech feature analysis unit 106, a pre-registered speech feature storage unit 108, and a random projection unit 109.

[0039] Furthermore, even when the configuration shown in FIG. 1 is implemented in one device, the portion including “the speech signal acquisition unit 101, the speech digital signal storage unit 103, the feature analysis unit 105, the feature storage unit 107, the pre-registered speech feature storage unit 108, and the target speaker task learning unit 110” may be called the learning device 200, and the portion including “the pre-registered speech speech signal acquisition unit 102, the pre-registered speech speech digital signal storage unit 104, the pre-registered speech feature analysis unit 106, the pre-registered speech feature storage unit 108, and the random projection unit 109” may be called the random projection device 300.

[0040] (Device Operation) Next, the flow of operation of the learning device 100 will be described with reference to a flowchart. Here, the operation of each of the learning device 200 shown in Fig. 2 and the random projection device 300 shown in Fig. 3 will be described. The operation of the learning device 200 and the operation of the random projection device 300 are performed in parallel.

[0041] <Operation of Random Projection Device 300> FIG. 4 is a flowchart showing the operation of the random projection device 300.

[0042] In S1, the pre-registered speech signal acquisition unit 102 acquires a speech signal of a pre-registered person (a person to be trained using a target speaker task model). This speech signal is, for example, a reproduced speech signal recorded with a PC microphone, an IC recorder, etc. The pre-registered speech signal acquisition unit 102 stores the acquired speech signal as a pre-registered speech digital signal in the pre-registered speech digital signal storage unit 104.

[0043] In S2, the pre-registered utterance feature analysis unit 106 reads the pre-registered utterance digital signal from the pre-registered utterance digital signal storage unit 104, performs feature analysis, and outputs a pre-registered utterance feature sequence. The pre-registered utterance feature sequence is stored in the pre-registered utterance feature storage unit 108.

[0044] In S3, the random projection unit 109 reads out the pre-registered utterance feature sequence from the pre-registered utterance feature storage unit 108, performs random projection, generates a target speaker expression, and outputs it.

[0045] <Operation of Learning Device 200> FIG. 5 is a flowchart showing the operation of the learning device 200.

[0046] In S11, the audio signal acquisition unit 101 acquires an audio signal that is a mixture of the audio signals of the pre-registered user and other users. This audio signal is, for example, a reproduced audio signal recorded by a PC microphone, an IC recorder, etc. The audio signal acquisition unit 101 stores the acquired audio signal as an audio digital signal in the audio digital signal storage unit 103.

[0047] It should be noted that the voice of the pre-registered person input to the random projection device 300 does not have to be the same as the voice of the pre-registered person input to the learning device 200 (which is mixed with the voices of other people).

[0048] In S12, the feature analysis unit 105 reads out the audio digital signal from the audio digital signal storage unit 103, performs feature analysis, and outputs a sequence of features. The sequence of features is stored in the feature storage unit 107.

[0049] In S13, the target speaker task training unit 110 trains a target speaker task model provided in the target speaker task training unit 110, using the acoustic features read out from the feature storage unit 107 and the target speaker expressions input from the random projection unit 109. A specific example of training will be described below.

[0050] The target speaker task is, for example, a task to extract (emphasize) and output the voice of only a specific person (pre-registered person) from a mixture of voices from multiple people.

[0051] In this case, the acoustic feature sequence from the feature accumulation unit 107 and the target speaker expression from the random projection unit 109 are input to a target speaker task model, which is, for example, a neural network, and an acoustic feature sequence is output from the target speaker task model. A correct acoustic feature sequence (of a pre-registered person) is prepared in advance, and the target speaker learning unit 110 optimizes the parameters of the target speaker task model so as to minimize the difference (loss) between the acoustic feature sequence output from the target speaker task model and the correct acoustic feature sequence.

[0052] Furthermore, when the target speaker task is, for example, a target speaker speech recognition task, the target speaker's speech text (transcription) for the input mixed speech is used as the correct answer data to recognize the target speaker's speech. Furthermore, when the target speaker task is, for example, a target speaker speech segment detection task, the target speaker's speech segment information is used as the correct answer data.

[0053] (Regarding execution of target speaker task) Figure 6 shows the device configuration when executing a target speaker task. A device with this configuration is called inference device 400. As shown in Figure 6, inference device 400 is obtained by replacing target speaker task learning unit 110 in learning device 100 (Figure 1) with inference unit 120. The operations of each unit other than inference unit 120 are as already explained. Inference unit 120 includes a trained target speaker task model. Note that learning device 100 and inference device 400 may be the same device. For example, the target speaker task learning unit 110 may include the function of inference unit 120.

[0054] As in the case of the learning device 100, an inference device 500 and a random projection device 600 may be provided, as shown in FIGS.

[0055] The operation during execution of the target speaker task will be described with reference to Fig. 6. Here, as an example, the target speaker task is assumed to be "a task to extract (emphasize) and output only the speech of a specific person (target speaker) from speech containing a mixture of speech from multiple people."

[0056] In this case, pre-recorded speech of a target speaker (a person whose voice is to be emphasized) is input to the pre-registration speech signal acquisition unit 102. A mixed voice signal of multiple people is input to the voice signal acquisition unit 101. Through the processes described above, the random projection unit 109 creates and outputs a target speaker representation.

[0057] The inference unit 120 receives the acoustic feature sequence from the feature accumulation unit 107 and the target speaker expression from the random projection unit 109. The inference unit 120 inputs the acoustic feature sequence and the target speaker expression to a trained target speaker task model, thereby acquiring an acoustic feature sequence output from the target speaker task model. The inference unit 120 generates a speech waveform from the acoustic feature sequence and outputs it as speech (target speaker speech).

[0058] (Hardware Configuration Example) Any of the devices described in this embodiment (learning device, random projection device, inference device) can be realized by, for example, running a program on a computer. This computer may be a physical computer or a virtual machine on the cloud.

[0059] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0060] Fig. 9 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 9 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0061] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0062] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0063] As described above, the technology described in this embodiment uses random projection (random matrix or random quantization) as a speaker model, which makes it possible to omit the training of the speaker model itself and reduce the training cost. In addition, it is possible to obtain robustness against degradation of task accuracy of the target speaker task for speakers who do not appear in the training.

[0064] The following additional notes are provided regarding the above-described embodiments.

[0065] <Additional Notes> (Additional Item 1) A learning device that trains a target speaker task model, comprising: a random projection unit that generates a target speaker representation by performing random projection on acoustic features of a target speaker's speech; and a learning unit that trains the target speaker task model using the target speaker representation and acoustic features of speech including the target speaker's speech and speech of other speakers. (Additional Item 2) The learning device according to Additional Item 1, wherein the random projection unit performs the random projection using a random matrix or random quantization. (Additional Item 3) An inference device that executes a target speaker task, comprising: a random projection unit that generates a target speaker representation by performing random projection on acoustic features of the target speaker's speech; and an inference unit that executes the target speaker task using the target speaker representation and acoustic features of speech including speech of one or more speakers. (Supplementary Item 4) The inference device according to Supplementary Item 3, wherein the inference unit executes the target speaker task using a target speaker task model trained by inputting a target speaker expression generated by randomly projecting acoustic features of the target speaker. (Supplementary Item 5) A training method executed by a training device that trains a target speaker task model, comprising: a random projection step of generating a target speaker expression by randomly projecting acoustic features of the target speaker's speech; and a training step of training the target speaker task model using the target speaker expression and acoustic features of speech including the target speaker's speech and speech of another speaker. (Supplementary Item 6) An inference method by an inference device that executes a target speaker task, comprising: a random projection step of generating a target speaker expression by randomly projecting acoustic features of the target speaker's speech; and an inference step of executing the target speaker task using the target speaker expression and acoustic features of speech including speech of one or more speakers. (Supplementary Item 7) A non-transitory storage medium storing a program for causing a computer to function as each unit in the learning device described in Supplementary Item 1 or 2. (Supplementary Item 8) A non-transitory storage medium storing a program for causing a computer to function as each unit in the inference device described in Supplementary Item 3 or 4.

[0066] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0067] 100, 200 Learning device 101 Speech signal acquisition unit 102 Pre-registered speech speech signal acquisition unit 103 Speech digital signal storage unit 104 Pre-registered speech speech digital signal storage unit 105 Feature analysis unit 106 Pre-registered speech feature analysis unit 107 Feature storage unit 108 Pre-registered speech feature storage unit 109 Random projection unit 110 Target speaker task learning unit 120 Inference unit 300, 600 Random projection device 400, 500 Inference device 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A learning device that trains a target speaker task model, comprising: a random projection unit that generates a target speaker representation by performing random projection on acoustic features of a target speaker's speech; and a learning unit that trains the target speaker task model using the target speaker representation and acoustic features of speech that includes the target speaker's speech and speech of other speakers.

2. The learning device according to claim 1, wherein the random projection unit performs the random projection using a random matrix or random quantization.

3. An inference device that executes a target speaker task, comprising: a random projection unit that generates a target speaker expression by performing random projection on acoustic features of the target speaker's speech; and an inference unit that executes the target speaker task using the target speaker expression and acoustic features of speech including speech of one or more speakers.

4. The inference device according to claim 3, wherein the inference unit executes the target speaker task using a target speaker task model trained by inputting a target speaker expression generated by performing random projection on the acoustic features of the target speaker.

5. A learning method executed by a learning device that trains a target speaker task model, comprising: a random projection step of generating a target speaker representation by performing random projection on acoustic features of the target speaker's speech; and a learning step of training the target speaker task model using the target speaker representation and acoustic features of speech that includes the target speaker's speech and speech of another speaker.

6. An inference method for an inference device that executes a target speaker task, comprising: a random projection step of generating a target speaker representation by performing random projection on acoustic features of the target speaker's speech; and an inference step of executing the target speaker task using the target speaker representation and acoustic features of speech including speech of one or more speakers.

7. A program for causing a computer to function as each part of the learning device according to claim 1 or 2.

8. A program for causing a computer to function as each part of the inference device according to claim 3 or 4.

Images