Voice extraction device and voice extraction method

Abstract

To improve performance of target voice extraction.SOLUTION: A basic encoder calculates features from mixed voices. An enhancement unit 112 calculates hierarchical features by upsampling and summing outputs from each of multiple layers contained in the SSL model to which the mixed voices are input. A speaker encoder uses the SSL model to compute embedding information based on the voice of the registered target sound source. An extraction network extracts the voice of the target sound source from the mixed voices, based on the features output by the basic encoder, the hierarchical features, and the embedding information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an audio extraction device and an audio extraction method. [Background technology]

[0002] Target Speech Extraction (TSE) is a well-known technology that extracts the speech of a target speaker from a mixed speech containing speech from multiple speakers. TSE uses a neural network that receives the mixed speech as input and is tuned by a signal from a pre-recorded target speaker's speech (hereinafter referred to as "registration speech").

[0003] For example, Non-Patent Document 1 describes a method of using embedded information of a target voice acquired using an auxiliary neural network when extracting the target voice from a mixed voice using a neural network. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] M. Delcroix et al., "Improving Speaker Discrimination of Target Speech Extraction With Time-Domain Speakerbeam," ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, ​​Spain, 2020, pp. 691-695, doi: 10.1109 / ICASSP40776.2020.9054683. Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional techniques have a problem in that speech other than the target speech may be erroneously extracted, making it impossible to improve the target speech extraction performance.

[0006] In recent years, it has been reported that speech models pre-trained using self-supervised learning (SSL) methods (hereafter referred to as SSL models) are useful for various speech processing tasks such as automatic speech recognition (ASR) and speaker verification.

[0007] The SSL model provides powerful and robust speaker representation and can provide useful information (e.g., phoneme information) for speech extraction. Therefore, if the SSL model can be applied to TSE, it is expected that the performance of TSE will be improved.

[0008] On the other hand, effective application of SSL models to TSE and other speech enhancement (SE) tasks has not been established because of the lack of a method to handle the difference in temporal resolution between SSL models and state-of-the-art (SOTA) models such as TSE. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems and achieve the object, the audio extraction device is characterized by having: a first calculation unit that calculates a first feature from mixed audio; a second calculation unit that calculates a second feature by upsampling and adding up outputs from multiple layers included in a neural network to which the mixed audio is input; a third calculation unit that uses the neural network to calculate embedded information based on audio of a registered target audio source; and an extraction unit that extracts the audio of the target audio source from the mixed audio based on the first feature, the second feature, and the embedded information. [Effects of the Invention]

[0010] According to the present invention, it is possible to improve the performance of target speech extraction. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a voice extraction device according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of a speaker encoder. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of an encoder. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of the strengthening unit. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of an upsample block. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of an upsample block. [Figure 7] FIG. 7 is a flowchart showing the flow of the training process. [Figure 8] FIG. 8 is a flowchart showing the flow of the inference process. [Figure 9] FIG. 9 shows the results of the experiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a computer that executes a program. DETAILED DESCRIPTION OF THE INVENTION

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A voice extraction device and a voice extraction method according to embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described below.

[0013] First, the configuration of a voice extraction device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a voice extraction device according to the first embodiment. The voice extraction device 1 receives input of a mixed voice and a registration voice, and outputs an extracted voice.

[0014] Furthermore, the audio here may be data that represents the waveform of an audio signal obtained by recording using amplitude, frequency, etc. Furthermore, the audio may be MFCC (Mel frequency cepstral coefficient), log Mel filter bank (FBANK: log Mel filter bank coefficients), ΔMFCC (first-order derivative of MFCC), ΔΔMFCC (second-order derivative of MFCC), log power, Δlog power (first-order derivative of log power), etc., calculated from the audio signal obtained by recording.

[0015] Mixed sounds are sounds that are a mixture of many types of sounds (hereinafter referred to as acoustic classes), such as human voices, car sounds, and dog barks. Mixed sounds can also be said to be sounds that are a mixture of sounds generated from multiple sound sources. The sound sources may be human or non-human.

[0016] The target speech is speech generated by a target speaker. The speaker may also be referred to as a sound source. The speech extraction device 1 performs processing to extract the target speech from the mixed speech. The extracted speech is an estimate of the target speech, and is obtained by processing by the speech extraction device 1. The enrolled speech is obtained, for example, by recording the speech of the target speaker.

[0017] 1, the speech extraction device 1 includes an extraction network 10 and a speaker encoder 20. The extraction network 10 and the speaker encoder 20 are both realized by neural networks. The extraction network 10 is an example of a third calculation unit.

[0018] The speaker encoder 20 calculates embedding information based on the input registration. The embedding vector is information that represents the characteristics of the target speaker, and is expressed as a vector, for example.

[0019] The extraction network 10 outputs extracted speech based on the input mixed speech and the embedded information received from the speaker encoder 20. The extraction network 10 includes an encoder 11, an extractor 12, a mask applicator 13, and a decoder 14.

[0020] The encoder 11 converts the mixed voice into a sequence of features. The encoder 11 will be described in detail later.

[0021] The extractor 12 calculates a mask based on the feature output from the encoder 11 and the embedded information output from the speaker encoder 20 .

[0022] The mask is information for extracting the feature of the target speech from the feature of the mixed speech. For example, the mask represents the occupancy rate of the target speaker's speech signal in the mixed speech signal at each time-frequency point as a weight. Furthermore, for example, the mask may be a binary value representing whether the target speaker's speech signal is dominant or not in the mixed speech signal at each time-frequency point.

[0023] The extractor 12 integrates the feature vector output from the encoder 11 and the embedding information vector output from the speaker encoder 20. For example, the extractor 12 may calculate the element-wise product of the vectors, the element-wise sum of the vectors, or perform vector concatenation.

[0024] Furthermore, the extractor 12 calculates a mask based on the vector obtained by the integration. For example, the extractor 12 linearly transforms the vector, and outputs the mask after confining the value range from 0 to 1 using an activation function (such as a Sigmoid function or ReLU function).

[0025] The mask application unit 13 applies a mask. For example, the mask application unit 13 outputs the element-wise product of the feature vector output from the encoder 11 and the mask vector as the masked feature.

[0026] The decoder 14 reconstructs the masked features and outputs the extracted speech. The decoder 14 is, for example, an iSTFT block or a trainable one-dimensional convolutional layer (1D Conv).

[0027] The configuration of the speaker encoder 20 will be described using Fig. 2. Fig. 2 is a diagram showing an example of the configuration of a speaker encoder. The speaker encoder 20 outputs speaker embedding information based on input enrollment speech. As shown in Fig. 2, the speaker encoder 20 includes an SSL model 21 and a pooling layer 22.

[0028] The SSL model 21 has been trained in advance using a large amount of speech data. The speech data used for training includes HuBERT, wav2vec, WaveLM, etc. Details of the SSL model 21 will be described later.

[0029] The pooling layer 22 pools the output of the SSL model 21. The pooling layer 22 calculates, for example, an average value.

[0030] The configuration of the encoder 11 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the configuration of the encoder. The encoder 11 outputs encoded features based on the input mixed audio. As shown in Fig. 3, the encoder 11 includes a base encoder 111, an enhancement unit 112, and a combination unit 113. The base encoder 111 is an example of a first calculation unit. The enhancement unit 112 is an example of a second calculation unit.

[0031] The basic encoder 111 encodes the mixed audio. For example, the basic encoder 111 is an STFT module or a trainable one-dimensional convolution layer (1D Conv).

[0032] The enhancement unit 112 is an adaptive input enhanced (AIE) block. The enhancement unit 112 converts the features of the mixed speech obtained from each of the multiple layers of the SSL model into a sequence of features. The enhancement unit 112 outputs the sequence of features as a vector (hierarchical feature h).

[0033] The combining unit 113 concatenates the vector output from the base encoder 111 and the vector output from the enhancing unit 112 .

[0034] In conventional TSE, the output of the base encoder 111 is used as the encoded feature. In contrast, in this embodiment, the output of the combining unit 113 is used as the encoded feature. That is, the speech extraction device 1 of this embodiment differs from the conventional technology in that it includes the enhancing unit 112 and the combining unit 113.

[0035] The configuration of the enhancement unit 112 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the enhancement unit. As shown in Fig. 4, the enhancement unit 112 includes a trained SSL model. The SSL model of the enhancement unit 112 is the same model as the SSL model 21 of the speaker encoder 20. In other words, the SSL model is used by both the speaker encoder 20 and the encoder 11.

[0036] 4, the SSL model includes a convolutional neural network (CNN) layer 112a and a transformer 112b. Furthermore, the enhancement unit 112 includes a linear layer 112c and an upsample layer 112d. The SSL model including the transformer and the CNN is described in References 1 and 2.

[0037] Reference 1: Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed, “Hubert: Selfsupervised speech representation learning by masked prediction of hidden units,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451-3460, 2021.

[0038] Reference 2: Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems, vol. 33, pp. 12449-12460, 2020.

[0039] As shown in FIG. 4, the CNN layer 112a includes multiple one-dimensional convolution layers (Conv1D Block). j cnn is the output of the j-th layer of the CNN Layer 112a.

[0040] The Transformer 112b has Self-Attention, Add & Norm, and FFN (Feed-Forward Network) layers. The Transformer 112b repeats the process N times. H i trf is the output of the i-th processing of the Transformer 112b.

[0041] The transformer 112b calculates the weighted sum shown in the parentheses on the right side of equation (1). Furthermore, the linear layer 112c performs a linear transformation of the weighted sum as shown in equation (1). Linear() is a function that performs a linear transformation. Furthermore, the weight ω i AIE is a learnable parameter.

[0042]

number

[0043] Here, layers other than the Transformer in the SSL model (e.g., CNN Layer) have higher time resolution than the Transformer. Therefore, the Upsample Layer 112d integrates the output of the Transformer 112b and the output of the Linear Layer 112c while performing upsampling as shown in Equation (2). Upsample() is a function that performs upsampling.

[0044]

number

[0045] The configuration of the Upsample Layer 112d will be described using Figures 5 and 6. Figures 5 and 6 are diagrams showing examples of the configuration of an Upsample Block. The Upsample Layer 112d includes a plurality of Upsample Blocks that perform upsampling. The configuration of the Upsample Block may be as shown in either Figure 5 or Figure 6. However, the configuration of the Upsample Block is not limited to that shown in Figures 5 and 6.

[0046] [Feature Pyramid Module] In the example of FIG. 5, as shown in equation (3), the upsample block uses the output H j cnnThe result of the convolution is then transferred to the output T of the previous Upsample Block. j+1 Then, the Upsample Block performs deconvolution on the result of the addition. Conv() is a function that performs convolution. DeConv() is a function that performs deconvolution.

[0047]

number

[0048] [U-net Module] In the example of FIG. 6, as shown in equation (4), the upsample block uses the output H j cnn and the output T of the previous Upsample Block j+1 Then, the Upsample Block performs deconvolution of the combined result. Concat() is a function that combines vectors.

[0049]

number

[0050] The flow of the training process will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of the training process. As shown in Fig. 7, first, the speech extraction device 1 loads parameters of the SSL model (step S101). In addition, the speech extraction device 1 randomly initializes parameters other than those of the SSL model (step S102).

[0051] The parameters other than the SSL model include the speaker encoder 20 and the parts of the encoder 11 other than the SSL model, and the parameters of the decoder 14 .

[0052] The speech extraction device 1 loads the mixed speech for training and verification, the enrollment speech, and the target speech (step S103). Then, the speech extraction device 1 performs the backpropagation algorithm (step S104). At this time, the parameters of the SSL model are fixed. That is, the speech extraction device 1 updates parameters other than the SSL model without updating the parameters of the SSL model. The backpropagation algorithm is performed on, for example, the extraction network 10.

[0053] The voice extraction device 1 calculates a loss for verification (for example, SDR: source-to-distortion ratio) (step S105). If the SDR has not converged (step S106, No), the voice extraction device 1 returns to step S104 and repeats the process. If the SDR has converged (step S106, Yes), the voice extraction device 1 proceeds to step S107.

[0054] If fine tuning is not to be performed (step S107, No), the speech extraction device 1 outputs the updated model parameters (step S113). On the other hand, if fine tuning is to be performed (step S107, Yes), the speech extraction device 1 proceeds to step S108 and performs fine tuning. Whether or not to perform fine tuning may be determined in advance by the user.

[0055] The speech extraction device 1 loads parameters of the SSL model, a trained encoder, a decoder, and an extractor (step S108), and then loads mixed speech for training and verification, enrollment speech, and target speech (step S109).

[0056] The training and validation audio loaded in step S109 may be different from the training and validation audio loaded in step S103. For example, a general-purpose audio is loaded in step S103, and an audio appropriate for the task to be actually performed is loaded in step S109.

[0057] Next, the voice extraction device 1 performs the backpropagation algorithm (step S110). At this time, similar to step S104, the parameters of the SSL model are passed through without being updated.

[0058] Then, the speech extraction device 1 calculates the verification loss (SDR) (step S111). If the SDR has not converged (step S112, No), the speech extraction device 1 returns to step S110 and repeats the process. If the SDR has converged (step S112, Yes), the speech extraction device 1 proceeds to step S113 and outputs the model parameters (step S113).

[0059] The flow of the inference process will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the flow of the inference process. As shown in Fig. 8, first, the speech extraction device 1 loads parameters of a speaker encoder including an SSL model (step S201).

[0060] Next, the speech extraction device 1 inputs the recorded speech of the target speaker (enrollment speech) into the model (step S202). The enrollment speech is input to the speaker encoder 20. The speech extraction device 1 calculates speaker embedding using the speaker encoder 20 (step S203).

[0061] Next, the voice extraction device 1 loads the parameters of each model including the SSL model (step S204). Here, the voice extraction device 1 loads the parameters of the extraction network 10.

[0062] The speech extraction device 1 receives mixed speech from multiple speakers and speaker embeddings (step S205). The mixed speech is input to the encoder 11. The speaker embeddings are output from the speaker encoder 20 and input to the extractor 12.

[0063] The voice extraction device 1 processes the mixed voice using the encoder 11 and the extractor 12 (step S206). A mask is obtained by the process of step S206.

[0064] The speech extraction device 1 applies a mask to the encoded features (step S207), and then reconstructs the extracted signal using the decoder 14 (step S208).

[0065] As explained above, the base encoder 111 calculates features from the mixed speech. The enhancement unit 112 calculates hierarchical features by upsampling and adding up the outputs of multiple layers included in the SSL model to which the mixed speech is input. The speaker encoder 20 uses the SSL model to calculate embedded information based on the registered target speech. The extraction network 10 extracts the target speech from the mixed speech based on the features, hierarchical features, and embedded information output by the base encoder 111.

[0066] This allows the high temporal resolution required for speech extraction to be achieved by upsampling the features derived from the pre-trained SSL model, enabling the SSL model to be applied effectively in speech extraction, thereby improving the performance of target speech extraction.

[0067] [Experimental Results] Here, the results of an experiment conducted to compare the embodiment with a conventional method will be described. Fig. 9 is a diagram showing the experimental results.

[0068] The experiment used LibriMix, a mixture of speech from multiple speakers obtained from LibriSpeech. The evaluation metrics used in the experiment were source-to-distortion ratio (SDR), scale-invariant SDR improvement (SI-SDRi), perceptual evaluation of speech quality (PESQ), short-time objective intelligibility (STOI), and failure rate (FR) (see Reference 3). FR is the percentage of samples with SI-SDRi below 1 dB. The better the model performance, the higher the metrics other than FR and the lower the FR.

[0069] Reference 3: Olaf Ronneberger, Philipp Fischer, and Thomas Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 2015, pp. 234-241.

[0070] The upper row (cross) of Proposed in Figure 9 corresponds to an embodiment without fine tuning. The lower row (check) of Proposed corresponds to an embodiment with fine tuning. TD-Speakerbeam is a method that excludes the part that uses the SSL model from the embodiment, and corresponds to the prior art.

[0071] As shown in Fig. 9, the model performance of this embodiment is better than that of TD-Speakerbeam. In particular, the model performance is improved when fine tuning is performed.

[0072] [System configuration, etc.] Furthermore, the components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of each device can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0073] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0074] The configuration of the SSL model is not limited to a combination of CNN and Transformer, and may be other configurations. Furthermore, the enhancement unit 112 of this embodiment, i.e., the AIE, may be used for tasks other than target speech extraction, such as noise reduction or separation without using registered speech. Furthermore, target speech extraction as in this embodiment can be performed using an SSL model trained on general speech.

[0075] [program] In one embodiment, the voice extraction device 1 can be implemented by installing a program that executes the above-described processes as package software or online software on a desired computer. For example, by having an information processing device execute the above-described program, the information processing device can function as the voice extraction device 1. The information processing device referred to here includes desktop and notebook personal computers. In addition, the information processing device also includes smartphones, tablet terminals, and the like.

[0076] The voice extraction device 1 can also be implemented as a server device that provides services related to the above processing to a client terminal device used by a user. For example, the server device is implemented as a server device that provides a voice extraction service that receives a mixed voice signal as input and extracts a target voice signal. In this case, the server device may be implemented as a web server or as a cloud that provides services related to the above processing by outsourcing.

[0077] 10 is a diagram showing an example of a computer that executes a program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0078] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0079] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define each process of the audio extraction device 1 are implemented as program modules 1093 in which computer-executable codes are written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to the functional configuration in the processes of the audio extraction device 1 are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD.

[0080] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0081] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0082] 1. Audio extraction device 11 Encoder 12 Extractor 13 Mask application area 14 Decoder 20 Speaker Encoder 21 SSL Model 22 Pooling Layer 111 Basic Encoder 112 Strengthening Department 112a CNN Layer 112b Transformer 112c Linear Layer 112d Upsample Layer 113 Joint

Claims

1. a first calculation unit that calculates a first feature from the mixed audio; a second calculation unit that calculates a second feature by upsampling and adding up outputs of a plurality of layers included in a neural network to which the mixed speech is input; a third calculation unit that calculates embedding information based on a registered target sound source voice using the neural network; an extraction unit that extracts the target sound source sound from the mixed sound based on the first feature amount, the second feature amount, and the embedded information; An audio extraction device comprising:

2. the second calculation unit has upsampling layers corresponding to each of a plurality of layers included in the neural network; Each of the upsampling layers calculates the second feature by outputting a deconvolution result obtained by adding together the convolution result of the output of the corresponding layer of the neural network and the output of the previous upsampling layer.

2. The audio extraction device according to claim 1.

3. the second calculation unit has upsampling layers corresponding to each of a plurality of layers included in the neural network; Each of the upsampling layers calculates the second feature by outputting a deconvolution result obtained by combining the output of the corresponding layer of the neural network with the output of the previous upsampling layer.

2. The audio extraction device according to claim 1.

4. A voice extraction method executed by a voice extraction device, comprising: a first calculation step of calculating a first feature from the mixed audio; a second calculation step of calculating a second feature amount by upsampling and adding up outputs of a plurality of layers included in the neural network to which the mixed speech is input; a third calculation step of calculating embedding information based on the registered target sound source speech using the neural network; an extraction step of extracting the target sound source sound from the mixed sound based on the first feature amount, the second feature amount, and the embedded information; 10. A method for extracting audio, comprising:

Images