Generative Speech Separation Method and Device Incorporating Fundamental Frequency Cues

Abstract

The present invention discloses a generative voice separation method and device that introduce fundamental frequency cues. The present invention innovatively uses fundamental frequency cues as conditions, combines the spectral detail information provided by the mixed speech, and uses a generative neural network to generate the target speaker's speech. Compared with the traditional method of extracting the target speech by estimating the time-frequency mask of the mixed speech, the quality and intelligibility of the separated speech are improved. Compared with the method of separating all speaker speeches simultaneously, the present invention solves the problems of uncertain number of speakers and difficulty in determining the correspondence between the network output and the real speaker in the existing methods by introducing fundamental frequency cues. Relying on the auditory processing mechanism and neural network structure, the present invention achieves comparable performance with the current optimal method, providing the possibility for machines to solve the voice separation task in complex scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech signal processing, relates to speech separation technology, and particularly relates to a generative speech separation method and device introducing fundamental frequency clues. Background Art

[0002] Speech, as an interaction medium, demonstrates the advantages of high efficiency and directness. With the progress and application of artificial intelligence algorithms, intelligent devices using speech as an interaction method have been rapidly applied and developed. Abroad, there are Siri of Apple, Google Home of Google, and Echo of Amazon; domestically, there are Xiaodu of Baidu, Xiaoai of Xiaomi, and Tmall Genie. However, real acoustic scenarios are usually complex, and environmental noise, human voice interference, as well as echo and reverberation will seriously reduce the intelligibility and sound quality of speech, affecting the functions of intelligent voice devices, such as speech recognition and speaker recognition, and further affecting the human-computer interaction experience. Therefore, in complex acoustic scenarios, separating the target speech from the mixed speech signal is crucial.

[0003] The human auditory system has a powerful anti-interference ability, that is, it can recognize the speech of a specific speaker in a complex acoustic scenario while ignoring the interference of other sounds (human voices or non-human voices). This phenomenon is also known as the "cocktail party problem". However, for machines, it is very difficult to have such an anti-interference ability, and speech interaction in complex acoustic scenarios still poses challenges.

[0004] In recent years, with the progress of deep learning, deep neural networks have begun to be applied to solve the above problems, which has accelerated the development of this field. Researchers have abstracted the above "cocktail party problem" as a speech separation task, and the goal is to construct a mapping from multi-speaker mixed speech to separated speech. Most existing methods follow the following paradigm: using multi-speaker mixed speech as input, and learning the speech representation of each speaker through a deep neural network, that is, separating the speech of each speaker. However, whether in terms of interpretability or generalization, the performance of current methods still has a certain gap compared with the human level.

[0005] In addition, most of the current neural network-based speech separation methods first transform the mixed speech to obtain the mixed speech spectrum as the input, use the time-frequency mask of the target speaker as the network output, and then multiply the time-frequency mask by the spectrum of the input mixed speech to obtain the spectral representation of the estimated target. The problem with this type of method is that in some time-frequency regions, when the foreground speech (the speech to be separated) is largely or completely masked by the background sound (noise and other interfering sounds), the foreground speech cannot be recovered from the mixed speech, and the background sound cannot be completely removed. On the one hand, the network uses the time-frequency mask rather than the speech signal directly as the estimation target, and the accuracy of the mask estimation will affect the performance of the reconstructed speech signal. On the other hand, the time-frequency mask needs to be multiplied by the spectrum of the mixed speech through an additional operation to obtain the spectrum of the target speech, which will further reduce the signal-to-noise ratio of the target speech.

[0006] Most speech separation methods follow the paradigm of separating each speaker. The separation system lacks conditions as a guide and cannot determine the speakers to be separated. It can only separate the speech of all speakers and output it at once. Then, through subsequent processing modules, such as speaker recognition, the speech of the target speaker to be separated is determined. Due to the lack of conditions, the target to be separated cannot be determined, making it difficult to determine the correspondence between the multiple outputs of the network and the target speaker labels. Some methods need to pre-determine the number of speakers and are difficult to handle the situation where the number of speakers is unknown. Summary of the Invention

[0007] In view of the shortcomings of the existing methods, the present invention proposes a generative speech separation method and device that introduces fundamental frequency cues.

[0008] The present invention proposes a speech separation framework conditioned on auditory cues (fundamental frequency): the input is the mixed speech spectrum and the fundamental frequency cue of the speaker to be separated, and this cue will be used as a condition to guide the separation of the corresponding speaker's speech. The speaker to be separated can be any one of the speakers included in the mixed speech, or can also be a specified speaker in practical applications.

[0009] The present invention innovatively uses the fundamental frequency cue as a condition, combines the spectral detail information provided by the mixed speech, and obtains the speech corresponding to the fundamental frequency in a generative manner rather than in the form of a time-frequency mask. By introducing the fundamental frequency cue, the problems of uncertain number of speakers and difficulty in determining the correspondence between the network output and the real speaker are solved. From an auditory perspective, it is to simulate the human ear to generate the target speech in the brain based on the representation of the auditory target and the salient features (here refers to the fundamental frequency) retained in memory. Therefore, the entire speech separation process not only includes the encoding process of the target speech (most of the existing methods only include this process), but also includes the fundamental frequency cue based on the given speaker.

[0010] The technical solution of the present invention is as follows:

[0011] 1) For a given mixed speech, perform frame segmentation, windowing, and short-time Fourier transform operations in sequence to obtain the short-time Fourier transform spectrum (STFT spectrum) of the mixed speech;

[0012] 2) Input the spectrum obtained in 1) and the fundamental frequency trajectory sequence of the target speaker into the generator G1, and generate the speech spectrum of the target speaker through the fundamental frequency condition-constrained generator;

[0013] 3) Input the speech spectrum of the target speaker obtained in 2) into the generator G2. The input spectrum passes through a one-dimensional transposed convolution module, an upsampling module, and a residual module, and then through a one-dimensional convolution and activation function layer to output a representation that satisfies the numerical range of the time-domain signal, that is, the speech of the target speaker;

[0014] 4) Input the speech of the target speaker generated in 3) and the fundamental frequency trajectory sequence of the target speaker into the discriminator. The discriminator determines whether the generated speech is the real speech of the target speaker (that is, outputs 1 / 0 to represent authenticity), calculates the optimization objective function values (that is, loss function values) of the generators G1 and G2 based on this result, and adjusts the parameters of the generators G1 and G2 based on this. At the same time, calculate the optimization objective function value of the discriminator based on the true labels of the training data, and adjust the discriminator parameters based on this;

[0015] 5) During training, loop through steps 1) to 4) until the loss function values of the generators G1 and G2 and the discriminator tend to be stable, that is, reach the convergence state. At this time, fix the parameter settings of the generators and the discriminator;

[0016] 6) During testing (that is, during actual application), for the mixed speech to be separated, using the parameter settings of the generators G1 and G2 in the convergence state as the standard, perform steps 1) to 3). The speech output in step 3) is the finally separated speech of the target speaker.

[0017] Furthermore, the input of the generator G1 is the mixed speech and the fundamental frequency condition. This fundamental frequency-constrained generator G1 generates a representation of the corresponding speaker's speech; the generator G1 first uses three layers of two-dimensional convolution with a convolution kernel size of 3 and a stride of 2. Each layer adopts a residual network structure with a residual connection, followed by 4 Transformer modules. The input feature dimension of each layer is 512, where the self-attention part uses 8 heads and the dimension of the feed-forward layer is 1024; a linear layer is added between the convolution module and the Transformer module to make the output dimension of the former match the input dimension of the latter.

[0018] Furthermore, the generator G2 upsamples the output magnitude spectrum of G1 to the dimension of the corresponding time-domain signal using a set of stacked transposed convolution modules and one-dimensional convolutions. The upsampling factor is determined by the frame shift of the input magnitude spectrum. A residual module is connected after each transposed convolution module. The residual module consists of three layers of one-dimensional convolutions with dilation. By setting different coefficients (1, 3, 9), a receptive field of size 27 is obtained to expand the receptive ability for the time dimension of the signal and better model the long-range dependencies in the time dimension. Finally, the generator G2 uses a single layer of one-dimensional convolution and the tanh activation function to output a representation that satisfies the numerical range of the time-domain signal. The output channel of this one-dimensional convolution is set to 1, corresponding to the time-domain signal of the full frequency band.

[0019] Furthermore, the discriminator D adopts a Transformer structure and performs discrimination at the block level. That is, the input time-frequency representation is regarded as a two-dimensional graph with two channels. The first channel represents the magnitude spectrum of the true or false target speech, and the second channel corresponds to the concatenated conditional fundamental frequency. The input is decomposed into several blocks and unfolded into a one-dimensional sequence through a linear layer. A learnable positional encoding and a classification symbol [cls] are added at the beginning of this sequence, and then it is input into a 3-layer Transformer module. Finally, the classification symbol [cls] is used to give the true / false discrimination result.

[0020] Furthermore, using the mixed speech sequence and the fundamental frequency trajectory sequence of the target speaker as input conditions, a conditional generative adversarial network is used to generate the speech of the target speaker corresponding to the fundamental frequency. The conditional adversarial generative network consists of a generator and a discriminator. The generator aims to generate a time-domain signal corresponding to the fundamental frequency condition from the magnitude spectrum of the mixed speech. The discriminator consists of multiple sub-discriminators acting on different frequency bands of the speech signal.

[0021] A generative speech separation device introducing fundamental frequency cues, comprising:

[0022] A training unit is used to train generator G1, generator G2, and discriminator. The training process includes: for a given mixed speech, perform frame segmentation, windowing, and short-time Fourier transform operations in sequence to obtain the short-time Fourier transform spectrum of the mixed speech; input the obtained spectrum and the fundamental frequency trajectory sequence of the target speaker into generator G1, and generate the speech spectrum of the target speaker through the fundamental frequency condition constraint generator; input the obtained speech spectrum of the target speaker into generator G2, the input spectrum passes through a one-dimensional transposed convolution module, an upsampling module, and a residual module, and then through a one-dimensional convolution and activation function layer to output a representation that satisfies the numerical range of the time-domain signal, that is, the speech of the target speaker; input the generated speech of the target speaker and the fundamental frequency trajectory sequence of the target speaker into the discriminator, and the discriminator determines whether the generated speech is the real speech of the target speaker. Based on this result, calculate the optimization objective function values of generators G1 and G2, and adjust the parameters of generators G1 and G2 accordingly. At the same time, calculate the optimization objective function value of the discriminator according to the true label of the training data, and adjust the discriminator parameters based on this; loop through the above steps until the loss function values of generators G1, G2, and the discriminator tend to be stable, that is, reach the convergence state. At this time, fix the parameter settings of the generators and the discriminator;

[0023] A speech separation unit is used to use generators G1 and G2 in the convergence state to obtain the separated speech of the target speaker for the mixed speech to be separated.

[0024] Compared with the prior art, the positive effects of the present invention are:

[0025] Based on auditory scene analysis as the theoretical basis and inspired by the conditional generative adversarial network at the implementation level, the present invention proposes a generative speech separation framework based on fundamental frequency cues: given the fundamental frequency trajectory of any speaker in the mixed speech as a condition, obtain the separated speech of the speaker through the "generation" process. This framework has the following advantages: (1) It avoids the problem in the existing speech separation framework that the speaker to be separated cannot be determined due to the lack of conditions. The present invention gives the fundamental frequency cue of a certain speaker as a condition and only separates the speech of the speaker corresponding to the condition, regardless of the number of speakers included in the input mixed speech. Therefore, the problems of output permutation and unknown number of speakers are solved; (2) It completely abandons the existing time-frequency masking-based framework, regards the process of obtaining separated speech as a generation process rather than an encoding process, and improves the quality and intelligibility of the separated speech. Description of the Drawings

[0026] Figure 1 is a schematic diagram of the idea flow of the present invention;

[0027] Figure 2 is a schematic diagram of the specific process framework of the present invention. Detailed Embodiment

[0028] The specific implementation manners of the present invention will be described in more detail below.

[0029] The conceptual framework diagram of the method of the present invention is as shown in Figure 1 shown, and the specific process framework diagram is as shown in Figure 2 shown. The specific implementation modules of the method of the present invention include a signal preprocessing module, a generator G1, a generator G2, a discriminator, etc. The composition structures and specific implementations of each module are as follows:

[0030] 1. Voice signal preprocessing

[0031] In this method, the mixed speech is first subjected to short-time Fourier transform (STFT) as the subsequent input. The signal is subjected to short-time Fourier transform using an analysis window w(n), a window length N, and a frame shift R. The transformation formula is as follows:

[0032]

[0033] where X(t,f) represents the value corresponding to the f-th frequency band of the t-th frame of the spectrum X, x(n) represents the value corresponding to the n-th sampling point of the signal x, n represents the index of the signal sampling point, and t and f respectively represent the indices of the frame and the frequency band. After the transformation, the STFT spectrum is obtained. In a specific implementation, the frame length used is 32 ms, the frame shift is 16 ms, and the window function is a Hamming window. The STFT spectrum contains the amplitude information and phase information of this segment of the speech signal in the time dimension and the frequency dimension (that is, the amplitude spectrum and phase spectrum of the speech can be deduced from the STFT spectrum).

[0034] 2. Generator G1

[0035] The input of the generator G1 is the mixed speech and the fundamental frequency condition, and this fundamental frequency will constrain the generator to generate the representation of the corresponding speaker's speech. For the input mixed speech amplitude spectrum X and the fundamental frequency Y of the speaker to be separated, the former contains T frames and F frequency bands, and the latter contains T frames and N frequency bands (the possible frequency range of the fundamental frequency is quantized into N discrete frequencies). According to the critical band criterion, if the fundamental frequency of this frame falls within the frequency band centered on f c (corresponding to each frequency point in X), then this fundamental frequency is assigned to the corresponding frequency band fc, so that Y can be extended to the same number of frequency bands as X. In addition, an additional channel VC (C = 1) is extended, so that the two are concatenated in the channel dimension to obtain Z (C = 2). The entire generator includes the following several modules: a convolution module, a Transformer module, and a linear layer.

[0036]

[0037]

[0038] Among them, CNN(·) is a series of stacked two-dimensional convolutions, aiming to locally model the time-frequency representation of the input speech, capture its local spectral characteristics, merge the channel dimension and frequency dimension of its output into one dimension (C0×F0), and obtain E CNN is regarded as a sequence with T0 frames and a feature dimension of C0×F0. Then, a four-layer Transformer module is used to model the time dimension of the sequence E CNN The Transformer module consists of two components, including a multi-head self-attention module and a feed-forward layer with a GELU non-linear function. Finally, the output of the Transformer passes through a linear layer to map the dimension D of the frequency feature to the dimension F corresponding to the target amplitude spectrum. In fact, the time dimension of the above three modules remains unchanged and is equal to the number of frames of the input spectrum, that is, T=T0=T1=T2.

[0039] In terms of network structure, the generator G1 first uses three layers of two-dimensional convolutions with a kernel size of 3 and a stride of 2. Each layer adopts a ResNet structure with a residual connection, followed by 4 layers of Transformer modules. The input feature dimension of each layer is 512. Among them, the self-attention part uses 8 sub-attention modules to form the total attention module, also known as the 8-head attention module, and the dimension of the feed-forward layer is 1024. A linear layer is added between the convolution module and the Transformer module to make the output dimension of the former match the input dimension of the latter. The ResNet structure and the Transformer are both well-known network modules in the field, and this method combines them for practical problems.

[0040] 3. Generator G2

[0041] The generator G2 uses a group of stacked deconvolution modules and one-dimensional convolutions to upsample the output amplitude spectrum of the first stage (i.e., G1) to the dimension corresponding to the time-domain signal. The upsampling factor is determined by the frame shift of the input amplitude spectrum. After each deconvolution module, there is a residual module, which consists of three layers of one-dimensional convolutions with dilation. By setting different coefficients (1, 3, 9), a receptive field of size 27 can be obtained to expand the ability to sense the time dimension of the signal and better model the long-range dependence relationship of the time dimension. The generator G2 finally uses a layer of one-dimensional convolution and a tanh activation function to output a representation that satisfies the numerical range of the time-domain signal. The output channel of this one-dimensional convolution is set to 1, corresponding to the time-domain signal of the full frequency band.

[0042] In terms of the network structure, the upsampling module of the generator G2 uses three upsampling layers to gradually upsample the input dimension to 64 times the original (determined by the length of the frame shift). The upsampling coefficients of each layer are [4x, 4x, 4x], and the output channel numbers are 256, 128, and 64 respectively. Each upsampling layer consists of a transposed convolution layer and a residual dilation convolution module (ResStack). Among them, the kernel size of the transposed convolution is twice the stride. The ResStack module consists of 4 one-dimensional convolutional layers with dilation. The kernel size is 3, and the dilation sizes are 1, 3, 9, and 27 as the number of layers increases. Finally, a receptive field corresponding to 81 frames can be obtained. Some studies have shown that expanding the receptive field of the convolution to a reasonable range can improve the quality of the generated speech.

[0043] 4. Discriminator

[0044] The framework diagram of the discriminator D is as Figure 2 shown. Specifically, the discriminator adopts a Transformer structure. Different from the common method of discrimination based on the entire time-frequency representation, here discrimination based on the patch level is adopted, that is, the input time-frequency representation is regarded as a two-dimensional graph with two channels. The first channel represents the amplitude spectrum of the true or false target speech, and the second channel corresponds to the concatenated conditional fundamental frequency. The input is decomposed into several patches, and through a linear layer, it is unfolded into a one-dimensional sequence. A learnable position encoding and a classification symbol [cls] are added at the beginning of the sequence, and it is input into a 3-layer Transformer module, whose composition structure is the same as the above generator. Finally, the classification symbol [cls] is used to give the true or false discrimination result.

[0045] Specifically, first use 2 layers of two-dimensional convolution to decompose the input spectrum into 12 * 5 patches. The kernel size and stride of the convolution are both 7 * 5. Then, through a linear expansion layer, the time dimension and the frequency dimension are merged to obtain a one-dimensional sequence with a sequence length of 12 * 5 and a feature dimension of the channel dimension after the above two-dimensional convolution. A position encoding and a symbol [cls] for discriminative classification are added at the beginning of the sequence. The above result is input into the Transformer layer.

[0046] 5. Definition of the objective function

[0047] A multi-scale strategy is adopted to characterize the objective function. A conditional generative adversarial network is used to generate the speech of the target speaker corresponding to the fundamental frequency. G1 and G2 will jointly serve as the generator G. At the same time, multiple discriminators are used instead of the traditional single discriminator for true / false discrimination. The speech generated by only using a single discriminator will have metallic noise. Since speech signals have different spectral characteristics in different frequency ranges, therefore, for different frequency bands, different discriminators will be used. Here, the concept of "multi-scale" refers to different frequency bands. Specifically, the multi-scale discriminators will share the same set of network structures and parameters, but act on different frequency bands of the speech signal. When K discriminators are adopted, the training objectives of the discriminator and the generator are respectively:

[0048] Discriminator:

[0049] Generator:

[0050] Among them, D k is the k-th discriminator, x is the magnitude spectrum of the input mixed speech, y is the given fundamental frequency condition, and s is the time-domain waveform of the speaker corresponding to this fundamental frequency; represents the mathematical expectation of the formula (D k (s|y) - 1) 2 under the condition that the true time-domain waveform signal is s; D k (s|y) represents the output value of the k-th discriminator given the fundamental frequency condition y and the time-domain waveform s of the speaker corresponding to this fundamental frequency; represents the mathematical expectation of the formula (D k (G(x|y))) 2 under the condition that the magnitude spectrum of the mixed speech is x; D k (G(x|y)) represents the output result of the k-th discriminator for the time-domain signal G(x|y) generated by the generator.

[0051] Based on the loss function of the generative adversarial network (GAN), this invention introduces the multi-resolution STFT loss. Previous studies have shown that this loss function can not only effectively measure the differences between true and false time-domain waveforms in the latent feature space, but also stabilize the training of GAN and accelerate convergence. For the loss of a single STFT, the objective is to minimize the spectral convergence error L between the estimated real target signal s and the signal estimated by the generator sc and the logarithmic magnitude spectrum error

[0052]

[0053] Among them, ‖·‖ FThe ‖·‖₂ and ‖·‖₁ are the Frobenius and L₁ normalizations respectively, |STFT(·)| represents the magnitude spectrum of the signal after STFT transformation, and N is the total number of elements in the magnitude spectrum.

[0054] The multi-resolution STFT loss consists of M individual STFT losses, where each STFT loss corresponds to different FFT sizes, window lengths, and frame shift parameters. Averaging these M losses gives the final multi-resolution STFT loss function:

[0055]

[0056] where, denotes that the true target signal is s, and under the condition that the signal estimated by the generator is the mathematical expectation of the result obtained by the formula denotes the spectral convergence error between the true target signal s and the signal estimated by the generator, denotes the log magnitude spectrum error between the true target signal s and the signal estimated by the generator.

[0057] Therefore, the objective function of the entire adversarial - generative network can be expressed as:

[0058]

[0059] where λ represents the network training weight parameter set manually according to experience.

[0060] A generative speech separation method based on fundamental frequency cues according to an embodiment of the present invention includes the following steps:

[0061] 1) For the given mixed speech, perform frame division, windowing, and short-time Fourier transform operations in sequence to obtain the short-time Fourier transform spectrum (STFT spectrum) of the mixed speech.

[0062] 2) Input the spectrum obtained in 1) and the fundamental frequency trajectory sequence of the target speaker into the generator G1. The fundamental frequency condition will constrain the generator to generate the speech spectrum corresponding to the speaker.

[0063] 3) Input the speech spectrum of the target speaker obtained in 2) into the generator G2. The input spectrum passes through a one-dimensional transposed convolution module, an upsampling module, and a residual module, and then through a one-dimensional convolution and activation function layer to output a representation that satisfies the numerical range of the time-domain signal, that is, the speech corresponding to the target speaker.

[0064] ​4) Input the target speaker's speech and the fundamental frequency trajectory sequence of the target speaker generated in 3) into the discriminator. The discriminator determines whether the generated speech is the real target speaker's speech (i.e., outputs 1 / 0 to indicate authenticity), calculates the optimization objective function values (i.e., loss function values) of generators G1 and G2 based on this result, adjusts the parameters of generators G1 and G2 accordingly, and simultaneously calculates the optimization objective function value of the discriminator based on the true labels of the training data, and adjusts the discriminator parameters accordingly.

[0065] 5) During training, loop through steps 1) - 4) until the loss function values of generators G1 and G2 and the discriminator respectively tend to be stable, i.e., reach the convergence state. At this time, fix the parameter settings of the generators and the discriminator.

[0066] 6) During testing (i.e., during actual application), using the parameter settings of generators G1 and G2 in the convergence state as the standard, execute steps 1) - 3). The speech output in step 3) is the finally separated target speaker's speech.

[0067] The advantages of the present invention are described below in conjunction with specific embodiments. The speech separation performance was tested on an experimental dataset using this method, and the performance of this method was compared with that of previous methods.

[0068] 1) Experimental settings

[0069] The experimental dataset is composed of two - speaker speech (WSJ0 - 2mix) and three - speaker speech (WSJ0 - 3mix) mixed based on the Wall Street Journal speech data (WSJ0). Among them, each type of mixed speech contains approximately 30 hours of training data, 10 hours of validation data, and 5 hours of test data, and the signal - to - noise ratio ranges from 0 dB to 10 dB. The sampling rate of the input speech is resampled from the original 16 kHz to 8 kHz. The frame length for extracting short - time Fourier transform (STFT) features is 32 ms, the frame shift is 16 ms, the window function is a Hamming window, and an amplitude spectrum with a frequency dimension of 129 is further obtained as the input feature. Additionally, the fundamental frequency is extracted from the speech of a single speaker using the Praat tool to obtain the fundamental frequency trajectory clue. The length of the fundamental frequency sequence, i.e., the number of frames, is the same as the number of frames of the input mixed - speech amplitude spectrum. Each frame has a specific fundamental frequency value, and using the critical - band criterion, this fundamental frequency value is assigned to the corresponding frequency band among 129 frequency bands, i.e., the frequency band corresponding to this fundamental frequency value will be set to 1, and the rest of the values are 0. Finally, a two - dimensional representation composed of 0s and 1s with the same dimension as the amplitude spectrum is obtained. The relative improvement value (SDRi) relative to the original speech, the objective speech quality index (PESQ), and the short - time objective intelligibility (STOI) are used as evaluation indicators for the speech separation task.

[0070] 2) Experimental results

[0071] The experimental results are shown in Table 1. The method (Pitch-cGAN) proposed in the present invention can directly output a time-domain signal by using an upsampling module in the generator, and can be compared with the current state-of-the-art method (TasNet series models) based on the time domain. At the same time, the performance of the present invention is stronger than that of the representative methods (uPIT and DPCL) that reconstruct the time-domain signal using the phase of the mixed speech. The possible reason is that this type of frequency-domain method represented by uPIT and DPCL usually uses a mask as the estimation target, and the training process aims to minimize the mean square error at the sample level (time-frequency points) between the estimation result and the true label. This training objective does not guarantee that it is suitable for all objective metrics, that is, there is still a difference between the training objective (loss function) and the evaluation metric. The present invention avoids the step of selecting a suitable loss function, but uses a discriminator to determine whether the output of the generator is close enough to the true label through a deep neural network. The results prove that the method of the present invention has been improved in various metrics, and it can be considered that the network implicitly learns the corresponding phase information.

[0072] Table 1. Comparison of the present invention with other methods in terms of speech separation performance

[0073]

[0074] Under the condition of two-speaker mixed speech, the present method is still inferior to the current state-of-the-art method (Conv-TasNet) in terms of the SDRi metric, probably because the latter directly optimizes using this type of metric as the loss function. However, in terms of the two objective metrics of sound quality and intelligibility, the present method is superior to Conv-TasNet, which is considered to be the advantage brought by using a discriminator to replace the artificially defined loss function. Under the condition of three-speaker mixed speech, the present method is superior to Conv-TasNet in all metrics, which benefits from the fact that the present method is independent of the number of speakers in the input mixed speech, but uses the given condition to output the separated speech of only one speaker each time.

[0075] From the result analysis, the following advantages of the method proposed in the present invention can be obtained. First, the discriminator is used to replace the role of the loss function, avoiding the problem that the artificially designed loss function is inconsistent with the average metric. Because the evaluation metric involves all aspects of the separated signal, including signal-to-noise ratio, intelligibility, and sound quality, it is difficult to define it with a simple weighted sum loss function, while the generative adversarial network does not require an artificially defined loss function. Second, the fundamental frequency is selected as the condition, rather than the common speaker features. The former retains the local changes of speech in time, while the latter is a global feature and loses the characteristics of time and frequency dimensions. Theoretically, for speakers with similar timbres or speakers not seen in training, the results obtained by the method using speaker features should be inferior to those obtained by using fundamental frequency information.

[0076] Based on the same inventive concept, another embodiment of the present invention provides a generative voice separation device for introducing fundamental frequency cues, which includes:

[0077] A training unit for training generator G1, generator G2, and discriminator. The training process includes: for a given mixed voice, performing frame division, windowing, and short-time Fourier transform operations in sequence to obtain the short-time Fourier transform spectrum of the mixed voice; inputting the obtained spectrum and the fundamental frequency trajectory sequence of the target speaker into generator G1, and generating the voice spectrum of the target speaker through the fundamental frequency condition constraint generator; inputting the obtained voice spectrum of the target speaker into generator G2, where the input spectrum passes through a one-dimensional transposed convolution module, an upsampling module, and a residual module, and then through a one-dimensional convolution and activation function layer to output a representation that satisfies the numerical range of the time-domain signal, that is, the voice of the target speaker; inputting the generated voice of the target speaker and the fundamental frequency trajectory sequence of the target speaker into the discriminator, and the discriminator determines whether the generated voice is the voice of the real target speaker. Based on this result, the optimization objective function values of generators G1 and G2 are calculated, and based on this, the parameters of generators G1 and G2 are adjusted. At the same time, the optimization objective function value of the discriminator is calculated according to the true label of the training data, and based on this, the parameters of the discriminator are adjusted; the above steps are cyclically executed until the loss function values of generators G1, G2, and the discriminator tend to be stable, that is, reach the convergence state. At this time, the parameter settings of the generators and the discriminator are fixed;

[0078] A voice separation unit for separating the voice of the target speaker from the mixed voice to be separated by using generators G1 and G2 in the convergence state.

[0079] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (such as a computer, server, smart phone, etc.), which includes a memory and a processor. The memory stores a computer program, and the computer program is configured to be executed by the processor. The computer program includes instructions for executing each step in the method of the present invention.

[0080] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disc), and the computer-readable storage medium stores a computer program. When the computer program is executed by the computer, each step of the method of the present invention is implemented.

[0081] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and implement it accordingly. Those of ordinary skill in the art can understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification, and the protection scope of the present invention is subject to the scope defined by the claims.

Claims

1. A generative voice separation method for introducing fundamental frequency cues, the steps of which include: 1) For the given mixed speech, perform frame division, windowing, and short-time Fourier transform operations in sequence to obtain the short-time Fourier transform spectrum of the mixed speech; 2) Input the spectrum obtained in 1) and the fundamental frequency trajectory sequence of the target speaker into the generator G1. The generator G1 generates the speech spectrum of the target speaker through fundamental frequency conditional constraints; 3) Input the speech spectrum of the target speaker obtained in 2) into the generator G2. The input spectrum passes through a one-dimensional transposed convolution module, an upsampling module, and a residual module, and then through a one-dimensional convolution and an activation function layer to output a representation that satisfies the numerical range of the time-domain signal, that is, the speech of the target speaker; 4) Input the speech of the target speaker generated in 3) and the fundamental frequency trajectory sequence of the target speaker into the discriminator. The discriminator determines whether the generated speech is the real speech of the target speaker, calculates the optimization objective function values of the generators G1 and G2 based on this result, and adjusts the parameters of the generators G1 and G2 based on this. At the same time, calculate the optimization objective function value of the discriminator based on the true label of the training data, and adjust the discriminator parameters based on this; 5) During training, loop through steps 1) to 4) until the loss function values of the generators G1 and G2 and the discriminator tend to be stable, that is, reach the convergence state. At this time, fix the parameter settings of the generators and the discriminator; 6) For the mixed speech to be separated, using the parameter settings of the generators G1 and G2 in the convergence state as the standard, perform steps 1) to 3). The speech output in step 3) is the finally separated speech of the target speaker.

2. The method according to claim 1, wherein The input of the generator G1 is the mixed speech and the fundamental frequency condition, and this fundamental frequency condition constrains the generator G1 to generate a representation of the corresponding speaker's speech; the generator G1 first uses three layers of two-dimensional convolution with a convolution kernel size of 3 and a stride of 2. Each layer adopts a residual network structure with a residual connection, followed by 4 Transformer modules. The input feature dimension of each layer is 512, where the self-attention part uses 8 heads and the dimension of the feed-forward layer is 1024; a linear layer is added between the convolution module and the Transformer module to make the output dimension of the former match the input dimension of the latter.

3. The method according to claim 1, characterized in that The generator G2 upsamples the output amplitude spectrum of G1 to the dimension of the corresponding time-domain signal using a set of stacked transposed convolution modules and one-dimensional convolution. The upsampling multiple is determined by the frame shift of the input amplitude spectrum; a residual module is connected after each transposed convolution module. The residual module consists of three layers of one-dimensional convolution with dilation. By setting different coefficients (1, 3, 9), a receptive field of size 27 is obtained to expand the ability to sense the signal in the time dimension and better model the long-range dependencies in the time dimension; the generator G2 finally uses a layer of one-dimensional convolution and the tanh activation function to output a representation that satisfies the numerical range of the time-domain signal. The output channel of this one-dimensional convolution is set to 1, corresponding to the time-domain signal of the full frequency band.

4. The method according to claim 1, wherein The discriminator adopts a Transformer structure and performs discrimination at the block level. That is, the input time-frequency representation is regarded as a two-dimensional graph with two channels. The first channel represents the amplitude spectrum of the true or false target speech, and the second channel corresponds to the concatenated conditional fundamental frequency. The input is decomposed into several blocks and unfolded into a one-dimensional sequence through a linear layer. A learnable position encoding and a classification symbol [cls] are added at the beginning of the sequence, and then it is input into a 3-layer Transformer module. Finally, the classification symbol [cls] is used to give the true / false discrimination result.

5. The method according to claim 1, characterized in that, Using the mixed speech sequence and the fundamental frequency trajectory sequence of the target speaker as input conditions, a conditional generative adversarial network is utilized to generate the speech of the target speaker corresponding to the fundamental frequency. The conditional adversarial generative network consists of a generator and a discriminator. The generator aims to generate a time-domain signal corresponding to the fundamental frequency condition from the amplitude spectrum of the mixed speech. The discriminator is composed of multiple sub-discriminators acting on different frequency bands of the speech signal. Their optimization objectives are respectively: Discriminator: Generator: Among them, G is a generator, which consists of G1 and G2, and D k is the k-th discriminator, x is the magnitude spectrum of the input mixed speech, y is the given fundamental frequency condition, and s is the speaker time-domain waveform corresponding to the fundamental frequency; represents the mathematical expectation of the formula (D k (s|y) - 1) 2 under the condition that the true time-domain waveform signal is s; D k (s|y) represents the output value of the k-th discriminator given y and s; represents the mathematical expectation of the formula (D k (G(x|y))) 2 under the condition of x; D k (G(x|y)) represents the output result of the k-th discriminator for the time-domain signal G(x|y) generated by the generator; K represents the number of discriminators.

6. The method according to claim 5, characterized in that, Based on the loss function of the conditional generative adversarial network, a multi-resolution STFT loss is introduced. The multi-resolution STFT loss is composed of M individual STFT losses. Each STFT loss corresponds to different FFT sizes, window lengths, and frame shift parameters. The M individual STFT losses are averaged to obtain the multi-resolution STFT loss function.

7. A generative voice separation device for introducing fundamental frequency cues, characterized in that, Including: A training unit for training the generator G1, generator G2, and discriminator. The training process includes: for a given mixed speech, perform frame division, windowing, and short-time Fourier transform operations in sequence to obtain the short-time Fourier transform spectrum of the mixed speech; input the obtained spectrum and the fundamental frequency trajectory sequence of the target speaker into the generator G1. The generator G1 is constrained by the fundamental frequency condition to generate the speech spectrum of the target speaker; input the obtained speech spectrum of the target speaker into the generator G2. The input spectrum passes through a one-dimensional transposed convolution module, an upsampling module, and a residual module, and then through a one-dimensional convolution and activation function layer to output a representation that satisfies the numerical range of the time-domain signal, that is, the speech of the target speaker; input the generated speech of the target speaker and the fundamental frequency trajectory sequence of the target speaker into the discriminator. The discriminator determines whether the generated speech is the real speech of the target speaker, calculates the optimization objective function values of the generators G1 and G2 based on this result, and adjusts the parameters of the generators G1 and G2 accordingly. At the same time, calculate the optimization objective function value of the discriminator based on the true label of the training data and adjust the discriminator parameters accordingly; repeat the above steps until the loss function values of the generators G1, G2, and the discriminator tend to be stable, that is, reach the convergence state. At this time, fix the parameter settings of the generator and the discriminator. A speech separation unit for using the generators G1 and G2 in the convergence state to obtain the separated speech of the target speaker for the mixed speech to be separated.

8. An electronic device, characterized in that, Comprising a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 6 is implemented.

Images