Speech separation method and device based on multi-channel full convolution time domain network

Abstract

The application discloses a speech separation method and device based on a multi-channel full convolution time domain network, comprising the following steps: (1) obtaining a plurality of noisy mixed multi-channel speech signals containing different sound sources, noise and reverberation, and taking corresponding pure single-channel speech signals as labels to form a training data set; (2) establishing a multi-channel full convolution time domain network, wherein the multi-channel full convolution time domain network comprises an encoder, a separator, a point multiplication module and a decoder, and the parameters of the encoder are fixed Gammatone filter coefficients; (3) inputting the training data set into the multi-channel full convolution time domain network for training; and (4) inputting a noisy mixed multi-channel speech signal to be separated into the multi-channel full convolution time domain network to obtain a pure single-channel speech signal after sound source separation. The application has better separation effect.

Description

Technical Field

[0001] This invention relates to speech separation technology, and more particularly to a speech separation method and device based on a multi-channel fully convolutional temporal network. Background Technology

[0002] Speech separation (SS) was originally a signal processing problem, but with the rapid development of neural networks and the reduction of computational costs, data-driven speech separation algorithms have developed rapidly. Traditional machine learning methods are severely limited in their ability to process large amounts of data and complex environments, while machine learning algorithms based on deep neural networks have a stronger modeling ability for handling complex nonlinear problems and are better able to extract valuable information from large datasets.

[0003] Multichannel speech separation based on deep learning can be broadly classified into two categories. The first category combines the spatial features of multichannel speech with the time-frequency features of a single channel, using these two types of features as input to a neural network to train and obtain the target sound source. The second category is deep learning-based beamforming. This type of algorithm implicitly incorporates the beamformer into the neural network without explicitly generating the beamforming filter. In this framework, the multichannel speech signal is directly input into the neural network, and the training objective is to learn the mapping between the multichannel input and the target speech. Therefore, the beamformer is implicitly included in the mapping function defined by the model.

[0004] Most of the training objectives based on masks and mappings mentioned above require a short-time Fourier transform (SFT) of the speech signal. However, this is not necessarily the optimal signal transformation method for speech separation tasks. The SFT causes decoupling between signal amplitude and phase, increasing the difficulty of phase estimation. In recent years, end-to-end speech separation algorithms, which have become a research hotspot, perform speech separation directly in the time domain, avoiding the extraction of time-domain feature parameters and phase estimation in the frequency domain. These methods provide a unique perspective for reconstructing the phase of the target signal. However, in environments with high reverberation and low signal-to-noise ratio (SNR), the SNR and speech quality intelligibility of these methods decrease significantly. Summary of the Invention

[0005] Purpose of the invention: This invention addresses the problems existing in the prior art by providing a speech separation method and device based on a multi-channel fully convolutional temporal network that achieves excellent speech separation performance even in environments with high reverberation and low signal-to-noise ratio.

[0006] Technical solution: The speech separation method based on a multi-channel fully convolutional temporal network described in this invention includes the following steps:

[0007] (1) Obtain several noisy mixed multi-channel speech signals containing different sound sources, noise and reverberation, and use the corresponding clean single-channel speech signals as labels to form a training dataset;

[0008] (2) Establish a multi-channel fully convolutional temporal network, which includes an encoder, a separator, a dot product module and a decoder. The encoder has fixed Gammatone filter coefficients and is used to extract intermediate audio expression features from noisy mixed multi-channel speech signals. The separator is used to calculate a mask matrix based on the intermediate audio expression features. The dot product module is used to multiply the intermediate audio expression features with the mask matrix to obtain mask features. The decoder is used to decode the mask features to obtain the separated clean single-channel speech signal.

[0009] (3) Input the training dataset into the multi-channel fully convolutional temporal network for training;

[0010] (4) Input the noisy mixed multi-channel speech signal to be separated into a multi-channel fully convolutional time-domain network to obtain a clean single-channel speech signal after source separation.

[0011] Furthermore, step (1) specifically includes:

[0012] (1.1) Acquire several clean single-channel speech signals;

[0013] (1.2) For each pure single-channel speech signal, the impulse response of a single sound source in a specified direction is generated using the mirror method, and convolved with the pure single-channel speech signal to obtain a single-source multi-channel speech signal in a specified direction. The single-source multi-channel speech signals in different directions are added together to obtain a mixed multi-channel speech signal containing multiple sound sources.

[0014] (1.3) For each hybrid multi-channel speech signal, noise with different signal-to-noise ratios and different types of reverberation are added to obtain several noisy hybrid multi-channel speech signals;

[0015] (1.4) Use all noisy mixed multi-channel speech signals as the training dataset, where the clean single-channel speech signal corresponding to each noisy mixed multi-channel speech signal is its label.

[0016] Furthermore, the parameters of the encoder are obtained through the following method:

[0017] A. Extract N frequency points from the preset frequency range as the center frequency f. i , i = 1, ..., N / 2, where N is an even number greater than 2;

[0018] B. For each center frequency f i Calculate the corresponding L phases j = 1, ..., L, where L phases are uniformly distributed on [0, π), and L is an integer greater than 2;

[0019] C. Based on the center frequency f i The time-domain impulse responses of (N / 2)×L filters are calculated based on the corresponding phases:

[0020]

[0021] In the formula, h i,j (t) represents the center frequency f i Phase is The time-domain impulse response of the filter, where A is the amplitude, n is the filter order, and b i It is the attenuation factor;

[0022] D. Take the inverse of the time-domain impulse response of the (N / 2)×L filters obtained in step C, and thus obtain the time-domain impulse response of the remaining (N / 2)×L filters;

[0023] E. Form an N×L matrix from the time-domain impulse responses of all N×L filters, and use it as the parameter matrix of the encoder.

[0024] Furthermore, the separator includes a first normalization layer, a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, a temporal convolutional network module, a first nonlinear function PReLU, a third two-dimensional convolutional layer, a fourth two-dimensional convolutional layer, and a sigmoid function connected in sequence.

[0025] Furthermore, the temporal convolutional network module includes several sequentially connected two-dimensional dilated convolutional blocks and a splicing module. Each two-dimensional dilated convolutional block is also connected to the splicing module in a skip manner. The splicing module splices the outputs of all two-dimensional dilated convolutional blocks and outputs them as the temporal convolutional network module.

[0026] Furthermore, the two-dimensional dilated convolutional block includes a fifth two-dimensional convolutional layer, a sixth two-dimensional convolutional layer, a second nonlinear function PReLU, a second normalized layer, a two-dimensional depthwise convolutional layer, a third nonlinear function PReLU, a third normalized layer, a seventh two-dimensional convolutional layer, an eighth two-dimensional convolutional layer, a ninth two-dimensional convolutional layer, a tenth two-dimensional convolutional layer, and a splicing unit. The fifth two-dimensional convolutional layer, the sixth two-dimensional convolutional layer, the second nonlinear function PReLU, the second normalized layer, the two-dimensional depthwise convolutional layer, the third nonlinear function PReLU, and the third normalized layer are connected sequentially. The seventh two-dimensional convolutional layer and the ninth two-dimensional convolutional layer are respectively connected to the output of the third normalized layer. The eighth two-dimensional convolutional layer and the tenth two-dimensional convolutional layer are respectively connected to the outputs of the seventh two-dimensional convolutional layer and the ninth two-dimensional convolutional layer. The eighth two-dimensional convolutional layer skips to the splicing module. The splicing unit splices the output of the tenth two-dimensional convolutional layer and the output of the sixth two-dimensional convolutional layer and connects them to the next connected two-dimensional dilated convolutional block. All two-dimensional convolutional layers are 1×1 two-dimensional convolutional layers.

[0027] Furthermore, the decoder includes a first one-dimensional transposed convolutional layer, an activation function, and a second one-dimensional transposed convolutional layer connected in sequence.

[0028] Furthermore, the loss function used during training of the multi-channel fully convolutional temporal network is:

[0029]

[0030]

[0031] Where L represents the loss function, C is the number of sound sources, perm(C) is all possible permutations of C sound sources, J is one of these permutations, jc is the permutation index, and s jc This represents the pure single-channel speech signal of the jc-th sound source when arranged in J. When the noisy mixed multi-channel speech signal in the training dataset is taken as input, the j-th output of the multi-channel fully convolutional temporal network is... c The single-channel speech signal is separated from each sound source. L1(·) is the error function based on the L1 norm, ‖‖ is the L1 norm, SISNR(·) is the scale-invariant signal-to-noise ratio function, and β and α are weighting parameters.

[0032] The speech separation device based on a multi-channel fully convolutional temporal network according to the present invention includes a processor and an executable program stored in a memory and capable of running on the processor. When the processor executes the executable program, it implements the above-mentioned method.

[0033] Compared with existing technologies, the advantages of this invention are as follows: This invention combines the encoder and Gammatone filter bank, reducing network parameters and eliminating the need for encoder parameters to be obtained through training, thus reducing computational load and shortening the computation time required by the model. This invention applies end-to-end speech separation technology to multi-channel systems, improving the signal-to-noise ratio and speech quality intelligibility. Experimental results under different acoustic environments show that this invention significantly improves separation performance in high reverberation and low signal-to-noise ratio environments, exhibits better generalization performance, and demonstrates excellent robustness. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the speech separation method based on a multi-channel fully convolutional temporal network provided by the present invention.

[0035] Figure 2 This is a schematic diagram of the structure of the multi-channel fully convolutional temporal network of the present invention;

[0036] Figure 3 This is a schematic diagram of the separator of the present invention;

[0037] Figure 4 This is a schematic diagram of the structure of the two-dimensional dilated convolution block of the present invention;

[0038] Figure 5 This is a schematic diagram of the decoder structure;

[0039] Figure 6 This is a schematic diagram of the speech separation device based on a multi-channel fully convolutional temporal network provided by the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Example 1

[0042] This embodiment provides a speech separation method based on a multi-channel fully convolutional temporal network, such as... Figure 1 As shown, it includes the following steps:

[0043] (1) Obtain several noisy mixed multi-channel speech signals containing different sound sources, noise and reverberation, and use the corresponding clean single-channel speech signals as labels to form a training dataset.

[0044] This step specifically includes:

[0045] (1.1) Obtain several clean single-channel speech signals; for example, clean single-channel speech signals can be randomly extracted from the TIMIT (The DARPATIMIT Acoustic-Phonetic Continuous Speech Corpus) single-channel speech library;

[0046] (1.2) For each pure single-channel speech signal, the impulse response of a single sound source in a specified direction is generated using the mirror method, and convolved with the pure single-channel speech signal to obtain a single-source multi-channel speech signal in a specified direction. The single-source multi-channel speech signals in different directions are added together to obtain a mixed multi-channel speech signal containing multiple sound sources.

[0047] (1.3) For each hybrid multichannel speech signal, noise with different signal-to-noise ratios and different types of reverberation are added to obtain several noisy hybrid multichannel speech signals; for example, noise with a signal-to-noise ratio of 10dB and 20dB, and reverberation of 0ms (no reverberation), 200ms or 600ms can be added to the hybrid multichannel speech signal to form noisy hybrid multichannel speech signals under 9 acoustic environments;

[0048] (1.4) Use all noisy mixed multi-channel speech signals as the training dataset, where the clean single-channel speech signal corresponding to each noisy mixed multi-channel speech signal is its label.

[0049] (2) Establish a multi-channel fully convolutional temporal network.

[0050] Among them, such as Figure 2 As shown, the multi-channel fully convolutional temporal network includes an encoder, a separator, a dot product module, and a decoder. The encoder has fixed Gammatone filter coefficients and is used to extract intermediate audio representation features from noisy mixed multi-channel speech signals. The separator is used to calculate a mask matrix based on the intermediate audio representation features. The dot product module is used to multiply the intermediate audio representation features by the mask matrix to obtain mask features. The decoder is used to decode the mask features to obtain the separated clean single-channel speech signal. In fact, the network structure proposed in this invention is not limited to the network structure of this embodiment; minor modifications also fall within the scope of multi-channel fully convolutional temporal networks.

[0051] The encoder functions similarly to a short-time Fourier transform, extracting the intermediate audio features of the signal. The characteristics of the Gammatone filter bank allow the encoder to focus more on the low- and mid-frequency components of speech, aligning with human auditory characteristics. The coefficients of the Gammatone filter are used as the coefficients of the fully convolutional time-domain network encoder, resulting in an encoder with fixed parameters. The specific process is as follows:

[0052] A. Extract N frequency points from a preset frequency range (e.g., [50Hz, 4kHz]) as the center frequency f. i i = 1, ..., N / 2, where N is an even number greater than 2, for example, it could be 26;

[0053] B. For each center frequency f i Calculate the corresponding L phases j = 1, ..., L, where L phases are uniformly distributed on [0, π), and L is an integer greater than 2, for example, 26;

[0054] C. Based on the center frequency f i The time-domain impulse responses of (N / 2)×L filters are calculated based on the corresponding phases:

[0055]

[0056] In the formula, h i,j (t) represents the center frequency f i Phase is The time-domain impulse response of the filter, where A is the amplitude, n is the filter order, and b i It is the attenuation factor;

[0057] D. Take the inverse of the time-domain impulse response of the (N / 2)×L filters obtained in step C, and thus obtain the time-domain impulse response of the remaining (N / 2)×L filters;

[0058] E. Form an N×L matrix from the time-domain impulse responses of all N×L filters, and use it as the parameter matrix of the encoder.

[0059] like Figure 3 As shown, the separator comprises a first normalization layer (LayerNorm, LN), a first two-dimensional convolutional layer (1×1 convolution), a second two-dimensional convolutional layer (1×1 convolution), a temporal convolutional network module (TCN), a first nonlinear function (PReLU), a third two-dimensional convolutional layer (1×1 convolution), a fourth two-dimensional convolutional layer (1×1 convolution), and a sigmoid function, all connected in sequence. The first two-dimensional convolutional layer compresses the feature dimension of the input, while the second two-dimensional convolutional layer expands the channel dimension. The third two-dimensional convolutional layer compresses the channel dimension to 1, and the fourth two-dimensional convolutional layer restores the feature dimension to match the input. The sigmoid function restricts the mask values ​​to the range [0,1], forming a mask matrix. The temporal convolutional network module comprises several sequentially connected two-dimensional dilated convolutional blocks and a concatenation module. Each two-dimensional dilated convolutional block is also connected to the concatenation module. The concatenation module concatenates the outputs of all two-dimensional dilated convolutional blocks as the output of the temporal convolutional network module, which is used to mine information from the feature dimension and channel dimension.

[0060] like Figure 4 As shown, the two-dimensional dilated convolutional block includes a fifth two-dimensional convolutional layer (1×1 convolution), a sixth two-dimensional convolutional layer (1×1 convolution), a second nonlinear function PReLU, a second normalized layer, a two-dimensional depthwise convolutional layer, a third nonlinear function PReLU, a third normalized layer, a seventh two-dimensional convolutional layer (1×1 convolution), an eighth two-dimensional convolutional layer (1×1 convolution), a ninth two-dimensional convolutional layer (1×1 convolution), a tenth two-dimensional convolutional layer (1×1 convolution), and a splicing unit. The fifth two-dimensional convolutional layer, the sixth two-dimensional convolutional layer, and the second nonlinear function PReLU... The first two layers are connected in sequence: the second normalized layer, the second two-dimensional deep convolutional layer, the third nonlinear function PReLU, and the third normalized layer. The seventh and ninth two-dimensional convolutional layers are respectively connected to the output of the third normalized layer. The eighth and tenth two-dimensional convolutional layers are respectively connected to the outputs of the seventh and ninth two-dimensional convolutional layers. The eighth two-dimensional convolutional layer is connected to the splicing module. The splicing unit splices the output of the tenth two-dimensional convolutional layer and the output of the sixth two-dimensional convolutional layer and connects them to the next connected two-dimensional dilated convolutional block. The fifth 2D convolutional layer expands the channel dimension of the input, enabling the network to provide more channel-dimensional information to subsequent deep convolutions. The sixth 2D convolutional layer expands the feature dimension to extract more details. The 2D deep convolutional layer uses zero padding to ensure that the dimensions of the input and output feature maps are consistent. The output of the deep convolution is divided into two paths after passing through the PReLU nonlinear function and normalization: a skip connection path and a residual path. The skip connection path is restored to the same channel and feature dimensions as the input through the seventh and eighth 2D convolutional layers. The residual path is restored to the same channel and feature dimensions as the input through the ninth and tenth 2D convolutional layers.

[0061] like Figure 5 As shown, the decoder includes a first one-dimensional transposed convolutional layer, an activation function, and a second one-dimensional transposed convolutional layer connected in sequence. The first one-dimensional transposed convolutional layer has a kernel size of 3 and a stride of 1, while the second one-dimensional transposed convolutional layer has a kernel size of 256 and a stride of 128, and uses ReLU as the activation function. The decoder decodes the mask features to obtain the separated, clean single-channel speech signal.

[0062] (3) Input the training dataset into the multi-channel fully convolutional temporal network for training.

[0063] During training, the loss function used, based on the forward propagation and back propagation algorithms, is:

[0064]

[0065]

[0066] Where L represents the loss function, C is the number of sound sources, perm(C) is all possible permutations of C sound sources, J is one of these permutations, jc is the permutation index, and s jc This represents the pure single-channel speech signal of the jc-th sound source when arranged in J. This represents the j-th output of a multi-channel fully convolutional temporal network when noisy mixed multi-channel speech signals from the training dataset are used as input. c The single-channel speech signal is separated from each sound source. L1(·) is the error function based on the L1 norm, ‖‖ is the L1 norm, SISNR(·) is the scale-invariant signal-to-noise ratio function, and β and α are weighting parameters.

[0067] SISNR(·) is the expression for calculating SI-SNR:

[0068]

[0069]

[0070]

[0071] (4) Input the noisy mixed multi-channel speech signal to be separated into a multi-channel fully convolutional time-domain network to obtain a clean single-channel speech signal after source separation.

[0072] To verify the effectiveness of the present invention, simulation verification was performed. The evaluation metrics used included source-to-distortion ratio (SDR), source-to-interference ratio (SIR), and SI-SNR.

[0073] Tables 1, 2, and 3 compare the SDR, SIR, and SI-SNR metrics of various algorithms under different acoustic environments. Simulation verification of the above methods yields the following final performance evaluation:

[0074] During the training phase, nine acoustic environments were simulated, including no noise, two signal-to-noise ratios (SNR) of 10dB and 20dB, and three reverberation durations of 0ms (no reverberation), 200ms, and 600ms. During the testing phase, speaker speech was randomly sampled from TIMIT. In addition to the reverberation durations and SNRs mentioned in the training phase, 800ms reverberation time and SNRs of 0dB, 3dB, 5dB, 7dB, 9dB, and 15dB were also tested. The quality was evaluated using SDR, SIR, and SI-SNR metrics, compared to the original mixed speech and a frequency-domain parameter-based multi-channel separation algorithm.

[0075] Table 1 Comparison of SDR values ​​before and after separation under various conditions

[0076]

[0077]

[0078] Table 2 Comparison of SIR values ​​before and after separation under various environments.

[0079]

[0080] Table 3 Comparison of SI-SNR values ​​before and after separation under various environments.

[0081]

[0082] As shown in Tables 1, 2, and 3, under reverberation-free and low-noise conditions, the SDR performance of the algorithm proposed in this invention is slightly lower than that of the multi-channel separation algorithm based on frequency domain parameters. Under high reverberation and high-noise conditions, the algorithm proposed in this invention exhibits the best performance. The SIR performance of the algorithm proposed in this invention is generally superior to that of the multi-channel separation algorithm based on frequency domain parameters. The SI-SNR performance of the algorithm proposed in this invention is higher than that of the multi-channel separation algorithm based on frequency domain parameters, and the advantages of the algorithm proposed in this invention are more pronounced under low signal-to-noise ratio and high reverberation conditions.

[0083] To verify the generalization ability of the algorithm of this invention, keeping other conditions unchanged, the reverberation time RT60 in the test phase was set to 800ms, which is higher than the maximum reverberation time in the training phase. The signal-to-noise ratio (SNR) in the test phase was set to an SNR not present in the training phase, and SDR, SIR, SI-SNR, STOI, and PESQ were analyzed. The Perceptual Evaluation of Speech Quality (PESQ) score, based on the ITU-T P.862 standard, is an objective speech quality assessment method that uses the original signal as a reference to measure the quality of degraded signals, returning a score in the range of -0.5 to 4.5. The Short Time Objective Intelligibility (STOI) score is an objective evaluation method for measuring the intelligibility of speech by the human auditory perception system, with a score between 0 and 1, using the original signal as a reference to measure the intelligibility of degraded signals. Table 4 shows the speech separation performance indicators of the algorithm of this invention at different SNRs with a reverberation time RT60 of 800ms.

[0084] Even in environments with higher signal-to-noise ratios and longer reverberation times that were not present during training, the algorithm proposed in this invention still performs well in terms of objective metrics and intelligibility metrics.

[0085] Table 4. Separation performance indicators of the algorithm of this invention under different signal-to-noise ratios with 800ms reverberation.

[0086] 0 12.85 24.68 11.33 0.771 2.305 3 13.46 25.86 11.92 0.786 2.360 5 13.83 26.52 12.22 0.794 2.391 7 14.12 27.02 12.48 0.799 2.414 9 14.33 27.42 12.73 0.803 2.433 10 14.40 27.60 12.84 0.804 2.442 15 14.65 28.16 13.16 0.809 2.468 20 14.71 28.34 13.26 0.811 2.476 No noise 14.88 28.71 13.35 0.818 2.493

[0087] Example 2

[0088] This embodiment provides a speech separation device based on a multi-channel fully convolutional temporal network, and provides services for implementing the method of Embodiment 1 above. For example... Figure 6 As shown, the device may include: a memory 301 storing a computer-executable program; a processor 302 coupled to the memory 301; the processor 302 calls the computer-executable program stored in the memory 301 to perform the steps in the method described in Embodiment 1.

[0089] Memory 301 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, memory 301 may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as a "hard disk drive"). A program / utility having a set (at least one) of program modules may be stored, for example, in memory 301. Such program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The computer-executable program of the program modules typically performs the functions and / or methods described in the embodiments of the present invention.

[0090] The code for performing the operations of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages.

[0091] The processor 302 executes various functional applications and data processing by running programs stored in the memory 301, such as implementing the method provided in Embodiment 1 of the present invention.

[0092] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. The reference to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

Claims

1. A speech separation method based on a multi-channel fully convolutional temporal network, characterized in that, Includes the following steps: (1) Obtain several noisy mixed multi-channel speech signals containing different sound sources, noise and reverberation, and use the corresponding clean single-channel speech signals as labels to form a training dataset; (2) Establish a multi-channel fully convolutional temporal network, which includes an encoder, a separator, a dot product module and a decoder. The encoder has fixed Gammatone filter coefficients and is used to extract intermediate audio expression features from noisy mixed multi-channel speech signals. The separator is used to calculate a mask matrix based on the intermediate audio expression features. The dot product module is used to multiply the intermediate audio expression features with the mask matrix to obtain mask features. The decoder is used to decode the mask features to obtain a clean single-channel speech signal after separation. (3) Input the training dataset into the multi-channel fully convolutional temporal network for training; (4) Input the noisy mixed multi-channel speech signal to be separated into a multi-channel fully convolutional time-domain network to obtain a clean single-channel speech signal after sound source separation; The parameters of the encoder are obtained through the following method: A. Extract N frequency points from the preset frequency range as the center frequency f. i , i=1,…,N / 2, N is an even number greater than 2; B. For each center frequency f i Calculate the corresponding L phases , j=1,…,L, L phases are uniformly distributed on [0, π), and L is an integer greater than 2; C. Based on the center frequency f i The time-domain impulse responses of (N / 2)×L filters are calculated based on the corresponding phases: , In the formula, The center frequency is f i Phase is The time-domain impulse response of the filter, where A is the amplitude. Let the filter order be . It is the attenuation factor; D. Take the inverse of the time-domain impulse response of the (N / 2)×L filters obtained in step C, and thus obtain the time-domain impulse response of the remaining (N / 2)×L filters; E. Form an N×L matrix from the time-domain impulse responses of all N×L filters, and use it as the parameter matrix of the encoder.

2. The speech separation method based on a multi-channel fully convolutional temporal network according to claim 1, characterized in that: Step (1) specifically includes: (1.1) Acquire several clean single-channel speech signals; (1.2) For each pure single-channel speech signal, the impulse response of a single sound source in a specified direction is generated using the mirror method, and convolved with the pure single-channel speech signal to obtain a single-source multi-channel speech signal in a specified direction. The single-source multi-channel speech signals in different directions are added together to obtain a mixed multi-channel speech signal containing multiple sound sources. (1.3) For each mixed multi-channel speech signal, noise with different signal-to-noise ratios and different types of reverberation are added to obtain several noisy mixed multi-channel speech signals; (1.4) Use all noisy mixed multi-channel speech signals as the training dataset, where the clean single-channel speech signal corresponding to each noisy mixed multi-channel speech signal is its label.

3. The speech separation method based on a multi-channel fully convolutional temporal network according to claim 1, characterized in that: The separator comprises a first normalization layer, a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, a temporal convolutional network module, a first nonlinear function PReLU, a third two-dimensional convolutional layer, a fourth two-dimensional convolutional layer, and a sigmoid function connected in sequence.

4. The speech separation method based on a multi-channel fully convolutional temporal network according to claim 3, characterized in that: The temporal convolutional network module includes several sequentially connected two-dimensional dilated convolutional blocks and a splicing module. Each two-dimensional dilated convolutional block is also connected to the splicing module in a skip manner. The splicing module splices the outputs of all two-dimensional dilated convolutional blocks and outputs them as the temporal convolutional network module.

5. The speech separation method based on a multi-channel fully convolutional temporal network according to claim 4, characterized in that: The two-dimensional dilated convolutional block includes a fifth two-dimensional convolutional layer, a sixth two-dimensional convolutional layer, a second nonlinear function PReLU, a second normalized layer, a two-dimensional depthwise convolutional layer, a third nonlinear function PReLU, a third normalized layer, a seventh two-dimensional convolutional layer, an eighth two-dimensional convolutional layer, a ninth two-dimensional convolutional layer, a tenth two-dimensional convolutional layer, and a splicing unit. The fifth two-dimensional convolutional layer, the sixth two-dimensional convolutional layer, the second nonlinear function PReLU, the second normalized layer, the two-dimensional depthwise convolutional layer, the third nonlinear function PReLU, and the third normalized layer are connected sequentially. The seventh two-dimensional convolutional layer and the ninth two-dimensional convolutional layer are respectively connected to the output of the third normalized layer. The eighth two-dimensional convolutional layer and the tenth two-dimensional convolutional layer are respectively connected to the output of the seventh two-dimensional convolutional layer and the ninth two-dimensional convolutional layer. The eighth two-dimensional convolutional layer skips to the splicing module. The splicing unit splices the output of the tenth two-dimensional convolutional layer and the output of the sixth two-dimensional convolutional layer and then connects them to the next connected two-dimensional dilated convolutional block.

6. The speech separation method based on a multi-channel fully convolutional temporal network according to claim 1, characterized in that: All two-dimensional convolutional layers are 1×1 two-dimensional convolutional layers.

7. The speech separation method based on a multi-channel fully convolutional temporal network according to claim 1, characterized in that: The decoder includes a first one-dimensional transposed convolutional layer, an activation function, and a second one-dimensional transposed convolutional layer connected in sequence.

8. The speech separation method based on a multi-channel fully convolutional temporal network according to claim 1, characterized in that: The loss function used during the training of the multi-channel fully convolutional temporal network is: , , in, Let J represent the loss function, C be the number of sound sources, perm(C) be all possible permutations of the C sound sources, J be one of these permutations, and jc be the permutation index. Indicates the arrangement as The clean single-channel speech time-domain signal from the jcth sound source. When the noisy mixed multi-channel speech signal in the training dataset is taken as input, the j-th output of the multi-channel fully convolutional temporal network is... c Separate single-channel speech signals from each sound source It is an error function based on the L1 norm. SISNR ( () is the scale-invariant signal-to-noise ratio function. These are the weight parameters.

9. A speech separation device based on a multi-channel fully convolutional temporal network, comprising a processor and an executable program stored in memory and capable of running on the processor, characterized in that: When the processor executes the executable program, it implements the method as described in any one of claims 1-8.

Images