Two-stage single-channel speech separation method based on strong constraint dictionary and deep neural network

Abstract

The application discloses a two-stage single-channel speech separation method based on a strong constraint dictionary and a deep neural network. First, the speech separation method based on dictionary learning often has a 'cross projection' problem, and the application proposes an optimization function with strong constraints, and a more distinctive joint dictionary is constructed under the constraints of the function to reduce the 'cross projection'. Second, in order to improve the separation effect of two similar signals, the application proposes a two-stage single-channel speech separation method, the first stage uses a strong constraint dictionary to realize speech separation and obtain a preliminary estimated signal. In the second stage, a deep neural network with strong mapping ability is used to realize the separation of speech and cross projection residuals through joint constraints, remove the influence of cross projection residuals, and obtain a fine estimated signal. The application improves the performance of the speech separation system, and the speech separated by the system is improved in five measurement indicators, and the application is suitable for intelligent human-computer interaction.

Description

Technical Field

[0001] This invention belongs to the field of speech separation technology, specifically relating to a two-stage single-channel speech separation method based on a strongly constrained dictionary and a deep neural network. Background Technology

[0002] Speech separation technology is increasingly used in everyday products, such as mobile phones, hearing aids, smart home control systems, and military walkie-talkies. With the rapid development of communication technology, people have higher demands for the voice quality of these products. How to extract the pure speech signal to the maximum extent possible from interfered speech or minimize the influence of interference signals is one of the key research areas in speech separation. Many scholars at home and abroad have studied the speech separation problem and proposed many different methods. Early traditional signal processing methods include spectral subtraction and Wiener filters, which are generally suitable for continuous, stable, high signal-to-noise ratio environments. Based on this, model-based methods have been proposed to construct a model based on the source signal and the mixing process, and to derive the estimated source signal mathematically. Methods based on Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) have achieved good results. In addition, the NMF method, by establishing a linear mapping from the mixed signal to the target signal, is also widely used in speech separation, but it cannot effectively represent complex signal structures.

[0003] Dictionary-based sparse signal representation methods are also commonly used to solve speech separation problems. Sparse signal representation uses as few overcomplete dictionary atoms as possible to represent the signal. Researchers mainly focus on dictionary construction methods and sparse signal decomposition algorithms. Aharon et al. first proposed the K-SVD algorithm, and some researchers have improved the separation performance by adding dictionary constraints. Sigg et al. trained a speech dictionary and an interference dictionary using clean speech signals and interference signals respectively, and then concatenated the two dictionaries into a joint dictionary for sparse coding, achieving good performance in separating clean speech from mixed speech. Zhang et al. studied the correlation between speech signals and the cross-interference between dictionaries, and proposed a speech enhancement method that jointly learns the speech dictionary and the interference dictionary, which can reduce source distortion and confusion, improve speech quality, and has a more significant enhancement effect when the input signal-to-noise ratio is low. Tian et al. proposed a source separation method based on a learned dictionary, merging common sub-dictionaries into a regular joint dictionary to ensure that the sub-dictionary of a specific source can capture the discriminative information of the corresponding source. They also designed a task-driven learning algorithm to optimize the proposed joint dictionary and the weights used to allocate common information. Experimental results show that this algorithm can achieve better separation performance than traditional algorithms.

[0004] In recent years, deep learning technology has become increasingly prominent in the field of speech separation due to its powerful learning capabilities. Deep learning-based speech separation learns the nonlinear mapping relationship between the mixed signal and the target signal through training. This method does not require a large amount of prior knowledge and still has good generalization ability in low signal-to-noise ratio or non-stationary signal environments. Professor Geoffrey Hinton optimized traditional neural network algorithms and was the first to propose the concept of deep neural networks. Han et al. proposed learning the amplitude spectrum of clean speech from the amplitude spectrum of damaged speech by training a DNN to achieve dereverberation and denoising. Different training objectives affect the separation performance of the network model during DNN training. Wang et al. analyzed and compared the enhancement effects of commonly used training objectives, including IBM and IRM, and found that masking techniques generally achieved good results. Li et al. proposed a DNN speech separation algorithm with a spectrum change-aware loss function, and the results showed that the proposed loss function can improve speech clarity and signal-to-noise ratio gain. Our team's proposed joint constraint algorithm not only penalizes the sum of squared residuals but also utilizes the joint relationship between outputs to train a dual-output DNN, achieving better performance compared to the basic loss function. Summary of the Invention

[0005] The technical problem this invention aims to solve is to overcome the shortcomings of existing technologies and provide a two-stage single-channel speech separation method based on a strongly constrained dictionary and a deep neural network. The first stage performs preliminary estimation, using a strongly constrained dictionary to learn and separate speech signals, obtaining a coarse estimate of the signal. The second stage performs refined estimation, using a DNN with strong mapping capabilities to enhance the signal separated and reconstructed in the first stage. Joint constraints are used to separate speech from the cross-projection residual, making the final estimated signal closer to the target signal, thereby improving the quality of the separated speech.

[0006] This invention provides a two-stage single-channel speech separation method based on a strongly constrained dictionary and a deep neural network, comprising the following steps:

[0007] Step S1. Extract the amplitude spectrum of the clean speech signal and the mixed speech signal, use the K-SVD algorithm to calculate the sub-dictionary D1 and D2 of the clean speech signal, concatenate the sub-dictionaries into a joint dictionary D = [D1, D2], and iteratively update the dictionary under the constraint of a strongly constrained optimization function.

[0008] Step S2. Project another set of training samples onto dictionary D to obtain the sparse coding matrix C = [C1, C2] of the mixed signal on D. T Reconstructing the amplitude spectrum of the speech signal separated in the first stage and

[0009] Step S3. and After obtaining the estimated ideal ratio masks M1 and M2 through a DNN, the masks are respectively compared with... and Obtain the amplitude spectrum of the second-stage fine estimation by performing the Hadamard product. and The speech signal is recovered by multiplying the amplitude spectrum and the phase of the mixed signal.

[0010] As a further technical solution of the present invention, the specific steps of step S1 are as follows:

[0011] Step S11. Sample each input time-domain continuous speech signal and then perform a short-time Fourier transform to obtain the preprocessed speech signal;

[0012] Step S12. Calculate the amplitude spectra S1 and S2 of the preprocessed speech signal, train and obtain the corresponding identity sub-dictionaries D1 and D2 based on the K-SVD algorithm, and concatenate D1 and D2 to obtain the initial joint dictionary D = [D1, D2];

[0013] Step S13. Fix D, and obtain the sparse coding matrix C of the mixed signal Y on D using the OMP algorithm under the constraint of the objective function;

[0014] Step S14. Fix C, use the L-BFGS algorithm to find the strong constraint optimization function to update the dictionary, and obtain the strongly constraint optimized dictionary.

[0015] Furthermore, in step S14, the specific method for updating the dictionary by obtaining the strongly constrained optimization function using the L-BFGS algorithm is as follows:

[0016] Step S141. Define the strongly constrained optimization function as follows:

[0017]

[0018] in, and Let S1 and S2 be the sparse coding matrices projected onto the joint dictionary D, respectively, representing the projections of individual clean signals S1 and S2. and for and The projection of the middle element onto its own sub-dictionary;

[0019] Step S142. Define the matrix and Where I is the identity matrix and O is a matrix of all zeros;

[0020] Step S143. The strongly constrained optimization function is rewritten as follows:

[0021]

[0022] The gradient function of the objective function is

[0023]

[0024] Step S144. After multiple iterations, an updated optimized dictionary is obtained, which enables the mixed signal to distinguish different source signals when projected onto the dictionary.

[0025] Furthermore, the specific steps of step S2 are as follows:

[0026] Step S21. Project the mixed signal onto the dictionary D to obtain the estimated sparse coding matrix.

[0027] Step S22. According to The reconstruction yields the amplitude spectrum of the speech signal separated in the first stage. and

[0028] Furthermore, the specific steps of step S3 are as follows:

[0029] Step S31. Construct a DNN network framework, which includes one input layer, three hidden layers, and one output layer;

[0030] Step S32. The ideal ratio mask of the target speech is used as the training objective of the DNN network, and a joint constraint loss function is defined to constrain the network;

[0031] Step S33. Analyze the amplitude spectrum of the speech signal separated in the first stage. and Input the DNN network and output the estimated ideal ratio masks M1 and M2;

[0032] Step S34. M1 and M2 respectively with and Obtain the amplitude spectrum of the second-stage fine estimation by performing the Hadamard product. and

[0033] Step S35. and The target speech signal is recovered by multiplying the phase of the mixed signal.

[0034] Furthermore, in step S32, the specific steps for defining the joint constraint loss function to constrain the network are as follows:

[0035] Step S321. Considering the loss function for masking error, it is: in, and M iLet be the prediction output matrix and the target mask matrix of the Lth layer output layer of the i-th source signal, respectively, where ||·||2 is the l2 norm;

[0036] Step S322. Using the relationship between IRM and signal amplitude spectrum, the joint constraint loss function is defined as follows:

[0037]

[0038] in, and S i These are the amplitude matrix of the initially estimated signal obtained from the first stage of reconstruction and the amplitude matrix of the i-th source signal, respectively.

[0039] Step S323. In the forward propagation algorithm, randomly initialize the weights and biases of each neuron node in the DNN layer to obtain the randomly predicted values. Then, the network parameters are fine-tuned under the joint constraint loss function through the backpropagation algorithm, so that the predicted estimate gradually approaches the true target value;

[0040] Step S324. After multiple iterations and updates, the network weights and biases are obtained. The trained DNN model is used for the separation of speech and cross-projection residues.

[0041] Furthermore, step S4 is included: performance evaluation of the proposed two-stage single-channel speech separation method based on a strongly constrained dictionary and a deep neural network.

[0042] The specific method is as follows:

[0043] Step S41. Compare the single-channel speech separation method based on strong constraint dictionary learning with the method based on traditional joint dictionary to verify the effectiveness of the strong constraint optimization function;

[0044] Step S42. A comparative experiment was conducted between the two-stage single-channel speech separation method based on strong constraint dictionary and deep neural network and the single-channel speech separation method based on strong constraint dictionary learning to verify the effectiveness of the two-stage speech separation method.

[0045] The advantages of this invention are that,

[0046] 1. This method reduces the “cross-projection” that occurs when the signal is represented in the joint dictionary by three parts: suppressing the error between the reconstructed signal and the target signal, constraining the reconstruction error of a single clean signal represented in the joint dictionary, and suppressing the projection of the clean signal onto other dictionaries and limiting the atomic correlation between dictionaries, thereby improving the performance of the single-channel speech separation system.

[0047] 2. In the first stage, this method uses a strongly constrained dictionary to achieve speech separation and obtain a preliminary estimated signal. In the second stage, a deep neural network with strong mapping capabilities is used to separate the speech from the cross-projection residual through joint constraints, resulting in a refined estimated signal. This makes the estimated signal closer to the target signal and improves the performance of the single-channel speech separation system. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the operation process of the present invention;

[0049] Figure 2 This is a schematic diagram of the second-stage DNN model structure of the present invention;

[0050] Figure 3 This is a comparison chart of the Loss1 algorithm and the MSE algorithm in an embodiment of the present invention;

[0051] Figure 4 This is a comparison diagram of Loss1-MMSE, Loss1-Loss2 and Loss1 algorithms in the embodiments of the present invention. Detailed Implementation

[0052] Please see Figure 1 This embodiment provides a two-stage single-channel speech separation method based on a strongly constrained dictionary and a deep neural network. In practical applications, acoustic features of the speech signal are typically extracted, input into a separation model for training, and then the trained model is used for speech separation. Dictionary-based sparse signal representation methods are commonly used to solve speech separation problems; therefore, this embodiment uses dictionary learning for the first stage of speech separation. In recent years, deep learning has achieved many results in data mining, pattern recognition, natural language processing, multimedia learning, speech, recommendation and personalization technologies, and other related fields. Its powerful feature extraction and modeling capabilities have greatly improved the performance of pattern recognition. Therefore, this embodiment applies a deep neural network (DNN) to speech separation for the second stage, enabling the construction of a higher-performance speech separation system.

[0053] To effectively improve speech separation performance, the first stage proposes a strongly constrained optimization function. Under this function, a more discriminative joint dictionary is constructed to reduce "cross-projection." This constraint consists of three parts: the first part suppresses the error between the reconstructed signal and the target signal; the second part constrains the error of the clean signal on the joint dictionary; and the third part suppresses the projection of the clean signal onto other dictionaries and limits the atomic correlation between dictionaries. In the second stage, to improve the separation effect between two similar signals, this embodiment proposes a two-stage single-channel speech separation method based on a strongly constrained dictionary and a deep neural network. The first stage uses the strongly constrained dictionary to achieve speech separation and obtain a preliminary estimated signal. The second stage utilizes a deep neural network with strong mapping capabilities to separate speech from cross-projection residues through joint constraints, thereby removing the influence of cross-projection residues and obtaining a refined estimated signal that is closer to the target signal. The following is a detailed description of the implementation method of this embodiment.

[0054] Step S1: Preprocess the input speech signal and update the dictionary.

[0055] Preprocess the speech signal and calculate its amplitude spectrum.

[0056] Because speech signals have short-term stationary characteristics, preprocessing is necessary before feature extraction to extract their feature information. Preprocessing operations mainly include pre-emphasis and frame-by-frame windowing. Next, the amplitude spectra S1 and S2 of the speech signal are calculated.

[0057] The training yields identity sub-dictionaries D1 and D2.

[0058] K-SVD is a dictionary learning algorithm in sparse representation. Its name comes from the fact that the algorithm requires K iterations, with each iteration using SVD decomposition. During dictionary updates, K-SVD updates only one atom and its corresponding sparse coding vector at a time. While updating this atom, the other atoms remain unchanged. Each iteration updates all the atoms in the dictionary, simultaneously updating the coefficient coding coefficients; this is called one iteration. Based on the K-SVD algorithm, corresponding identity sub-dictionaries D1 and D2 are trained.

[0059] Get the initial union dictionary

[0060] Concatenate D1 and D2 to obtain the initial joint dictionary D = [D1, D2].

[0061] Sparse coding

[0062] The OMP algorithm selects an atom (i.e., a column) from a dictionary matrix D (also known as an overcomplete atom library) that best matches the signal Y, constructs a sparse approximation, calculates the signal residual, and then continues to select the atom that best matches the signal residual, iterating repeatedly. The signal Y can be represented by the linear sum of these atoms plus the final residual value. In each iteration, a Schmidt orthogonalization operation is performed on all selected atoms to ensure that the result of each loop is the optimal solution. With D fixed, the OMP algorithm is used on the objective function... The sparse coding matrix C of the mixed signal Y on D is obtained under constraints.

[0063] Update the dictionary under the guidance of a strongly constrained optimization function.

[0064] The specific steps are as follows:

[0065] A. Define a strongly constrained optimization function:

[0066]

[0067] in, and Let S1 and S2 be the sparse coding matrices projected onto the joint dictionary D, respectively, representing the projections of individual clean signals S1 and S2. and for and The projection of the dictionary onto its own sub-dictionary.

[0068] The strongly constrained optimization function can be analyzed in three parts. The first part contains the first three terms. The first term reduces the error when the mixed signal is represented by the joint dictionary. The second and third terms constrain the projection error of the clean speech signal onto the corresponding sub-dictionary, ensuring that the speech signal is represented by its own sub-dictionary as much as possible. The second part contains the fourth and fifth terms, which further strengthen the constraint that when the clean speech signal is projected onto the joint dictionary, it should be projected onto the corresponding sub-dictionary as much as possible, rather than onto other sub-dictionaries. The difference between the first and second parts is that the second part starts from the signal itself and further strengthens the constraint that when a single clean speech signal is projected onto the joint dictionary, it should be projected onto its own sub-dictionary as much as possible. The third part contains the sixth, seventh, and eighth terms. The sixth and seventh terms suppress the sparse representation of the clean speech signal on other sub-dictionaries, reduce the cross-projection of the speech signal on other sub-dictionaries, and minimize the cross-interference between dictionaries. The eighth term suppresses the atomic correlation between sub-dictionaries by minimizing the atomic inner product between sub-dictionaries, increasing the discriminability between two sub-dictionaries, thereby further reducing cross-projection.

[0069] The L-BFGS algorithm is the most commonly used quasi-Newton method for solving unconstrained nonlinear programming problems. It performs an approximate BFGS algorithm within limited memory, resulting in low memory overhead, high computational efficiency, and significant advantages when dealing with high feature dimensions. With a fixed value C, the dictionary is updated by using the L-BFGS algorithm to find the strong optimization constraint function, thus obtaining a strongly constrained optimized dictionary.

[0070] B. Define the matrix and Where I represents the identity matrix and O represents a matrix of all zeros.

[0071] C. The strongly constrained optimization function is rewritten as:

[0072]

[0073] The gradient function of the objective function is:

[0074]

[0075] D. After multiple iterations, an updated optimized dictionary is obtained. This dictionary is more differentiated from the initial joint dictionary, which allows the mixed signal to be better distinguished from the different source signals when projected onto this dictionary.

[0076] Step S2: Obtain the amplitude spectrum of the first-stage separated speech.

[0077] The estimated sparse coding matrix is ​​obtained by projecting the mixed signal onto the dictionary D.

[0078] according to The reconstruction yields the amplitude spectrum of the speech signal separated in the first stage. and

[0079] Step S3: Obtain the separated target speech through the second-stage DNN.

[0080] Building a DNN network framework

[0081] The DNN framework consists of one input layer, three hidden layers, and one output layer, as follows: Figure 2 As shown. The neuron configuration is as follows: 257-1024-1204-1024-257. The input and output layers have 257 nodes each, and the hidden layers have three layers with 1024 nodes each.

[0082] Define the joint constraint loss function

[0083] Using the preliminary estimated signal features as input and the ideal ratio mask of the target speech as supervision, we define the ideal ratio mask for the second stage in the two-stage method of this embodiment as:

[0084]

[0085] Among them, S i (t,f) represents the amplitude spectrum of the i-th source signal at time t and frequency f. This indicates the estimated amplitude spectrum of the signal after the first stage of separation, which still contains some cross-projection residue, M. i (t,f) represents the mask of the i-th source signal at time t and frequency f.

[0086] Loss function considering masking error:

[0087]

[0088] in, and M i Let be the prediction output matrix and the target mask matrix of the L-th output layer of the i-th source signal, respectively, and ||·||2 represent the l2 norm. The above loss function only considers the relationship between the estimated signal and the target signal. To improve the approximation between the estimated signal and the target signal, the joint constraint loss function is defined using the relationship between the IRM and the signal amplitude spectrum:

[0089]

[0090] in, and S i These are the amplitude matrices of the initially estimated signal obtained from the first stage of reconstruction and the amplitude matrix of the i-th source signal, respectively. The joint constraint loss function J... Loss2 In addition to the constraints of IRM, an amplitude spectrum error constraint corresponding to the IRM estimation is added.

[0091] Using the gradient function to solve for W and b of each layer, the predicted output of the output layer satisfies:

[0092]

[0093] At this point, the loss function changes to:

[0094]

[0095] The gradients of W and b in the Lth layer are:

[0096]

[0097] Where ⊙ represents the Hadamard product, for z L The derivative part represents the residual of the Lth layer, denoted as:

[0098]

[0099] Then, according to the forward propagation algorithm, Wl and b l The following relationship exists:

[0100]

[0101] Calculate W l and b l gradient:

[0102]

[0103] By inductive reasoning, we can deduce that:

[0104] δ l =(W l+1 ) T δ l+1 ⊙σ'(z l )

[0105] For W l and b l The update is as follows:

[0106]

[0107] Where α represents the learning rate.

[0108] Training Network

[0109] In the forward propagation algorithm, the weights and biases of each neuron node in each layer of the DNN are randomly initialized to obtain random predictions. Then, the network parameters are fine-tuned under the constraints of the joint loss function through backpropagation, so that the predicted values ​​gradually approach the true target values. After multiple iterations, the network weights and biases are obtained, and the trained DNN model is used to separate speech from cross-projection residues, thereby enhancing the separated speech.

[0110] The amplitude spectrum was obtained by separation

[0111] The amplitude spectrum of the speech signal separated in the first stage and Input into the DNN network, output the estimated ideal ratio masks M1 and M2.

[0112] Reconstructing speech

[0113] M1 and M2 respectively with and The amplitude spectrum of the second-stage fine estimation is obtained by performing the Hadamard product. and and The target speech signal is recovered by multiplying the phase of the mixed signal.

[0114] Step S4: Evaluate the performance of the proposed two-stage single-channel speech separation method based on a strongly constrained dictionary and a deep neural network.

[0115] Database and experimental setup.

[0116] The experimental hardware consisted of an Intel(R) Core(TM) i5-4210U CPU at 1.70GHz and 2.40GHz, and the program was written using Python 3.7 and Matlab. The speech samples used in this experiment came from the GRID corpus, which comprises high-quality audio and video recordings, with a total of 34 speakers (18 males and 16 females), 1000 records per speaker, and each sentence approximately 2 seconds long. Furthermore, the deep learning frameworks used were Keras 2.3.1 and PyTorch 1.10.0. With the release of Tensorflow 2.0, Tensorflow includes the Keras learning library; this embodiment used Tensorflow version 2.1.0.

[0117] Experiments were conducted using two male and two female speakers selected from the GRID corpus. The mixed speech of any two speakers was separated. Of the 1000 sentences for each speaker, 400 were used as training set 1 for the first stage of dictionary learning, 400 as training set 2 for the second stage of DNN training, and the remaining 200 as the test set. The sampling frequency of the speech signal was downsampled from 25kHz to 16kHz. The DNN framework used was 257-1024-1204-1024-257, with 257 nodes in the input and output layers and three hidden layers with 1024 nodes. During DNN training, the number of iterations was set to 50 epochs, the initial learning rate was 0.01, the ReLU activation function was used between hidden layers, and the sigmoid activation function was used in the output layer. This embodiment uses SDR, SIR, SAR, PESQ, and STOI as indicators to evaluate speech quality, and speech quality is positively correlated with the indicator values.

[0118] Experimental performance evaluation

[0119] First, the effectiveness of the single-channel speech separation method based on strongly constrained dictionary learning was verified. Experiments were conducted using six speech mixing methods, with two female speakers (F1 and F2) and two male speakers (M1 and M2) randomly selected. Figure 3 The performance comparison between the strongly constrained dictionary algorithm and the traditional joint dictionary algorithm is shown. Figure 3As shown, compared with the traditional joint dictionary algorithm (MSE algorithm), the speech separation method using a strongly constrained optimization function to train the dictionary (Loss1 algorithm) shows improvements in SDR, SIR, SAR, PESQ, and STOI, indicating enhanced speech intelligibility and accuracy. This suggests that the strongly constrained optimization function effectively limits the interference between different signals. Regardless of the metric, the overall performance of separating mixed signals of opposite genders is better than that of mixed signals of the same gender. This is because the similarity of speech between the same gender is higher, making the separation task more difficult. When separating mixed F1-F2 speech in females, compared with the traditional joint dictionary algorithm, SDR, SIR, SAR, PESQ, and STOI were improved by 0.692dB, 1.632dB, 1.014dB, 0.046, and 0.055, respectively; when separating mixed M1-M2 speech in males, SDR, SIR, SAR, PESQ, and STOI were improved by 1.637dB, 0.73dB, 1.565dB, 0.03, and 0.028, respectively. The reason why the separation performance of female combinations is worse than that of male combinations is that females have higher fundamental frequencies and higher similarity, making separation more difficult. The mixed signal separation results of opposite-sex combinations are better than those of same-sex combinations because the similarity between male and female speech is lower. Taking F2-M1 as an example, SDR improved by 0.642dB, SIR by 0.571dB, SAR by 1.178dB, PESQ by 0.079, and STOI by 0.057. The experimental results verify that the dictionary obtained by the strongly constrained optimization function proposed in this embodiment is effective for speech separation. It also shows that the gender combination of mixed speech has an impact on the experimental results.

[0120] Next, we will verify the effectiveness of the two-stage single-channel speech separation method using a strongly constrained dictionary and a deep neural network. Figure 4 This paper compares the separation performance of a two-stage single-channel speech separation method using strongly constrained dictionaries and deep neural networks (Loss1-MMSE algorithm, Loss1-Loss2 algorithm) with a single-channel speech separation method based on strongly constrained dictionary learning (Loss1 algorithm). Figure 4As can be seen, firstly, the Loss1-MMSE and Loss1-Loss2 algorithms show improvement in all evaluation metrics compared to the Loss1 algorithm, indicating that the two-stage speech separation method combining dictionary learning and DNN can effectively improve speech separation performance. Secondly, the overall separation performance of the Loss1-Loss2 algorithm is further improved compared to the Loss1-MMSE algorithm, especially in the case of opposite-sex combinations. Taking F2-M1 as an example of opposite-sex mixing, the Loss1-Loss2 speech separation method improves SDR, SIR, SAR, PESQ, and STOI by 2.545dB, 0.77dB, 2.236dB, 0.089, and 0.058 respectively compared to the Loss1 speech separation method. When female F1-F2 is mixed, the corresponding SDR, SIR, SAR, PESQ, and STOI are improved by 1.776dB, 0.32dB, 0.22dB, 0.042, and 0.048 respectively. When male M1-M2 combinations are mixed, the corresponding SDR, SIR, SAR, PESQ, and STOI are improved by 0.451dB, 0.325dB, 0.317dB, 0.036, and 0.036, respectively. The values ​​of each index for opposite-sex combinations are higher than those for female F1-F2 combinations or male M1-M2 combinations. Furthermore, the improvement values ​​for each index from the Loss1 algorithm to the Loss1-Loss2 algorithm are also greater. This is mainly because the two speech samples are more similar when combined with the same sex, making the separation task more difficult. Experimental results verify that the two-stage speech separation method proposed in this embodiment, which combines dictionary learning with DNN, can effectively improve the intelligibility and clarity of the separated speech.

[0121] The above results demonstrate that the two-stage single-channel speech separation method based on a strongly constrained dictionary and a deep neural network proposed in this embodiment offers two approaches to improve the performance of the speech separation system, focusing on the optimization function of dictionary learning and the construction of the separation model. Experimental results verify the rationality and effectiveness of these two methods. Compared with other methods, the method in this embodiment achieves improvements in five measurement metrics and can overcome the shortcomings of ordinary methods in achieving incomplete speech separation to a certain extent, making it a valuable reference in practical applications.

[0122] The foregoing has shown and described the basic principles, main features, and advantages of this embodiment. Those skilled in the art should understand that this embodiment is not limited to the specific embodiments described above. The specific embodiments and descriptions in the specification are merely for further illustrating the principles of this embodiment. Various changes and modifications can be made to this embodiment without departing from the spirit and scope of this embodiment, and all such changes and modifications fall within the scope of this embodiment as claimed. The scope of protection of this embodiment is defined by the claims and their equivalents.

Claims

1. A two-stage single-channel speech separation method based on a strongly constrained dictionary and a deep neural network, characterized in that, It comprises the following steps, Step S1. Extracting the amplitude spectrum of the clean speech signal and the mixed speech signal, calculating the sub-dictionary D1 and D2 of the clean speech signal by using the K-SVD algorithm, splicing the sub-dictionary into a joint dictionary D = [D1, D2], and iteratively updating the dictionary under the constraint of a strong constraint optimization function; Step S2. Projecting another training sample on the dictionary D to obtain the sparse coding matrix C = [C1, C2] of the mixed signal on D T , reconstructing the speech signal amplitude spectrum separated in the first stage and Step S3. and The estimated ideal ratio masks M1 and M2 are obtained through the DNN, and the masks are multiplied with and The second-stage fine-estimated amplitude spectrum is obtained by performing Hadamard product and The amplitude spectrum and the phase of the mixed signal are multiplied to recover the speech signal; The specific steps of the step S1 are, Step S11. Sampling each input time-domain continuous speech signal, and then performing short-time Fourier transform to obtain a preprocessed speech signal; Step S12. Calculating the amplitude spectrum S1 and S2 of the preprocessed speech signal, training and obtaining the corresponding identity sub-dictionary D1 and D2 based on the K-SVD algorithm, and splicing D1 and D2 to obtain an initial joint dictionary D = [D1, D2]; Step S13. Fixing D, the sparse coding matrix C of the mixed signal Y on D is obtained by the OMP algorithm under the constraint of the objective function; Step S14. Fixing C, the dictionary is updated by solving the strong constraint optimization function by the L-BFGS algorithm, and the dictionary after strong constraint optimization is obtained; In the step S14, the specific method for updating the dictionary by solving the strong constraint optimization function by the L-BFGS algorithm is that step S141. The strong constraint optimization function is defined as wherein, and are sparse coding matrices of the individual clean signals S1 and S2 projected on the joint dictionary D, and are and the projection part of the individual clean signals S1 and S2 on their own sub-dictionary; Step S142. Defining a matrix and where I is the identity matrix and O is the zero matrix. Step S143. The strong constraint optimization function is rewritten as The gradient function of the objective function is Step S144. After multiple iterations, the updated optimization dictionary is obtained, so that the mixed signal can be projected on the dictionary to distinguish different source signals.

2. The two-stage single-channel speech separation method based on a hard-constrained dictionary and a deep neural network according to claim 1, characterized in that, The specific steps of the step S2 are, Step S21. The mixed signal is projected on the dictionary D to obtain an estimated sparse coding matrix Step S22. According to reconstructing the amplitude spectrum of the speech signal separated in the first stage and 3. The two-stage single-channel speech separation method based on a strongly constrained dictionary and a deep neural network according to claim 1, characterized in that, The specific steps of the step S3 are, Step S31. A DNN network framework is constructed, which includes an input layer, three hidden layers and an output layer; Step S32. The ideal ratio mask of the target speech is used as the training target of the DNN network, and a joint constraint loss function is defined to constrain the network; Step S33. computing the amplitude spectrum of the speech signal separated in the first stage and inputting the DNN network, and outputting the estimated ideal ratio masks M1 and M2; Step S34. M1 and M2 are multiplied with and Performing Hadamard product to obtain the second stage fine estimated amplitude spectrum and Step S35. and The target speech signal is recovered by multiplying the mixed signal by the phase.

4. The two-stage single-channel speech separation method based on a strongly constrained dictionary and a deep neural network according to claim 3, characterized in that, In the step S32, the specific steps of defining the joint constraint loss function to constrain the network are, Step S321. The loss function considering the mask error is where, and M i are the predicted output matrix of the Lth output layer and the target mask matrix of the ith source signal, respectively, and ||·||2is the l2norm. Step S322. Using the relationship between IRM and signal amplitude spectrum, the joint constraint loss function is defined as wherein, and S i are the amplitude matrix of the preliminary estimate signal and the amplitude matrix of the i-th source signal, respectively, obtained from the first stage of reconstruction. Step S323. In the forward propagation algorithm, the weights and biases of each layer of neurons of the DNN are randomly initialized, and a random prediction is obtained The network parameters are further fine-tuned under the joint constraint loss function through the back propagation algorithm, so that the predicted estimated value gradually approaches the true target value. Step S324. After multiple iterations, the network weight and bias are obtained, and the trained DNN model is used for the separation of speech and cross projection residual.

5. The two-stage single-channel speech separation method based on a hard-constrained dictionary and a deep neural network according to claim 1, characterized in that, It also comprises step S4. Performance evaluation is performed on the proposed two-stage single-channel speech separation method based on the strong constraint dictionary and the deep neural network; The specific method is, Step S41. Comparing the single-channel speech separation method based on the strong constraint dictionary learning with the method based on the traditional joint dictionary, and verifying the effectiveness of the strong constraint optimization function; Step S42. Comparing the two-stage single-channel speech separation method based on the strong constraint dictionary and the deep neural network with the single-channel speech separation method based on the strong constraint dictionary learning, and verifying the effectiveness of the two-stage speech separation method.

Images