A single-channel speech separation method based on deep neural network

Abstract

The application discloses a single-channel speech separation method based on a multiple joint constraint double-output deep neural network, first proposes three joint constraint loss functions, integrates them, and proposes a multiple joint constraint loss function for training a double-output deep neural network to solve a single-channel speech separation problem. The multiple joint constraint loss function fully utilizes the relationship between network output values, the relationship between output values and target speech characteristics, and the relationship between estimated target speech characteristics and mixed speech characteristics, and the deep neural network trained under the guidance of the loss function can output more accurate prediction values. In addition, the application solves the optimal weight coefficients corresponding to each constraint term by using an optimization idea, so that the constraint degree of each term is more accurate, and the performance of the separation system is further improved. Experimental results show that compared with a method based on a traditional loss function, the performance of the method is more superior, and the target speech can be separated more effectively.

Description

TECHNICAL FIELD

[0001] The application relates to a single-channel speech separation method, in particular to a single-channel speech separation method based on a multiple joint constraint double-output deep neural network, and belongs to the technical field of speech separation. BACKGROUND

[0002] In daily life, speech as a carrier of human communication cannot be ignored. Speech separation aims to recover high-quality and high-definition target speech signals from mixed speech signals. The technology can be applied to front-end processing of machine translation, advanced hearing aids and automatic speech recognition, and helps to improve their performance. If multiple speakers speak at the same time, people can easily understand what they say, but machines cannot easily identify the language of the speakers, and the user's communication will be very poor in human-computer dialogue. Therefore, it is very important and practical for intelligent machines to have the ability to extract clean speech signals from mixed speech signals. According to the number of input sound sources, speech separation can be divided into single-channel and multi-channel speech separation. Among them, single-channel usually refers to mixing audio signals from different directions and recording them by a recording device. Compared with multi-channel speech separation, single-channel speech separation is more challenging.

[0003] For the difficult single-channel speech separation problem, researchers have proposed many solutions. According to different target tasks, they can be mainly divided into three categories: statistical methods, clustering methods and factorization methods. In the statistical method, it is assumed that the interference speech signal is statistically independent of the target speech signal, and a probability distribution model such as complex Gaussian or independent component analysis is used to model the target speech signal. In the clustering algorithm, the pitch of the target speech signal and the continuity characteristics of the signal are estimated by observation, so as to distinguish the target speech from the interference speech. For the factorization method, such as non-negative matrix factorization (NMF), the speech separation problem is described as a matrix decomposition problem, the time-frequency representation of the mixed speech signal is decomposed into the combination of the basic signal and the excitation signal, and then the learned excitation signal of each basic signal is used to reconstruct the target speech signal.

[0004] In recent years, deep learning technology has made significant progress in speech separation, particularly in speech enhancement and music separation, where deep learning-based methods have achieved better results than traditional methods. Depending on the training objective, deep learning-based speech separation methods can be divided into spectral mapping-based methods and time-frequency masking-based methods. Spectral mapping-based methods separate mixed speech signals by training a neural network model to learn a mapping function from the features of the mixed speech signal to the features of the clean speech signal. Masking-based methods separate mixed speech signals by training a neural network model to learn a mapping function from the features of the mixed speech signal to the mask. This method is generally used to estimate Ideal Binary Mask (IBM) and Ideal Ratio Mask (IRM), as well as Phase Sensitive Mask (PSM) and Complex Ideal Ratio Mask (cIRM). Typically, the training objective of the neural network is either IBM or IRM. For IBM, time-frequency units are assigned to 0 or 1 by comparing the signal-to-noise ratio (SNR) with a local threshold. If the target speech is dominant, it is marked as 1; conversely, if the interfering speech is dominant, it is marked as 0. IBM only uses two values, 0 and 1, which is too crude in its processing of mixed speech. If a large amount of interfering speech is introduced during processing, it cannot effectively improve speech quality. IRM, on the other hand, directly characterizes the ratio of clean speech energy to mixed speech energy within a time-frequency unit, and is a continuous value distributed between 0 and 1. This allows IRM to effectively improve both speech quality and intelligibility simultaneously, making it a widely used masking method. Therefore, this invention uses IRM as the training target for the model.

[0005] In deep learning-based single-channel speech separation methods, the loss function measures the quality of the model's predictions and has a significant impact on the performance of the separation system. Traditional single-channel speech separation methods based on dual-output DNNs typically use loss functions that only consider the error between the estimated and true values, which cannot effectively constrain the training of the neural network, resulting in considerable room for improvement in separation performance. Summary of the Invention

[0006] To address the problems existing in current speech separation techniques, three joint constraint loss functions are proposed and integrated to propose a single-channel speech separation method based on a multi-joint constraint dual-output deep neural network, which overcomes the shortcomings of traditional single-channel speech separation methods based on dual-output deep neural networks.

[0007] To achieve the above objectives, the technical solution adopted by this invention is as follows: a single-channel speech separation method based on a multi-joint constraint dual-output deep neural network, the main implementation steps of which include the following:

[0008] Step 1: Randomly select the speech signals of speaker 1 and speaker 2 from the corpus for preprocessing to obtain a mixed speech signal. Extract the amplitude spectrum features of the clean speech signal and the mixed speech signal, and calculate the training target IRM.

[0009] Step 2: Using the amplitude spectrum features of the mixed speech signal as input to the deep neural network (DNN), and the training objective IRM as output of the DNN, the DNN model is trained under the guidance of joint constraint loss function 1, joint constraint loss function 2, and joint constraint loss function 3, respectively. JC_1 DNN JC_2 DNN JC_3 .

[0010] Step 3: Input the amplitude spectrum features of the test mixed speech signal into the DNN. JC_1 DNN JC_2 and DNN JC_3 The corresponding IRM estimate is obtained, the amplitude spectrum characteristics of the estimated target speech signal are calculated, and the target speech signal is reconstructed.

[0011] Step 4: Using the amplitude spectrum features of the mixed speech signal as input to the deep neural network (DNN), and the training objective IRM as output of the DNN, the DNN model is trained under the guidance of a multi-joint constraint loss function. JC_multi .

[0012] Step 5: Input the amplitude spectrum features of the test mixed speech signal into the DNN JC_multi The corresponding IRM estimate is obtained, the amplitude spectrum features of the estimated target speech signal are calculated, the target speech signal is reconstructed, and the performance of the proposed single-channel speech separation method based on a dual-output deep neural network with multiple joint constraints is evaluated.

[0013] To achieve better separation performance, the multi-constraint loss function of this invention considers not only the relationships between estimated masking values ​​and actual masking values ​​of different speech signals, and between estimated masking values ​​and target speech features, but also the relationships between target speech features and mixed speech features. Furthermore, this invention employs optimization to solve for the optimal weighting coefficients of the corresponding constraint terms, making the constraint strength of each term more precise. The neural network trained under the guidance of this loss function can output more accurate estimates, thereby improving the quality of separated speech.

[0014] Furthermore, step 1 includes:

[0015] Step 1-1: Sample the input time-domain continuous speech signal 1 and speech signal 2 at 16kHz, and then perform pre-emphasis, framing, and windowing operations to obtain the preprocessed speech signal.

[0016] Steps 1-2: Perform short-time Fourier transform on the preprocessed speech signal and extract amplitude spectrum features.

[0017] Steps 1-3: Calculate the training target IRM based on the extracted amplitude spectrum features, using the following formula:

[0018]

[0019] Among them, M i (t,f) represents the ideal ratio masking of the i-th target speech signal at time t and frequency f, S i (t,f) represents the amplitude spectrum of the i-th target speech signal at time t and frequency f. ε is a very small positive value, which is used to prevent the denominator from being 0.

[0020] Steps 1-4: To facilitate data processing, the amplitude spectrum features of the input mixed speech signal are normalized to obtain the normalized amplitude spectrum features Y of the training mixed speech signal. train .

[0021] Furthermore, step 2 includes:

[0022] Step 2-1: Calculate the amplitude spectrum features Y of the training mixed speech signal train As input to the DNN, the IRM serves as the training target. Guided by joint constraint loss function 1, joint constraint loss function 2, and joint constraint loss function 3, the DNN is trained iteratively multiple times to obtain a fully trained DNN. JC_1 DNN JC_2 DNN JC_3 Used to train DNN JC_1 DNN JC_2 DNN JC_3 The loss functions are as follows:

[0023]

[0024] Where T represents the total number of time frames, M 1t and M represents the true IRM value and the estimated IRM value of the target speech signal 1 at time t, respectively; 2t and S represents the true IRM value and the estimated IRM value of the target speech signal 2 at time t, respectively. 1t and S represents the true amplitude spectrum feature and the estimated amplitude spectrum feature of the target speech signal 1 at time t, respectively. 2t and Y represents the true amplitude spectrum feature and the estimated amplitude spectrum feature of the target speech signal 2 at time t, respectively. t This represents the amplitude spectrum characteristics of the mixed speech signal. α, β, and γ are regularization coefficients, all ranging from [0,1]. Different values ​​indicate different levels of constraint.

[0025] Step 2-2: The training of the DNN model consists of two stages: forward propagation and back propagation. In the forward propagation stage, weights and biases are randomly initialized to obtain a rough estimate of the output. In the back propagation stage, the model is computed backward from the back, optimizing the loss function using the gradient descent algorithm and updating the weights and biases.

[0026] Furthermore, step 3 includes:

[0027] Step 3-1: First, preprocess the test mixed speech signal, then perform a short-time Fourier transform to extract the amplitude spectrum and phase spectrum features of the test mixed speech signal, and normalize the amplitude spectrum features to obtain Y. test .

[0028] Step 3-2: Test the amplitude spectrum characteristics Y of the mixed speech signal test Input into DNN respectively JC_1 DNN JC_2 and DNN JC_3 The corresponding IRM estimate is obtained, and the estimated target speech amplitude spectrum features are calculated to reconstruct the target speech signal.

[0029] Furthermore, step 4 includes:

[0030] Step 4-1: Integrate the above four prediction methods into a combined prediction method, and use an optimized algorithm to solve for the optimal weight coefficient K. * =[k1 * k2 * k3 * ,k4 * ] T , which serves as the regularization coefficient for the loss function under multiple joint constraints.

[0031] Step 4-2: Train the amplitude spectrum features Y of the mixed speech signal train Using the IRM as input to the DNN and the IRM as the training target, the DNN is trained iteratively through multiple iterations under the guidance of a multi-joint constraint loss function, resulting in a fully trained DNN. JC_multi Used to train DNN JC_multi The loss function is:

[0032]

[0033] in, The optimal weight coefficients obtained in the above steps are denoted as .

[0034] Furthermore, step 5 includes:

[0035] Step 5-1: Test the amplitude spectrum characteristics Y of the mixed speech signal test Input to DNN JC_multi The corresponding IRM estimate is obtained, the amplitude spectrum characteristics of the estimated target speech signal are calculated, and the target speech signal is reconstructed.

[0036] Step 5-2: Evaluate the performance of the separation system using PESQ, STOI, SDR, SIR, and SAR indices.

[0037] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention has certain theoretical research value and practical application value. The method considers the inherent relationship between different source signals from multiple perspectives, jointly constrains the network output value and speech signal features, and adjusts the constraint strength of corresponding terms through weight coefficients to improve the accuracy of the separation model. The neural network trained under this loss function can more accurately estimate the corresponding output value. Furthermore, this invention uses optimization to solve for the optimal weighting coefficients of corresponding constraint terms in the multiple joint constraint loss function, making the constraint strength of the constraints more precise and further improving the performance of the separation system. Attached Figure Description

[0038] Figure 1 This is a block diagram of the single-channel speech separation system based on a multi-joint constraint dual-output DNN in this invention.

[0039] Figure 2 The diagram shows the performance of the single-term joint constraint separation system under different regularization coefficients. Among them, (a) is a schematic diagram of PESQ of JC1 under different regularization coefficients, (b) is a schematic diagram of PESQ of JC2 under different regularization coefficients, and (c) is a schematic diagram of PESQ of JC3 under different regularization coefficients.

[0040] Figure 3 These are schematic diagrams illustrating the SIR, SDR, and SAR methods for separating target speech. (a) is a schematic diagram of SIR, (b) is a schematic diagram of SDR, and (c) is a schematic diagram of SAR.

[0041] Figure 4These are waveforms of the target speech separated by different methods. Among them, (a) is the original female speech waveform, (b) is the original male speech waveform, (c) is the mixed speech waveform, (d) is the estimated female speech waveform separated using the Basic-IRM method, (e) is the estimated male speech waveform separated using the Basic-IRM method, (f) is the estimated female speech waveform separated using the JC4 method, and (g) is the estimated male speech waveform separated using the JC4 method. Detailed Implementation

[0042] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0043] This embodiment proposes a single-channel speech separation method based on a multi-joint constraint dual-output deep neural network. In the past decade, deep neural networks have become increasingly widely used in supervised learning tasks due to their powerful data fitting capabilities, such as image classification, handwriting recognition, automatic speech recognition, and speech separation. Deep learning-based speech separation methods achieve speech separation by training a neural network model to learn the nonlinear relationship between the mixed speech signal and the target speech signal. Compared with traditional methods based on spectral subtraction or Wiener filters, the single-channel speech separation method based on deep neural networks achieves better separation results. In deep learning-based single-channel speech separation algorithms, the loss function measures the degree of difference between the predicted value and the actual data, thus significantly impacting system performance. Traditional DNN-based single-channel speech separation methods typically only consider the relationship between the output value and the corresponding true value, neglecting the intrinsic connections between the output values, resulting in a need to improve the quality of the separated speech. Therefore, this embodiment starts with the loss function. Based on the traditional loss function, it considers the inherent relationship between different signals and jointly constrains the masking values ​​and features of different speech signals, which improves the accuracy of model prediction and makes the network output value closer to the true value, thereby improving the clarity and intelligibility of the separated speech.

[0044] The overall process of the single-channel speech separation algorithm based on a multi-joint constraint dual-output deep neural network proposed in this embodiment is as follows: Figure 1As shown, the speech separation algorithm consists of two phases: a training phase and a testing phase. The purpose of the training phase is to train a DNN with more accurate network parameters under the guidance of a multi-joint constraint loss function. The purpose of the testing phase is to use the fully trained DNN to extract the target speech signal from the mixed speech signal through a series of linear and nonlinear transformations. First, in the training phase, the target speech signal and the mixed speech signal need to be preprocessed to extract the corresponding amplitude spectrum features and calculate the corresponding training target IRM. Then, the normalized amplitude spectrum of the mixed speech signal is used as the input to the DNN, and the training target IRM is used as the output of the DNN. The DNN is trained under the guidance of the multi-joint constraint loss function, and after several adjustments to the network parameters, a fully trained DNN is obtained. In the testing phase, a Short-Time Fourier Transform (STFT) is first performed on the test mixed speech signal to extract amplitude spectrum features. The normalized amplitude spectrum is then input into the fully trained DNN to obtain the estimated mask value of the target speech signal. Next, the estimated mask value corresponding to the target speaker is multiplied by the amplitude spectrum of the mixed speech signal to obtain the estimated amplitude spectrum of the target speech signal. Finally, the estimated target signal amplitude spectrum is combined with the mixed phase spectrum, and the target speech signal is reconstructed using the Inverse Short-Time Fourier Transform (ISTFT). Experiments show that this algorithm can improve the performance of the speech separation system compared with the DNN model using the traditional loss function.

[0045] The following is a detailed description of specific embodiments of the present invention:

[0046] Step 1: Preprocess the speech signal and extract amplitude spectrum features.

[0047] Since the human vocal organs themselves and the equipment used to acquire speech signals can affect the quality of the speech signal, preprocessing operations such as pre-emphasis, framing, and windowing must be performed before analyzing and processing the speech signal. Preprocessing provides better parameters for subsequent processing, improving the quality of the processed speech. In this embodiment, the default sampling rate for both training and test speech signals is 25kHz. To make the time-domain signal sparser, each speech signal is downsampled to 16kHz in this embodiment. Of course, in other embodiments, the speech signal sampling rate can be set to other values.

[0048] The preprocessed target speech signal is mixed to obtain a mixed speech signal. A 512-point STFT is then applied to both the target and mixed speech signals to obtain their spectra. Next, the absolute value of the speech signal spectra is taken to obtain the amplitude spectrum features of the target and mixed speech signals, with a dimension of 257×1. The training target IRM is calculated based on the obtained amplitude spectrum features. Since the amplitude spectrum of the speech signal ranges from [0, +∞), excessively large values ​​are detrimental to DNN training. To improve the convergence speed and model accuracy, the amplitude spectrum features of the speech signal are normalized. The formula for calculating the training target IRM is:

[0049]

[0050] Among them, M i (t,f) represents the ideal ratio masking of the i-th target speech signal at time t and frequency f, S i (t,f) represents the magnitude of the amplitude spectrum of the i-th target speech signal at time t and frequency f. ε is a very small positive value, which is used to prevent the denominator from being 0.

[0051] Step 2: Train a single-term joint constraint dual-output deep neural network.

[0052] The DNN used in this embodiment contains one input layer, three hidden layers, and one output layer. The number of neurons in the input, hidden, and output layers are 257, 1024, and 514, respectively. There are 257 single-source output nodes, therefore the total number of dual-source output nodes is 514 (257*2). For the hidden layers, we choose ReLU as the activation function, which is responsible for summarizing the input neuron information and converting it into new output information, which is then passed to the next neuron. Since both outputs of the DNN take values ​​in the range [0,1], we choose sigmoid as the activation function for the output layer. The optimizer used for training is stochastic gradient descent, with a learning rate of 0.01 and 200 iterations.

[0053] The normalized amplitude spectrum features of the mixed speech signal are used as the input to the DNN, and the training objective IRM is used as the output of the DNN. Guided by joint constraint loss function 1, joint constraint loss function 2, and joint constraint loss function 3, the DNN is trained iteratively multiple times to obtain a fully trained DNN. JC_1 DNN JC_2 and DNN JC_3 Among them, the part used to train the DNN JC_1 DNN JC_2 and DNN JC_3 The loss functions are as follows:

[0054]

[0055] Where T represents the total number of time frames, M 1t and M represents the true IRM value and the estimated IRM value of the target speech signal 1 at time t, respectively; 2t and S represents the true IRM value and the estimated IRM value of the target speech signal 2 at time t, respectively. 1t and S represents the true amplitude spectrum feature and the estimated amplitude spectrum feature of the target speech signal 1 at time t, respectively. 2t and Y represents the true amplitude spectrum feature and the estimated amplitude spectrum feature of the target speech signal 2 at time t, respectively. t This represents the amplitude spectrum characteristics of the mixed speech signal. α, β, and γ are regularization coefficients, all ranging from [0,1]. Different values ​​indicate different levels of constraint.

[0056] DNN model training consists of two stages: forward propagation (FP) and backward propagation (BP).

[0057] The FP phase is a process of randomly initializing weights and biases to obtain a rough estimate of the output, which can be represented as:

[0058]

[0059] Among them, w i x represents the weights between neurons in layer i and layer i+1. i Let b represent the output of the i-th layer, b represent the bias, z represent the input of the (i+1)-th layer, and x represent the bias. i+1 Let σ represent the output of the (i+1)th layer, and σ represent the activation function.

[0060] The difference between the backpropagation (BP) and front-processing (FP) stages lies in their starting point. BP starts from the output layer and performs backward computation. The main mechanism controlling the backward parameter tuning is the loss function. The gradient descent algorithm is used to optimize the loss function until it converges, thereby constraining the output error value. The output error affects the parameters w of the i-th layer of the neural network. i The derivative, obtained by recursion in reverse, is as follows:

[0061]

[0062] Next, update the weights and biases of the neural network:

[0063]

[0064] Where η, t, and l represent the learning rate, number of iterations, and number of layers of the neural network, respectively.

[0065] Step 3: Test the single-term joint constraint dual-output deep neural network

[0066] First, the test mixed speech signal is preprocessed, then subjected to STFT, and amplitude spectrum and phase spectrum features are extracted. The amplitude spectrum features are then normalized. The normalized amplitude spectrum features are then input into a DNN. JC_1 DNN JC_2 and DNN JC_3 The corresponding estimated IRM value is obtained. and The amplitude spectrum of the estimated target speech signal is calculated using the following formula:

[0067]

[0068] in, Y represents the amplitude spectrum estimate of the i-th target speech. t Let represent the amplitude spectrum of the mixed speech, and ⊙ represent the Hadamard product. The estimated target speech amplitude spectrum is combined with the mixed speech phase spectrum to obtain the estimated time-domain target speech signal. The calculation process is as follows:

[0069]

[0070] Step 4: Train a deep neural network with multiple joint constraints and dual outputs

[0071] The normalized amplitude spectrum features of the mixed speech signal are used as the input to the DNN, and the training objective IRM is used as the output of the DNN. Under the guidance of a multi-joint constraint loss function, a fully trained DNN is obtained through multiple iterations of training. JC_multi Used for training DNNs JC_multi The loss functions are as follows:

[0072]

[0073] From the above equation, JC4 can be broken down into the following equations:

[0074]

[0075] Then equation (11) can be written as:

[0076] JC4=k1L origin +k2L1+k3L2+k4L3 (16)

[0077] Therefore, the JC4-based prediction method can be viewed as a combined prediction method. The key to combined prediction is how to accurately determine the weighting coefficients of each prediction method. To solve for the optimal weighting coefficients, we adopted an optimization approach and performed the following series of tasks.

[0078] The amplitude spectrum features Y of the training mixed speech signal train As input to the DNN, the IRM serves as the training target, and is respectively in L origin Guided by L1, L2, and L3 algorithms, and after multiple iterations of training, a fully trained DNN was obtained. origin DNN L1 DNN L2 and DNN L3 The amplitude spectrum characteristics Y of the mixed speech signal will be tested. test Input into DNN respectively L1 DNN L2 and DNN L3 The corresponding IRM estimate is obtained. and

[0079] The predicted value of the combined prediction method is:

[0080]

[0081] The prediction error of the i-th prediction method is:

[0082]

[0083] The prediction error of the combined prediction method is:

[0084]

[0085] The sum of squared prediction errors of the combined prediction method is:

[0086]

[0087] Let the weighted coefficient vector be K = [k1, k2, k3, k4] T The prediction error vector of the i-th single-item prediction method is The sum of squared prediction errors of the combined prediction method can also be expressed as:

[0088]

[0089] Remember R P =[1,1,1,1] T The constraint condition for the weighting coefficients of the combined prediction method is R. P T K P=1. Our goal is to find the weighted coefficient vector that minimizes the sum of squared prediction errors of the combined prediction method under this constraint. This problem can then be expressed as a nonlinear programming problem:

[0090]

[0091] Introducing Lagrange multipliers, the sum of squared prediction errors of the combined prediction method is:

[0092] J = K P T E (P) K P +λ(R P T K P -1) (23)

[0093] The condition for the above expression to take its minimum value is:

[0094]

[0095] Right now:

[0096] 2E (P) K P +λR P =0 (25)

[0097] Multiply both sides of equation (22) by E on the left (P) -1 We can obtain:

[0098] 2K P +λE (P) -1 R P =0 (26)

[0099] Multiply both sides of equation (23) by R on the left P T We can obtain:

[0100] 2R P T K P +λR P T E (P) -1 R P =0 (27)

[0101] Because R P T K P =1, substituting this condition into equation (24), we can access the Lagrange multipliers as follows:

[0102]

[0103] Substituting equation (25) into equation (23), we can obtain the optimal weighting coefficients as follows:

[0104]

[0105] The optimal weighting coefficient can be expressed as:

[0106]

[0107] To ensure consistency with the format in JC4, K * Multiply by 1 / k1 * ,Right now

[0108] Then used to train DNN JC_multi The loss function is:

[0109]

[0110] Among them, k1 * k2 * k3 * ,k4 * These are the optimal weighting coefficients obtained from the above solution.

[0111] Step 5: Test the multi-joint-constraint dual-output deep neural network

[0112] The normalized amplitude spectrum features of the test mixed speech signal are input into the DNN. JC_multi The corresponding IRM estimate is obtained. The amplitude spectrum of the target speech signal is calculated according to equation (9), and the estimated time domain signal of the target speech is obtained according to equation (10).

[0113] Performance evaluation

[0114] In our experiments, both training and testing speech signals were derived from the GRID corpus, which contains 18 males and 16 females, each with 1000 clean speech samples, each lasting approximately one second. Speech samples from two males and two females were randomly selected from the corpus as experimental data, resulting in three gender combinations (F+F, F+M, M+M). 700 speech samples from each speaker were randomly selected as training target speech samples and mixed to form the training mix. Then, 50 samples were randomly selected from the remaining 300 and mixed to form the test mix. To obtain more representative results, all experimental results are statistical averages. The sampling rate of each speech sample was downsampled from 25kHz to 16kHz. The amplitude spectrum of the speech was obtained using a 512-point STFT with a Hamming window of length 512 and a frame shift of 256.

[0115] This invention employs multiple speech metrics to measure the accuracy and effectiveness of the proposed algorithm, including Perceptual Evaluation of Speech Quality (PESQ), Short-Time Objective Intelligibility (STOI), Signal-to-Interference Ratio (SIR), Signal-to-Distortion Ratio (SDR), and Sources-to-Artifacts Ratio (SAR). Higher values ​​for these metrics indicate better quality speech separation.

[0116] First, this invention investigates the impact of regularization coefficients on the separation performance of the single-factor joint constraint method under different gender combinations. For JC1, the regularization coefficient α ranges from 0 to 1, increasing by 0.1 each time. The result when α = 0 represents the performance of the traditional loss function. Experimental results are as follows: Figure 2 As shown in (a), when α is less than 0.5, the PESQ of all three gender combinations increases to varying degrees with the increase of α. For F+M, the PESQ increases rapidly when α is 0–0.3, and slowly when α is 0.4–0.5. Since the mixed speech features of different gender combinations are different, the optimal weighting coefficients are different. For F+F, the separation system performs best when α is 0.7. When α is greater than 0.7, the PESQ curve decreases with the increase of α, but the separation performance when α is set to 1 is still better than that when α is set to 0, proving the effectiveness of the proposed joint constraint algorithm. Similarly, we conducted the same experiments on JC2 and JC3. The PESQ of JC2 and JC3 under different regularization coefficients are shown in the figure. Figure 2 (b) and Figure 2 As shown in (c). For MM and FM, the separation system performs best when β and γ in JC2 and JC3 are 0.5. For FF, the separation system performs best when β and γ in JC2 and JC3 are 0.6. Therefore, in the comparative experiments below, when separating F+M and M+M based on JC1, α is set to 0.5, and when separating F+F, α is set to 0.7; when separating F+M and M+M based on JC2 and JC3, β and γ are set to 0.5, and when separating F+F, β and γ are set to 0.6.

[0117] Next, the performance of the dual-output DNN speech separation systems based on JC1, JC2, and JC3 was evaluated and compared with traditional mapping-based methods (Basic-TMS) and traditional masking-based methods (Basic-IRM). The experimental results are shown in Table 1.

[0118]

[0119] Table 1. Comparison of separation performance between the single-item joint constraint method and the traditional method.

[0120] From the experimental results, we can draw the following conclusions: First, the methods based on JC1, JC2, and JC3 all outperform traditional methods, proving that jointly constraining the estimates and speech signal features of different source signals from different aspects is effective. Second, compared to the JC2 and JC3 methods, the speech signal reconstructed by the JC1 method is closer to the clean speech signal. This can be explained by the fact that we ultimately need to reconstruct the amplitude spectrum of the target speech signal to achieve speech separation. Therefore, jointly constraining the masking value and amplitude spectrum can make the prediction more accurate. Third, compared with mapping-based methods, masking-based methods are more effective, especially when solving the same-sex combination problem. In addition, the proposed method outperforms cross-sex combinations in separating same-sex combinations, because the similarity of same-sex speech signals is higher, making speech signal separation more difficult.

[0121] Next, this invention evaluates the separation performance based on the multi-constraint algorithm. First, to obtain the optimal result of the JC4 algorithm, this invention studies the impact of the regularization coefficients α, β, and γ in the multi-constraint loss function on the separation performance. Based on the above analysis, for different gender combinations, JC1 performs better when α is close to 0.5, and JC1 has better separation performance than JC2 and JC3. Therefore, in the following experiments, α is set to 0.5, and β and γ are set from 0 to 0.5, increasing by 0.1 each time, to sequentially optimize β and γ. The experimental results are shown in Table 2.

[0122]

[0123] Table 2 Performance of the multi-constraint separation system under different regularization coefficients

[0124] As can be seen from the table, PESQ, STOI, and SIR are all affected to varying degrees under different constraint strengths. For example, when α = 0.5, γ = 0, and β ranges from 0 to 0.3, PESQ, STOI, and SIR gradually increase; when β ranges from 0.4 to 0.5, PESQ, STOI, and SIR gradually decrease. Therefore, α and β are set to 0.5 and 0.4 respectively, and γ is optimized further. It can be seen that when γ ranges from 0 to 0.2, PESQ increases with the increase of γ. When γ is greater than 0.2, PESQ decreases with the increase of γ, but the separation performance is still better than when α, β, and γ are all set to 0, proving the effectiveness of the multiple joint constraint method. The experimental results show that the weighting coefficients have a significant impact on speech separation performance. Therefore, solving for the optimal weighting coefficients is of great significance for further improving the performance of the separation system.

[0125] To evaluate the effectiveness of the optimal weighted coefficient vector obtained based on optimization, we compared the speech separation performance of the optimal weighted coefficient combination with that of different weighted combinations of α, β, and γ. Table 2 shows that the optimal weighted coefficient combination method achieves the best separation performance. Specifically, compared to the traditional method, the optimal weighted coefficient combination method improves PESQ by 0.27, STOI by 5%, and SIR by 4.29 dB. Clearly, the results demonstrate that the optimal weighted coefficient vector obtained based on nonlinear programming optimization is more effective than the weighted combination selected in the experiment.

[0126] Furthermore, to evaluate the speech separation performance of dual-output DNNs based on multiple joint constraint loss functions, we conducted a series of experiments using the JC4 method under different gender combinations and compared it with Basic-IRM, JC1, JC2, and JC3 methods. The comparison results are as follows: Figure 3 As shown. From Figure 3 The results from (a) to (c) show that JC4 performs better than other separation methods in different gender combinations.

[0127] To visually demonstrate the separation performance of each method, we selected the test speech waveform of the FM combination to illustrate the performance of the separation system. The results are as follows: Figure 4 As shown in the figure, (a) and (b) are the original speech waveforms of the target female and the target male, respectively. (c) is the mixed speech waveform. (d) and (e) are the estimated female and male speech waveforms separated using the Basic-IRM method. (f) and (g) are the estimated female and male speech waveforms separated using the JC4 method. It can be seen that the speech waveform recovered using the JC4 method is closest to the original reference signal.

[0128] The above results show that the single-channel speech separation method based on a multi-joint constraint dual-output deep neural network proposed in this invention has a significantly improved separation effect compared with traditional speech separation methods, and has certain reference value for practical applications.

[0129] It should be noted that the above embodiments are illustrative of the present invention and not restrictive of the present invention, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims.

Claims

1. A single-channel speech separation method based on deep learning, characterized in that: The single-channel speech separation method is based on a multi-joint-constraint dual-output deep neural network, wherein the deep neural network (DNN) contains one input layer, three hidden layers, and one output layer. The hidden layers use ReLU as the activation function, and the output layer uses sigmoid as the activation function. The method includes the following specific steps: Step 1: Randomly select the speech signals of speaker 1 and speaker 2 from the corpus for preprocessing to obtain a mixed speech signal. Extract the amplitude spectrum features of the clean speech signal and the mixed speech signal, and calculate the training target IRM. Step 2: Using the amplitude spectrum features of the mixed speech signal extracted in the previous steps as the input of the deep neural network (DNN), and the training objective IRM as the output of the DNN, the DNN model is trained under the guidance of joint constraint loss function 1, joint constraint loss function 2, and joint constraint loss function 3, respectively. JC_1 DNN JC_2 DNN JC_3 ; Step 3: Input the amplitude spectrum features of the test mixed speech signals into the DNN. JC_1 DNN JC_2 and DNN JC_3 The corresponding IRM estimate is obtained, the amplitude spectrum characteristics of the estimated target speech signal are calculated, and the target speech signal is reconstructed. Step 4: Integrate the methods based on joint constraint loss function 1, joint constraint loss function 2, and joint constraint loss function 3 into a combined prediction method. Use an optimized algorithm to solve for the optimal weight coefficients as the regularization coefficients of the multiple joint constraint loss functions. Then, use the amplitude spectrum features of the mixed speech signal as the input to the deep neural network (DNN), and the training objective IRM as the output of the DNN. Under the guidance of the multiple joint constraint loss functions, train the DNN model. JC_multi ; Step 5: Input the amplitude spectrum features of the test mixed speech signal into the DNN. JC_multi The corresponding IRM estimate is obtained, the amplitude spectrum features of the estimated target speech signal are calculated, the target speech signal is reconstructed, and the performance of the proposed single-channel speech separation method based on a dual-output deep neural network with multiple joint constraints is evaluated.

2. The single-channel speech separation method based on deep learning according to claim 1, characterized in that: Step 1 includes the following specific process: Step 1-1: Sample the speech signals of speaker 1 and speaker 2 from the previous steps at 16kHz, and then perform pre-emphasis, framing, and windowing operations to obtain the preprocessed speech signal. Steps 1-2: Perform short-time Fourier transform on the preprocessed speech signal and extract amplitude spectrum features; Steps 1-3: Calculate the training target IRM based on the extracted amplitude spectrum features, using the following formula: ; in, Let f be the ideal ratio masking for the i-th target speech signal at time t and frequency f. This represents the amplitude spectrum of the i-th target speech signal at time t and frequency f. It is a very small positive value to prevent the denominator from being 0; Steps 1-4: To facilitate data processing, the amplitude spectrum features of the input mixed speech signal are normalized to obtain the normalized amplitude spectrum features of the training mixed speech signal.

3. The single-channel speech separation method based on deep learning according to claim 1, characterized in that: Step 2 includes the following specific process: Step 2-1: Analyze the amplitude spectrum features of the training mixed speech signal obtained in the previous steps. Using the IRM as input to the DNN and the joint constraint loss function 1, joint constraint loss function 2, and joint constraint loss function 3 as training targets, a fully trained DNN is obtained after 200 iterations. JC_1 DNN JC_2 DNN JC_3 Used to train DNN JC_1 DNN JC_2 DNN JC_3 The loss functions are as follows: ； in, T Indicates the total number of time frames. and These represent the true IRM value and the estimated IRM value of the target speech signal 1 at time t, respectively. and These represent the true IRM value and the estimated IRM value of the target speech signal 2 at time t, respectively. and Let represent the true amplitude spectrum characteristics and the estimated amplitude spectrum characteristics of the target speech signal 1 at time t, respectively. and Let represent the true amplitude spectrum characteristics and the estimated amplitude spectrum characteristics of the target speech signal 2 at time t, respectively. The amplitude spectrum characteristics of the mixed speech signal are represented; α, β, and γ are regularization coefficients, and their values ​​range from [value missing]. Different values ​​mean different levels of constraint. Step 2-2: The training of the DNN model includes two stages: forward propagation and backward propagation. In the forward propagation stage, a rough estimate of the output is obtained by randomly initializing the weights and biases. In the backward propagation stage, the loss function is optimized by using the gradient descent algorithm, and the weights and biases are updated.

4. The single-channel speech separation method based on deep learning according to claim 1, characterized in that: Step 3 includes the following specific process: Step 3-1: First, preprocess the test mixed speech signal, then perform a short-time Fourier transform to extract the amplitude spectrum and phase spectrum features of the test mixed speech signal, and normalize the amplitude spectrum features to obtain... ; Step 3-2: Analyze the amplitude spectrum characteristics of the test mixed speech signal. Input into DNN respectively JC_1 DNN JC_2 and DNN JC_3 The corresponding IRM estimate is obtained, and the estimated target speech amplitude spectrum features are calculated to reconstruct the target speech signal.

5. The single-channel speech separation method based on deep learning according to claim 1, characterized in that: Step 4 includes the following specific process: Step 4-1: Integrate the methods based on joint constraint loss function 1, joint constraint loss function 2, and joint constraint loss function 3 into a combined prediction method, and use an optimized algorithm to solve for the optimal weight coefficients. , which serves as the regularization coefficient for the loss function under multiple joint constraints; Step 4-2: Train the amplitude spectrum features of the mixed speech signal Using the IRM as input to the DNN and the IRM as the training target, the DNN is trained through 200 iterations under the guidance of a multi-joint constraint loss function, resulting in a fully trained DNN. JC_multi Used to train DNN JC_multi The loss function is: ,in, , , , The optimal weight coefficients obtained in the above steps are denoted as .

6. The single-channel speech separation method based on deep learning according to claim 1, characterized in that: Step 5 includes the following specific process: Step 5-1: Test the amplitude spectrum characteristics of the mixed speech signal Input to DNN JC_multi The corresponding IRM estimate is obtained, the amplitude spectrum characteristics of the estimated target speech signal are calculated, and the target speech signal is reconstructed. Step 5-2: Use PESQ, STOI, SDR, SIR, and SAR indices to evaluate the performance of the separation system and obtain the estimated time-domain signal of the target speech.

Images