Speech enhancement method of amplitude-phase mixed feature cross

Abstract

The application discloses a deep learning speech enhancement method based on amplitude-phase mixed feature intersection; according to the collected noisy speech signal, an enhanced mixed intersection feature is obtained; according to the collected clean speech signal and the corresponding noisy speech signal, a label intersection compressed complex mask used for training of an amplitude-phase noise reduction network APNSN is calculated; the enhanced mixed intersection feature is input into the trained APNSN network to obtain an estimated intersection compressed complex mask; according to the estimated intersection compressed complex mask and the spectrum of the noisy speech signal, a time domain reconstructed signal is obtained; compared with a single feature method, such as amplitude spectrum mapping and time-frequency masking based on amplitude spectrum features, the method can further improve speech quality and intelligibility under the condition of the same model size; under a relatively small model, the method can obtain speech quality and intelligibility comparable to the single feature method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of noise cancellation in speech enhancement, and particularly relates to a deep learning speech enhancement method based on mixed amplitude-phase cross features. BACKGROUND

[0002] The classic speech enhancement methods, such as spectral subtraction, Wiener filtering, minimum mean square error estimation, etc., show good noise suppression effect to some extent, but derive the widely known musical noise. In particular, most of the classic speech enhancement methods assume that the noise is additive noise, the noise is a slowly varying process, or the distribution of the noise is a certain specific distribution; when the actual noise does not meet the assumption, the noise reduction performance of these classic methods will be greatly reduced.

[0003] In recent years, deep learning-based speech enhancement methods have shown better speech enhancement effect than classic methods and good application prospects. According to the different learning goals of the network, the deep learning-based speech enhancement methods can be divided into three categories: frequency spectrum mapping-based speech enhancement method, time-frequency masking-based speech enhancement method and signal approximation-based speech enhancement method. However, most of these deep learning-based speech enhancement methods are based on amplitude information to construct network architecture, and the phase information has not been fully developed. In fact, the phase information of speech has a great influence on speech quality and speech intelligibility. Therefore, it is of great application significance to use amplitude and phase as input features at the same time in order to fully utilize the feature information of noisy speech signals. SUMMARY

[0004] The purpose of the present application is to provide a deep learning speech enhancement method based on mixed amplitude-phase cross features, which can further improve speech quality and intelligibility under the condition of the same model size compared with single feature methods such as amplitude spectrum mapping and time-frequency masking based on amplitude spectrum features; and can obtain speech quality and intelligibility comparable to single feature methods under a relatively small model.

[0005] The specific application scheme is as follows:

[0006] A deep learning speech enhancement method based on mixed amplitude-phase cross features, comprising:

[0007] a) obtaining an enhanced mixed cross feature O according to a discrete time domain noisy speech signal y with a length of N;

[0008] b) obtaining a label cross-compressed complex mask L used for training an amplitude-phase noise suppression network (APNSN) according to a discrete time domain clean speech signal s with a length of N and a discrete time domain noisy speech signal y corresponding thereto;

[0009] c) inputting the enhanced hybrid cross feature O into the trained amplitude-phase noise reduction network APNSN to obtain an estimated cross compression complex mask

[0010] d) obtaining a time-domain reconstructed signal according to the estimated cross compression complex mask wherein, the step a) further comprises:

[0011] a1). after transforming the discrete time-domain noisy speech signal y with length N into time-frequency domain, obtaining a noisy signal spectrum Y

[0012] in the noisy signal spectrum Y, T = floor(N / N h )+1 represents the number of frames after transforming into time-frequency domain, wherein N h is the frame shift length of the short time Fourier transform STFT, and floor() represents the down rounding operation on elements;

[0013] in the noisy signal spectrum Y, F = N STFT / 2+1 represents the frequency point number of the noisy speech signal after transforming into time-frequency domain, wherein N STFT is the length of the STFT (Short Time Fourier Transform, STFT) transform;

[0014] a2). according to the noisy signal spectrum Y, obtaining the logarithmic power spectrum of the discrete time-domain noisy speech signal y by using the following formula and phase i.e.

[0015]

[0016] wherein, A[i,j] represents the element of the i-th row and the j-th column of the matrix A; Re(·) and Im(·) represent the real part and the imaginary part of the complex number respectively; and θ[i,j] represents the element of the i-th row and the j-th column of the matrix θ;

[0017] a3). sequentially cross-arranging each column of A and θ to obtain a hybrid cross feature O i.e.

[0018]

[0019] wherein, may be expressed as

[0020]

[0021] wherein, the symbol (·) T represents the transpose operation of the matrix or the vector;

[0022] ​a4). The enhanced hybrid cross feature H is obtained according to the hybrid cross feature H using the following formula i.e.

[0023]

[0024] wherein T' = T - m + 1, F' = 2*m*F, m is a feature expansion coefficient, can be expressed as

[0025]

[0026] wherein can be expressed as

[0027]

[0028] wherein H[x, : ] represents all column elements of the xth row.

[0029] wherein the step b) of generating the training label cross-compression complex mask L of the APNSN network further comprises:

[0030] b1). The discrete time domain clean speech signal s with length N is transformed into time-frequency domain to obtain the clean signal spectrum

[0031] b2). The real part of the complex mask and the imaginary part of the complex mask can be obtained according to the noisy signal spectrum Y and the clean signal spectrum S using the following formula

[0032]

[0033] wherein Y r [i,j] represents the real part of the element of the i th row and the j th column of the noisy signal spectrum Y; Y i [i,j] represents the imaginary part of the element of the i th row and the j th column of the noisy signal spectrum Y; S r [i,j] represents the real part of the element of the i th row and the j th column of the clean signal spectrum S; S i [i,j] represents the imaginary part of the element of the i th row and the j th column of the clean signal spectrum S;

[0034] b3). The compressed complex mask real part R r and the compressed complex mask imaginary part R i are calculated according to the complex mask real part M r and the complex mask imaginary part M i i.e.

[0035]

[0036] wherein, i={1, 2…T}, j={1, 2…F}; subscript x represents r or i; R x ∈[-K, K] is the real part or imaginary part of the compressed complex mask; C is the compression coefficient controlling the steepness of the curve;

[0037] b4). According to the manner of a3) in step a), the real part R r and the imaginary part R i of the compressed complex mask of each column are sequentially crossed to obtain

[0038] b5). According to the following formula, the cross-compressed complex mask of the label used for APNSN training can be obtained i.e.

[0039]

[0040] wherein, m'=(m-1)2, represents the elements of all columns from the m'+1th row to the T-m'th row of the matrix .

[0041] wherein, the network architecture of the amplitude-phase noise reduction network APNSN in step c) and the network training are as follows:

[0042] Network architecture: the amplitude-phase noise reduction network APNSN network is composed of an input block, an intermediate block and an output block;

[0043] The input block is composed of 1 layer of fully connected layer, batch normalization layer and ELU activation function, and the number of nodes is 2N;

[0044] The intermediate block is composed of a feature compression block, a feature reconstruction block and a feature restoration block; wherein, the feature compression block is composed of 3 input blocks, and the number of nodes is N, N / 2, N / 4 in turn; the feature reconstruction module is composed of 2 layers of LSTM, and the number of nodes is N / 4; the feature restoration block is composed of 3 input blocks, and the number of nodes is N / 4, N / 2, N in turn; wherein, the feature compression block and the feature restoration block adopt residual connection;

[0045] The output block is composed of 1 layer of fully connected layer, and the number of nodes is F;

[0046] The training process includes:

[0047] Collect the set of noisy signals and clean signals {y, s} and calculate the corresponding training sample set {O, L} to train the APNSN network, save the network model and its parameters after error convergence;

[0048] The training input of the APNSN network is the enhanced hybrid cross feature O;

[0049] ​The training label of the APNSN network is a cross-compression complex mask L;

[0050] The noisy signal and the clean signal set {y, s} are recorded according to actual application or artificially synthesized.

[0051] Wherein, the step d) of reconstructing the time domain signal further comprises:

[0052] d1). According to the estimated cross-compression complex mask The estimated compression complex mask real part And the estimated compression complex mask imaginary part That is

[0053]

[0054] Wherein, Indicates the matrix All row elements of the pth column; Indicates the matrix All row elements of the qth column; Indicates the matrix All row elements of the uth column; Indicates the matrix All row elements of the vth column;

[0055] d2). According to the estimated compression complex mask real part And the estimated compression complex mask imaginary The estimated complex mask real part And the estimated complex mask imaginary part That is

[0056]

[0057] Wherein, i={1, 2, …, T′}, j={1, 2, …, F}; The subscript x represents r or i;

[0058] d3). According to the estimated complex mask real part And the estimated complex mask imaginary part And the noisy signal spectrum Y, the time-frequency domain estimated signal That is

[0059]

[0060] Wherein, i={1, 2, …, T′}, j={1, 2, …, F}, Y[m′+1:T-m′,:] indicates all column elements of the matrix Y from the m′+1th row to the T-m′th row;

[0061] d4). The time-frequency domain estimated signal After inverse short-time Fourier transform, the time-domain reconstructed signal can be obtained

[0062] The present application has the advantages that, under the condition of the same model size, compared with the single feature method such as amplitude spectrum mapping and time-frequency masking based on amplitude spectrum features, the present application can further improve the speech quality and intelligibility, and can reduce the size of the model to a certain extent. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 The present application is a flowchart;

[0064] Figure 2 The present application is an enhanced hybrid cross feature extraction flowchart;

[0065] Figure 3 The present application is a cross-compression complex mask calculation flowchart;

[0066] Figure 4 The present application is a time-domain reconstruction flowchart. DETAILED DESCRIPTION

[0067] The technical solutions of the present application will be described in further detail below in combination with the drawings, but the protection scope of the present application is not limited to the following description.

[0068] As shown in the following formula, according to the length N of the discrete time-domain noisy signal y, the enhanced hybrid cross feature O can be obtained: Figure 1 The enhanced hybrid cross feature O extraction flowchart is shown in the following formula: Figure 2 The specific implementation is as follows:

[0069] a1). After the length N of the discrete time-domain noisy speech signal y is transformed into the time-frequency domain, the noisy signal spectrum Y is obtained

[0070] In the noisy signal spectrum Y, T = floor(N / N h )+1 represents the number of frames after transformation into the time-frequency domain, wherein N h is the frame shift length of the short-time Fourier transform STFT, and floor() represents the down rounding of elements;

[0071] In the noisy signal spectrum Y, F = N STFT / 2+1 represents the frequency point number of the noisy speech signal after transformation into the time-frequency domain, wherein N STFT is the length of the STFT (Short Time Fourier Transform, STFT) transformation;

[0072] a2). According to the noisy signal spectrum Y, the logarithmic power spectrum of the discrete time-domain noisy speech signal y is obtained by using the following formula and phase Right now

[0073]

[0074] Where A[i,j] represents the element in the i-th row and j-th column of matrix A; Re(·) and Im(·) represent the real and imaginary parts of the complex number, respectively; θ[i,j] represents the element in the i-th row and j-th column of matrix θ;

[0075] a3). By sequentially interleaving each column of A and θ, a mixed interleaving feature is obtained. Right now

[0076]

[0077] in, It can be represented as

[0078]

[0079] Among them, the symbol (·) T This represents the transpose operation of a matrix or vector.

[0080] a4). Using the following formula, based on the hybrid crossover feature H, the enhanced hybrid crossover feature is obtained. Right now

[0081]

[0082] Where T′=T-m+1, F′=2*m*F, and m is the characteristic expansion coefficient. It can be represented as

[0083]

[0084] in, It can be represented as,

[0085]

[0086] Here, H[x,:] represents the elements of all columns in the x-th row.

[0087] Example 1: Step a) is shown below:

[0088] Assumptions: A discrete-time noisy speech signal y = [y1, y2, ..., y] of length N = 1024 1024 Characteristic expansion coefficient m = 3, STFT transform length N STFT =256, STFT frame shift length N h =128, and the spectrum of the noisy signal after STFT. for

[0089]

[0090] The obtained log power spectrum from the noisy signal spectrum Y and phase are respectively

[0091]

[0092]

[0093] From the log power spectrum A and phase θ, further, the hybrid cross feature H can be obtained is

[0094]

[0095] From the hybrid cross feature H, the enhanced hybrid cross feature H can be obtained is

[0096]

[0097] As shown in Figure 1 , the label cross compression complex mask L used in the amplitude-phase noise reduction network APNSN training of step b) is calculated as shown in Figure 3 , and the specific implementation is shown as follows:

[0098] b1). Transform the discrete time-domain clean speech signal s with length N to the time-frequency domain to obtain the clean signal spectrum

[0099] b2). According to the noisy signal spectrum Y and the clean signal spectrum S, the real part of the complex mask and the imaginary part of the complex mask can be obtained by using the following formula, that is

[0100]

[0101] wherein Y r [i,j] represents the real part of the element in the i-th row and j-th column of the noisy signal spectrum Y; Y i [i,j] represents the imaginary part of the element in the i-th row and j-th column of the noisy signal spectrum Y; S r [i,j] represents the real part of the element in the i-th row and j-th column of the clean signal spectrum S; S i [i,j] represents the imaginary part of the element in the i-th row and j-th column of the clean signal spectrum S;

[0102] b3). According to the real part of the complex mask M r and the imaginary part of the complex mask M i , the compressed complex mask real part R r and the compressed complex mask imaginary part R ii.e.

[0103]

[0104] where i={1,2…T}, j={1,2…F}; subscript x represents r or i; R x ∈[-K,K] is the compressed complex mask real part or imaginary part; C is the compression coefficient controlling the steepness of the curve;

[0105] b4). According to the manner of a3) in step a), the compressed complex mask real part R r and the compressed complex mask imaginary part R i are obtained by sequentially crossing each column of the compressed complex mask real part R

[0106] b5). According to the following formula, the cross-compressed complex mask for the APNSN training label can be obtained from i.e.

[0107]

[0108] where m'=(m-1) / 2, represents the elements of all columns from the m'+1th row to the T-m'th row of the matrix .

[0109] Example 2: The step b) is shown as follows:

[0110] Assumption: the length of the discrete time domain clean speech signal s=[s1, s2, …, s 1024 N=1024, constant K=10, compression coefficient C=0.1, STFT transform related parameters and noisy signal spectrum Y are as in Example 1, the clean signal spectrum S after STFT transform is

[0111]

[0112] According to the noisy signal spectrum Y and the clean signal spectrum S, the complex mask real part M and the complex mask imaginary part M are respectively

[0113]

[0114]

[0115] According to the complex mask real part M r and the complex mask imaginary part M i , the compressed complex mask real part R and the compressed complex mask imaginary part R are respectively ​​

[0116]

[0117]

[0118] According to the compressed complex mask real part R r and the compressed complex mask imaginary part R i , the APNSN network training label cross-compressed complex mask L can be obtained , that is

[0119]

[0120] According to , the APNSN network training label cross-compressed complex mask L can be obtained , that is

[0121]

[0122] As shown in Figure 1 , the network architecture and training method of the amplitude-phase noise reduction network APNSN in step c) are respectively:

[0123] The network architecture: the amplitude-phase noise reduction network APNSN network is composed of an input block, an intermediate block and an output block;

[0124] The input block is composed of 1 layer of full connection layer, node number is 2N, batch normalization layer and ELU activation function;

[0125] The intermediate block is composed of a feature compression block, a feature reconstruction block and a feature restoration block; wherein the feature compression block is composed of 3 input blocks, the node numbers are N, N / 2, N / 4 in turn, the feature reconstruction module is composed of 2 layers of LSTM, and the node numbers are N / 4; The feature restoration block is composed of 3 input blocks, and the node numbers are N / 4, N / 2 and N in turn; wherein the feature compression block and the feature restoration block adopt residual connection;

[0126] The output block is composed of 1 layer of full connection layer, and the node number is F;

[0127] The training process includes:

[0128] Collect the noisy signal and clean signal set {y, s} and calculate the corresponding training sample set {O, L} to train the APNSN network, and save the network model and its parameters after error convergence;

[0129] The training input of the APNSN network is the enhanced mixed cross feature O;

[0130] The training label of the APNSN network is the cross-compressed complex mask L;

[0131] The noisy signal and the clean signal set {y, s} are recorded according to practical application or artificially synthesized.

[0132] As shown in Figure 1 , the calculation process of the time domain reconstructed signal of step d) is shown in Figure 4 , and the specific implementation is as follows:

[0133] d1). The estimated compressed complex mask real part and the estimated compressed complex mask imaginary part are calculated according to the estimated cross-compressed complex mask , that is

[0134]

[0135] wherein, represents all row elements of the pth column of the matrix ; and represents all row elements of the qth column of the matrix ; and represents all row elements of the u-th column of the matrix ; and represents all row elements of the v-th column of the matrix ; and

[0136] d2). The estimated complex mask real part and the estimated complex mask imaginary part can be obtained according to the estimated compressed complex mask real part and the estimated compressed complex mask imaginary part , that is

[0137]

[0138] wherein, i={1, 2, …, T′}, j={1, 2, …, F}; the subscript x represents r or i;

[0139] d3). The time-frequency domain estimated signal can be obtained according to the estimated complex mask real part , the estimated complex mask imaginary part , and the noisy signal spectrum Y, that is

[0140]

[0141] wherein, i={1, 2, …, T′}, j={1, 2, …, F}, Y[m′+1:T-m′,:] represents all column elements of the m′+1th row to the T-m′th row of the matrix Y;

[0142] d4). The time-frequency domain estimated signal After inverse short-time Fourier transform, the time-domain reconstructed signal can be obtained

[0143] Example 3: the step d) is shown as follows:

[0144] Assumption: constant K, compression coefficient C, characteristic expansion coefficient m, STFT transform related parameters and noisy signal spectrum Y, which are based on Example 1 and Example 2, estimate the cross-compression complex mask is

[0145]

[0146] According to the estimated cross-compression complex mask the estimated compression complex mask real part is obtained and the estimated compression complex mask imaginary part is obtained respectively

[0147]

[0148] According to and the estimated complex mask real part is further obtained and the estimated complex mask imaginary part is obtained namely

[0149]

[0150]

[0151] According to and the noisy signal spectrum Y, the time-frequency domain estimated signal is obtained is

[0152]

[0153] The time-frequency domain estimated signal After inverse short-time Fourier transform, the time-domain reconstructed signal can be obtained

[0154] It should be noted that those skilled in the art will realize that the embodiments described herein are to help the reader to understand the implementation method of the present application, and should be understood as the protection scope of the present application is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the present application without departing from the essence of the present application, and these modifications and combinations are still within the protection scope of the present application.

Claims

1. A deep learning speech enhancement method based on amplitude-phase mixed feature crossing, characterized in that, The method comprises the following steps: a) obtaining an enhanced mixed cross feature O according to a discrete time domain noisy speech signal y with a length of N; comprising: a1). Transforming the discrete time domain noisy speech signal y of length N into time-frequency domain, obtaining noisy signal spectrum In the noisy signal spectrum Y, T = floor(N / N h )+1 represents the frame number after transformation to the time-frequency domain, where N h is the frame shift length of the short-time Fourier transform STFT, and floor(·) represents rounding down to the element. In the noisy signal spectrum Y, F = N STFT / 2+1 represents the frequency point number of the noisy speech signal transformed into the time-frequency domain, wherein N STFT is the length of the STFT transformation; a2). Obtain the log power spectrum of the discrete time noisy speech signal y from the noisy signal spectrum Y using the following equation and phase i.e. Wherein, A[i,j] represents the element of the i-th row and the j-th column of the matrix A; Re(·) and Im(·) represent the real part and the imaginary part of the complex number respectively; θ[i,j] represents the element of the i-th row and the j-th column of the matrix θ; a3). Each column of A and Θ is arranged in a cross pattern to obtain a mixed cross feature That is wherein may be represented as wherein the notation (·) T denotes the transpose operation of a matrix or vector; a4). Using the following equation, the enhanced hybrid cross feature H is obtained from the hybrid cross feature H i.e. where T' = T - m + 1, F' = 2*m*F, m is a feature expansion coefficient, is represented as wherein denotes that, Wherein, H[x,:] represents the element of all columns of the x-th row; b) obtaining a label cross-compressed complex mask L used for training of an amplitude-phase noise reduction network APNSN according to a discrete time domain clean speech signal s with a length of N and a discrete time domain noisy speech signal y corresponding thereto; c) inputting the enhanced hybrid cross feature O to the trained amplitude- phase noise reduction network APNSN to obtain an estimated cross-compression complex mask d) estimating the cross-compression complex mask obtaining the time-domain reconstructed signal 2. The deep learning speech enhancement method based on amplitude-phase mixed feature intersection according to claim 1, characterized in that, The generating step of the label cross-compressed complex mask L used for training of the APNSN network in step b) further comprises: b1). Transform the discrete time-domain clean speech signal s of length N into the time-frequency domain to obtain the clean signal spectrum b2). Obtain the real part of the complex mask from the noisy signal spectrum Y and the clean signal spectrum S using the following formula and the imaginary part of the complex mask i.e. where Y r [i,j] denotes the real part of the element in the i-th row and j-th column of the noisy signal spectrum Y; Y i [i,j] denotes the imaginary part of the element in the i-th row and j-th column of the noisy signal spectrum Y; S r [i,j] denotes the real part of the element in the i-th row and j-th column of the clean signal spectrum S; S i [i,j] denotes the imaginary part of the element in the i-th row and j-th column of the clean signal spectrum S; b3) Calculate the compressed complex mask real part R r and the compressed complex mask imaginary part R i from the complex mask real part M r and the complex mask imaginary part M i i.e. where i = {1, 2...T}, j = {1, 2...F}; subscript x represents r or i; R x ∈ [-K, K] is the compressed complex mask real or imaginary part; C is the compression coefficient controlling the steepness of the curve; b4). The compressed complex mask real part R r and the compressed complex mask imaginary part R i are arranged in each column in turn to obtain b5). Obtain the label cross-compression complex mask for APNSN training from L' using the following formula i.e. L = L'[m'+1:T-m', : ] Wherein, m'= (m-1) / 2, L'[m'+1:T-m',:] represents the element of all columns of the m'+1-th row to the T-m' row of the matrix L'.

3. The deep learning speech enhancement method based on amplitude-phase mixed feature intersection according to claim 2, characterized in that, The network architecture and network training of the amplitude-phase noise reduction network APNSN in step c) are as follows: The network architecture: the amplitude-phase noise reduction network APNSN network is composed of an input block, an intermediate block and an output block; The input block is composed of 1 layer of full connection layer, node number is 2N, batch normalization layer and ELU activation function; The intermediate block is composed of a feature compression block, a feature reconstruction block and a feature restoration block; wherein, the feature compression block is composed of 3 input blocks, the node numbers are N, N / 2, N / 4 in turn, the feature reconstruction module is composed of 2 layers of LSTM, the node numbers are N / 4 in turn, the feature restoration block is composed of 3 input blocks, the node numbers are N / 4, N / 2, N in turn; wherein, the feature compression block and the feature restoration block adopt residual connection; The output block is composed of 1 layer of full connection layer, the node number is F; The training process comprises: Collecting a noisy signal and clean signal set {y,s} and calculating a corresponding training sample set {O,L} to train the APNSN network, saving the network model and its parameters after error convergence; The training input of the APNSN network is the enhanced mixed cross feature O; The training label of the APNSN network is the cross-compressed complex mask L; The noisy signal and clean signal set {y,s} is obtained according to actual application records and artificial synthesis.

4. The deep learning speech enhancement method based on amplitude-phase mixed feature intersection according to claim 3, characterized in that, Step d) further comprises: d1). Estimate the cross-compression complex mask from the estimated cross-compression real and imaginary parts using the following formula Compute the estimated compression complex mask real part and the estimated compression complex mask imaginary part i.e. wherein represents a matrix all row elements of the pth column d2). Obtain an estimated compressed complex mask real part and an estimated compressed complex mask imaginary part from the estimated compressed complex mask real part and the estimated compressed complex mask imaginary part i.e. Wherein, i={1,2,...,T'}, j={1,2,...,F}, the subscript x represents r or i; d3) using the following equation to estimate the real part of the complex mask and the imaginary part of the complex mask and the noisy signal spectrum Y, to obtain the time-frequency domain estimated signal i.e. wherein i = {1, 2,..., T'}, j = {1, 2,..., F}, Y[m'+1:T-m',:] denotes the elements of all columns from the (m'+1)th row to the (T-m')th row of the matrix Y. d4). The time-frequency domain estimated signal The time domain reconstructed signal is obtained after inverse short-time Fourier transform

Images