Speech processing method and system based on Khatri-Rao subspace

Abstract

The invention relates to the technical field of audio signals, and discloses a voice processing method and system based on Khatri-Rao subspace, and the method comprises the steps: collecting a to-be-processed voice signal; processing the voice signal to be processed by using a Khatri-Rao subspace method to obtain a processed voice signal; constructing and training a speech enhancement model, and adjusting parameters of the speech enhancement model by using a loss function in the training process to obtain a trained speech enhancement model; and inputting the processed voice signal into the trained voice enhancement model to obtain an enhanced voice signal. According to the invention, the noise and echo can be effectively removed, and the naturalness and definition of the voice signal can be maintained and recovered.

Description

Technical Field

[0001] The present invention relates to the technical field of audio signals, and in particular to a speech processing method and system based on Khatri-Rao subspace. Background Art

[0002] In intelligent conference systems and other voice interaction applications, the quality of voice signals is crucial to the clarity and efficiency of communication. However, voice signals in real environments are often interfered by various noise sources, such as background noise, echo, reverberation, etc. These interferences seriously affect the intelligibility and quality of voice signals. In order to improve this problem, researchers have developed a variety of voice activity detection (VAD) technologies and speech enhancement technologies.

[0003] Existing VAD techniques are mainly based on threshold processing and statistical models, which perform well in quiet or low-noise environments, but their performance drops sharply in high-noise or complex acoustic environments. In addition, these methods usually lack consideration of long-term contextual information of speech signals, limiting their effectiveness in practical applications. Although deep learning methods have made progress in VAD, subsequent speech enhancement processing remains a challenge. Traditional speech enhancement techniques, such as spectral subtraction and Wiener filtering, although they can reduce noise, often introduce speech distortion, especially in non-stationary noise environments. Therefore, there is an urgent need for a new speech processing method that can not only effectively remove noise but also maintain or restore the naturalness and clarity of speech signals. Summary of the invention

[0004] The primary purpose of the present invention is to overcome the problems existing in the prior art and provide a speech processing method based on Khatri-Rao subspace, which can effectively remove noise and echo, and can maintain and restore the naturalness and clarity of speech signals.

[0005] As another object of the present invention, a system adapted to the method based on the aforementioned object is provided.

[0006] As another object of the present invention, a non-volatile storage medium suitable for storing a computer program implemented according to the method described is provided.

[0007] In order to achieve the above object, the present invention provides a speech processing method based on Khatri-Rao subspace, the method comprising the following steps: Collecting speech signals to be processed; Processing the speech signal to be processed using the Khatri-Rao subspace method to obtain a processed speech signal; Constructing and training a speech enhancement model, and using a loss function to adjust parameters of the speech enhancement model during the training process to obtain a trained speech enhancement model; The processed speech signal is input into the trained speech enhancement model to obtain an enhanced speech signal.

[0008] Furthermore, the acquisition of the speech signal to be processed is specifically as follows: assuming that the microphone array has array elements, and the array element spacing is , and each array element is the same omnidirectional microphone, there is far-field signal is incident, then the microphone array The output at the moment, that is, the speech signal to be processed as follows: : in, Indicates time Target source complex amplitude vector; represents the spatial noise, The incident angle is The array response matrix of the angle is , is the angular direction of the far-field signal, and its value range is ,and , is the direction vector function, is the far-field signal wavelength.

[0009] Further, the using of the Khatri-Rao subspace method to process the speech signal to be processed to obtain the processed speech signal comprises: Assuming that each far-field signal is generalized quasi-stationary with the frame length, calculate the energy of each far-field signal as follows:

[0010] Obtaining a local covariance matrix of the speech signal to be processed by local averaging; Calculating a data matrix based on the local covariance matrix and the energy of each far-field signal; Performing singular value decomposition on the data matrix to obtain a first decomposition result, wherein the decomposition result includes a left singular matrix, a singular value diagonal matrix, and a right singular matrix; From the left singular matrix select the largest The eigenvectors corresponding to the singular values ​​constitute the signal subspace, and the largest The singular values ​​are taken from the singular value matrix; Solving the angle according to the signal subspace; Design a filter based on the angle, input the speech signal to be processed into the filter, and obtain Processed speech signal.

[0011] Furthermore, the local covariance matrix of the speech signal to be processed is obtained by local averaging, and the calculation method is as follows:

[0012] Where L is the frame length.

[0013] Further, solving the angle according to the signal subspace specifically includes: Extract the previous Line and after OK, take out the front The first matrix is ​​recorded as the first row, and the latter is taken out The rows are denoted as the second matrix, According to the first matrix and the second matrix, a least square method is used to calculate to obtain a third matrix; Perform eigenvalue decomposition on the third matrix to obtain a second decomposition result, wherein the second decomposition result includes eigenvalues; Based on the The eigenvalues ​​are used to calculate the angle.

[0014] Furthermore, the angle is calculated based on the characteristic value, and the calculation method is as follows:

[0015] in, is the characteristic value.

[0016] Furthermore, the speech enhancement model is constructed and trained, and a loss function is used during the training process to adjust the parameters of the speech enhancement model to obtain a trained speech enhancement model, including: Constructing a speech enhancement model, wherein the speech enhancement model includes an LSTM-RNN network and a linear transformation module; Collecting a data set, the data set includes original acoustic features of a speech signal, and dividing the data set into a training set and a validation set in a ratio of 7:3; The training set is input into the speech enhancement model to obtain the acoustic features after denoising, and based on the original acoustic features and the acoustic features after denoising, the error is calculated using the loss function, and the parameters of the speech enhancement model are optimized and adjusted using the back propagation method with minimizing the error as the optimization goal, and the above iterative training process is continuously repeated until a preset number of iterative training times is reached or the loss value converges, thereby obtaining a trained speech enhancement model; The validation set is used to validate the performance of the trained speech model.

[0017] Furthermore, the loss function is as follows:

[0018] in, is the training set, is the original acoustic feature, is the acoustic feature after denoising.

[0019] In order to achieve another object of the present invention, the present invention further provides a speech processing system based on Khatri-Rao subspace, the system is based on the speech processing method based on Khatri-Rao subspace, comprising: Acquisition module: used to collect the voice signal to be processed; A first processing module: used for processing the speech signal to be processed by using the Khatri-Rao subspace method to obtain a processed speech signal; Training module: used to build and train a speech enhancement model. During the training process, a loss function is used to adjust the parameters of the speech enhancement model to obtain a trained speech enhancement model. The second processing module is used to input the processed speech signal into the trained speech enhancement model to obtain an enhanced speech signal.

[0020] In order to achieve another object of the present invention, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer-stored program is executed by a processor, the speech processing method based on Khatri-Rao subspace is implemented.

[0021] Compared with the prior art, the present invention has the following beneficial effects: The present invention uses the Khatri-Rao subspace to process speech signals, and can accurately estimate the direction of arrival, so as to better distinguish the target signal from the echo signal, and effectively eliminate the echo signal; and also constructs a speech enhancement model, which is trained with the goal of minimizing the error between the original acoustic features and the acoustic features after denoising, so as to obtain a trained model, and inputs the speech signal processed by the Khatri-Rao subspace into the trained model, so as to effectively remove noise, and maintain and restore the naturalness and clarity of the speech signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of a speech processing method based on Khatri-Rao subspace according to Embodiment 1 of the present invention; Figure 2is a block diagram of a speech processing system based on Khatri-Rao subspace according to Embodiment 2 of the present invention; Figure 3 This is a diagram of a microphone array according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0023] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0024] In the description of the present invention, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0025] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0026] Furthermore, in the description of the present invention, unless otherwise specified, “plurality” means two or more.

[0027] In order to facilitate ordinary technicians in this field to read this application, some professional terms are explained first: The Khatri-Rao product is the Kronecker product of the corresponding column vectors of two matrices with the same number of columns. If there are two matrices A and B, where A is an I×K matrix and B is a J×K matrix, then their Khatri-Rao product A⊙B is a (IJ)×K matrix, defined as: A⊙B=[a1⊗b1,a2⊗b2,…,a K ⊗b K ], where a i and b iare the i-th column vectors of A and B respectively, ⊗ represents the Kronecker product, and ⊙ represents the Kronecker product of matrices. The Khatri-Rao product is a special matrix operation.

[0028] Represents the operation of converting a matrix into a vector. Specifically, if a matrix ,So .

[0029] Example 1 like Figure 1 As shown, a speech processing method based on Khatri-Rao subspace according to a preferred embodiment of the present invention comprises: S1: Collect the speech signal to be processed; In a feasible embodiment, a microphone array is used to collect a mixed signal including the target speech and the echo, specifically, as Figure 3 As shown in the microphone array diagram, assume that the microphone array has array elements, and the array element spacing is , and each array element is the same omnidirectional microphone, there is far-field signal is incident, then the microphone array The output at the moment, that is, the speech signal to be processed as follows: : in, Indicates time Target source complex amplitude vector; represents the spatial noise, The incident angle is The array response matrix of the angle is , is the angular direction of the far-field signal, and its value range is ,and , is the direction vector function, is the far-field signal wavelength.

[0030] S2: Processing the speech signal to be processed using the Khatri-Rao subspace method to obtain a processed speech signal; In a feasible embodiment, S2 specifically includes: S2.1: Assume that each far-field signal is generalized quasi-stationary with the frame length, and calculate the energy of each far-field signal as follows:

[0031] in, is the frame length, and , is the number of frames, For the A far-field signal, namely the source signal.

[0032] S2.2: Obtaining a local covariance matrix of the speech signal to be processed by local averaging; The calculation method is as follows:

[0033] Where L is the frame length.

[0034] S2.3: Calculate a data matrix based on the local covariance matrix and the energy of each far-field signal; Calculate the data matrix based on the local covariance matrix and the energy of each far-field signal , Represents the operation of converting a matrix into a vector. The specific process is as follows: The Khatri-Rao product of vec has the following properties:

[0035] From this we can get:

[0036] because[ ] = Y, so

[0037] in, = [1,...,1]T∈ ,

[0038] S2.4: performing singular value decomposition on the data matrix to obtain a first decomposition result, wherein the decomposition result includes a left singular matrix, a singular value diagonal matrix, and a right singular matrix; Specifically, for the data matrix Perform singular value decomposition (SVD), ,in, is a left singular matrix, is a diagonal matrix of singular values, is a right singular matrix S2.5: From the left singular matrix Select the one with the largest The eigenvectors corresponding to the singular values ​​constitute the signal subspace, and the largest The singular values ​​are taken from the singular value diagonal matrix ; S2.6: solving the angle according to the signal subspace; The calculation is as follows:

[0039] in, is the characteristic value.

[0040] S2.7: Design a filter based on the angle, input the speech signal to be processed into the filter, and obtain a processed speech signal.

[0041] S3: construct and train a speech enhancement model, and use a loss function to adjust the parameters of the speech enhancement model during the training process to obtain a trained speech enhancement model; In a feasible embodiment, S3 specifically includes: S3.1: construct a speech enhancement model, wherein the speech enhancement model includes an LSTM-RNN network and a linear transformation module; S3.2: Collect a data set, the data set including original acoustic features of a speech signal, and divide the data set into a training set and a validation set in a ratio of 7:3; S3.3: Input the training set into the speech enhancement model to obtain the denoised acoustic features. Based on the original acoustic features and the denoised acoustic features, use the loss function to calculate the error. Minimizing the error is the optimization goal. The back propagation method is used to optimize and adjust the parameters of the speech enhancement model. The above iterative training process is repeated until a preset number of iterative training times is reached or the loss value converges, and a trained speech enhancement model is obtained. Specifically, the original acoustic features are first input into the LSTM-RNN network, and the processing process is as follows:

[0042] in, , is the context vector, which contains the effective information for estimating the denoising features of the tth frame; then, Input linear transformation module, which includes a linear transformation function ,The specific processing process is as follows:

[0043] in, is the acoustic feature after denoising, are model parameters; The loss function is as follows:

[0044] in, is the training set, is the original acoustic feature, is the acoustic feature after denoising.

[0045] S3.4: Use the validation set to verify the performance of the trained speech model.

[0046] S4: Inputting the processed speech signal into the trained speech enhancement model to obtain an enhanced speech signal.

[0047] Example 2 like Figure 2 As shown, a speech processing system based on Khatri-Rao subspace according to an embodiment of the present invention includes: Acquisition module: used to collect the voice signal to be processed; A first processing module: used for processing the speech signal to be processed by using the Khatri-Rao subspace method to obtain a processed speech signal; Training module: used to build and train a speech enhancement model. During the training process, a loss function is used to adjust the parameters of the speech enhancement model to obtain a trained speech enhancement model. The second processing module is used to input the processed speech signal into the trained speech enhancement model to obtain an enhanced speech signal.

[0048] The system proposed in this embodiment is based on the speech processing method based on Khatri-Rao subspace proposed in Embodiment 1. Therefore, the optional items proposed in Embodiment 1 are also applicable to this embodiment and will not be described in detail here.

[0049] This embodiment uses the Khatri-Rao subspace to process the speech signal, so that the direction of arrival can be accurately estimated, so as to better distinguish the target signal from the echo signal and effectively eliminate the echo signal; and also constructs a speech enhancement model, which is trained with the goal of minimizing the error between the original acoustic features and the acoustic features after denoising, so as to obtain a trained model. The speech signal processed by the Khatri-Rao subspace is input into the trained model, so that the noise can be effectively removed, and the naturalness and clarity of the speech signal can be maintained and restored.

[0050] Example 3 An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer-stored program is executed by a processor, the speech processing method based on Khatri-Rao subspace is implemented.

[0051] In summary, the embodiments of the present invention provide a speech processing method and system based on Khatri-Rao subspace, which can accurately estimate the direction of arrival by using Khatri-Rao subspace to process speech signals, thereby better distinguishing target signals from echo signals and effectively eliminating echo signals; and also by constructing a speech enhancement model, which is trained with the goal of minimizing the error between original acoustic features and denoised acoustic features, thereby obtaining a trained model, and inputting the speech signal processed by Khatri-Rao subspace into the trained model, which can effectively remove noise, and can maintain and restore the naturalness and clarity of the speech signal.

[0052] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A speech processing method based on Khatri-Rao subspace, characterized in that: The method comprises the following steps: Collecting speech signals to be processed; Processing the speech signal to be processed using the Khatri-Rao subspace method to obtain a processed speech signal; Constructing and training a speech enhancement model, and using a loss function to adjust parameters of the speech enhancement model during the training process to obtain a trained speech enhancement model; The processed speech signal is input into the trained speech enhancement model to obtain an enhanced speech signal.

2. A speech processing method based on Khatri-Rao subspace according to claim 1, characterized in that: The method of collecting the speech signal to be processed is specifically as follows: assuming that the microphone array has array elements, and the array element spacing is , and each array element is the same omnidirectional microphone, there is far-field signal is incident, then the microphone array The output at the moment, that is, the speech signal to be processed as follows: ： in, Indicates time Target source complex amplitude vector; represents the spatial noise, The incident angle is The array response matrix of the angle is , is the angular direction of the far-field signal, and its value range is ,and , is the direction vector function, is the far-field signal wavelength.

3. A speech processing method based on Khatri-Rao subspace according to claim 2, wherein The characteristic is that The method of using the Khatri-Rao subspace method to process the speech signal to be processed to obtain the processed speech signal comprises: Assuming that each far-field signal is generalized quasi-stationary with the frame length, calculate the energy of each far-field signal as follows: Obtaining a local covariance matrix of the speech signal to be processed by local averaging; Calculating a data matrix based on the local covariance matrix and the energy of each far-field signal; Performing singular value decomposition on the data matrix to obtain a first decomposition result, wherein the decomposition result includes a left singular matrix, a singular value diagonal matrix, and a right singular matrix; From the left singular matrix select the largest The eigenvectors corresponding to the singular values ​​constitute the signal subspace, and the largest The singular values ​​are taken from the singular value matrix; Solving the angle according to the signal subspace; Design a filter based on the angle, input the speech signal to be processed into the filter, and obtain Processed speech signal.

4. A speech processing method based on Khatri-Rao subspace according to claim 3, It is characterized in that The local covariance matrix of the speech signal to be processed is obtained by local averaging, and the calculation method is as follows: Where L is the frame length.

5. A speech processing method based on Khatri-Rao subspace according to claim 3, It is characterized in that The solving the angle according to the signal subspace specifically includes: Extract the previous Line and after OK, take out the front The first matrix is ​​recorded as the first row, and the latter is taken out The rows are denoted as the second matrix, According to the first matrix and the second matrix, a least square method is used to calculate to obtain a third matrix; Perform eigenvalue decomposition on the third matrix to obtain a second decomposition result, wherein the second decomposition result includes eigenvalues; Based on the The eigenvalues ​​are used to calculate the angle.

6. A speech processing method based on Khatri-Rao subspace according to claim 4, It is characterized in that The angle is calculated based on the characteristic value, and the calculation method is as follows: in, is the characteristic value.

7. The speech processing method based on Khatri-Rao subspace according to claim 1, characterized in that: The speech enhancement model is constructed and trained, and a loss function is used during the training process to adjust the parameters of the speech enhancement model to obtain a trained speech enhancement model, including: Constructing a speech enhancement model, wherein the speech enhancement model includes an LSTM-RNN network and a linear transformation module; Collecting a data set, the data set includes original acoustic features of a speech signal, and dividing the data set into a training set and a validation set in a ratio of 7:3; The training set is input into the speech enhancement model to obtain the acoustic features after denoising, and based on the original acoustic features and the acoustic features after denoising, the error is calculated using the loss function, and the parameters of the speech enhancement model are optimized and adjusted using the back propagation method with minimizing the error as the optimization goal, and the above iterative training process is continuously repeated until a preset number of iterative training times is reached or the loss value converges, thereby obtaining a trained speech enhancement model; The validation set is used to validate the performance of the trained speech model.

8. The method for speech processing based on Khatri-Rao subspace according to claim 7, characterized in that: The loss function is as follows: in, is the training set, is the original acoustic feature, is the acoustic feature after denoising.

9. A speech processing system based on Khatri-Rao subspace, the system being based on a speech processing method based on Khatri-Rao subspace according to any one of claims 1 to 8, comprising: Acquisition module: used to collect the voice signal to be processed; A first processing module: used for processing the speech signal to be processed by using the Khatri-Rao subspace method to obtain a processed speech signal; Training module: used to build and train a speech enhancement model. During the training process, a loss function is used to adjust the parameters of the speech enhancement model to obtain a trained speech enhancement model. The second processing module is used to input the processed speech signal into the trained speech enhancement model to obtain an enhanced speech signal.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer stored program is executed by a processor, the speech processing method based on Khatri-Rao subspace described in any one of claims 1 to 8 is implemented.

Images