A speech processing method and system based on Khatri-Rao subspace

Abstract

The present invention relates to the field of audio signal technology and discloses a speech processing method and system based on Khatri-Rao subspace. The method comprises: acquiring a speech signal to be processed; processing the speech signal using the Khatri-Rao subspace method to obtain a processed speech signal; constructing and training a speech enhancement model, using a loss function to adjust parameters of the speech enhancement model during the training process to obtain a trained speech enhancement model; and inputting the processed speech signal into the trained speech enhancement model to obtain an enhanced speech signal. The present invention can effectively remove noise and echo while maintaining and restoring the naturalness and clarity of the speech signal.

Description

Technical Field

[0001] The present invention relates to the technical field of audio signals, and in particular to a speech processing method and system based on Khatri-Rao subspace. Background Art

[0002] In intelligent conferencing systems and other voice interaction applications, voice signal quality is crucial for communication clarity and efficiency. However, in real-world environments, voice signals are often disrupted by various noise sources, such as background noise, echo, and reverberation. These interferences severely impact voice signal intelligibility and quality. To address this issue, researchers have developed various voice activity detection (VAD) and speech enhancement techniques.

[0003] Existing VAD technologies are mainly based on threshold processing and statistical models. They perform well in quiet or low-noise environments, but their performance drops sharply in high-noise or complex acoustic environments. In addition, these methods generally lack consideration of the long-term contextual information of the speech signal, limiting their effectiveness in practical applications. Although deep learning methods have made progress in VAD, subsequent speech enhancement processing remains a challenge. Traditional speech enhancement techniques, such as spectral subtraction and Wiener filtering, although they can reduce noise, often introduce speech distortion, especially in non-stationary noise environments. Therefore, there is an urgent need for a new speech processing method that can not only effectively remove noise but also maintain or restore the naturalness and clarity of the speech signal. Summary of the Invention

[0004] The primary purpose of the present invention is to overcome the problems existing in the prior art and provide a speech processing method based on Khatri-Rao subspace, which can effectively remove noise and echo, and can maintain and restore the naturalness and clarity of the speech signal.

[0005] As another object of the present invention, a system adapted to the method according to the aforementioned object is also provided.

[0006] As another object of the present invention, a non-volatile storage medium suitable for storing a computer program implemented according to the method described is provided.

[0007] In order to achieve the above object, the present invention provides a speech processing method based on Khatri-Rao subspace, the method comprising the following steps:

[0008] Collecting voice signals to be processed;

[0009] The method of processing the speech signal to be processed by using the Khatri-Rao subspace method to obtain the processed speech signal specifically includes:

[0010] Assuming that each far-field signal is generalized quasi-stationary with the frame length, calculate the energy of each far-field signal as follows:

[0011] ;

[0012] in, For the A far-field signal, the source signal, is the frame length;

[0013] Obtaining a local covariance matrix of the speech signal to be processed by local averaging;

[0014] Calculating a data matrix based on the local covariance matrix and the energy of each far-field signal;

[0015] Performing singular value decomposition on the data matrix to obtain a first decomposition result, wherein the decomposition result includes a left singular matrix, a singular value diagonal matrix, and a right singular matrix;

[0016] From the left singular matrix, select the largest The eigenvectors corresponding to the singular values ​​constitute the signal subspace, and the largest The singular values ​​are taken from the singular value diagonal matrix;

[0017] solving for an angle based on the signal subspace;

[0018] Design a filter based on the angle, input the speech signal to be processed into the filter, and obtain

[0019] processed speech signal;

[0020] Constructing and training a speech enhancement model, and using a loss function to adjust the parameters of the speech enhancement model during the training process to obtain a trained speech enhancement model;

[0021] The processed speech signal is input into the trained speech enhancement model to obtain an enhanced speech signal.

[0022] Furthermore, the acquisition of the speech signal to be processed is specifically as follows: assuming that the microphone array has array elements, and the array element spacing is , and each array element is the same omnidirectional microphone, there is far-field signal is incident, the microphone array is The output at the moment, that is, the speech signal to be processed as follows:

[0023] :

[0024] in, Indicates time Target source complex amplitude vector; represents the spatial noise, The incident angle is The array response matrix of the angle is , is the angular direction of the far-field signal, and its value range is ,and , is the direction vector function, is the far-field signal wavelength.

[0025] Furthermore, the local covariance matrix of the speech signal to be processed is obtained by local averaging, and the calculation method is as follows:

[0026]

[0027] Where L is the frame length.

[0028] Furthermore, solving the angle according to the signal subspace specifically includes:

[0029] Extract the previous Line and back OK, take out the front The first matrix is ​​recorded as the first row, and the The rows are denoted as the second matrix, is the number of frames;

[0030] Calculate the first matrix and the second matrix using the least square method to obtain a third matrix;

[0031] Perform eigenvalue decomposition on the third matrix to obtain a second decomposition result, wherein the second decomposition result includes eigenvalues;

[0032] Based on the Calculate the angle of the eigenvalues.

[0033] Furthermore, the angle is calculated based on the eigenvalue in the following manner:

[0034]

[0035] in, is the characteristic value.

[0036] Furthermore, the speech enhancement model is constructed and trained, and a loss function is used during the training process to adjust the parameters of the speech enhancement model to obtain a trained speech enhancement model, including:

[0037] Constructing a speech enhancement model, wherein the speech enhancement model includes an LSTM-RNN network and a linear transformation module;

[0038] Collecting a data set, the data set including original acoustic features of a speech signal, and dividing the data set into a training set and a validation set in a ratio of 7:3;

[0039] Inputting the training set into the speech enhancement model to obtain denoised acoustic features, calculating errors based on the original acoustic features and the denoised acoustic features using a loss function, optimizing and adjusting the parameters of the speech enhancement model using a backpropagation method with minimizing the error as an optimization goal, and continuously repeating the iterative training process until a preset number of iterative training times is reached or the loss value converges, thereby obtaining a trained speech enhancement model;

[0040] The performance of the trained speech model is verified using the validation set.

[0041] Furthermore, the loss function is as follows:

[0042]

[0043] in, is the training set, is the original acoustic feature, is the acoustic feature after denoising, , are model parameters.

[0044] In order to achieve another object of the present invention, the present invention further provides a speech processing system based on Khatri-Rao subspace, wherein the system is based on the speech processing method based on Khatri-Rao subspace, and comprises:

[0045] Acquisition module: used to collect voice signals to be processed;

[0046] A first processing module is configured to process the speech signal to be processed using a Khatri-Rao subspace method to obtain a processed speech signal;

[0047] Training module: used to build and train the speech enhancement model. During the training process, the loss function is used to adjust the parameters of the speech enhancement model to obtain a trained speech enhancement model.

[0048] The second processing module is used to input the processed speech signal into the trained speech enhancement model to obtain an enhanced speech signal.

[0049] In order to achieve another object of the present invention, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer-stored program is executed by a processor, the speech processing method based on Khatri-Rao subspace is implemented.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The present invention uses the Khatri-Rao subspace to process speech signals, thereby accurately estimating the direction of arrival, thereby better distinguishing between target signals and echo signals, and effectively eliminating echo signals. Furthermore, by constructing a speech enhancement model that is trained with the goal of minimizing the error between original acoustic features and denoised acoustic features, thereby obtaining a trained model, the speech signal processed by the Khatri-Rao subspace is input into the trained model, thereby effectively removing noise and maintaining and restoring the naturalness and clarity of the speech signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flowchart of a speech processing method based on Khatri-Rao subspace according to embodiment 1 of the present invention;

[0053] Figure 2 This is a block diagram of a speech processing system based on Khatri-Rao subspace according to embodiment 2 of the present invention;

[0054] Figure 3 This is a diagram of the microphone array of Example 1 of the present invention. DETAILED DESCRIPTION

[0055] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0056] In the description of the present invention, it should be noted that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" and the like, indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate the description of the present invention and simplify the description. They are not intended to indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0057] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0058] Furthermore, in the description of the present invention, unless otherwise specified, “plurality” means two or more.

[0059] To facilitate those skilled in the art to read this application, some professional terms are explained first:

[0060] The Khatri-Rao product is the permutation of the Kronecker product of the corresponding column vectors of two matrices with the same number of columns. If there are two matrices A and B, where A is an I×K matrix and B is a J×K matrix, then their Khatri-Rao product A⊙B is a (IJ)×K matrix, defined as: A⊙B=[a1⊗b1,a2⊗b2,…,a K ⊗b K ], where a i and b i are the i-th column vectors of A and B respectively, ⊗ represents the Kronecker product, and ⊙ represents the Kronecker product of matrices. The Khatri-Rao product is a special matrix operation.

[0061] Represents the operation of converting a matrix into a vector. Specifically, if a matrix ,So .

[0062] Example 1

[0063] like Figure 1 As shown, a speech processing method based on Khatri-Rao subspace according to a preferred embodiment of the present invention includes:

[0064] S1: collect the voice signal to be processed;

[0065] In a feasible embodiment, a microphone array is used to collect a mixed signal containing target speech and echo, specifically, as Figure 3 As shown in the microphone array diagram, assume that the microphone array has array elements, and the array element spacing is , and each array element is the same omnidirectional microphone, there is far-field signal is incident, the microphone array is The output at the moment, that is, the speech signal to be processed as follows:

[0066] :

[0067] in, Indicates time Target source complex amplitude vector; represents the spatial noise, The incident angle is The array response matrix of the angle is , is the angular direction of the far-field signal, and its value range is ,and , is the direction vector function, is the far-field signal wavelength.

[0068] S2: Processing the speech signal to be processed using the Khatri-Rao subspace method to obtain a processed speech signal;

[0069] In a feasible embodiment, S2 specifically includes:

[0070] S2.1: Assume that each far-field signal is generalized quasi-stationary with respect to the frame length. Calculate the energy of each far-field signal as follows:

[0071]

[0072] in, is the frame length, and , is the number of frames, For the A far-field signal, namely the source signal.

[0073] S2.2: Obtaining a local covariance matrix of the speech signal to be processed by local averaging;

[0074] The calculation method is as follows:

[0075]

[0076] Where L is the frame length.

[0077] S2.3: Calculate a data matrix based on the local covariance matrix and the energy of each far-field signal;

[0078] Calculate the data matrix based on the local covariance matrix and the energy of each far-field signal , Represents the operation of converting a matrix into a vector. The specific process is as follows:

[0079] The Khatri-Rao product of vec has the following properties:

[0080]

[0081] From this we can get:

[0082]

[0083] because[ ] = Y, so

[0084]

[0085] in, = [1,...,1]T∈ ,

[0086]

[0087] S2.4: Perform singular value decomposition on the data matrix to obtain a first decomposition result, wherein the decomposition result includes a left singular matrix, a singular value diagonal matrix, and a right singular matrix;

[0088] Specifically, for the data matrix Perform singular value decomposition (SVD), ,in, is a left singular matrix, is a diagonal matrix of singular values, is a right singular matrix

[0089] S2.5: From the left singular matrix Select the one with the largest The eigenvectors corresponding to the singular values ​​constitute the signal subspace, and the largest The singular values ​​are taken from the singular value diagonal matrix ;

[0090] S2.6: Determine the angle based on the signal subspace;

[0091] The calculation is as follows:

[0092]

[0093] in, is the characteristic value.

[0094] S2.7: Design a filter based on the angle, input the speech signal to be processed into the filter, and obtain a processed speech signal.

[0095] S3: Build and train a speech enhancement model, and use a loss function to adjust the parameters of the speech enhancement model during the training process to obtain a trained speech enhancement model;

[0096] In a feasible embodiment, S3 specifically includes:

[0097] S3.1: Construct a speech enhancement model, wherein the speech enhancement model includes an LSTM-RNN network and a linear transformation module;

[0098] S3.2: Collect a dataset, the dataset including original acoustic features of speech signals, and divide the dataset into a training set and a validation set in a ratio of 7:3;

[0099] S3.3: Input the training set into the speech enhancement model to obtain denoised acoustic features. Based on the original acoustic features and the denoised acoustic features, calculate the error using a loss function. Minimizing the error is the optimization goal. The parameters of the speech enhancement model are optimized and adjusted using a backpropagation method. The above iterative training process is repeated until a preset number of iterative training times is reached or the loss value converges, thereby obtaining a trained speech enhancement model. Specifically, the original acoustic features are first input into the LSTM-RNN network, and the processing process is as follows:

[0100]

[0101] in, , is the context vector, which contains the effective information for estimating the denoising features of the t-th frame; then, Input linear transformation module, which includes a linear transformation function ,The specific processing process is as follows:

[0102]

[0103] in, is the acoustic feature after denoising, are model parameters;

[0104] The loss function is as follows:

[0105]

[0106] in, is the training set, is the original acoustic feature, is the acoustic feature after denoising.

[0107] S3.4: Use the validation set to verify the performance of the trained speech model.

[0108] S4: Inputting the processed speech signal into the trained speech enhancement model to obtain an enhanced speech signal.

[0109] Example 2

[0110] like Figure 2 As shown, a speech processing system based on Khatri-Rao subspace according to an embodiment of the present invention includes:

[0111] Acquisition module: used to collect voice signals to be processed;

[0112] A first processing module is configured to process the speech signal to be processed using a Khatri-Rao subspace method to obtain a processed speech signal;

[0113] Training module: used to build and train the speech enhancement model. During the training process, the loss function is used to adjust the parameters of the speech enhancement model to obtain a trained speech enhancement model.

[0114] The second processing module is used to input the processed speech signal into the trained speech enhancement model to obtain an enhanced speech signal.

[0115] The system proposed in this embodiment is based on the speech processing method based on Khatri-Rao subspace proposed in Example 1. Therefore, the options proposed in Example 1 are also applicable to this embodiment and will not be repeated here.

[0116] This embodiment uses the Khatri-Rao subspace to process speech signals, enabling accurate direction of arrival estimation, thereby better distinguishing target signals from echo signals and effectively eliminating echo signals. Furthermore, by constructing a speech enhancement model that is trained to minimize the error between original acoustic features and denoised acoustic features, thereby obtaining a trained model, and inputting the speech signal processed by the Khatri-Rao subspace into the trained model, noise can be effectively removed while maintaining and restoring the naturalness and clarity of the speech signal.

[0117] Example 3

[0118] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer-stored program is executed by a processor, the speech processing method based on Khatri-Rao subspace is implemented.

[0119] In summary, the embodiments of the present invention provide a speech processing method and system based on the Khatri-Rao subspace. By using the Khatri-Rao subspace to process speech signals, the method and system can accurately estimate the direction of arrival, thereby better distinguishing the target signal from the echo signal and effectively eliminating the echo signal. The method also constructs a speech enhancement model that is trained with the goal of minimizing the error between the original acoustic features and the denoised acoustic features, thereby obtaining a trained model. The speech signal processed by the Khatri-Rao subspace is input into the trained model, thereby effectively removing noise and maintaining and restoring the naturalness and clarity of the speech signal.

[0120] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A speech processing method based on Khatri-Rao subspace, characterized in that: The method comprises the following steps: Collecting voice signals to be processed; The method of processing the speech signal to be processed by using the Khatri-Rao subspace method to obtain the processed speech signal specifically includes: Assuming that each far-field signal is generalized quasi-stationary with the frame length, calculate the energy of each far-field signal as follows: ； in, For the A far-field signal, the source signal, is the frame length; Obtaining a local covariance matrix of the speech signal to be processed by local averaging; Calculating a data matrix based on the local covariance matrix and the energy of each far-field signal; Performing singular value decomposition on the data matrix to obtain a first decomposition result, wherein the decomposition result includes a left singular matrix, a singular value diagonal matrix, and a right singular matrix; From the left singular matrix, select the largest The eigenvectors corresponding to the singular values ​​constitute the signal subspace, and the largest The singular values ​​are taken from the singular value diagonal matrix; solving for an angle based on the signal subspace; Design a filter based on the angle, input the speech signal to be processed into the filter, and obtain processed speech signal; Constructing and training a speech enhancement model, and using a loss function to adjust the parameters of the speech enhancement model during the training process to obtain a trained speech enhancement model; The processed speech signal is input into the trained speech enhancement model to obtain an enhanced speech signal.

2. A speech processing method based on Khatri-Rao subspace according to claim 1, characterized in that: The method of collecting the speech signal to be processed is as follows: assuming that the microphone array has array elements, and the array element spacing is , and each array element is the same omnidirectional microphone, there is far-field signal is incident, the microphone array is The output at the moment, that is, the speech signal to be processed as follows: ： in, Indicates time Target source complex amplitude vector; represents the spatial noise, The incident angle is The array response matrix of the angle is , is the angular direction of the far-field signal, and its value range is ,and , is the direction vector function, is the far-field signal wavelength.

3. A speech processing method based on Khatri-Rao subspace according to claim 2, It is characterized in that The local covariance matrix of the speech signal to be processed is obtained by local averaging, and the calculation method is as follows: Where L is the frame length.

4. A speech processing method based on Khatri-Rao subspace according to claim 3, It is characterized in that Solving the angle according to the signal subspace specifically includes: Extract the previous Line and back OK, take out the front The first matrix is ​​recorded as the first row, and the The rows are denoted as the second matrix, is the number of frames; Calculate the first matrix and the second matrix using the least square method to obtain a third matrix; Perform eigenvalue decomposition on the third matrix to obtain a second decomposition result, wherein the second decomposition result includes eigenvalues; Based on the Calculate the angle of the eigenvalues.

5. A speech processing method based on Khatri-Rao subspace according to claim 4, It is characterized in that The angle is calculated based on the characteristic value, and the calculation method is as follows: in, is the characteristic value.

6. The speech processing method based on Khatri-Rao subspace according to claim 1, characterized in that: The speech enhancement model is constructed and trained, and a loss function is used to adjust the parameters of the speech enhancement model during the training process to obtain a trained speech enhancement model, including: Constructing a speech enhancement model, wherein the speech enhancement model includes an LSTM-RNN network and a linear transformation module; Collecting a data set, the data set including original acoustic features of a speech signal, and dividing the data set into a training set and a validation set in a ratio of 7:3; Inputting the training set into the speech enhancement model to obtain denoised acoustic features, calculating errors based on the original acoustic features and the denoised acoustic features using a loss function, optimizing and adjusting the parameters of the speech enhancement model using a backpropagation method with minimizing the error as an optimization goal, and continuously repeating the iterative training process until a preset number of iterative training times is reached or the loss value converges, thereby obtaining a trained speech enhancement model; The performance of the trained speech model is verified using the validation set.

7. The speech processing method based on Khatri-Rao subspace according to claim 6, characterized in that: The loss function is as follows: in, is the training set, is the original acoustic feature, is the acoustic feature after denoising, , are model parameters.

8. A speech processing system based on Khatri-Rao subspace, the system being based on the speech processing method based on Khatri-Rao subspace according to any one of claims 1 to 7, comprising: Acquisition module: used to collect voice signals to be processed; A first processing module is configured to process the speech signal to be processed using a Khatri-Rao subspace method to obtain a processed speech signal; Training module: used to build and train the speech enhancement model. During the training process, the loss function is used to adjust the parameters of the speech enhancement model to obtain a trained speech enhancement model. The second processing module is used to input the processed speech signal into the trained speech enhancement model to obtain an enhanced speech signal.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer stored program is executed by a processor, the speech processing method based on Khatri-Rao subspace according to any one of claims 1 to 7 is implemented.

Images