Multi-speaker voice separation method and system based on low-rank decomposition and graph regularization

Abstract

The invention discloses a multi-speaker voice separation method and system based on low-rank decomposition and graph regularization, and the method comprises the steps: carrying out the sampling of a mixed voice signal, carrying out the short-time Fourier transform of a sampling signal, and obtaining a time-frequency domain complex matrix; taking the amplitude spectrum of the complex matrix as a characteristic matrix; decomposing the feature matrix by using a low-rank matrix to obtain a feature matrix of each speaker, establishing an optimized objective function, and applying graph regularization on the low-rank feature matrix of each speaker; and performing iterative optimization to obtain a feature matrix of each speaker, and obtaining a voice signal of each speaker. The low-rank characteristic and graph regularization are combined, so that the characteristic matrix is kept smooth and consistent in a complex environment (such as noise and multi-speaker environment), the voice of multiple speakers can be effectively separated, and the method has relatively high noise immunity and robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech signal processing, and the present invention relates to a multi-speaker speech separation method and system based on low-rank decomposition and graph regularization. The method is suitable for multi-speaker speech separation applications in complex environments, such as cocktail party effect, far-field speech recognition, real-time speech separation systems, etc. Background Art

[0002] With the popularity of intelligent voice devices and the widespread application of voice interaction systems, speech signal processing technology has become one of the core research directions in the field of artificial intelligence. In real life, speech signals are often in a complex acoustic environment with multiple noise interferences and multiple speakers speaking at the same time. In such scenarios, how to effectively separate the speech of different speakers has become a key issue in speech signal processing, especially in the "cocktail party effect", that is, speech separation in an environment where multiple speakers are speaking at the same time.

[0003] Traditional speech separation methods mainly rely on filter banks and blind source separation (BSS) techniques, such as Independent Component Analysis (ICA) or Non-negative Matrix Factorization (NMF). These methods work well when dealing with a single sound source, but perform poorly in multiple sound sources and noisy environments. In addition, with the rise of deep learning, neural network-based methods have also been applied to speech separation tasks, such as Time-Frequency Masking and Deep Separation Networks. However, these methods usually require a large amount of training data and have performance bottlenecks when facing dynamic noise environments or multiple speakers.

[0004] Research has found that speech signals have strong structure in the time-frequency domain, especially the speech features of a single speaker can usually be represented by a lower-dimensional subspace, which means that its feature matrix has a low-rank characteristic. At the same time, there is correlation and redundancy between different frequency components, and this information can be captured by the graph structure. Based on this feature, research in recent years has gradually combined low-rank decomposition with graph learning methods, extracting the main components of speech through low-rank decomposition, and using graph regularization technology to maintain the smoothness of the feature matrix in the frequency and time dimensions. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-speaker speech separation method and system based on low-rank decomposition and graph regularization, which combines low-rank characteristics and graph regularization to keep the feature matrix smooth and consistent in complex environments (such as noise and multi-speaker environments). It can not only effectively separate the speech of multiple speakers, but also has strong noise resistance and robustness.

[0006] The technical solution to achieve the purpose of the present invention is:

[0007] A multi-speaker speech separation method based on low-rank decomposition and graph regularization comprises the following steps:

[0008] S01: sampling the mixed speech signal and performing short-time Fourier transform on the sampled signal to obtain a complex matrix in the time-frequency domain;

[0009] S02: Use the magnitude spectrum of the complex matrix as the feature matrix;

[0010] S03: Decompose the feature matrix using the low-rank matrix to obtain the feature matrix of each speaker, establish the optimization objective function, and apply graph regularization on the low-rank feature matrix of each speaker;

[0011] S04: Iteratively optimize to obtain the feature matrix of each speaker and obtain the speech signal of each speaker.

[0012] In the preferred technical solution, the time-frequency domain complex matrix in step S01 is:

[0013]

[0014] Where f is the frequency, t is the center of the time window, w[nt] is the window function used to localize the signal within the time window, y[n] is the nth sampling point of the mixed speech signal, N is the number of sampling points in each time window, that is, the window length, and the rows of the complex matrix D(f,t) correspond to the frequencies f, and the columns correspond to the time windows.

[0015] In the preferred technical solution, step S02 uses the amplitude spectrum of the complex matrix as the characteristic matrix including:

[0016] Compute the magnitude spectrum of the complex matrix D(f,t):

[0017]

[0018] in, and are the real and imaginary parts of D(f,t), respectively. The dimension of |D(f,t)| is M×T, where M is the number of frequency components and T is the number of time windows.

[0019] Feature matrix Y(f,t)=|D(f,t)|.

[0020] In the preferred technical solution, the optimization objective function established in step S03 is:

[0021]

[0022] Among them, K is the total number of speakers, X k is the low-rank feature matrix corresponding to the k-th speaker, Y is the feature matrix, is the Frobenius norm of the matrix, ||X k || * is the nuclear norm and λ1 is the low-rank regularization parameter.

[0023] In the preferred technical solution, in step S03, graph regularization is applied to the low-rank feature matrix of each speaker as follows:

[0024]

[0025] Where Tr(·) represents the trace of the matrix, λ s represents the weight parameter on the sth scale, the number of S scales, L s is the graph Laplacian matrix at each scale s.

[0026] In the preferred technical solution, the graph Laplacian matrix at each scale s is:

[0027] L s =D s -W s

[0028] Among them, D s is the degree matrix, the degree matrix D s The diagonal elements of represents the degree of node i,

[0029] N s is the number of nodes at the sth scale, is the weighted adjacency matrix W corresponding to scale s s Elements of is the edge weight between nodes i and j of scale s.

[0030] In the preferred technical solution, in step S04, the feature matrix of each speaker is obtained by iterative optimization through the alternating multiplier method, and the specific steps are:

[0031] S41: fix the feature matrix of other speakers and update the feature matrix of the current speaker;

[0032] S42: Recalculate the graph Laplacian matrix L at different scales s , and apply regularization at multiple scales;

[0033] S43: Iterate the loop until the loss function converges.

[0034] The present invention also discloses a multi-speaker speech separation system based on low-rank decomposition and graph regularization, comprising:

[0035] A sampling and transformation module samples the mixed speech signal and performs short-time Fourier transform on the sampled signal to obtain a complex matrix in the time-frequency domain;

[0036] The feature matrix extraction module uses the amplitude spectrum of the complex matrix as the feature matrix;

[0037] The low-rank decomposition and graph regularization module uses the low-rank matrix to decompose the feature matrix to obtain the feature matrix of each speaker, establishes an optimization objective function, and applies graph regularization on the low-rank feature matrix of each speaker;

[0038] The iterative optimization module iteratively optimizes and obtains the feature matrix of each speaker, thereby obtaining the speech signal of each speaker.

[0039] In the preferred technical solution, the optimization objective function established in the low-rank decomposition and graph regularization module is:

[0040]

[0041] Among them, K is the total number of speakers, X k is the low-rank feature matrix corresponding to the k-th speaker, Y is the feature matrix, is the Frobenius norm of the matrix, ||X k || * is the nuclear norm and λ1 is the low-rank regularization parameter.

[0042] The present invention further discloses a computer storage medium on which a computer program is stored. When the computer program is executed, the multi-speaker speech separation method based on low-rank decomposition and graph regularization is implemented.

[0043] Compared with the prior art, the present invention has the following significant advantages:

[0044] 1. High robustness: Combining low-rank properties and graph regularization, the feature matrix remains smooth and consistent in complex environments (such as noise and multi-speaker environments). By combining low-rank decomposition and graph regularization, efficient speech separation can be achieved.

[0045] 1. Strong adaptability: This method can adapt to different numbers of speakers and various environmental noises, and optimize the separation effect by adjusting parameters.

[0046] 3. Real-time processing capability: Through iterative optimization methods (such as ADMM), quasi-real-time speech separation processing can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of the multi-speaker speech separation method based on low-rank decomposition and graph regularization of the present invention;

[0048] Figure 2 It is a principle block diagram of the multi-speaker speech separation system based on low-rank decomposition and graph regularization of the present invention. DETAILED DESCRIPTION

[0049] The principle of the present invention is that speech signals have low-rank characteristics, that is, in the feature space, the speech features of the same speaker can be represented as a low-rank subspace, and the mixed speech signal is composed of the superposition of multiple low-rank subspaces. At the same time, in the time-frequency domain, the similarity between different frequency components can be modeled by a graph. Therefore, by combining low-rank decomposition and graph regularization, efficient speech separation can be achieved.

[0050] Embodiment 1:

[0051] like Figure 1 As shown, a multi-speaker speech separation method based on low-rank decomposition and graph regularization includes the following steps:

[0052] S01: sampling the mixed speech signal and performing short-time Fourier transform on the sampled signal to obtain a complex matrix in the time-frequency domain;

[0053] S02: Use the magnitude spectrum of the complex matrix as the feature matrix;

[0054] S03: Decompose the feature matrix using the low-rank matrix to obtain the feature matrix of each speaker, establish the optimization objective function, and apply graph regularization on the low-rank feature matrix of each speaker;

[0055] S04: Iteratively optimize to obtain the feature matrix of each speaker and obtain the speech signal of each speaker.

[0056] In a preferred embodiment, the time-frequency domain complex matrix in step S01 is:

[0057]

[0058] Where f is the frequency, t is the center of the time window, w[nt] is the window function used to localize the signal within the time window, y[n]y[n] is the nth sampling point of the mixed speech signal, N is the number of sampling points in each time window, that is, the window length, and the rows of the complex matrix D(f,t) correspond to the frequencies f, and the columns correspond to the time windows.

[0059] In a preferred embodiment, step S02 uses the amplitude spectrum of the complex matrix as a feature matrix including:

[0060] Compute the magnitude spectrum of the complex matrix D(f,t):

[0061]

[0062] in, and are the real and imaginary parts of D(f,t), respectively. The dimension of |D(f,t)| is M×T, where M is the number of frequency components and T is the number of time windows.

[0063] Feature matrix Y(f,t)=|D(f,t)|.

[0064] In a preferred embodiment, the optimization objective function established in step S03 is:

[0065]

[0066] Among them, K is the total number of speakers, X k is the low-rank feature matrix corresponding to the k-th speaker, Y is the feature matrix, is the Frobenius norm of the matrix, ||X k || * is the nuclear norm and λ1 is the low-rank regularization parameter.

[0067] In a preferred embodiment, in step S03, graph regularization is applied to the low-rank feature matrix of each speaker as follows:

[0068]

[0069] Where Tr(·) represents the trace of the matrix, λ s represents the weight parameter on the sth scale, S is the number of scales, L s is the graph Laplacian matrix at each scale s.

[0070] In a preferred embodiment, the graph Laplacian matrix at each scale s is:

[0071] L s =D s -W s

[0072] Among them, D s is the degree matrix, the degree matrix D s The diagonal elements of represents the degree of node i,

[0073] N s is the number of nodes at the sth scale, is the weighted adjacency matrix W corresponding to scale s s Elements of is the edge weight between nodes i and j of scale s.

[0074] In a preferred embodiment, in step S04, the feature matrix of each speaker is obtained by iterative optimization using the alternating multiplier method, and the specific steps are:

[0075] S41: fix the feature matrix of other speakers and update the feature matrix of the current speaker;

[0076] S42: Recalculate the graph Laplacian matrix L at different scales s , and apply regularization at multiple scales;

[0077] S43: Iterate the loop until the loss function converges.

[0078] In another embodiment, a computer storage medium stores a computer program, which, when executed, implements the multi-speaker speech separation method based on low-rank decomposition and graph regularization.

[0079] The multi-speaker speech separation method based on low-rank decomposition and graph regularization can adopt any of the multi-speaker speech separation methods based on low-rank decomposition and graph regularization mentioned above, and the specific implementation is not repeated here.

[0080] In another embodiment, Figure 2 As shown, a multi-speaker speech separation system based on low-rank decomposition and graph regularization includes:

[0081] The sampling and transformation module 10 samples the mixed speech signal and performs short-time Fourier transformation on the sampled signal to obtain a complex matrix in the time-frequency domain;

[0082] A feature matrix extraction module 20 uses the amplitude spectrum of the complex matrix as a feature matrix;

[0083] A low-rank decomposition and graph regularization module 30, which uses a low-rank matrix to decompose the feature matrix to obtain a feature matrix of each speaker, establishes an optimization objective function, and applies graph regularization on the low-rank feature matrix of each speaker;

[0084] The iterative optimization module 40 iteratively optimizes and obtains the feature matrix of each speaker, thereby obtaining the speech signal of each speaker.

[0085] Specifically, the workflow of a multi-speaker speech separation system based on low-rank decomposition and graph regularization is described below by taking a preferred embodiment as an example:

[0086] The main steps include:

[0087] Step 1: Let the mixed speech signal received by the speech receiving system be y(t), then first sample y(t) and perform short-time Fourier transform (STFT) on the sampled signal to obtain the time-frequency domain complex matrix

[0088]

[0089] Where f is the frequency and t is the center of the time window. w[nt] is the window function used to localize the signal within the time window. N is the number of sampling points in each time window, that is, the window length. Through this transformation, the speech signal is represented in the time-frequency domain as a complex matrix D(f,t), whose rows correspond to frequencies f and whose columns correspond to time windows t.

[0090] Step 2: Calculate the magnitude spectrum of the complex matrix D(f,t)

[0091]

[0092] in and are the real and imaginary parts of D(f,t), respectively. The dimension of |D(f,t)| is M×T, where M is the number of frequency components (i.e., the number of frequency points in the discrete Fourier transform), and T is the number of time windows, corresponding to the number of windows into which the signal is divided.

[0093] Step 3: Define the feature matrix

[0094] Y(f,t)=|D(f,t)|,f=0,1,...,M-1,t=0,1,...,T-1 (3)

[0095] Step 4: Construct multiple graphs at different frequency resolutions based on the feature matrix Y to capture the local and global characteristics of the signal. Define different scales through frequency clustering, downsampling or smoothing:

[0096] Fine-grained scale: retain all frequency components and construct a high-resolution graph that reflects the similarity between adjacent frequency components.

[0097] Coarse-grained scale: Reduce the number of frequency components by frequency clustering or downsampling, and construct a low-resolution map to capture the global relationship between frequency bands.

[0098] Intermediate scale: Multiple intermediate scale graphs are constructed between fine-grained and coarse-grained, gradually transitioning from local to global.

[0099] The feature matrix Y is considered as a graph G = (V, E), where each node represents a frequency component. For scale s, the edge weight between nodes i and j is Calculated by a similarity metric such as Euclidean distance or cosine similarity:

[0100]

[0101] where σ s is a scale parameter that controls the sensitivity of the similarity measure or the decay rate of the distance at scale s. represents the Euclidean distance between nodes i and j.

[0102] Step 5: Define the weighted adjacency matrix W corresponding to each scale s , whose elements are specifically defined as:

[0103]

[0104] If N is the total number of nodes, then the weighted adjacency matrix W s The dimension is N s ×N s , N s is the number of nodes at the sth scale.

[0105] Step 6: Calculate the degree matrix D at each scale s s , D s Is a diagonal matrix used to represent the degree of each node in the graph (that is, the sum of the weights of the edges connected to it). s The diagonal elements of Represents the degree of node i, which is specifically defined as:

[0106]

[0107] Step 7: Calculate the graph Laplacian matrix at each scale s

[0108] L s =D s -W s (7)

[0109] Step 8: Low-rank matrix decomposition. Assume that the feature matrix Y of the mixed speech signal is composed of multiple low-rank subspaces:

[0110]

[0111] Where K is the total number of speakers, X k is the low-rank feature matrix corresponding to the k-th speaker.

[0112] Use low-rank sparse decomposition (such as Robust PCA or SVD) to separate the feature matrix of each speaker. The optimization objective function is:

[0113]

[0114] where ||X k ||* is the nuclear norm, which is used to maintain the low-rank property of each speaker’s feature matrix, and λ1 is the low-rank regularization parameter.

[0115] Step 9: Graph regularization, graph regularization is applied on the low-rank feature matrix of each speaker to maintain the local similarity between frequency components.

[0116]

[0117] where Tr(·) represents the trace of the matrix, λ s Represents the weight parameter on the sth scale.

[0118] Step 10: Use iterative optimization methods to iteratively optimize X k , for example, the alternating multiplier method (ADMM), the specific steps are:

[0119] Step 10.1: Fix the feature matrix of other speakers and update the feature matrix of the current speaker.

[0120] Step 10.2: Recalculate the graph Laplacian matrix L at different scales s , and apply regularization at multiple scales.

[0121] Step 10.3: Repeat the loop until the loss function converges.

[0122] Step 11: For each speaker’s feature matrix X k , combined with the original phase information ∠D(f,t) to reconstruct the spectrum:

[0123]

[0124] The separated time-frequency signals are converted back to the time domain through the inverse STFT (ISTFT) to obtain the speech signals of each speaker.

[0125] It can be applied to multi-speaker speech separation applications in complex environments, such as cocktail party effect, far-field speech recognition, real-time speech separation systems, etc.

[0126] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention shall be equivalent replacement modes and shall be included in the protection scope of the present invention.

Claims

1. A multi-speaker speech separation method based on low-rank decomposition and graph regularization, characterized in that: The following steps are involved: S01: sampling the mixed speech signal and performing short-time Fourier transform on the sampled signal to obtain a complex matrix in the time-frequency domain; S02: Use the magnitude spectrum of the complex matrix as the feature matrix; S03: Decompose the feature matrix using the low-rank matrix to obtain the feature matrix of each speaker, establish the optimization objective function, and apply graph regularization on the low-rank feature matrix of each speaker; S04: Iteratively optimize to obtain the feature matrix of each speaker and obtain the speech signal of each speaker.

2. The multi-speaker speech separation method based on low-rank decomposition and graph regularization according to claim 1, characterized in that: The time-frequency domain complex matrix in step S01 is: Where f is the frequency, t is the center of the time window, and w[nt] is the window function used to localize the signal within the time window. y [ n ] is the nth sampling point of the mixed speech signal, N is the number of sampling points in each time window, that is, the window length, the rows of the complex matrix D(f,t) correspond to the frequency f, and the columns correspond to the time window.

3. The multi-speaker speech separation method based on low-rank decomposition and graph regularization according to claim 2, characterized in that: Step S02 uses the amplitude spectrum of the complex matrix as a characteristic matrix including: Compute the magnitude spectrum of the complex matrix D(f,t): in, and are the real and imaginary parts of D(f,t), respectively. The dimension of |D(f,t)| is M×T, where M is the number of frequency components and T is the number of time windows. Feature matrix Y(f,t)=|D(f,t)|.

4. The multi-speaker speech separation method based on low-rank decomposition and graph regularization according to claim 1, characterized in that: The optimization objective function established in step S03 is: Among them, K is the total number of speakers, X k is the low-rank feature matrix corresponding to the k-th speaker, Y is the feature matrix, is the Frobenius norm of the matrix, ||X k || * is the nuclear norm and λ1 is the low-rank regularization parameter.

5. The multi-speaker speech separation method based on low-rank decomposition and graph regularization according to claim 4, characterized in that: In step S03, graph regularization is applied to the low-rank feature matrix of each speaker as follows: Where Tr(·) represents the trace of the matrix, λ s represents the weight parameter on the sth scale, S is the number of scales, L s is the graph Laplacian matrix at each scale s.

6. The multi-speaker speech separation method based on low-rank decomposition and graph regularization according to claim 5, characterized in that: The graph Laplacian matrix at each scale s is: L s =D s -W s Among them, D s is the degree matrix, the degree matrix D s The diagonal elements of represents the degree of node i, N s is the number of nodes at the sth scale, is the weighted adjacency matrix corresponding to scale s W s Elements of is the edge weight between nodes i and j of scale s.

7. The multi-speaker speech separation method based on low-rank decomposition and graph regularization according to claim 6, characterized in that: In step S04, the feature matrix of each speaker is obtained by iterative optimization through the alternating multiplier method. The specific steps are: S41: fix the feature matrix of other speakers and update the feature matrix of the current speaker; S42: Recalculate the graph Laplacian matrix L at different scales s , and apply regularization at multiple scales; S43: Iterate the loop until the loss function converges.

8. A multi-speaker speech separation system based on low-rank decomposition and graph regularization, characterized in that: include: A sampling and transformation module samples the mixed speech signal and performs short-time Fourier transform on the sampled signal to obtain a complex matrix in the time-frequency domain; The feature matrix extraction module uses the amplitude spectrum of the complex matrix as the feature matrix; The low-rank decomposition and graph regularization module uses the low-rank matrix to decompose the feature matrix to obtain the feature matrix of each speaker, establishes an optimization objective function, and applies graph regularization on the low-rank feature matrix of each speaker; The iterative optimization module iteratively optimizes and obtains the feature matrix of each speaker, thereby obtaining the speech signal of each speaker.

9. The multi-speaker speech separation system based on low-rank decomposition and graph regularization according to claim 8, characterized in that: The optimization objective function established in the low-rank decomposition and graph regularization module is: Among them, K is the total number of speakers, X k is the low-rank feature matrix corresponding to the k-th speaker, Y is the feature matrix, is the Frobenius norm of the matrix, ‖X k ‖ * is the nuclear norm and λ1 is the low-rank regularization parameter.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the multi-speaker speech separation method based on low-rank decomposition and graph regularization described in claims 1-7 is implemented.

Images