Multi-speaker speech separation method and system based on low-rank decomposition and graph regularization
Patent Information
- Application Number
- CN202411502073.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2026-08-28
- Estimated Expiration
- 2044-10-25
AI Technical Summary
然而，这些方法通常需要大量的训练数据，并且在面对动态噪声环境或多说话人时存在性能瓶颈
[0044] 1. High robustness: Combining low-rank properties and graph regularization ensures that the feature matrix remains smooth and consistent in complex environments (such as noisy and multi-speaker environments). Efficient speech separation can be achieved by combining low-rank decomposition and graph regularization.
Smart Images
Figure CN119152875B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of speech signal processing technology, and specifically relates to a multi-speaker speech separation method and system based on low-rank decomposition and graph regularization. It is applicable to multi-speaker speech separation applications in complex environments, such as the cocktail party effect, far-field speech recognition, and real-time speech separation systems. Background Technology
[0002] With the popularization of intelligent voice devices and the widespread application of voice interaction systems, speech signal processing technology has become one of the core research directions in the field of artificial intelligence. In real life, speech signals are often in complex acoustic environments, with various noise interferences and situations where multiple speakers are speaking simultaneously. In such scenarios, how to effectively separate the speech of different speakers has become a key problem in speech signal processing, especially in the "cocktail party effect," that is, speech separation in an environment where multiple speakers are speaking at the same time.
[0003] Traditional speech separation methods primarily rely on filter banks and blind source separation (BSS) techniques, such as Independent Component Analysis (ICA) or Non-negative Matrix Factorization (NMF). These methods perform well when dealing with single sound sources, but perform poorly in multi-source or noisy environments. Furthermore, with the rise of deep learning, neural network-based methods have also been applied to speech separation tasks, such as time-frequency masking and deep separation networks. However, these methods typically require large amounts of training data and suffer from performance bottlenecks when facing dynamic noise environments or multiple speakers.
[0004] Research has found that speech signals possess strong structure in the time-frequency domain. In particular, the speech features of a single speaker can usually be represented by a low-dimensional subspace, meaning that its feature matrix has low-rank properties. Simultaneously, correlations and redundancies exist between different frequency components, information that can be captured through graph structures. Based on this characteristic, recent research has increasingly combined low-rank decomposition with graph learning methods. Low-rank decomposition extracts the main components of speech, while graph regularization techniques maintain the smoothness of the feature matrix across the frequency and time dimensions. Summary of the Invention
[0005] The purpose of this invention is to provide a multi-speaker speech separation method and system based on low-rank decomposition and graph regularization. By combining low-rank characteristics and graph regularization, the feature matrix can maintain smoothness and consistency in complex environments (such as noisy and multi-speaker environments). It can not only effectively separate multi-speaker speech, but also has strong noise resistance and robustness.
[0006] The technical solution to achieve the purpose of this invention is as follows:
[0007] A multi-speaker speech separation method based on low-rank decomposition and graph regularization includes the following steps:
[0008] S01: Sample the mixed speech signal and perform a short-time Fourier transform on the sampled signal to obtain a complex matrix in the time-frequency domain;
[0009] S02: Use the amplitude spectrum of the complex matrix as the characteristic matrix;
[0010] S03: Use the low-rank matrix decomposition of the feature matrix to obtain the feature matrix of each speaker, establish the optimization objective function, and apply graph regularization to the low-rank feature matrix of each speaker;
[0011] S04: Iterative optimization yields the feature matrix for each speaker, resulting in the speech signal for each speaker.
[0012] In the preferred technical solution, the time-frequency domain complex matrix in step S01 is:
[0013]
[0014] Where f is the frequency, t is the center of the time window, w[nt] is the window function used to localize the signal within the time window, y[n] is the nth sampling point of the mixed speech signal, N is the number of sampling points in each time window, i.e. the window length, and the rows of the complex matrix D(f,t) correspond to the frequency f, and the columns correspond to the time window.
[0015] In the preferred technical solution, step S02, which uses the amplitude spectrum of the complex matrix as the characteristic matrix, includes:
[0016] Calculate the amplitude spectrum of the complex matrix D(f,t):
[0017]
[0018] in, and Let f and t be the real and imaginary parts of D(f,t), respectively. The dimension of |D(f,t)| is M×T, where M is the number of frequency components and T is the number of time windows.
[0019] The characteristic matrix Y(f,t) = |D(f,t)|.
[0020] In the preferred technical solution, the objective function established in step S03 is:
[0021]
[0022] Where K is the total number of speakers, X k Let Y be the low-rank feature matrix corresponding to the k-th speaker, and let Y be the feature matrix. It is the Frobinius norm of the matrix, ||X k || * λ is the nuclear norm, and λ1 is the low-rank regularization parameter.
[0023] In the preferred technical solution, step S03 applies graph regularization to the low-rank feature matrix of each speaker as follows:
[0024]
[0025] Where Tr(·) represents the trace of the matrix, λ s Let L represent the weight parameters at the s-th scale, where S is the number of scales and L is the number of scales. s The graph Laplacian matrix at each scale s.
[0026] In the preferred technical solution, the graph Laplacian matrix at each scale s is:
[0027] L s =D s -W s
[0028] Among them, D s Let D be the degree matrix. s diagonal elements Indicates the degree of node i.
[0029] N s It is the number of nodes at the s-th scale. W is the weighted adjacency matrix corresponding to scale s. s elements, Let be the edge weight between nodes i and j at scale s.
[0030] In the preferred technical solution, step S04 involves iterative optimization using the alternating multiplier method to obtain the feature matrix of each speaker. The specific steps are as follows:
[0031] S41: Fix the feature matrices of other speakers and update the feature matrix of the current speaker;
[0032] S42: Recalculate the graphical Laplacian matrix L at different scales s And apply regularization at multiple scales;
[0033] S43: Iterate continuously until the loss function converges.
[0034] This invention also discloses a multi-speaker speech separation system based on low-rank decomposition and graph regularization, comprising:
[0035] The sampling and transformation module samples the mixed speech signal and performs a short-time Fourier transform on the sampled signal to obtain a complex matrix in the time-frequency domain.
[0036] The feature matrix extraction module uses the amplitude spectrum of the complex matrix as the feature matrix;
[0037] The low-rank decomposition and graph regularization module uses the low-rank matrix to decompose the feature matrix to obtain the feature matrix of each speaker, establishes an optimization objective function, and applies graph regularization to the low-rank feature matrix of each speaker.
[0038] The iterative optimization module iteratively optimizes to obtain the feature matrix of each speaker, thus obtaining the speech signal of each speaker.
[0039] In the preferred technical solution, the optimization objective function established in the low-rank decomposition and graph regularization module is:
[0040]
[0041] Where K is the total number of speakers, X k Let Y be the low-rank feature matrix corresponding to the k-th speaker, and let Y be the feature matrix. It is the Frobinius norm of the matrix, ||X k || * λ is the nuclear norm, and λ1 is the low-rank regularization parameter.
[0042] The present invention also discloses a computer storage medium storing a computer program, which, when executed, implements the above-described multi-speaker speech separation method based on low-rank decomposition and graph regularization.
[0043] Compared with the prior art, the significant advantages of this invention are:
[0044] 1. High robustness: Combining low-rank properties and graph regularization ensures that the feature matrix remains smooth and consistent in complex environments (such as noisy and multi-speaker environments). Efficient speech separation can be achieved by combining low-rank decomposition and graph regularization.
[0045] 1. High adaptability: This method can adapt to different numbers of speakers and various environmental noises, and the separation effect can be optimized by adjusting the parameters.
[0046] 3. Real-time processing capability: Near real-time speech separation processing is achieved through iterative optimization methods (such as ADMM). Attached Figure Description
[0047] Figure 1 This is a flowchart of the multi-speaker speech separation method based on low-rank decomposition and graph regularization of the present invention;
[0048] Figure 2 This is a block diagram illustrating the principle of the multi-speaker speech separation system based on low-rank decomposition and graph regularization of the present invention. Detailed Implementation
[0049] The principle of this invention is that speech signals possess low-rank characteristics; that is, in the feature space, the speech features of the same speaker can be represented as a low-rank subspace, while mixed speech signals are composed of the superposition of multiple low-rank subspaces. Simultaneously, in the time-frequency domain, the similarity between different frequency components can be modeled using a graph. Therefore, by combining low-rank decomposition and graph regularization, efficient speech separation can be achieved.
[0050] Example 1:
[0051] like Figure 1 As shown, a multi-speaker speech separation method based on low-rank decomposition and graph regularization includes the following steps:
[0052] S01: Sample the mixed speech signal and perform a short-time Fourier transform on the sampled signal to obtain a complex matrix in the time-frequency domain;
[0053] S02: Use the amplitude spectrum of the complex matrix as the characteristic matrix;
[0054] S03: Use the low-rank matrix decomposition of the feature matrix to obtain the feature matrix of each speaker, establish the optimization objective function, and apply graph regularization to the low-rank feature matrix of each speaker;
[0055] S04: Iterative optimization yields the feature matrix for each speaker, resulting in the speech signal for each speaker.
[0056] In a preferred embodiment, the time-frequency domain complex matrix in step S01 is:
[0057]
[0058] Where f is the frequency, t is the center of the time window, w[nt] is the window function used to localize the signal within the time window, y[n] is the nth sampling point of the mixed speech signal, N is the number of sampling points in each time window, i.e. the window length, and the rows of the complex matrix D(f,t) correspond to the frequency f, and the columns correspond to the time window.
[0059] In a preferred embodiment, step S02, which uses the amplitude spectrum of the complex matrix as the characteristic matrix, includes:
[0060] Calculate the amplitude spectrum of the complex matrix D(f,t):
[0061]
[0062] in, and Let f and t be the real and imaginary parts of D(f,t), respectively. The dimension of |D(f,t)| is M×T, where M is the number of frequency components and T is the number of time windows.
[0063] The characteristic matrix Y(f,t) = |D(f,t)|.
[0064] In a preferred embodiment, the objective function established in step S03 is:
[0065]
[0066] Where K is the total number of speakers, X k Let Y be the low-rank feature matrix corresponding to the k-th speaker, and let Y be the feature matrix. It is the Frobinius norm of the matrix, ||X k || * λ is the nuclear norm, and λ1 is the low-rank regularization parameter.
[0067] In a preferred embodiment, step S03 applies graph regularization to the low-rank feature matrix of each speaker as follows:
[0068]
[0069] Where Tr(·) represents the trace of the matrix, λ s Let L represent the weight parameters at the s-th scale, where S is the number of scales and L is the number of scales. s The graph Laplacian matrix at each scale s.
[0070] In a preferred embodiment, the graph Laplacian matrix at each scale s is:
[0071] L s =D s -W s
[0072] Among them, D s Let D be the degree matrix. s diagonal elements Indicates the degree of node i.
[0073] N s It is the number of nodes at the s-th scale. W is the weighted adjacency matrix corresponding to scale s. s elements, Let be the edge weight between nodes i and j at scale s.
[0074] In a preferred embodiment, step S04 involves iterative optimization using the alternating multiplier method to obtain the feature matrix for each speaker. The specific steps are as follows:
[0075] S41: Fix the feature matrices of other speakers and update the feature matrix of the current speaker;
[0076] S42: Recalculate the graphical Laplacian matrix L at different scales s And apply regularization at multiple scales;
[0077] S43: Iterate continuously until the loss function converges.
[0078] In another embodiment, a computer storage medium stores a computer program that, when executed, implements the aforementioned multi-speaker speech separation method based on low-rank decomposition and graph regularization.
[0079] The multi-speaker speech separation method based on low-rank decomposition and graph regularization can adopt any of the aforementioned multi-speaker speech separation methods based on low-rank decomposition and graph regularization; the specific implementation will not be elaborated here.
[0080] In another embodiment, such as Figure 2 As shown, a multi-speaker speech separation system based on low-rank decomposition and graph regularization includes:
[0081] The sampling transformation module 10 samples the mixed speech signal and performs a short-time Fourier transform on the sampled signal to obtain a complex matrix in the time-frequency domain.
[0082] The feature matrix extraction module 20 uses the amplitude spectrum of the complex matrix as the feature matrix;
[0083] The low-rank decomposition and graph regularization module 30 uses the low-rank matrix to decompose the feature matrix to obtain the feature matrix of each speaker, establishes an optimization objective function, and applies graph regularization to the low-rank feature matrix of each speaker.
[0084] The iterative optimization module 40 iteratively optimizes to obtain the feature matrix of each speaker and the speech signal of each speaker.
[0085] Specifically, the workflow of a multi-speaker speech separation system based on low-rank decomposition and graph regularization is described below using a preferred embodiment as an example:
[0086] It mainly includes the following steps:
[0087] Step 1: Let the mixed speech signal received by the speech receiving system be y(t). First, sample y(t) and perform a short-time Fourier transform (STFT) on the sampled signal to obtain a complex matrix in the time-frequency domain.
[0088]
[0089] Where f is the frequency and t is the center of the time window. w[nt] is the window function used to localize the signal within the time window. N is the number of sampling points in each time window, i.e., the window length. Through this transformation, the speech signal is represented in the time-frequency domain as a complex matrix D(f,t), where the rows correspond to the frequency f and the columns correspond to the time window t.
[0090] Step 2: Calculate the amplitude spectrum of the complex matrix D(f,t).
[0091]
[0092] in and These are the real and imaginary parts of D(f,t), respectively. The dimension of |D(f,t)| is M×T, where M is the number of frequency components (i.e., the number of frequency points in the Discrete Fourier Transform), and T is the number of time windows, corresponding to the number of windows into which the signal is divided.
[0093] Step 3: Define the feature matrix
[0094] Y(f,t)=|D(f,t)|,f=0,1,...,M-1,t=0,1,...,T-1 (3)
[0095] Step 4: Construct multiple graphs based on the feature matrix Y at different frequency resolutions to capture the local and global characteristics of the signal. Define different scales through frequency clustering, downsampling, or smoothing.
[0096] Fine-grained scale: Preserves all frequency components and constructs a high-resolution graph that reflects the similarity between adjacent frequency components.
[0097] Coarse-grained scale: By reducing the number of frequency components through frequency clustering or downsampling, a low-resolution graph is constructed to capture the global relationships between frequency bands.
[0098] Intermediate scale: Construct multiple intermediate scale graphs between fine-grained and coarse-grained scales, gradually transitioning from local to global.
[0099] The feature matrix Y can be viewed as a graph G = (V, E), where each node represents a frequency component. For scale s, the edge weights between nodes i and j are... Calculated using similarity metrics such as Euclidean distance or cosine similarity:
[0100]
[0101] Where σ s It is a scale parameter that controls the sensitivity of the similarity metric or the rate of distance decay at scale s. This represents the Euclidean distance between nodes i and j.
[0102] Step 5: Define the weighted adjacency matrix W for each scale. s The specific definition of its elements is:
[0103]
[0104] If N is the total number of nodes, then the weighted adjacency matrix W s The dimension is N s ×N s N s It is the number of nodes at the s-th scale.
[0105] Step 6: Calculate the degree matrix D at each scale s. s D s It is a diagonal matrix used to represent the degree of each node in the graph (i.e., the sum of the weights of the edges connected to it). Degree matrix D s diagonal elements The degree of node i is specifically defined as:
[0106]
[0107] Step 7: Calculate the graph Laplacian matrix at each scale s
[0108] L s =D s -W s (7)
[0109] Step 8: Low-rank matrix decomposition. Assume that the feature matrix Y of the mixed speech signal is composed of multiple low-rank subspaces:
[0110]
[0111] Where K is the total number of speakers, X k Let be the low-rank feature matrix corresponding to the k-th speaker.
[0112] Low-rank sparse decomposition (such as Robust PCA or SVD) is used to separate the feature matrices of each speaker. The optimization objective function is:
[0113]
[0114] Where ||X k ||* λ1 is the nuclear norm, used to maintain the low-rank property of each speaker's feature matrix, and λ1 is the low-rank regularization parameter.
[0115] Step 9: Graph regularization. Apply graph regularization to the low-rank feature matrix of each speaker to preserve local similarity between frequency components.
[0116]
[0117] Where Tr(·) represents the trace of the matrix, λ s This represents the weight parameter at the s-th scale.
[0118] Step 10: Use an iterative optimization method to iteratively optimize X. k For example, the Alternating Multiplier Method (ADMM) has the following specific steps:
[0119] Step 10.1: Fix the feature matrices of other speakers and update the feature matrix of the current speaker.
[0120] Step 10.2: Recalculate the graph Laplacian matrix L at different scales s Regularization is applied at multiple scales.
[0121] Step 10.3: Iterate repeatedly until the loss function converges.
[0122] Step 11: For the feature matrix X of each speaker k The spectrum is reconstructed by combining the original phase information ∠D(f,t):
[0123]
[0124] The separated time-frequency signals are converted back to the time domain by inverse STFT (ISTFT) to obtain the speech signals of each speaker.
[0125] It can be applied to multi-speaker speech separation applications in complex environments, such as the cocktail party effect, far-field speech recognition, and real-time speech separation systems.
[0126] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A multi-speaker speech separation method based on low-rank decomposition and graph regularization, characterized in that, Includes the following steps: S01: Sample the mixed speech signal and perform a short-time Fourier transform on the sampled signal to obtain a complex matrix in the time-frequency domain; S02: Use the amplitude spectrum of the time-frequency domain complex matrix as the characteristic matrix. According to the feature matrix Multiple graphs are constructed at different frequency resolutions to capture the local and global characteristics of the signal. Different scales are defined through frequency clustering, downsampling, or smoothing. feature matrix Viewed as a graph, where each node represents a frequency component, for scale compute nodes and Edge weights between , Calculated using similarity metrics: in, It is a scale parameter that controls the scale. The sensitivity of similarity measures or the rate of distance decay. Represents a node and The Euclidean distance between them; Define the weighted adjacency matrix for each scale. The specific definition of its elements is: like Given the total number of nodes, the weighted adjacency matrix is... The dimension is , It is the first Number of nodes at each scale; Calculate each scale degree matrix on , It is a diagonal matrix used to represent the degree of each node in the graph; the degree matrix. diagonal elements Represents a node The degree, specifically defined as: Calculate each scale The graph Laplace matrix above: ； S03: Obtain the low-rank feature matrices of each speaker by decomposing the feature matrices using low-rank matrix decomposition. Establish an optimization objective function, and apply graph regularization to the low-rank feature matrix of each speaker as follows: in, The total number of speakers, It is the Frobinius norm of the matrix. It is the nuclear norm. For low-rank regularization parameters, Represents the trace of a matrix. Indicates the first Weight parameters at each scale S For the number of scales, For each scale The Laplace matrix of the graph; S04: The low-rank feature matrix of each speaker is obtained through iterative optimization using the alternating multiplier method, specifically including: S41: Fix the low-rank feature matrix of other speakers, and update the low-rank feature matrix of the current speaker; S42: Recalculate the Graph Laplacian Matrix at Different Scales And apply regularization at multiple scales; S43: Iterate repeatedly until the loss function converges; S05: For the low-rank feature matrix of each speaker, the corresponding spectrum is reconstructed by combining the original phase information. The separated time-frequency signal is converted back to the time domain by inverse short-time Fourier transform to obtain the speech signal of each speaker.
2. The multi-speaker speech separation method based on low-rank decomposition and graph regularization according to claim 1, characterized in that, The time-frequency domain complex matrix in step S01 is: in, It's frequency. It is the center of the time window. It is a window function used to localize signals within a time window. It is the mixed speech signal n One sampling point, It is the number of sampling points within each time window, i.e., the window length, a complex matrix. The corresponding frequency of the row The column corresponds to the time window.
3. The multi-speaker speech separation method based on low-rank decomposition and graph regularization according to claim 2, characterized in that, Step S02, which uses the amplitude spectrum of the complex matrix as the characteristic matrix, includes: Calculate complex matrices Amplitude spectrum: in, and They are respectively The real and imaginary parts, The dimension is , The number of frequency components. It refers to the number of time windows; Feature matrix .
4. A multi-speaker speech separation system based on low-rank decomposition and graph regularization, used to implement the multi-speaker speech separation method as described in any one of claims 1-3, characterized in that, include: The sampling and transformation module samples the mixed speech signal and performs a short-time Fourier transform on the sampled signal to obtain a complex matrix in the time-frequency domain; The feature matrix extraction module uses the amplitude spectrum of the complex matrix as the feature matrix; The low-rank decomposition and graph regularization module uses the low-rank matrix to decompose the feature matrix to obtain the feature matrix of each speaker, establishes an optimization objective function, and applies graph regularization to the low-rank feature matrix of each speaker. The iterative optimization module iteratively optimizes to obtain the feature matrix of each speaker, thus obtaining the speech signal of each speaker.
5. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the multi-speaker speech separation method based on low-rank decomposition and graph regularization as described in any one of claims 1-3.