A multi-channel speech enhancement method, system, and medium

Abstract

The application relates to a multi-channel speech enhancement method, system and medium, which comprises the following steps: improving a minimum variance distortion response (MVDR) beamformer to obtain a single-channel enhanced speech signal and complete the whole spatial filtering process; designing a deep complex convolution recurrent network (DCCRN), proposing a double-dimension adaptive mechanism, and redesigning a frequency domain reconstruction link to improve the flexibility, robustness and deployment efficiency of the network; and proposing a perception-oriented loss function group taking a signal distortion ratio (SDR) and a perception multi-scale spectrum quality error (PMSQE) weighted fusion as the core, breaking through the limitation of fixed single-loss training through a dynamic switching mechanism. The application closely combines data-driven frequency domain refinement enhancement and model-driven spatial filtering, and effectively solves the problems of insufficient utilization of spatial information and unreliable noise covariance matrix estimation in traditional multi-channel speech enhancement.

Description

Technical Field

[0001] This application relates to the field of multi-channel speech enhancement, and in particular to a multi-channel speech enhancement method, system, and medium. Background Technology

[0002] Traditional speech enhancement primarily relies on single-channel signal processing techniques, utilizing the statistical characteristics of frequency-domain noise through methods such as Wiener filtering and spectral subtraction to achieve noise reduction. However, limited by the core limitation of single-channel methods—the inability to distinguish between sound sources at different spatial locations—the performance of these methods deteriorates sharply in scenarios with strong noise, non-stationary noise, or multi-source interference. In recent years, the widespread application of microphone arrays has driven research focus to multi-channel speech enhancement—a technique that utilizes the spatial differences of sound sources to separate target speech from noise, and has become a research hotspot in the field. Combining beamforming-based spatial filtering techniques with the refined frequency-domain modeling capabilities of deep learning allows multi-channel methods to overcome the performance limitations of single-channel solutions, achieving superior noise reduction results.

[0003] Despite progress in multi-channel speech enhancement technology, two key challenges remain. First, while the current best-performing deep model—the Deep Complex Convolutional Recurrent Network (DCCRN)—excels in single-channel frequency domain reconstruction, its application to multi-channel scenarios rarely incorporates dedicated spatial filtering mechanisms, resulting in insufficient exploitation and utilization of the spatial information inherent in the microphone array. Second, the mainstream spatial filtering technique—the classic Minimum Variance Distortionless Response (MVDR) beamformer—requires strict adherence to the premise that "the first and last frames of the signal contain only noise" to accurately estimate the noise covariance matrix. However, in real-world scenarios, noise often exhibits non-stationary characteristics (such as transient and time-varying noise), making this assumption difficult to uphold. This leads to incorrect covariance estimations and severely weakens the spatial filtering performance of the beamformer. Summary of the Invention

[0004] The purpose of this application is to provide a multi-channel speech enhancement method, system, and medium that closely combines data-driven frequency domain refinement enhancement with model-driven spatial filtering, effectively solving the problems of insufficient utilization of spatial information and unreliable noise covariance matrix estimation in traditional multi-channel speech enhancement.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] In a first aspect, embodiments of this application provide a multi-channel speech enhancement method, comprising the following steps:

[0007] An improved minimum variance distortionless response MVDR beamformer was used to obtain a single-channel enhanced speech signal and complete the entire spatial filtering process.

[0008] We designed a deep complex convolutional recurrent network (DCCRN), proposed a two-dimensional adaptation mechanism, and redesigned the frequency domain reconstruction link to improve the network's flexibility, robustness, and deployment efficiency.

[0009] A perception-guided loss function set is proposed, which is based on the weighted fusion of signal distortion ratio (SDR) and perceptual multi-scale spectral centroid error (PMSQE). The dynamic switching mechanism breaks through the limitation of fixed single-loss training.

[0010] The improved minimum variance distortionless response MVDR beamformer obtains a single-channel enhanced speech signal, completing the entire spatial filtering process as follows:

[0011] The system inputs multi-channel noisy speech and spatial geometric information, including the three-dimensional position of the microphone array, the azimuth and elevation angles of the sound source;

[0012] An improved method is used to robustly estimate the noise covariance matrix. And calculate the three-dimensional spatial guidance vector. ;

[0013] The frequency domain weight vector w(f) is solved by the core formula of the minimum variance distortionless response MVDR beamformer, and a small regularization term is added to avoid the problem of singular values ​​in the matrix inversion process.

[0014] The multi-channel noisy signal is converted to the time-frequency domain by using Short Time Fourier Transform (STFT), and spatial filtering is completed by multiplying the complex spectrum with the weight vector w(f).

[0015] The filtered complex spectrum is converted back to the time domain by the inverse short-time Fourier transform (ISTFT) to obtain a single-channel enhanced speech signal, thus completing the entire spatial filtering process.

[0016] The core objective of the minimum variance distortionless response (MVDR) beamformer is to find the frequency domain weight vector w(f) that minimizes the output noise power while maintaining a distortionless response to the desired signal. Its calculation formula is as follows:

[0017] in, The guide vector is determined by the frequency f, azimuth angle θ, and elevation angle ϕ. The noise covariance matrix (NCM) is a core parameter that determines beamforming performance. This represents the conjugate transpose operation;

[0018] A dynamic frame selection criterion based on single-frame energy statistics is proposed, which completely abandons the traditional fixed frame selection paradigm. The specific implementation process is as follows: the multi-channel signal is divided into frames, with 512 sampling points per frame, an overlap rate of 50%, and a step size of 256 sampling points.

[0019] For each frame, calculate the mean square value of all channels to obtain the scalar energy index of that frame;

[0020] Set an energy percentile threshold of 20. Frames with energy below the 20th percentile are identified as mainly noise frames. Adaptive screening of noisy frames is achieved based on the prior knowledge that low-energy segments are mostly noise.

[0021] This data-driven selection method dynamically tracks the temporal variations of noise. The noise covariance matrix is ​​estimated solely from the filtered noise-dominant frames, and its calculation formula is as follows:

[0022] Where K is the number of noise-dominant frames obtained through screening. Let be the multi-channel complex short-time Fourier transform (STFT) vector of the k-th frame;

[0023] The guide vector model is extended to three-dimensional space to achieve precise localization and enhancement of sound sources with arbitrary azimuth angle θ and pitch angle ϕ. The specific steps are as follows:

[0024] First, the angle is converted from degrees to radians to eliminate unit differences in subsequent calculations. The formula for calculating the sound source direction vector d is as follows:

[0025] ,

[0026] in, , These are the pitch and azimuth angles in radians, based on the microphone's three-dimensional position. Given the sound source direction vector d, calculate the phase delay of the incident signal at each microphone: Where c = 340 m / s is the speed of sound, the guide vector is constructed accordingly, and its m-th element is defined by the phase delay mentioned above. The entire vector is then normalized to eliminate the interference of amplitude differences on the calculation of beamforming weights.

[0027] The proposed Deep Complex Convolutional Recurrent Network (DCCRN) utilizes a two-dimensional adaptation mechanism to redesign the frequency domain reconstruction link, thereby enhancing the network's flexibility, robustness, and deployment efficiency. The specific steps are as follows:

[0028] A two-dimensional adaptation mechanism is proposed, as follows:

[0029] Reconfigurable temporal backbone: Complex / real LSTM is switched by runtime flags, and the number of network layers and hidden units are set as external hyperparameters, which can be flexibly adjusted according to the actual noise scenario;

[0030] Diverse mask generation: The decoder can output three types of masks—energy mask, complex mask, and real mask, which can be flexibly selected through gating parameters. Among them, the energy mask uses tanh output nonlinearity to limit the amplitude to the [0,1] interval, effectively avoiding the numerical overflow distortion problem common in traditional CRM.

[0031] The frequency domain reconstruction link has been redesigned, with the following specific optimizations:

[0032] Noisy Phase Compensation Operator: A phase compensation mechanism is embedded in the complex masking decoder, and the calculation formula is as follows:

[0033]

[0034] This mechanism fuses the original noisy phase with the masking correction amount, rather than directly replacing the original phase, effectively preserving the temporal continuity of the speech.

[0035] Amplitude limiting layer: The reconstructed waveform is hard-limited to the [−1,1] interval to eliminate clipping distortion caused by numerical overflow, while ensuring that the amplitude statistics of the DCCRN output are consistent with those of the MVDR beamformer, and significantly reducing scale mismatch artifacts in the “space-frequency domain” cascaded link;

[0036] Standardized input / output interface: DCCRN can directly receive single-channel augmented signals from beamforming outputs without the need for manually designed spatial feature fusion modules, forming a complete link that can be optimized end-to-end.

[0037] The proposed perceptual-guided loss function set, which uses a weighted fusion of signal distortion ratio (SDR) and perceptual multi-scale spectral centroid error (PMSQE) as its core, overcomes the limitations of fixed single-loss training through a dynamic switching mechanism. The specific steps are as follows:

[0038] Signal Distortion Ratio (SDR) measures the fidelity of an enhanced signal relative to a clean reference signal, and its calculation formula is as follows:

[0039]

[0040] The regularization term ε is used to avoid low-energy frames (where division by zero occurs), ensuring numerical stability throughout training. Since the model optimization objective is to minimize the loss, it is defined as follows: A composite loss function is constructed through weighted fusion, and the calculation formula is as follows:

[0041]

[0042] In the formula, the weights , The validation set grid search determines that the perception-driven PMSQE is assigned a higher weight, allowing the optimizer to focus on improving the subjective perceived quality of the speech; SDR is used to ensure the overall signal fidelity and avoid producing audible artifacts by excessively pursuing perceptual effects.

[0043] Secondly, an embodiment of this application provides a multi-channel speech enhancement system, including a memory and a processor. The memory includes a program for a multi-channel speech enhancement method. When the program for the multi-channel speech enhancement method is executed by the processor, it implements the steps of the multi-channel speech enhancement method as described above.

[0044] Thirdly, an embodiment of this application provides a computer-readable storage medium storing program code, which, when executed by a processor, implements the steps of the multi-channel speech enhancement method described above.

[0045] Compared with existing technologies, the advantages of this invention are as follows: It proposes a DCCRN architecture that integrates MVDR beamforming, tightly combining data-driven frequency domain refinement enhancement with model-driven spatial filtering, effectively solving the problems of insufficient utilization of spatial information and unreliable noise covariance matrix estimation in traditional multi-channel speech enhancement. By replacing the heuristic "first and last frame" noise estimator with an energy percentile adaptive selector and extending the steering vector modeling to an omnidirectional-pitch sphere, the improved MVDR front-end can still provide reliable spatial cues in strongly non-stationary noise scenarios. Integrating this spatial cue into the reconstructed DCCRN (which features switchable LSTM type, multi-mode masking, phase compensation reconstruction, and perceptual alignment composite loss), the proposed model achieves continuous improvement in various objective indicators of the DNS 2023 speech enhancement task while maintaining computational efficiency suitable for real-time applications. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of the method provided in an embodiment of the present invention.

[0048] Figure 2 This is a diagram showing the masking and mapping filtering results of a noisy speech segment provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0050] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0051] The terms “first,” “second,” etc., are used only to distinguish one entity or operation from another, and should not be construed as indicating or implying relative importance, nor as requiring or implying any such actual relationship or order between these entities or operations.

[0052] This embodiment is achieved through the following technical solution: a multi-channel speech enhancement method, comprising the following steps:

[0053] An improved minimum variance distortionless response MVDR beamformer was used to obtain a single-channel enhanced speech signal and complete the entire spatial filtering process.

[0054] We designed a deep complex convolutional recurrent network (DCCRN), proposed a two-dimensional adaptation mechanism, and redesigned the frequency domain reconstruction link to improve the network's flexibility, robustness, and deployment efficiency.

[0055] A perception-guided loss function set is proposed, which is based on the weighted fusion of signal distortion ratio (SDR) and perceptual multi-scale spectral centroid error (PMSQE). The dynamic switching mechanism breaks through the limitation of fixed single-loss training.

[0056] Improved MVDR beamforming module:

[0057] While traditional MVDR beamforming can suppress interference signals in multi-channel speech enhancement using spatial cues, its performance is highly dependent on the accuracy of the noise covariance matrix (NCM) estimation. Classical NCM estimators require strict assumption that the first and last frames of the signal contain only noise, but this assumption is easily invalidated in real acoustic environments with non-stationary noise and overlapping speech and interference in the time domain. The resulting NCM mismatch significantly reduces the effectiveness of spatial filtering. To overcome this bottleneck, this application proposes an improved MVDR module, mainly introducing two improvements: (1) a robust NCM estimator based on energy percentiles; and (2) a three-dimensional steering vector model, which extends the applicability of MVDR to more complex three-dimensional acoustic environments.

[0058] From a mathematical perspective, the core objective of MVDR beamforming is to find the frequency domain weight vector w(f) that minimizes the output noise power while maintaining a distortion-free response to the desired signal. The calculation formula is as follows:

[0059]

[0060] in, The guide vector is determined by the frequency f, azimuth angle θ, and elevation angle ϕ. The noise covariance matrix (NCM) is a core parameter that determines beamforming performance; This represents the Hermitian (conjugate transpose) operation. Traditional methods estimate the Hermitian transpose by selecting a fixed number of frames from the beginning and end of the signal. This static frame selection method cannot track the time-varying characteristics of non-stationary noise, which is the main reason for the decrease in the robustness of the beamformer. Therefore, this application focuses on the study of an adaptive and robust noise covariance matrix estimation method.

[0061] To address this, this application proposes a dynamic frame selection criterion based on single-frame energy statistics, completely abandoning the traditional fixed-frame selection paradigm. The specific implementation process is as follows: First, the multi-channel signal is processed into frames (512 sampling points per frame, 50% overlap, 256 sampling points per step); second, the mean square value of all channels is calculated for each frame to obtain the scalar energy index of that frame; subsequently, an energy percentile threshold is set (threshold percentile is set to 20), and frames with energy below the 20th percentile are determined to be noise-dominant frames. Adaptive screening of noise frames is achieved based on the prior knowledge that "low-energy segments are mostly noise." This data-driven selection method can dynamically track the temporal changes of noise, and the noise covariance matrix is ​​estimated only from the screened noise-dominant frames. Its calculation formula is:

[0062]

[0063] Where K is the number of noise-dominant frames obtained through screening. Let K be the multi-channel complex short-time Fourier transform (STFT) vector of the k-th frame. To ensure the robustness of the algorithm, this application sets up a backoff mechanism: if the dynamically selected K=0 (i.e. no noise-dominant frames are selected), the estimator will automatically switch to the traditional scheme, select the first and last 10 frames of the signal to calculate the noise covariance matrix, and avoid the algorithm from failing due to the failure of noise frame selection.

[0064] To overcome the limitations of traditional MVDR applications, which are confined to the horizontal plane (pitch angle ϕ=0), this application extends the guide vector model to three-dimensional space, enabling precise localization and enhancement of sound sources with arbitrary azimuth angle θ and pitch angle ϕ. First, the angle is converted from degrees to radians to eliminate unit differences in subsequent calculations. The formula for calculating the sound source direction vector d is as follows:

[0065]

[0066] in, , These are the pitch and azimuth angles in radians, respectively. Based on the microphone's three-dimensional position. Given the m-th microphone and the sound source direction vector d, calculate the phase delay of the incident signal at each microphone: Where c = 340 m / s is the speed of sound. Based on this, a steering vector is constructed, the m-th element of which is defined by the phase delay mentioned above, and the entire vector is normalized to eliminate the interference of amplitude differences on the calculation of beamforming weights.

[0067] The complete engineering process of the proposed improved MVDR beamformer is as follows: First, the system inputs multi-channel noisy speech and spatial geometry information (including the three-dimensional position of the microphone array, the azimuth and elevation angles of the sound source); second, the improved method proposed in this application is used to robustly estimate the noise covariance matrix. And calculate the three-dimensional spatial guidance vector. Subsequently, the frequency domain weight vector w(f) is solved using the MVDR core formula, and a small regularization term (1e−6) is added to avoid singular value problems during matrix inversion. Next, the multi-channel noisy signal is converted to the time-frequency domain using Short Time Fourier Transform (STFT), and spatial filtering is completed by multiplying the complex spectrum with the weight vector w(f). Finally, the filtered complex spectrum is converted back to the time domain using Inverse Short Time Fourier Transform (ISTFT) to obtain a single-channel enhanced speech signal, thus completing the entire spatial filtering process.

[0068] DCCRN Enhanced Network Design:

[0069] The complex spectrogram of speech contains both amplitude and phase information: amplitude represents the energy distribution of the signal, and phase determines the fine temporal structure of speech. The fidelity of both affects the perceptual quality and intelligibility of speech. The original DCCRN first achieved full complex domain modeling through complex convolution and complex LSTM [4], but its fixed complex ratio masking (CRM) paradigm, coarse feature fusion method and rigid hyperparameter settings limit its adaptability in diverse acoustic scenarios and multi-module pipelines. The DCCRN designed in this application retains the complex domain processing advantages of the original model, and at the same time, through architectural innovation, it significantly improves the flexibility, robustness and deployment efficiency of the network.

[0070] Multi-modal adaptation of temporal modeling and mask generation:

[0071] The core drawback of traditional DCCRN is its reliance on a single, fixed Complex Ratio Masking (CRM) strategy and a single complex LSTM modeling the temporal context, resulting in poor performance in noisy scenarios. To address this issue, this application proposes a two-dimensional adaptation mechanism, as follows:

[0072] Temporal backbone reconfigurable: Complex / real LSTM can be switched via runtime flags. The number of network layers (rnn_layers) and the number of hidden units (rnn_units) are both set as external hyperparameters, which can be flexibly adjusted according to the actual noise scenario. In stationary noise scenarios, a single-layer 128-unit real LSTM can meet the noise reduction requirements, reducing the computational cost by 40% compared to complex LSTM while maintaining good enhancement quality.

[0073] Diverse Mask Generation: The decoder can output three types of masks—(i) energy mask (E), (ii) complex mask (C), and (iii) real mask (R), which can be flexibly selected through gating parameters. Among them, the energy mask uses tanh output nonlinearity to limit the amplitude to the [0,1] interval, effectively avoiding the numerical overflow distortion problem common in traditional CRM. In low signal-to-noise ratio scenarios, this strategy achieves a stable PESQ gain of 0.2~0.3 compared to a fixed CRM baseline, eliminating the dependence of traditional DCCRN on a single mask mode.

[0074] Frequency domain reconstruction and multi-module collaboration for engineering purposes:

[0075] Traditional DCCRN treats frequency domain reconstruction as an independent step: the predicted complex spectrum is directly transformed back to the time domain via ISTFT, without considering the requirements of upstream spatial filtering and downstream normalization, which easily leads to scale mismatch between modules. This application redesigns the frequency domain reconstruction link at the system level, with the following specific optimizations:

[0076] 1. Noisy Phase Compensation Operator: A phase compensation mechanism is embedded in the complex masking decoder, and the calculation formula is as follows:

[0077]

[0078] This mechanism fuses the original noisy phase with the masking correction, rather than directly replacing the original phase, effectively preserving the temporal continuity of the speech.

[0079] 2. Amplitude Limiting Layer: The reconstructed waveform is hard-limited to the [−1,1] interval (16-bit linear PCM normalization interval), eliminating clipping distortion caused by numerical overflow, while ensuring that the amplitude statistics of the DCCRN output are consistent with those of the MVDR beamformer, and significantly reducing scale mismatch artifacts in the "space-frequency domain" cascaded link.

[0080] 3. Standardized Input / Output Interfaces: DCCRN can directly receive single-channel augmented signals from beamforming outputs, eliminating the need for manually designed spatial feature fusion modules and forming a complete, end-to-end optimizable link. This is more compact than the Spatial-DCCRN variant, which requires dedicated angle feature embedding. All intermediate tensors (including real / complex masking and estimated spectra) are integrated into the loss calculation graph, supporting the construction of multidimensional loss functions and effectively compensating for the engineering application deficiencies of the original DCCRN.

[0081] Perception-guided loss function set:

[0082] The ultimate goal of speech enhancement is to minimize both objective signal distortion and perceptual degradation. Single-criterion loss functions (such as MSE and SI-SNR) cannot simultaneously address both: MSE primarily penalizes spectral amplitude differences but ignores the frequency domain sensitivity of the human ear; SI-SNR quantifies source separation but has weak correlation with speech intelligibility and naturalness. Therefore, this application proposes a perceptual-guided loss function set centered on the weighted fusion of Signal Distortion Ratio (SDR) and Perceptual Multi-Scale Spectral Centroid Error (PMSQE), overcoming the limitations of fixed single-loss training through a dynamic switching mechanism.

[0083] The signal distortion ratio (SDR) measures the fidelity of the enhanced signal relative to the clean reference signal, and it is calculated using the following formula:

[0084]

[0085] The regularization term ε is used to avoid division-by-zero errors in low-energy frames (such as silent frames), ensuring numerical stability throughout training. Since the model optimization objective is to minimize the loss, it is defined as follows: A composite loss function is constructed through weighted fusion, and the calculation formula is as follows:

[0086]

[0087] In the formula, the weights , The validation set was determined through grid search. Perception-driven PMSQE was assigned a higher weight, allowing the optimizer to focus on improving the subjective perceived quality of the speech; SDR was used to ensure the overall signal fidelity and avoid producing audible artifacts by excessively pursuing perceptual effects.

[0088] Dataset

[0089] This application first conducts extensive experiments on the DNS-2023 challenge dataset to verify the effectiveness of the proposed MVDR module. The goal of the multi-channel speech enhancement task is to recover clean speech collected in a reverberant office environment using a six-microphone uniform circular array (10cm in diameter, 16kHz sampling rate). This dataset contains over 130,000 virtual 3D acoustic scenes, each lasting 3 seconds, for a total duration exceeding 100 hours. The clean speech is taken from a clean subset of DNS-2023, and the single-channel noise clips (8,000 clips covering over 150 noise categories) are also from the challenge. Spatialization processing is achieved by generating the room impulse response (RIR) through circular convolution of the analytical signal and a time-reversal probe, followed by convolution with a monophonic waveform. The wall reflectance coefficient is set to range from 0.30 to 0.97. In the experiment, the fixed training set and validation set contained 112,000 (93.33 hours) and 12,000 (10 hours) sentences, respectively, with the signal-to-noise ratio evenly distributed between -5dB and +15dB. The experiment used two industry standard evaluation indicators: Perceptual Speech Quality Assessment (PESQ, range -0.5 to 4.5) and Short-Time Objective Intelligibility (STOI, range 0 to 1)

[24] . The higher the scores of the two indicators, the better the speech enhancement effect.

[0090] Training settings and baseline model

[0091] The proposed system uses a 25ms Hanning window, a 6.25ms stride, and a 75% overlap rate; the FFT points are set to 512, and the batch size is 24. The model is trained for 90 rounds on 114 hours of 3D data, with an initial learning rate of 0.0008, and the learning rate is adjusted using a cosine annealing strategy. The encoder channel configuration is {32,64,128,256,256,256}, the convolution kernel size is (5,2), the stride is (2,1), and two 256-unit LSTM layers are followed by the convolutional layers [4]. To verify the superiority of the proposed model, MVDR-DCCRN and its ablation variants are compared with mainstream baseline models.

[0092] Experimental Results and Discussion

[0093] Table I summarizes all experimental results. Using only the improved MVDR module, compared to the unprocessed noisy signal, PESQ is improved by 0.06 and STOI by 0.02. This result verifies the effectiveness of the energy percentile-based noise covariance estimator and the 3D steering vector. Replacing the MVDR front-end with the traditional CGMM-MVDR reduces PESQ by 0.04, further highlighting the importance of robust noise covariance estimation.

[0094]

[0095] Table 1 Comparison of Speech Enhancement Performance

[0096] Among all components, the improved MVDR's built-in energy percentile noise covariance matrix estimator contributes the most critical marginal gain: its 0.19 PESQ improvement accounts for 29% of the total DCCRN gain, far exceeding the gain of using only the 3D steering vector (+0.02 PESQ) or baseline MVDR (+0.04 PESQ). This advantage stems from breaking the rigid assumptions of traditional noise covariance estimation: traditional methods (such as CGMM-MVDR) rely on "pure noise frames at the beginning and end" or Gaussian modeling, both of which fail in non-stationary noise scenarios. The energy percentile strategy proposed in this application adaptively selects low-energy noise-dominant frames, reducing covariance estimation bias by 15% and achieving more accurate MVDR weight calculation; simultaneously, off-axis interference suppression capability is improved by 20%, providing a cleaner input signal for subsequent DCCRN modules. Replacing the improved noise covariance estimation with CGMM-MVDR reduces PESQ by 0.04, further validating the necessity and superiority of the design in this application.

[0097] The 3D steering vector and end-to-end co-optimization further amplify the aforementioned advantages. By modeling global surface geometry, the 3D steering vector effectively suppresses vertically incident interference such as from ceiling fans, reducing the residual noise processing burden on the DCCRN module and providing an additional 0.02 PESQ gain. Simultaneously, the joint optimization strategy proposed in Chapter 2 aligns the MVDR weight update with the DCCRN frequency domain loss, avoiding the decoupling artifacts inherent in simple cascading and ensuring the preservation of critical phase information required for complex domain modeling during spatial filtering. Ultimately, the integrated system achieves synergistic amplification of spatial and frequency domain enhancements, maintaining excellent performance even in harsh acoustic environments.

[0098] Figure 2 shows the typical speech spectra of the proposed 3D MVDR and subsequent MVDR-DCCRN cascaded processing. After 3D MVDR processing (Figure 2(c)), the background noise and post-reverberation in the spectra have been significantly suppressed initially; the subsequent MVDR-DCCRN module (Figure 2(d)) further refines the processing results, effectively suppressing residual distortion and making the harmonic structure of the speech clearer, thus verifying the synergistic effect of the two-stage enhancement.

[0099] This application proposes a DCCRN architecture that integrates MVDR beamforming, tightly combining data-driven frequency domain refinement enhancement with model-driven spatial filtering. This effectively solves the problems of insufficient utilization of spatial information and unreliable noise covariance matrix estimation in traditional multi-channel speech enhancement. By replacing the heuristic "first and last frame" noise estimator with an energy percentile adaptive selector and extending the steering vector modeling to an omnidirectional-pitch sphere, the improved MVDR front-end can still provide reliable spatial cues in strongly non-stationary noise scenarios. By integrating this spatial cue into the reconstructed DCCRN (which features switchable LSTM type, multi-mode masking, phase compensation reconstruction, and perceptual alignment composite loss), the proposed model achieves continuous improvement in various objective indicators of the DNS2023 speech enhancement task while maintaining computational efficiency suitable for real-time applications.

[0100] This application provides a multi-channel speech enhancement system, including a memory and a processor. The memory includes a program for a multi-channel speech enhancement method. When the program for the multi-channel speech enhancement method is executed by the processor, it implements the steps of the multi-channel speech enhancement method as described above.

[0101] This application provides a computer-readable storage medium storing program code that, when executed by a processor, implements the steps of the multi-channel speech enhancement method described above.

[0102] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0106] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0107] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0108] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined in this application, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0109] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A multi-channel speech enhancement method, characterized in that, Includes the following steps: An improved minimum variance distortionless response MVDR beamformer was used to obtain a single-channel enhanced speech signal and complete the entire spatial filtering process. We designed a deep complex convolutional recurrent network (DCCRN), proposed a two-dimensional adaptation mechanism, and redesigned the frequency domain reconstruction link to improve the network's flexibility, robustness, and deployment efficiency. A perception-guided loss function set is proposed, which is based on the weighted fusion of signal distortion ratio (SDR) and perceptual multi-scale spectral centroid error (PMSQE). The dynamic switching mechanism breaks through the limitation of fixed single-loss training.

2. The multi-channel speech enhancement method according to claim 1, characterized in that, The improved minimum variance distortionless response MVDR beamformer obtains a single-channel enhanced speech signal, completing the entire spatial filtering process as follows: The system inputs multi-channel noisy speech and spatial geometric information, including the three-dimensional position of the microphone array, the azimuth and elevation angles of the sound source; An improved method is used to robustly estimate the noise covariance matrix. And calculate the three-dimensional spatial guidance vector. ; The frequency domain weight vector w(f) is solved by the core formula of the minimum variance distortionless response MVDR beamformer, and a small regularization term is added to avoid the problem of singular values ​​in the matrix inversion process. The multi-channel noisy signal is converted to the time-frequency domain by using Short Time Fourier Transform (STFT), and spatial filtering is completed by multiplying the complex spectrum with the weight vector w(f). The filtered complex spectrum is converted back to the time domain by the inverse short-time Fourier transform (ISTFT) to obtain a single-channel enhanced speech signal, thus completing the entire spatial filtering process.

3. The multi-channel speech enhancement method according to claim 2, characterized in that, The core objective of the minimum variance distortionless response (MVDR) beamformer is to find the frequency domain weight vector w(f) that minimizes the output noise power while maintaining a distortionless response to the desired signal. Its calculation formula is as follows: ， in, The guide vector is determined by the frequency f, azimuth angle θ, and elevation angle ϕ. The noise covariance matrix (NCM) is a core parameter that determines beamforming performance. This represents the conjugate transpose operation; A dynamic frame selection criterion based on single-frame energy statistics is proposed, which completely abandons the traditional fixed frame selection paradigm. The specific implementation process is as follows: the multi-channel signal is divided into frames, with 512 sampling points per frame, an overlap rate of 50%, and a step size of 256 sampling points. For each frame, calculate the mean square value of all channels to obtain the scalar energy index of that frame; Set an energy percentile threshold of 20. Frames with energy below the 20th percentile are identified as mainly noise frames. Adaptive screening of noisy frames is achieved based on the prior knowledge that low-energy segments are mostly noise. This data-driven selection method dynamically tracks the temporal variations of noise. The noise covariance matrix is ​​estimated solely from the filtered noise-dominant frames, and its calculation formula is as follows: ， Where K is the number of noise-dominant frames obtained through screening. The multi-channel complex short-time Fourier transform (STFT) vector of the k-th frame; The guide vector model is extended to three-dimensional space to achieve precise localization and enhancement of sound sources with arbitrary azimuth angle θ and pitch angle ϕ. The specific steps are as follows: First, the angle is converted from degrees to radians to eliminate unit differences in subsequent calculations. The formula for calculating the sound source direction vector d is as follows: ， in, , These are the pitch and azimuth angles in radians, based on the microphone's three-dimensional position. Given the sound source direction vector d, calculate the phase delay of the incident signal at each microphone: Where c = 340 m / s is the speed of sound, the guide vector is constructed accordingly, and its m-th element is defined by the phase delay mentioned above. The entire vector is then normalized to eliminate the interference of amplitude differences on the calculation of beamforming weights.

4. The multi-channel speech enhancement method according to claim 2, characterized in that, The proposed Deep Complex Convolutional Recurrent Network (DCCRN) utilizes a two-dimensional adaptation mechanism to redesign the frequency domain reconstruction link, thereby enhancing the network's flexibility, robustness, and deployment efficiency. The specific steps are as follows: A two-dimensional adaptation mechanism is proposed, as follows: Reconfigurable temporal backbone: Complex / real LSTM is switched by runtime flags, and the number of network layers and hidden units are set as external hyperparameters, which can be flexibly adjusted according to the actual noise scenario; Diverse mask generation: The decoder can output three types of masks—energy mask, complex mask, and real mask, which can be flexibly selected through gating parameters. Among them, the energy mask uses tanh output nonlinearity to limit the amplitude to the [0,1] interval, effectively avoiding the numerical overflow distortion problem common in traditional CRM. The frequency domain reconstruction link has been redesigned, with the following specific optimizations: Noisy Phase Compensation Operator: A phase compensation mechanism is embedded in the complex masking decoder, and the calculation formula is as follows: ， This mechanism fuses the original noisy phase with the masking correction, rather than directly replacing the original phase, effectively preserving the temporal continuity of the speech. Amplitude limiting layer: The reconstructed waveform is hard-limited to the [−1,1] interval to eliminate clipping distortion caused by numerical overflow, while ensuring that the amplitude statistics of DCCRN output are consistent with those of MVDR beamformer, and greatly reducing scale mismatch artifacts in the "space-frequency domain" cascaded link; Standardized input / output interface: DCCRN can directly receive single-channel augmented signals from beamforming outputs without the need for manually designed spatial feature fusion modules, forming a complete link that can be optimized end-to-end.

5. The multi-channel speech enhancement method according to claim 2, characterized in that, The proposed perceptual-guided loss function set, which uses a weighted fusion of signal distortion ratio (SDR) and perceptual multi-scale spectral centroid error (PMSQE) as its core, overcomes the limitations of fixed single-loss training through a dynamic switching mechanism. The specific steps are as follows: Signal Distortion Ratio (SDR) measures the fidelity of an enhanced signal relative to a clean reference signal, and its calculation formula is as follows: ， The regularization term ε is used to avoid low-energy frames (where division by zero occurs), ensuring numerical stability throughout training. Since the model optimization objective is to minimize the loss, it is defined as follows: A composite loss function is constructed through weighted fusion, and the calculation formula is as follows: ， In the formula, the weights , The validation set grid search determines that the perception-driven PMSQE is assigned a higher weight, allowing the optimizer to focus on improving the subjective perceived quality of the speech; SDR is used to ensure the overall signal fidelity and avoid producing audible artifacts by excessively pursuing perceptual effects.

6. A multi-channel speech enhancement system, characterized in that, The device includes a memory and a processor. The memory includes a program for a multi-channel speech enhancement method. When the program for the multi-channel speech enhancement method is executed by the processor, it implements the steps of the multi-channel speech enhancement method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that, when executed by a processor, implements the steps of the multi-channel speech enhancement method as described in any one of claims 1-5.

Images