Speech separation model training method, electronic device, and storage medium

Abstract

The embodiment of the application discloses a voice separation model training method, an electronic device and a storage medium, wherein the method comprises the following steps: performing convolution operation on the pure de-reverberation signal output by the de-reverberation module and the first room impulse response filter and the second room impulse response filter output by the room impulse response estimation module in the short-time Fourier transform domain to reconstruct an estimated signal with reverberation; calculating a reconstruction error between the estimated signal with reverberation and the initial separation signal; and simultaneously optimizing the network parameters of the separation module, the de-reverberation module and the room impulse response estimation module based on the reconstruction error through a back propagation algorithm to train the model.

Description

Technical Field

[0001] This application belongs to the field of speech separation technology, and in particular relates to speech separation model training methods, electronic devices and storage media. Background Technology

[0002] The "cocktail party problem" refers to the human ability to selectively focus on a single sound source in a noisy party environment (containing multiple simultaneous speaking sources, background music, television audio, and reverberation from room walls). Enabling machines to possess the same ability has been a long-standing challenge in speech processing. Solving the cocktail party problem is particularly difficult in a single-channel (single-microphone) environment due to the lack of multi-channel spatial filtering information.

[0003] Among related technologies, speech separation methods include traditional statistical modeling and signal processing methods such as Independent Component Analysis (ICA), Non-negative Matrix Factorization (NMF), and heuristic spatial filtering. In addition, there are recent data-driven methods based on Deep Neural Networks (DNNs), such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Transformer architecture models (such as SepFormer, WaveSplit, DPTNet, etc.), and the more recent SPMamba model.

[0004] Single-channel speech separation systems in related technologies primarily treat the separation problem as the extraction of speaker-related features (such as timbre and speech content). These systems typically receive a single-channel mixed audio signal and use a deep network architecture to model the spectral or temporal features of the signal, thereby predicting and separating different target speech streams.

[0005] The inventors discovered that traditional signal processing methods exhibit limited separation performance and poor generalization ability in real-world environments with strong nonlinearity, complex acoustic characteristics, and a dynamically changing number of sound sources. Existing deep learning models still fall short when dealing with real-world cocktail party recordings containing strong background noise and complex room reverberation. Most existing models are primarily designed for multi-speaker scenarios with superimposed noise, and their performance significantly degrades in real-world scenarios containing heterogeneous interference such as television audio and music. Summary of the Invention

[0006] This application provides a speech separation model training method, an electronic device, and a storage medium to at least solve one of the above-mentioned technical problems.

[0007] In a first aspect, embodiments of this application provide a speech separation model training method, wherein the speech separation model includes a separation module and a dereverberation module and a room impulse response estimation module arranged in parallel. The separation module is used to receive a single-channel mixed speech signal and output multiple initial separation signals with reverberation; the dereverberation module is used to receive multiple initial separation signals and output corresponding clean dereverberation signals; the room impulse response estimation module is used to extract a first feature embedding of each initial separation signal and extract a second feature embedding of the corresponding clean dereverberation signal, fuse the first feature embedding and the second feature embedding, and output a predicted first room impulse response filter and a second room impulse response filter. The training method includes: performing a convolution operation in the short-time Fourier transform domain on the clean dereverberation signal output by the dereverberation module and the first room impulse response filter and the second room impulse response filter output by the room impulse response estimation module to reconstruct a reverberant estimated signal; calculating the reconstruction error between the reverberant estimated signal and the initial separation signal; and simultaneously optimizing the network parameters of the separation module, the dereverberation module, and the room impulse response estimation module using a backpropagation algorithm based on the reconstruction error to train the model.

[0008] Secondly, embodiments of this application provide a method for single-channel speech separation based on a model trained according to the method described in the first aspect, comprising: inputting a single-channel mixed speech signal into the separation module and outputting multiple initial separation signals with reverberation; sending the initial separation signals into the dereverberation module and outputting a clean dereverberation signal corresponding to each sound source; fusing the feature embeddings of the initial separation signals with the feature embeddings of the clean dereverberation signals and inputting the fusion signals into the room impulse response estimation module and outputting a predicted room impulse response filter corresponding to each initial separation signal.

[0009] Thirdly, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of the speech separation model training method of any embodiment of this application.

[0010] Fourthly, embodiments of this application also provide an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the method described in the first or second aspect.

[0011] Fifthly, embodiments of this application also provide a storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the steps of the method described in the first or second aspect.

[0012] The method in this application, by explicitly modeling the room impulse response in a single-channel environment, endows a single device with the ability to effectively perceive and utilize the physical acoustic spatial information of a room. This is not only effective in pure speech mixing, but also greatly enhances the robustness of the system in complex real-world scenarios containing heterogeneous reverberation interference from television, music, dialogue, etc. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 An architecture diagram of a speech separation model provided in an embodiment of this application; Figure 2 A speech separation model training method is provided in one embodiment of this application; Figure 3 The sampling distribution of room impulse response (RIR) parameters in a HETMIXR provided in an embodiment of this application; Figure 4 A comparison of WHAMR! as a function of model size, provided for one embodiment of this application; Figure 5 The separation performance of HETMIXR and specific content subsets provided in one embodiment of this application; Figure 6 A comparison of the various categories of HETMIXR provided for an embodiment of this application; Figure 7 Ablation study of HETMIXR provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] The inventors discovered that one or more defects in the aforementioned related technologies are mainly caused by the following reasons: these technologies often ignore the role of room acoustic structure in separation and fail to utilize the structural cues related to the propagation path provided by reverberation. In single-channel separation, the weak spatial information cues provided by reverberation have long been underutilized, and there is a lack of explicit modeling of the relationship between room impulse response (RIR) and sound sources.

[0017] When faced with the aforementioned shortcomings, industry professionals typically try to artificially enhance separation capabilities by significantly increasing the number of model parameters (e.g., using SepFormer with up to 26M parameters or WaveSplit with 29M parameters), or by using a multi-microphone array to obtain spatial filtering information.

[0018] The novelty of this proposed solution lies in its departure from the traditional understanding that spatial information is difficult to obtain in a single-channel environment. It cleverly introduces a parallel RIR estimation branch and couples the dereverberation and separation processes through an RIR-perceptual reconstruction objective. This approach, which forces the single-channel model to learn using a physically inspired room-perceptual loss function and leverages reverberation propagation cues as spatial guidance, is unprecedented in the field of single-channel speech separation.

[0019] This application proposes a RAM (RIR-Aware Mamba-based Separation) framework, which uses SPMamba, a network with linear time complexity, as its backbone. In addition to the separation module, RAM adds a dereverberation module to refine the separated signals into reverberation-free estimates, and a parallel RIR estimation module to predict the room impulse response for each target.

[0020] This application introduces an RIR-sensory convolutional reconstruction loss function, which mandates that the déresonated estimated signal can be re-rendered back to the reverberated target signal through the predicted RIR room filter, thereby guiding the model to use reverberation-induced propagation cues as weak spatial guidance for separating sound sources at different locations.

[0021] In a specific example, the solution of this application embodiment includes the following steps: Step 1 (Signal Separation): The single-channel mixed signal y(t) is input into the separation module based on the SPMamba architecture, which outputs two separate waveform signals x1(t) and x2(t) with reverberation.

[0022] Step 2 (De-reverberation processing): The separated reverberant signal is sent to the de-reverberation module. This module uses cross-band blocks to aggregate frequency band information to reduce the spectral tail caused by reverberation, and uses narrow-band blocks to emphasize in-band time-domain modeling, outputting the clean de-reverberation signals s1(t) and s2(t) corresponding to the target.

[0023] Step 3 (RIR Parameter Estimation): The feature embeddings of the reverberant and dereverberant signals are fused and input into a parallel RIR estimation module. This module generates a content-aware representation through a narrowband branch, predicts time-frequency importance weights through a weighted branch, performs element-level modulation and time-axis aggregation, and outputs the predicted room impulse response (RIR) filters h1 and h2 through a CTF decoder.

[0024] Step 4 (Model Training Constraints): During the training phase, using the reconstructed target with RIR perception, the dereverberation estimation signal and the estimated RIR filter are convolved in the short-time Fourier transform (STFT) domain to reconstruct the reverberated target signal and calculate the reconstruction error, thereby supervising the model's ability to extract spatial cues.

[0025] The RAM model in this application is extremely lightweight, with only 7.2M parameters, yet it achieves a SI-SDRi (Scale-Invariant Signal-to-Distortion Ratio Improvement) and a SDRi (Signal-to-Distortion Ratio Improvement) of 16.7 dB on the publicly available benchmark set WHAMR!, surpassing advanced baseline models such as SepFormer and WaveSplit, which have nearly 30M parameters. Furthermore, this scheme is the first to explicitly model the room impulse response in a single-channel environment, enabling a single device to effectively perceive and utilize the room's physical acoustic spatial information. This is effective not only in pure speech mixing but also significantly enhances the system's robustness in complex real-world scenarios containing heterogeneous reverberation interference from television, music, and conversations. Its extremely small parameter size and strong noise reduction and dereverberation performance make it highly valuable for commercial deployment on resource-constrained edge devices such as smartphones and hearing aids.

[0026] Please refer to Figure 1 and Figure 2 , Figure 1 This paper illustrates an architecture diagram of a speech separation model provided in one embodiment of this application. Figure 2 A flowchart of a speech separation model training method provided in an embodiment of this application is shown.

[0027] Please refer to Figure 1The speech separation model includes a separation module and a dereverberation module and a room impulse response estimation module arranged in parallel. The separation module is used to receive a single-channel mixed speech signal and output multiple initial separation signals with reverberation. The dereverberation module is used to receive multiple initial separation signals and output corresponding clean dereverberation signals. The room impulse response estimation module is used to extract a first feature embedding of each initial separation signal and extract a second feature embedding of the corresponding clean dereverberation signal, fuse the first feature embedding and the second feature embedding, and output a predicted first room impulse response filter and a second room impulse response filter.

[0028] Please refer to Figure 2 The training method for the speech separation model includes the following steps: Step 201: Convolve the clean dereverberation signal output by the dereverberation module with the first room impulse response filter and the second room impulse response filter output by the room impulse response estimation module in the short-time Fourier transform domain to reconstruct the reverberated estimated signal. Step 202: Calculate the reconstruction error between the reverberant estimated signal and the initial separation signal; Step 203: Based on the reconstruction error, the network parameters of the separation module, the dereverberation module, and the room impulse response estimation module are simultaneously optimized using the backpropagation algorithm to train the model.

[0029] The method in this application explicitly models the room impulse response in a single-channel environment, enabling a single device to effectively perceive and utilize the room's physical acoustic spatial information. This is effective not only in pure speech mixing but also significantly enhances the system's robustness in complex real-world scenarios involving heterogeneous reverberation interference from television, music, and dialogue. Furthermore, the reconstruction error is simultaneously backpropagated to the separation module, dereverberation module, and RIR estimation module, enabling the three modules to evolve collaboratively. This avoids the overall performance degradation caused by the local optima of any single module. By using the spatial propagation cues introduced by reverberation as implicit supervision signals, the traditional bottleneck of single-channel speech separation lacking spatial information can be overcome. Convolution operations are performed in the short-time Fourier transform domain, which significantly reduces computational complexity compared to time-domain convolution while maintaining the integrity of frequency domain information.

[0030] In some optional embodiments, the separation module uses the SPMamba architecture as its backbone network. The SPMamba architecture includes two-dimensional convolutional blocks, intra-frame bidirectional Mamba modules, inter-frame bidirectional Mamba modules, and two-dimensional deconvolutional blocks. Specifically, SPMamba uses bidirectional Mamba modules instead of the Transformer's self-attention mechanism, resulting in computational complexity that is linearly related to the input sequence length, significantly reducing the processing cost of long-duration speech sequences. Furthermore, compared to other architectures such as SepFormer and WaveSplit, SPMamba reduces the total number of parameters in the RAM framework to 7.2M, making it suitable for deployment on resource-constrained edge devices. Moreover, the combination of two-dimensional convolutional blocks and intra / inter-frame bidirectional Mamba modules can simultaneously capture the dependencies along both the frequency and temporal axes, improving the representation capability of reverberant speech.

[0031] In some optional embodiments, the dereverberation module includes a two-dimensional convolutional block, a cross-band block, a narrowband block, and a two-dimensional deconvolutional decoder. The dereverberation module utilizes the cross-band block to aggregate frequency band information to mitigate spectral tailing caused by reverberation, and utilizes the narrowband block to emphasize intra-band time-domain modeling, outputting a clean dereverberated signal corresponding to the target. Thus, by aggregating information between different frequency bands through the cross-band block, the spectral tailing phenomenon caused by reverberation can be effectively mitigated. Furthermore, the narrowband block performs independent time-domain modeling within each sub-band, emphasizing the temporal structure of the speech signal at each frequency component, preserving the harmonic characteristics and fundamental frequency information of the speech. This complementary design of the cross-band and narrowband blocks enables the dereverberation module to suppress reverberant tails without damaging the original timbre of the speech.

[0032] In some optional embodiments, the room impulse response estimation module includes a fusion layer, parallel narrowband blocks and weight blocks, a T-summation, and a convolutional transfer function decoder. The room impulse response estimation module generates a content-aware representation using the narrowband blocks, predicts time-frequency importance weights using the weight blocks, performs element-level modulation and time-axis aggregation using the T-summation, and outputs a predicted room impulse response filter through the convolutional transfer function decoder. The content-aware representation generated by the narrowband blocks allows the RIR estimation to be dynamically adjusted according to the speech content, rather than outputting a fixed room filter. The time-frequency importance weights predicted by the weight blocks allow the model to focus on moments and frequencies where reverberation is significant, improving the accuracy of the RIR estimation. Furthermore, the T-summation (time-axis summation) operation aggregates the modulated features along the time dimension, preserving key temporal information while reducing the computational burden on the subsequent CTF decoder. Moreover, the convolutional transfer function decoder outputs the RIR in the CTF domain, which is easier to convolve with the speech signal compared to the time-domain RIR and exhibits higher numerical stability.

[0033] In some optional embodiments, the speech separation model is trained using four complementary losses: a purified permutation-invariant training loss (PIT loss), a reverberation loss, a room impulse response (RIR) supervised learning loss, and a reconstruction loss that considers the RIR. The purified PIT loss addresses the ambiguity between the separation output and the source arrangement, ensuring a correct correspondence between the separation result and the real source. The RIR loss directly supervises the output of the dredging module, improving speech intelligibility. The RIR supervised learning loss provides direct supervision when real RIR labels are present, accelerating the convergence of the RIR estimation module. The RIR-perceptual reconstruction loss enforces self-consistency constraints, guiding the model to utilize spatial cues. Thus, through this multi-loss joint training, the model can maintain stable separation performance even when faced with unseen room acoustics and interference types.

[0034] In some alternative embodiments, training the model includes: based on the four complementary losses, forcing the clean dereverberant signal to be re-rendered back to the reverberant target signal through a predicted room impulse response filter, thereby guiding the model to use reverberation-induced propagation cues as weak spatial guidance for separating sound sources at different locations. This transforms the implicit reverberation propagation path into an explicit RIR filter representation, enabling the model to understand the spatial relationship between the sound source and the microphone, and, under single-channel conditions, learns to use reverberation-induced propagation differences as weak spatial guidance for distinguishing sound sources at different locations through rendering consistency constraints. Furthermore, this constraint only requires mixed and clean speech to take effect, without the need for expensive manual annotation of spatial locations or RIR measurement data.

[0035] In some embodiments, this application also provides a method for single-channel speech separation using a speech separation model trained by the aforementioned method, comprising: inputting a single-channel mixed speech signal into the separation module and outputting a plurality of initial separation signals with reverberation; sending the initial separation signals into the dereverberation module and outputting a clean dereverberation signal corresponding to each sound source; The feature embeddings of the initial separated signals and the feature embeddings of the clean dereverberation signals are fused and input into the room impulse response estimation module, which outputs the predicted room impulse response filter corresponding to each initial separated signal. During the inference stage (when the model is applied to speech separation), only the separation and dereverberation modules are enabled, while the RIR estimation module and loss calculation unit are disabled, significantly reducing computational overhead and memory usage. The clean dereverberation signal output by the dereverberation module is directly used as the final result and can be used for downstream tasks without post-processing.

[0036] The following specific example and related experimental data will enable those skilled in the art to better understand the solution of this application.

[0037] Advances in monophonic speech separation for recordings of real-world cocktail party scenarios remain insufficient, as these recordings often contain both strong background noise and complex room reverberation. Previous work has largely focused on multi-speaker mixed speech with additive noise, while the roles of reverberation and room acoustic structure have not been explicitly addressed for classification tasks. To address this limitation, this application proposes RAM (RIR-Aware Mamba-based Separation), a Mamba-based separation method that considers RIR. This method combines the separation task with a parallel RIR (Room Impulse Response) estimator and an RIR-aware objective function, enabling the model to utilize propagation cues induced by reverberation as spatial guidance. For comprehensive evaluation under heterogeneous cocktail party conditions, this application further constructs the HETMIXR dataset, a dual-source dataset containing a mixture of television audio, music, and conversational speech with noise and reverberation. Experiments show that RAM consistently outperforms strong baseline methods on both the HETMIXR dataset and the publicly available WHAMR! benchmark.

[0038] Significant progress has been made in monophonic speech separation in noisy and reverberant environments in recent years, largely thanks to the emergence of realistic benchmark sets. Traditional methods rely primarily on statistical modeling and signal processing techniques, such as Independent Component Analysis (ICA), Nonnegative Matrix Factorization (NMF), and heuristic spatial filtering. While effective in specific scenarios, these methods often exhibit limited separation performance and poor generalization in real-world environments with strong nonlinearity, complex acoustic characteristics, and dynamically varying numbers of sound sources. In recent years, data-driven paradigms, represented by Deep Neural Networks (DNNs), have fundamentally reshaped the field and significantly improved technical capabilities. Representative architectures include Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and more recent variants based on self-attention mechanisms (such as the Transformer model), as well as generative models such as Generative Adversarial Networks (GANs) and diffusion models. Despite these advances, most existing research still focuses solely on pure speech mixtures, viewing separation primarily as a problem utilizing speaker-dependent features such as timbre and speech content. However, in a real room, reverberation carries propagation-related structure and can provide faint spatial information, but this clue is still not fully utilized in mono separation.

[0039] To better utilize reverberation-induced spatial cues in solving the cocktail party problem in a real room, this application proposes RAM (RIR-Aware Mamba-based Separation), a framework based on the SPMamba backbone network with room acoustic perception capabilities. This framework introduces a selective state-space model, achieving linear time expansion of sequence length while maintaining strong capacity. RAM enhances SPMamba through a dreverberation module that optimizes the separated output into reverberation-free estimates; it also introduces a parallel RIR estimation branch to predict the RIR of each target component. Furthermore, this application introduces a convolutional reconstruction objective that considers RIR, forcing the dreverberation estimate to be re-rendered as a reverberant target through the estimated room filter. This encourages the model to utilize reverberation-induced propagation cues as weak spatial information to improve source discrimination.

[0040] Existing benchmarks reflect this focus on pure speech scenarios, including WSJ0-2mix, WHAM!, WHAMR!, and LibriMix. However, real-world cocktail party problems are often more challenging because realistic acoustic scenarios typically involve heterogeneous interference sources (e.g., a mixture of speech and television audio or music) and may contain inherent background noise and complex room effects. HETMIXR (Heterogeneous Reverberation Dataset) is a dual-source benchmark dataset containing noise and reverberation, covering four source categories: television audio, music, clean monologue, and dialogue. Television and dialogue recordings inherently contain background noise and slight echoes, making them closer to recordings in real-world environments. This application further renders all sources by simulating room impulse responses to create paired unecho and reverberant reference samples, enabling controlled evaluation of separation and dereverberation effects in heterogeneous noise-reverberation mixtures. On the HETMIXR and WHAMR! datasets, RAM achieved competitive performance with only 7.2 million parameters, outperforming several state-of-the-art systems with fewer than 30 million parameters, demonstrating its effectiveness in heterogeneous noise reverberation "cocktail party" scenarios.

[0041] The contributions of this application are summarized as follows: This is the first study to explicitly model the room impulse response (RIR) and inter-source relationships in a monophonic environment, thereby achieving source separation based on room information. This application constructs a dataset that, by integrating television, music, real-world dialogue, and speech, and combining it with room reverberation, more realistically reflects the mixed sound field of a cocktail party. Compared to existing state-of-the-art models with up to 30 million parameters, the method in this application outperforms both the self-built HETMIXR dataset and the publicly available WHAMR! benchmark. HETMIXR is a heterogeneous reverberation dataset.

[0042] To bridge the gap between traditional pure speech separation benchmarks and real cocktail party recordings, this application proposes HETMIXR (Heterogeneous Mixing Model with Reverb).

[0043] Figure 3 Sampling distribution of room impulse response (RIR) parameters in HETMIXR. All distance units are meters; T60 is seconds; angle units are radians. The English and Chinese translations are as follows: Room: Room dimensions, L: Length, W: Width, H: Height, T60: Reverberation time, high: High, med.: Medium, low: Low, Mic.Center: Microphone center, Mic.Array: Microphone array, Sources: Sound sources.

[0044] This application focuses on mono separation, such as using only the left channel. All components are spatialized to obtain paired anechoic and echolable rendering results. The anechoic direct signal is used as the dreverberation target, thus achieving dreverberation-conscious training without explicitly modeling the relative delay caused by spatialization.

[0045] Source Corpus HETMIXR's source material comes from four content categories: Dialogue: Two-person dialogues from the open-source NCSSD corpus, where the voices of the two speakers are provided as independent audio tracks. Clean speech (pure human voice): Readings from WSJ0, serving as clean, near-echo-free speech sources. Music (humming): Non-verbal melodic vocalizations from the HummingWav portion of the Tencent Music Lyra-QBH 2 dataset, used as challenging non-verbal interference sources. Television (YouTube programs): VoxCeleb audio clips extracted from open-source media, typically containing background noise and diverse acoustic environments. Since each sound source in the mixed signal may belong to any of the four categories mentioned above, HETMIXR covers 16 combinations of sound source types.

[0046] Dialogue preprocessing NCSSD provides two-person dialogues presented as discrete waveforms. To better reflect overlap in natural conversation, continuous recordings are first spliced ​​into a longer two-channel data stream. This data stream is then divided into fixed-length 4-second time windows, with windows containing only single-person speech being removed—meaning one channel remains silent for the entire 4-second interval. The remaining segments contain two-person dialogues with overlapping and alternating dynamics and are used as dialogue source categories in HETMIXR.

[0047] Co-localization dialogue rendering combined with ambient noise A key choice in the HETMIXR design involves dialogue categories. Because it assumes both speakers are in the same dialogue location, the same RIR is applied to both speakers when synthesizing the dialogue sound sources. In contrast, competing mix components are rendered with different RIRs to simulate a second sound source located in a different location, such as a distant television or music playback. By incorporating the background noise inherent in categories like music, television audio, and dialogue, this design generates a mixed signal that more realistically reflects the characteristics of a cocktail party recording, while still supporting pairwise supervised learning.

[0048] RAM: RIR-based mono sound source separation Problem Definition The "cocktail party problem" aims to extract multiple sound sources from a mono mix. It assumes two sound sources originate from different locations and arrive at the microphone via different acoustic paths. The observed mix can be modeled as follows: Where y(t) represents the microphone signal, s i (t) is the signal from the i-th sound source (which may contain background noise), h i (t) represents the corresponding room impulse response (RIR), and * denotes convolution. The goal of mono source separation is to learn a deep neural network f. θ The network maps the mixed signal to estimates of the potential sound sources: Overall process Continue to refer to Figure 1 The RAM consists of a separation module, a dereverberation module, and an RIR estimation module, which are coupled to each other through an RIR-sensing objective function. The separation module follows the SPMamba architecture, taking y(t) as input and generating two separate reverberation waveforms ˆx1(t) and x2(t). Subsequently, the dereverberation module de-evaluates each ˆx1(t) and generates two separate reverberation waveforms ˆx1(t) and x2(t). i (t) is optimized into a clearer estimate sˆ i (t). Finally, the RIR estimation module is used for (ˆx) i , sˆ i The fusion representation of ) is processed to predict ˆh i During the training process, ˆh was used i The reverberation signal is reconstructed from the dereverberated estimate, thus ensuring room acoustic consistency.

[0049] Reverb Cancellation Module The dereverberation module maps each separated reverberation stream to a clearer estimate. A cross-band module mitigates spectral blurring caused by reverberation by aggregating information from each frequency bin, while a narrowband module focuses on in-band time-domain modeling to suppress late-reverberation tails. The module outputs the dereverberated waveform sˆ i (t).

[0050] RIR estimation module To utilize the room acoustic structure in a mono setup, this application employs a state-of-the-art (SOTA) RIR estimator to infer a room filter for a specific sound source for each separated output. Let z i rev and z i clean They respectively represent from ˆx i and sˆ i The extracted features are embedded (e.g., encoder domain features). The fusion layer combines them into... Where α and β are learnable scalars. For example... Figure 1 As shown, the fusion embedding z i The input is fed into two parallel branches. The narrow-band branch generates the content-aware representation u. i (t, f), while the weighted branch predicts the time-frequency importance weight w i (t, f). This application employs element-level modulation operations to obtain ˜u. i (t, f) = u i (t, f) ⊙ w i (t, f), where ⊙ represents the element-wise inner product. Subsequently, the T-sum pooling layer aggregates the modulated features along the time axis: Finally, the CTF decoder will ¯u i (f) Mapping to room filter estimate h i This design prioritizes frames that provide the most informational description of room characteristics while generating compact RIR estimates for downstream consistency constraints.

[0051] Figure 1 An overview of the RAM is shown. The separation module outputs two reverb streams {ˆx i These streams, after being processed by the dreverb module, result in {sˆ} i The RIR estimation module uses the fused reverberation / deverberation features (learnable α, β) to predict {ˆh}. i}, while the RIR-aware loss function ensures sˆ during training. i∗ ˆh i ≈ ˆx i The English-Chinese translation is as follows: input: input signal (single-channel mixed speech signal), Separation module: separation module, Dereverberation module: de-reverberation module, RIR estimation module: room impulse response estimation module, Conv2D Block: 2D convolutional block, GridNet Block: grid network module, Intra Bi-Mamba Block: intra-frame bidirectional Mamba module, Mamba: state-space sequence modeling network, Inter Bi-Mamba Block: inter-frame bidirectional Mamba module, Scaling factor: scaling factor, Query: query, Value: value, Soft-max: soft maximization, Matrix: matrix, ConvTrans2D Block: 2D deconvolutional block, Fusion Layer: fusion layer, Cross-band Block: cross-band block, ConvTrans2D Decoder: 2D deconvolutional decoder, Narrow-band Block: narrowband block, T-sum: T summation (time axis summation), Weight Block: weight block, CTF Decoder: convolutional transfer function decoder, Time: time / time axis, Output RIR: output room impulse response.

[0052] loss function The RAM is trained and supervised by four complementary objectives: (i) echo-free domain separation, (ii) reverberation spectrum prediction, (iii) direct RIR supervision, and (iv) RIR-based reconstruction. The last objective is the main contribution of this application. By constraining the dereverberation estimate to resynthesize the reverberation objective through the estimated room filters, the model is guided to utilize room-dependent reverberation features from different propagation paths as weak spatial cues for monophonic source separation.

[0053] Let S i ∈ C T ×F and X i ∈ C T ×F Let Si represent the true anechoic and reverberant short-time Fourier transform (STFT) spectra of sound sources i∈1,2, respectively. The model outputs the de-reverberation spectrum Sˆ. i The separated reverberation spectrum ˆX i And the room filter representation in the STFT domain ˆH iThe overall loss function is: Where λ rvb , λ recon γ and γ are weight hyperparameters.

[0054] Purified PIT loss. This application applies Permutation Invariant Training (PIT) to the dereverberated output to eliminate uncertainty in the sound source order: Where Π2 represents all permutations of {1, 2}, L cln (·, ·) is the SI-SDR loss in the waveform domain.

[0055] Reverberation loss. This application directly supervises the reverberation spectrum predicted by the separation module: Loss in RIR supervised learning. When real CTF labels are available, the estimated room filter is directly learned under supervised supervision in the following way: Considering the reconstruction loss of RIR, to combine dereverberation with room filter estimation, this application reconstructs the reverberant target signal by convolving the estimated CTF with the clean target signal in the STFT domain: in, This represents the CTF convolution of each frequency bin along the time direction. This constraint helps to achieve a factorization that takes into account room acoustics, thereby utilizing the propagation differences caused by reverberation as spatial guidance for monochannel separation. L RI+Mag LRI+Mag is a frequency domain reconstruction loss function that sums the amplitude of the complex spectrum along with the L1 errors of its real and imaginary parts, thereby facilitating accurate complex spectrum matching. For detailed information on LRI+Mag, please refer to the relevant technical documentation.

[0056] 4. Experimental Setup Model Setup. The separation module runs in the STFT domain, using a Hann window with an FFT size of Nsep = 256 and a jump size of Hsep = 64. The separation backbone network uses a hidden layer size of Chid = 256, nh = 4 attention heads, and an approximate QK dimension of dqk = 512. 2D embeddings use emb dim = 16, a patch size of emb ks = 8, and a jump stride of emb hs = 1. The PReLU activation function is used, and the normalization layer uses... = 10-5.

[0057] Training and Evaluation. Training data consisted of a 4-second hybrid segment randomly sampled from 8 kHz minimum segments of WHAMR! and HETMIXR. The Adam algorithm was used with a learning rate of 0.001 and a gradient norm upper bound of 5. Training was stopped if the validation set performance did not improve for 5 consecutive epochs. The evaluation report included SI-SNRi and SDRi metrics on the test set.

[0058] Results and Analysis Comparison with state-of-the-art baseline methods Figure 4 WHAMR! Comparison of changes with model size. The English and Chinese translations are as follows: SI-SDRi: Improved signal-to-noise ratio, SDRi: Improved signal-to-distortion ratio, Params: Number of parameters, RAM: This method.

[0059] Figure 5 : The separation performance of HETMIXR and specific content subsets.

[0060] Among them, Contains-TV (containing television (interference)), Contains-Music (containing music (interference)), Contains-Dialog (containing dialogue), and Pure-Speech (pure speech) are all specific content subsets, which are the four content categories of the HETMIXR dataset.

[0061] This application evaluates RAM on the publicly available WHAMR! benchmark dataset and the HETMIXR dataset proposed in this application. Figure 4 Under practical model size constraints, RAM was compared with representative baseline methods on the publicly available WHAMR! benchmark dataset. Despite having only 7.2 million parameters, RAM achieved scores of 16.7 SI-SDRi (Scale-Invariant Signal-to-Distortion Ratio improvement, a performance metric for speech separation) and 15.3 SDRi, matching or surpassing several methods with nearly 30 million parameters, including SepFormer and WaveSplit. Explicit acoustic modeling allows RAM to achieve strong separation without a large number of parameters, making it ideal for resource-constrained deployments.

[0062] Subsequently, the same batch of models were deployed on the HETMIXR dataset for evaluation, which is designed to better simulate cocktail party mixed scenarios containing heterogeneous interference sources. Figure 5Results are reported on the full HETMIXR test set and four content-defined subsets. While SPMamba performed strongly on WHAMR!, its performance declined on HETMIXR, indicating a mismatch between pure speech benchmarks and realistic mixed scenarios including TV audio, music, and dialogue. In contrast, RAM performed best across all subsets, consistently outperforming SPMamba and TIGER. On the full HETMIXR test set, RAM improved SPMamba's SDRi and SI-SNRi metrics from 13.45 and 13.70 to 14.36 and 14.78, respectively. This performance improvement remained consistent across different mixed content. Compared to SPMamba, RAM improved SDRi and SISNRi metrics by 0.88 and 1.05 in the "with TV" subset, 0.87 and 0.98 in the "with music" subset, and 0.95 and 1.08 in the "with dialogue" subset. RAM performance in pure speech scenarios also improved from 14.92 and 15.91 to 15.93 and 17.20, indicating that explicit room modeling is beneficial for both heterogeneous mixed signals and pure speech mixed signals.

[0063] Figure 6 HETMIXR category comparison. The radar chart shows the average SDRi (left) and average SI-SNRi (right) for RAM, SPMamba, and TIGER across various source type mix categories. Here, D, M, S, and T represent dialogue, music, pure speech, and television audio, respectively, while & represents source type pairs, such as D&M for dialogue-music.

[0064] Figure 6 Further finer-grained decomposition by mixed signal category revealed that RAM maintained a stable advantage over SPMamba in most categories. (Note: Mean SDRi: Average Improved Signal-to-Distortion Ratio; Mean SI-SDRi: Average Improved Signal-to-Noise Ratio.)

[0065] Ablation Research Figure 7 An ablation study for HETMIXR is reported, which isolates the contribution of each training loss term. The pure PIT baseline method achieves 13.45 dB SDRi and 13.70 dB SI-SNRi. Adding reverberation spectral supervision improves performance to 14.10 dB SDRi and 14.46 dB SI-SNRi, demonstrating the benefit of explicitly constrained reverberation estimation in noisy and reverberant mixed signals.

[0066] Figure 7The ablation study of HETMIXR. The English and Chinese translations are as follows: Variant, Clean, Rvb, Recon, Loss term, Metric.

[0067] To analyze the contribution of spatially-aware supervision, ablation experiments were conducted against the proposed target. Adding RIR loss only to the reverberation supervision yielded 14.02 dB SDRi and 14.36 dB SI-SNRi, indicating that regularization of the estimated spatial filter is beneficial. In contrast, adding only the reconstruction term without RIR loss reduced performance to 13.43 dB SDRi and 13.85 dB SI-SNRi, suggesting that constraints from the reconstruction term alone are insufficient and may tend to match the reverberation target rather than improve separation. Combining reconstruction with RIR loss yielded optimal performance, reaching 14.36 dB SDRi and 14.78 dB SI-SNRi, outperforming all ablation variants. This trend supports the argument of this application: reconstruction can only provide a reliable room-aware training signal when the estimated room filter is sufficiently constrained, thereby utilizing propagation cues induced by reverberation to improve monochannel source separation.

[0068] In this application, a cascaded monophonic source separation framework called RAM is proposed. This framework, built on an SPMamba backbone network, combines déreverberation with RIR estimation. To the best of our knowledge, RAM is the first single-channel source separation method to explicitly model the room impulse response (RIR). It utilizes a physics-inspired RIR perception objective function, taking advantage of reverberation-induced propagation cues as weak spatial information. For comprehensive evaluation, this application introduces HETMIXR, a heterogeneous noise-reverberation cocktail party dataset that better reflects mixed sound fields in the real world. Experiments on the HETMIXR and WHAMR! datasets demonstrate that, in noisy and reverberant environments, this method exhibits a significant performance improvement over robust baseline methods.

[0069] In other embodiments, this application also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the speech separation model training method in any of the above method embodiments; As one implementation, the non-volatile computer storage medium of this application stores computer-executable instructions, which are configured as follows: The clean dereverberation signal output by the dereverberation module is convolved with the first room impulse response filter and the second room impulse response filter output by the room impulse response estimation module in the short-time Fourier transform domain to reconstruct the reverberated estimation signal. Calculate the reconstruction error between the reverberated estimated signal and the initial separated signal; Based on the reconstruction error, the network parameters of the separation module, the dereverberation module, and the room impulse response estimation module are simultaneously optimized using a backpropagation algorithm to train the model.

[0070] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created according to the speech separation model training method and the use of the system, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the speech separation model training method via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0071] This application also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the above-described speech separation model training methods.

[0072] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 8 As shown, the device includes: one or more processors 710 and memory 720. Figure 8 Taking a processor 710 as an example, the device for the speech separation model training method and system may further include an input device 730 and an output device 740. The processor 710, memory 720, input device 730, and output device 740 can be connected via a bus or other means. Figure 8Taking a bus connection as an example, the memory 720 is the aforementioned non-volatile computer-readable storage medium. The processor 710 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 720, thereby implementing the speech separation model training method described in the above embodiment. The input device 730 can receive input numerical or character information and generate key signal inputs related to user settings and function control of the large language model routing device. The output device 740 may include a display screen or other display device.

[0073] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0074] In one implementation, the above-described electronic device is applied in a large-scale language model routing device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: The clean dereverberation signal output by the dereverberation module is convolved with the first room impulse response filter and the second room impulse response filter output by the room impulse response estimation module in the short-time Fourier transform domain to reconstruct the reverberated estimation signal. Calculate the reconstruction error between the reverberated estimated signal and the initial separated signal; Based on the reconstruction error, the network parameters of the separation module, the dereverberation module, and the room impulse response estimation module are simultaneously optimized using a backpropagation algorithm to train the model.

[0075] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones, multimedia phones, feature phones, and low-end phones, etc.

[0076] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, etc.

[0077] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0078] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0079] (5) Other electronic devices with data interaction functions.

[0080] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training a speech separation model, wherein, The speech separation model includes a separation module and a dereverberation module and a room impulse response estimation module arranged in parallel. The separation module is used to receive a single-channel mixed speech signal and output multiple initial separation signals with reverberation. The dereverberation module is used to receive multiple initial separation signals and output corresponding clean dereverberation signals. The room impulse response estimation module is used to extract a first feature embedding of each initial separation signal and extract a second feature embedding of the corresponding clean dereverberation signal, fuse the first feature embedding and the second feature embedding, and output a predicted first room impulse response filter and a second room impulse response filter. The training method includes: The clean dereverberation signal output by the dereverberation module is convolved with the first room impulse response filter and the second room impulse response filter output by the room impulse response estimation module in the short-time Fourier transform domain to reconstruct the reverberated estimation signal. Calculate the reconstruction error between the reverberated estimated signal and the initial separated signal; Based on the reconstruction error, the network parameters of the separation module, the dereverberation module, and the room impulse response estimation module are simultaneously optimized using a backpropagation algorithm to train the model.

2. The method according to claim 1, characterized in that, The separation module adopts the SPMamba architecture as the backbone network, which includes a two-dimensional convolutional block, an intra-frame bidirectional Mamba module, an inter-frame bidirectional Mamba module, and a two-dimensional deconvolutional block.

3. The method according to claim 1, characterized in that, The dereverberation module includes a two-dimensional convolutional block, a cross-band block, a narrowband block, and a two-dimensional deconvolutional decoder. The dereverberation module uses the cross-band block to aggregate frequency band information to reduce the spectral tail caused by reverberation, and uses the narrowband block to emphasize in-band time-domain modeling to output a clean dereverberation signal corresponding to the target.

4. The method according to claim 1, characterized in that, The room impulse response estimation module includes a fusion layer, parallel narrowband blocks and weight blocks, a T-summation, and a convolutional transfer function decoder. The room impulse response estimation module generates a content-aware representation through the narrowband blocks, predicts time-frequency importance weights through the weight blocks, performs element-level modulation and time-axis aggregation of the T-summation, and outputs a predicted room impulse response filter through the convolutional transfer function decoder.

5. The method according to claim 1, characterized in that, The speech separation model is trained using four complementary losses: purified permutation-invariant training loss, reverberation loss, loss from room impulse response supervised learning, and reconstruction loss that takes room impulse response into account.

6. The method according to claim 5, characterized in that, Training the model includes: Based on the four complementary losses, the clean dereverberation signal is required to be re-rendered back to the reverberant target signal through the predicted room impulse response filter, thereby guiding the model to use reverberation-induced propagation cues as weak spatial guidance for separating sound sources at different locations.

7. A method for single-channel speech separation using a speech separation model trained according to any one of claims 1 to 6, comprising: A single-channel mixed speech signal is input to the separation module, which outputs multiple initial separation signals with reverberation. The initial separation signal is sent to the dereverberation module, which outputs a clean dereverberation signal for each sound source. The feature embeddings of the initial separated signals and the feature embeddings of the clean dereverberation signals are fused and input into the room impulse response estimation module, which outputs the predicted room impulse response filter corresponding to each initial separated signal.

8. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images