A full-modal sound source separation method supporting extraction and cancellation modes
By using a time-frequency neural network based on subband structure and conditional modulation, the problems of insufficient frequency band modeling and poor reconstruction quality in existing sound source separation methods are solved, achieving high-quality target extraction and elimination, and enhancing the multimodal adaptability and flexibility of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INST OF ACOUSTICS CHINESE ACAD OF SCI
- Filing Date
- 2026-04-07
- Publication Date
- 2026-07-14
Smart Images
Figure CN122392560A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of audio processing, and in particular to a method for full-modal sound source separation that supports extraction and elimination modes. Background Technology
[0002] Query-driven sound source separation refers to separating or removing a target sound source from a mixed audio stream based on a user-provided query signal. This technology can be applied to tasks such as audio retrieval, target enhancement, audio editing, video post-production, and controllable content generation. Existing methods mostly estimate the target mask in the time-frequency domain and then reconstruct it using the phase of the mixed signal. These methods typically suffer from the following problems: First, the separation network often uses U-Net and its variants, which lacks sufficient modeling for the differences between different frequency bands, making it difficult to simultaneously consider local frequency band details and cross-band consistency. Second, reconstruction methods based primarily on amplitude masks have limited utilization of phase information, leading to transient blurring, harmonic distortion, and a decline in auditory quality. Third, existing systems are mostly designed for target extraction, and target elimination is usually achieved through complementary masks or indirect inference, making it difficult to stably support two opposite operations within the same model. Therefore, it is necessary to improve the separation of the backbone network, conditional control, and complex spectrum reconstruction so that the same model can complete target extraction and target elimination based on multimodal queries, and improve the quality of the separation results. This is a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0003] To address the shortcomings of existing separation networks, which mostly use U-Net and its variants, insufficient modeling of different frequency bands, making it difficult to simultaneously consider local frequency band details and cross-band consistency; the limited use of phase information in amplitude mask-based reconstruction methods, resulting in transient blurring, harmonic distortion, and degraded listening experience; and the fact that existing systems are mostly designed for target extraction, with target elimination typically achieved through complementary masks or indirect inference, making it difficult to stably support two opposite operations within the same model. The purpose of this invention is to overcome the shortcomings of the prior art and to propose a full-modal sound source separation method that supports extraction and elimination modes.
[0004] In view of this, the present invention proposes a full-modal sound source separation method that supports extraction and elimination modes, comprising: Step 1: Obtain the mixed audio to be separated, and perform a short-time Fourier transform on the mixed audio to obtain the complex spectrum; Step 2: Input the complex spectrum, user-input query information, and operation instructions into the pre-built and trained separation model. Based on the operation instructions, if extraction or elimination is required, output the target extracted audio or the target eliminated audio. The separation model is a time-frequency neural network based on subband structure and conditional modulation, including: Conditional modulation module: includes query condition branch and operation condition branch, used to encode input conditions to generate corresponding modulation parameters, and output the modulation parameters to other modules to conditionally modulate the feature representation; Subband partitioning module: used to perform non-uniform subband partitioning and compressed encoding on the input complex spectrum, obtain subband features, and perform feature splicing of each region; Attention-Cyclic Time-Frequency Module: Composed of multiple cascaded attention-cyclic time-frequency backbone blocks, used for time-frequency spatial modeling of sub-band features, enabling joint control based on query information and operation indicators; and Subband merging module: It uses parallel decoding of real and imaginary branches, then splices them into a full-band complex spectrum, and obtains the time-domain audio through inverse short-time Fourier transform.
[0005] As an improvement to the above method, the short-time Fourier transform in step 1 uses a Hann window with a window length of 1024 and a frame shift of 256 to obtain the complex spectrum and then construct the complex spectrum input tensor.
[0006] As an improvement to the above method, the query information in step 2 includes at least one multimodal query content among text, image, video and reference audio. A pre-trained multimodal coding model is used to map the query information to a unified semantic space to obtain a query condition vector.
[0007] As an improvement to the above method, the query condition vector in the query condition branch, after being activated by SiLU, is transformed into multiple sets of dedicated modulation parameters for the adaptation separation network through multiple linear mapping heads. These dedicated modulation parameters include: backbone block internal condition modulation parameters. Modulation parameters at the encoding end and the two sets of modulation parameters at the decoding end and The operation condition branch uses a binary operation indicator to switch between extraction and elimination modes. The operation indicator is mapped to an operation vector through an embedding layer, thereby generating operation modulation parameters. .
[0008] As an improvement to the above method, the query condition branch includes a lightweight adaptive condition injection module and condition modulation parameters within the backbone block. With operating modulation parameters They are respectively fed into the lightweight adaptive conditional injection module. After processing, the parameters are split into multiple sets, which are used to perform translation, scaling, and gating modulation on the four sub-modules of the attention-cycle time-frequency module: time multi-head attention, time bidirectional cycle, feedforward, and frequency bidirectional cycle.
[0009] As an improvement to the above method, the subband partitioning module divides the input complex spectrum tensor into three non-uniform frequency regions, performs two-dimensional convolution and layer normalization processing on each frequency region, and then processes the input... First, perform inter-band normalization to obtain the input. Modulation parameters at the encoding end Rearrangement as translation parameter With scaling parameters Then and As a conditional injection: ; And obtain the output of the sub-band partitioning encoding module ; in, The input is after band normalization. This represents element-wise multiplication. This indicates a characteristic linear modulation operation.
[0010] As an improvement to the above method, the attention-cyclic time-frequency backbone block adopts a staged conditional modulation mechanism. Conditional modulation is performed on the input features during the backbone block input stage and the processing stage of each sub-module. Specifically, it includes: first, normalization and operation modulation are performed on the block input features to obtain the initial modulation result; then, normalization and query modulation are performed before the four sub-modules of time multi-head attention, time bidirectional loop, feedforward, and frequency bidirectional loop respectively; after the calculation of each sub-module is completed, the sub-band features output by the fusion of the gated residual and the main branch are input into the sub-band merging module for merging and decoding.
[0011] As an improvement to the above method, the subband merging module uses parallel decoding of real and imaginary branches. After normalizing and conditionally modulating the separation features, it splits them into three groups of subbands and sends them to the real and imaginary branches respectively. After reconstruction by two-dimensional point convolution, GELU activation, and transposed convolution, the outputs of the real branches of the three groups of subbands are spliced along the frequency dimension to obtain the estimated real part, and the outputs of the imaginary branches of the three groups of subbands are spliced along the frequency dimension to obtain the estimated imaginary part. Then, the reconstruction results of the three groups of subbands are spliced along the frequency dimension to obtain the full-band estimated complex spectrum, and the target time domain output is obtained by inverse short-time Fourier transform.
[0012] As an improvement to the above method, the separation model training adopts a direct complex spectrum reconstruction training method, and the loss function is a logarithmic magnitude loss. Generator loss Real-to-virtual part loss , Feature matching loss The weighted sum, where, The phase loss employs omnidirectional phase loss, the real-to-imaginary part loss employs omnidirectional real-to-imaginary loss, and these are combined with logarithmic magnitude loss, generative adversarial loss, and feature matching loss for network optimization; multiple perspectives constrain the separation model together, resulting in a complete training objective function. for: ; in, , , , and These are the weighting coefficients for the corresponding loss terms.
[0013] The advantages of this invention compared to existing technologies are: 1. This invention achieves the synergistic effect of multi-stage conditional information by modulating query parameters and operation parameters of the sub-band division module, attention-cycle time-frequency module, and sub-band merging module, thereby effectively improving the target extraction and target elimination effects; 2. By combining non-uniform subband division with direct reconstruction of complex spectrum, on the one hand, fine modeling of local structure in different frequency bands is carried out and cross-frequency band information interaction is taken into account, and on the other hand, the dependence on mixed phase is reduced, thereby improving the overall sound quality of the generated audio. 3. By introducing operation indicators, target extraction and target elimination functions can be implemented simultaneously in the same model; 4. This invention can also uniformly process query conditions in the form of text, images, videos, and reference audio and their combinations, thereby improving the multimodal adaptability and application flexibility of the model; 5. This invention is supported by experimental data, and its extraction and separation effects are far superior to similar methods. Attached Figure Description
[0014] Figure 1 This is a schematic diagram of the overall process of the present invention; Figure 2 This is a schematic diagram of the sub-band partitioning module and the sub-band merging module. Figure 3 This is a schematic diagram of the attention-cycle time-frequency backbone structure; Figure 4 This is a schematic diagram of omnidirectional phase convolution calculation; Figure 5 This is a quantitative analysis graph of target extraction operations performed on the VGGSOUND-CLEAN+ benchmark set. Detailed Implementation
[0015] The technical solutions provided in this application are further illustrated below with reference to the embodiments.
[0016] The process of this invention is as follows Figure 1As shown, the present invention will be further described below with reference to specific embodiments: Example 1: Embodiment 1 of the present invention proposes a full-modal sound source separation method that supports extraction modes.
[0017] Step 1: Prepare training and testing data. This invention uses an open-source dataset for training, with an input audio length of 4 seconds and a sampling rate of 16kHz. Text queries use the video titles of samples in the dataset; image queries use 4 frames of images sampled from the video at 1-second intervals; reference audio queries use the complete audio of the corresponding samples during the training phase, and 5 reference samples for each class during the testing phase.
[0018] Step Two: Constructing the Separation Network. This invention employs a sub-band structure-based separation network, consisting of an attention-cyclic time-frequency module, a sub-band partitioning module, and a sub-band merging module. Given mixed audio... and user query The model outputs the target signal as .in, Indicates the input mixed waveform; Indicates the first One source signal; Indicates the number of mixed sources; This represents a query corresponding to the target source. The separation network is represented. The query encoder uses a pre-trained multimodal coding model to map text, images, videos, and reference audio to a unified semantic space, obtaining a query condition vector. In this embodiment, Imagebind is selected as the pre-trained multimodal coding model. A short-time Fourier transform (STFT) is performed on the input audio using a Hann window with a window length of 1024 and a frame shift of 256, yielding a complex spectrum. And construct the complex spectrum input tensor In the formula, Indicates the size of the frequency dimension; Indicates the total number of time frames; This indicates the operation of taking the real part; This indicates the imaginary part operation. The binary control flag is... The condition is mapped to an operation vector through the embedding layer. The condition modulation module consists of a query condition branch and an operation condition branch. Query condition vector First, the core block is activated by SiLU, and then four linear mapping heads are input to obtain the internal conditional modulation parameters. Modulation parameters at the encoding end and the two sets of modulation parameters at the decoding end and ; Operations on embedding vectors The operating modulation parameters are obtained after independent linear mapping. These modulation parameters are used to generate the scaling, translation, and gating parameters for each module. Specifically, in the query condition branch... The output dimension is , The output dimension is , and The output dimensions are all In the aforementioned operating condition branch, The output dimension is . Indicates the channel dimension, when At that time, the five output dimensions mentioned above are 3072, 512, 512, 512 and 768 respectively.
[0019] The internal structure of the sub-band partitioning coding module is as follows: Figure 2 As shown in the left figure, this module divides the input complex spectral tensor along the frequency dimension into For each of the three non-uniform frequency regions, perform two-dimensional convolution and layer normalization processing to obtain the compressed sub-band features. , And splice the features of each region into , in , Indicates the total number of sub-bands. Indicates the number of frequency regions. Indicates the first Input in one frequency range; Represents two-dimensional convolution; Representation layer normalization; Indicates the channel dimension; Indicates the first Number of subbands after compression in each region; Indicates a splicing operation; This represents the regional characteristics of the first to the Rth frequency regions. This invention divides the frequency dimension into three non-uniform regions, with the boundary points located at frequency indices 144, 336, and 513, respectively. The region inputs are as follows: , , The kernel size for each region is set to... The corresponding step size is set to After three-region convolutional encoding, 12, 8, and 4 sub-band features are obtained respectively, and the total number of sub-bands after concatenation is 24. Modulation parameters at the encoding end. Rearranged into two groups of length The vectors are used as translation parameters for the encoded output features. With scaling parameters Input to the time-frequency joint modeling network First, perform inter-band normalization, then inject the modulation parameters as conditions: , in FiLM represents element-wise multiplication and characteristic linear modulation operation. This represents the input after layer normalization.
[0020] Each attention-recurrent time-frequency backbone block of the time-frequency joint modeling network includes, in sequence, a time multi-head attention module, a time bidirectional recurrent module, a feedforward module, and a frequency bidirectional recurrent module, with the internal structure as follows: Figure 3 As shown. The channel dimension in this invention... The number of cascaded backbone blocks is 8. Given the input... First, temporal multi-head attention is performed along the temporal dimension for each sub-band to explicitly model the global temporal correlation within the same sub-band. The number of temporal multi-head attention heads is set to 8, and the number of channels per head is set to 32. The temporal bidirectional recurrent module supplements the local continuous frame modeling capability in the temporal dimension and reduces deep training degradation through residual connections. The intermediate layer channel dimension is set to 128. The processing flow of the feedforward sub-unit is as follows: after normalization, it passes through the first layer... Pointwise convolution expands the channels to 512, is activated by GELU, and then passed through a second layer. Pointwise convolutions are fed back to the original channel dimension and finally added to the residual branch. The frequency bidirectional cyclic module models by sub-band dimension, performing bidirectional cyclic operations on each time frame to recover the correlation between frequency regions after sub-band division. The intermediate layer channel dimension is also set to 128. In the time-frequency joint modeling network, the conditional modulation parameters... With operating modulation parameters The lightweight adaptive conditional injection module is input separately; this embodiment uses the AdaLN-SOLA module. Low-rank parameter rank and scaling factor All are set to 32. The AdaLN-SOLA module uses a low-rank adaptation method to generate additional modulation increments. Let the low-rank projection matrix be... The upper projection matrix is The increment generated by the query condition vector can be written as , The final modulation vector is formed by adding it to the basic modulation parameters. This structure reduces the number of new parameters while maintaining conditional expressiveness. Modulation parameters within the backbone block. After processing by the AdaLN-SOLA module, it was split into 12 groups with a length of The sub-vectors correspond to the modulation parameters of the four sub-modules: temporal multi-head attention, temporal bidirectional loop, feedforward, and frequency bidirectional loop.
[0021] Input features for any submodule The conditional modulation result is .
[0022] in, Representation layer normalization, Indicates the operation of the current submodule. Indicates the scaling parameter; Indicates the translation parameter; Indicates gating parameters. Operating modulation parameters. After processing by the AdaLN-SOLA module, it is split into three groups with a length of Subvectors: , The conditional modulation sequence for each backbone block is as follows: First, normalization and operational modulation are performed on the block input features to obtain the initial modulation result; then, normalization and query modulation are performed before the four sub-modules: temporal multi-head attention, temporal bidirectional loop, feedforward, and frequency bidirectional loop; after each sub-module has completed its calculation, the results are fused with the main branch through gating residuals. Thus, the operational conditions determine the "extraction / elimination" direction of the current block, and the query conditions determine the target semantics, forming a well-defined joint control within the same block.
[0023] The internal structure of the sub-band merging decoding module is as follows: Figure 2 As shown in the right figure, this module uses parallel decoding with real and imaginary branches. First, the output... Normalization and conditional modulation are performed, and then the backbone output is decomposed into... , and The real and imaginary branches are fed into the data respectively, and two-dimensional pointwise convolution, GELU activation, and transposed convolution are performed on each region to obtain the region reconstruction results: , in, For GELU activation function, This is the result of the region reconstruction. The transposed convolution size is set to... The corresponding step size is set to The estimated real part is obtained by concatenating the outputs of the real branches in the three regions along the frequency dimension, and the estimated imaginary part is obtained by concatenating the outputs of the imaginary branches in the three regions along the frequency dimension. The reconstruction results of the three regions are then concatenated along the frequency dimension to obtain the estimated complex spectrum for the entire frequency band. Then perform the inverse short-time Fourier transform (iSTFT) to obtain the target time-domain output. .
[0024] The separation network designed in this invention has approximately 24.95M parameters and a computational complexity of approximately 9.73 GMacs per second. During training, the batch size is set to 16, the total number of generator iterations is 1,000,000, and the AdamW optimizer is used. The learning rate is initialized to 0.0001, and an exponential decay strategy is used to adjust the learning rate, with the decay rate set to 0.999.
[0025] Step 3: Loss Function Selection and Training. This invention employs a direct complex spectrum reconstruction training method. Omnidirectional phase loss and omnidirectional real-imaginary part loss are used for phase loss and real-imaginary part loss, respectively. These are combined with logarithmic amplitude loss, generative adversarial loss, and feature matching loss for network optimization.
[0026] Omnidirectional phase loss is calculated by omnidirectional phase convolution to estimate the omnidirectional difference between the phase and the target phase. The process is as follows: Figure 4 As shown. The convolution kernel with fixed parameters is designed as follows: Used to calculate the omnidirectional phase difference after convolution. ,in Indicates the estimated phase spectrum; Represents the target phase spectrum; This represents the convolution operation; and These represent the differential mappings of the estimated phase and the target phase in the local direction, respectively. The phase loss is defined as... , in, Defined as an anti-wrapping function, and These represent the frequency index and the time frame index, respectively. Indicates the rounding operation; Pi is a constant.
[0027] The omnidirectional real-imaginary part loss is first decoupled from the amplitude and phase, and then the omnidirectional operation is applied to the phase part, defined as:
[0028] in, and These are the network output complex spectrum and the target complex spectrum, respectively. Indicates from arrive Broadcast extensions.
[0029] Log-amplitude loss is the mean-square error (MSE) between the log-estimated amplitude and the target amplitude spectrum, defined as follows:
[0030] in, At frequency index With time frame index The value at time, and These represent the frequency dimension size and the number of time frames, respectively. This represents the logarithmic operation. express Norm.
[0031] For adversarial loss, this invention employs a Multi-resolution Spectrogram Discriminator (MRSD) for adversarial learning. The MRSD comprises three sub-discriminators, with their window length, frame shift, and number of Fourier transform points set as follows: The adversarial loss, employing the Hinge form, is defined as follows:
[0032]
[0033] Feature matching loss is defined as:
[0034] in, Indicates the total number of sub-discriminators. Indicates the first Individual discriminator output; Represents the target audio waveform; Indicates the network-estimated waveform; This indicates the operation of retrieving the maximum value; Indicates the first Individual Discriminator Intermediate features of the layer; E represents the number of layers used for feature matching; express norm The complete training objective function is , in and These are the weighting coefficients for the corresponding loss terms. The weight values for each loss term are as follows: .
[0035] Step 4: Use the trained model for target extraction. When the operation indicator is set to extract, the model outputs the target sound source corresponding to the query semantics, which is suitable for text query extraction, image query extraction, reference audio query extraction, and multimodal joint query extraction.
[0036] Experimental results for target extraction on the VGGSOUND-CLEAN+ benchmark set are as follows: Figure 5 As shown, the metrics include: signal distortion ratio (SDR), scale-invariant signal distortion ratio (SI-SDR), logarithmic spectral distance (LSD), peak signal-to-noise ratio (PSNR), and omnidirectional phase-aware signal-to-noise ratio (GOMPSNR). Perceptual consistency metrics: Learnable perceptual audio block similarity (LPAPS), CLAP-based audio consistency score (CLAP-A), and CLAP-based text consistency score (CLAP-T). Distribution similarity index: Kullback-Leibler divergence (KL); The experimental data show that all experimental indicators of this invention are superior to those of commonly used existing methods.
[0037] Example 2 Embodiment 2 of the present invention proposes a full-modal sound source separation method that supports elimination mode.
[0038] This embodiment uses the same network structure, training objective, and parameter settings as Embodiment 1, the only difference being that the operation indicator is set to "eliminate". In this case, the model output is the remaining scene audio after removing the target sound source. Therefore, it is clear that there is no need to build a separate elimination model; simply switching the operation indicator can output the elimination result under the same parameter set, making it suitable for query-driven audio editing scenarios.
[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A method for full-modal sound source separation supporting extraction and elimination modes, comprising: Step 1: Obtain the mixed audio to be separated, and perform a short-time Fourier transform on the mixed audio to obtain the complex spectrum; Step 2: Input the complex spectrum, user-input query information, and operation instructions into the pre-built and trained separation model. Based on the operation instructions, if extraction or elimination is required, output the target extracted audio or the target eliminated audio. The separation model is a time-frequency neural network based on subband structure and conditional modulation, including: Conditional modulation module: includes query condition branch and operation condition branch, used to encode input conditions to generate corresponding modulation parameters, and output the modulation parameters to other modules to conditionally modulate the feature representation; Subband partitioning module: used to perform non-uniform subband partitioning and compressed encoding on the input complex spectrum, obtain subband features, and perform feature splicing of each region; Attention-Cyclic Time-Frequency Module: Composed of multiple cascaded attention-cyclic time-frequency backbone blocks, used for time-frequency spatial modeling of sub-band features, enabling joint control based on query information and operation indicators; and Subband merging module: It uses parallel decoding of real and imaginary branches, then splices them into a full-band complex spectrum, and obtains the time-domain audio through inverse short-time Fourier transform.
2. The full-modal sound source separation method supporting extraction and elimination modes according to claim 1, characterized in that, The short-time Fourier transform in step 1 uses a Hann window with a window length of 1024 and a frame shift of 256 to obtain the complex spectrum and then construct the complex spectrum input tensor.
3. The full-modal sound source separation method supporting extraction and elimination modes according to claim 1, characterized in that, The query information in step 2 includes at least one multimodal query content among text, image, video and reference audio. A pre-trained multimodal coding model is used to map the query information to a unified semantic space to obtain a query condition vector.
4. The full-modal sound source separation method supporting extraction and elimination modes according to claim 3, characterized in that, The query condition vector in the query condition branch, after being activated by SiLU, is transformed into multiple sets of dedicated modulation parameters for the adaptation separation network via multiple linear mapping heads. These dedicated modulation parameters include: backbone block internal condition modulation parameters. Modulation parameters at the encoding end and the two sets of modulation parameters at the decoding end and The operation condition branch uses a binary operation indicator to switch between extraction and elimination modes. The operation indicator is mapped to an operation vector through an embedding layer, thereby generating operation modulation parameters. .
5. The full-modal sound source separation method supporting extraction and elimination modes according to claim 1 or 4, characterized in that, The query condition branch includes a lightweight adaptive condition injection module and condition modulation parameters within the backbone block. With operating modulation parameters They are respectively fed into the lightweight adaptive conditional injection module. After processing, the parameters are split into multiple sets, which are used to perform translation, scaling, and gating modulation on the four sub-modules of time multi-head attention, time bidirectional loop, feedforward, and frequency bidirectional loop in the attention-loop time-frequency backbone block.
6. The full-modal sound source separation method supporting extraction and elimination modes according to claim 1, characterized in that, The subband partitioning module divides the input complex spectrum tensor into three non-uniform frequency regions. For each frequency region, it performs corresponding two-dimensional convolution and layer normalization processing, and then processes the input... First, perform inter-band normalization to obtain the input. ; Modulation parameters at the encoding end Rearrangement as translation parameter With scaling parameters Then and As a conditional injection: ； And obtain the output of the sub-band partitioning encoding module ; in, The input is after band normalization. This represents element-wise multiplication. This indicates a characteristic linear modulation operation.
7. The full-modal sound source separation method supporting extraction and elimination modes according to claim 6, characterized in that, The attention-cyclic time-frequency backbone block adopts a staged conditional modulation mechanism. Conditional modulation is performed on the input features during the backbone block input stage and the processing stage of each sub-module. Specifically, it includes: first, normalization and operation modulation are performed on the block input features to obtain the initial modulation result; then, normalization and query modulation are performed before the four sub-modules of time multi-head attention, time bidirectional cyclic, feedforward and frequency bidirectional cyclic respectively; after the calculation of each sub-module is completed, the sub-band features output by the fusion of the gated residual and the main branch are input into the sub-band merging module for merging and decoding.
8. The full-modal sound source separation method supporting extraction and elimination modes according to claim 1, characterized in that, The subband merging module employs parallel decoding of real and imaginary branches. After normalizing and conditionally modulating the separation features, it splits them into three subbands, which are then fed into the real and imaginary branches respectively. After reconstruction via two-dimensional point convolution, GELU activation, and transposed convolution, the outputs of the real branches of the three subbands are concatenated along the frequency dimension to obtain the estimated real part, and the outputs of the imaginary branches of the three subbands are concatenated along the frequency dimension to obtain the estimated imaginary part. The reconstruction results of the three subbands are then concatenated along the frequency dimension to obtain the estimated complex spectrum of the entire frequency band, which is then subjected to inverse short-time Fourier transform to obtain the target time-domain output.
9. The full-modal sound source separation method supporting extraction and elimination modes according to claim 1, characterized in that, In the training of the separation model, a direct complex spectrum reconstruction training method is adopted, and the loss function is the logarithmic magnitude loss. Generator loss Real-to-virtual part loss Omnidirectional phase loss Feature matching loss The weighted sum, where, The phase loss employs omnidirectional phase loss, the real-to-imaginary part loss employs omnidirectional real-to-imaginary loss, and these are combined with logarithmic magnitude loss, generative adversarial loss, and feature matching loss for network optimization; multiple perspectives constrain the separation model together, resulting in a complete training objective function. for: ； in, , , , and These are the weighting coefficients for the corresponding loss terms.