Implementing improved audio source separation

Abstract

This disclosure describes techniques for achieving improved audio source separation. The complex spectrum X is divided into K frequency bands along a frequency axis by applying a frequency band division operation to the complex spectrum X. The complex spectrum is a time-frequency representation of the audio signal. Each of the K frequency bands is represented as Xk, k = 1,..., K. Each frequency band Xk comprises one or more frequency intervals. Each individual multi-layer perceptron is applied to each frequency band Xk to extract potential representations and obtain an output Hk0. A time domain transformer and a frequency domain transformer are applied to the stacked representation H0. The time domain transformer and the frequency domain transformer are repeatedly applied L times in an interleaved manner to obtain an HL output from the transformer block. The HL is input into a multi-band mask estimation sub-model. The complex ideal ratio mask is generated based on the output from the multiband mask estimation sub-model.

Description

Background Art

[0001] Audio source separation aims to extract one or more source signals of interest from an audio recording containing a mixture of multiple sound sources. This technique plays a vital role in the success of hearing aids, speakerphones, voice commands and other noise-resistant audio analysis systems, and music post-production software. Therefore, it is desirable to improve audio source separation. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The following detailed description may be better understood when read in conjunction with the accompanying drawings.For illustrative purposes, example embodiments of various aspects of the disclosure are shown in the drawings; however, the disclosure is not limited to the specific methods and instrumentalities disclosed.

[0003] Figure 1 An example system for implementing audio source separation according to the present disclosure is shown.

[0004] Figure 2 An example system for implementing audio source separation according to the present disclosure is shown.

[0005] Figure 3 An example transformer layer according to the present disclosure is shown.

[0006] Figure 4 An example algorithm according to the present disclosure is shown.

[0007] Figure 5 An example process for implementing audio source separation according to the present disclosure is shown.

[0008] Figure 6 An example process for implementing audio source separation according to the present disclosure is shown.

[0009] Figure 7 An example process for implementation by a converter according to the present disclosure is shown.

[0010] Figure 8 An example process for implementing audio source separation according to the present disclosure is shown.

[0011] Fig. 9 An example table of signal distortion ratio scores illustrating improved audio source separation using techniques according to the present disclosure is shown.

[0012] Fig.10 An example table illustrating that the system according to the present disclosure performs better than existing systems is shown.

[0013] Fig.11 An example computing device is shown that can be used to perform any of the techniques disclosed herein. DETAILED DESCRIPTION

[0014] Audio source separation aims to extract one or more source signals of interest from an audio recording containing a mixture of multiple sound sources. When audio separation technology is applied to music recordings, users are interested in separating the audio into multiple musically different audio stems. This task is called music source separation (MSS). For example, MSS can be used to separate a music recording into different instruments, such as vocals, bass, drums, etc. MSS is a basic task in music information retrieval (MIR) and has many applications in downstream MIR tasks, such as vocal level estimation, music transcription, etc. MSS also has many industrial applications. For example, MSS can be used to separate a piece of music into vocals and accompaniment for music editing and composition.

[0015] Neural networks can be used to accomplish MSS tasks. MSS systems can use frequency domain methods or time domain methods to perform source separation on audio. MSS systems using frequency domain methods can first transform the audio recording into a representation (e.g., a complex spectrum). Then, a fully connected neural network, a convolutional neural network (CNN), or a recurrent neural network (RNN) can be applied to the complex spectrum. High-frequency signals and low-frequency signals have different patterns, but this frequency domain method tends to apply the same kernel across all frequencies. This method cannot capture the different frequency patterns of the signal. Therefore, improved audio source separation techniques are desired.

[0016] This article describes an improved audio source separation technique. The technique described herein can separate both the amplitude information and the phase information of (multiple) target sources. The system described herein may be referred to as a band-split Transformer with Rotary Position Encoding (BS-RoFormer). First, the BS-RoFormer system provides an improvement over existing methods by splitting the representation of the audio (e.g., complex spectrum) into finer sub-bands. Each sub-band can be input into a separate multi-layer perceptron (MLP) layer to extract potential representations. Second, the BS-RoFormer system uses a band-split transformer block to replace the RNN in the existing method. The transformer block has better representation capabilities than RNN in natural language tasks and visual tasks. Third, the BS-RoFormer system uses interlaced transformers along the frequency axis and the time axis to capture audio information from different aspects. Fourth, BS-RoFormer uses a Rotary Position Encoding (RoPE) method for each transformer in the transformer block. RoPE has better position encoding capabilities in natural language tasks. The time position encoding and frequency position encoding can be explicitly inserted into each transformer, making the BS-RoFormer system aware of the different time steps and frequency bands. The BS-RoFormer system far outperforms existing audio source separation systems.

[0017] Figure 1 An example system 100 according to the present disclosure is shown. The system 100 may be used to implement audio source separation. The system 100 may include a BS-RoFormer system 105. The BS-RoFormer system 105 may include a band segmentation submodel 106, a transformer block 108, and a multi-band mask estimation submodel 110.

[0018] Audio 102 may include any audio segment. This segment of audio may include a music audio segment (e.g., a music recording). This segment of music audio may include a song or a portion of a song. This segment of music audio may include human voice and / or a variety of different instrument sounds. Human voice may represent human speech (e.g., singing and / or speaking). Each of the multiple different instrument sounds may correspond to a different instrument present in this segment of music audio. The instrument may include one or more of a guitar, piano, drums, bass, cello, violin, piccolo, flute, trumpet, harmonica, xylophone, saxophone, harp, and / or any other instrument. Short-time Fourier transform (STFT) 104 may be configured to transform audio 102 into a time-frequency representation, such as a complex spectrum or a two-dimensional spectrogram. STFT is a Fourier-related transform for determining the local sinusoidal frequency content and phase content of a signal when the signal changes over time. The time-frequency representation may be referred to as "X" in this article.

[0019] The band segmentation submodel 106 may receive a time-frequency representation as input. The time-frequency representation is non-uniform in different frequency bins. For example, the frequency patterns of instruments in a low frequency bin and instruments in a high frequency bin may be different. Thus, unlike previous MSS systems that directly apply neural networks to time-frequency representations, the band segmentation submodel 106 may apply a band segmentation operation to the time-frequency representation. The band segmentation submodel 106 may be configured to project the time-frequency representation into a plurality of sub-band representations according to a predetermined mapping scheme. The mapping scheme may indicate which frequency bins should be grouped into sub-bands. The mapping scheme may include a Mel band mapping or a handcrafted mapping, regardless of whether there is frequency overlap. For example, the band segmentation submodel 106 may be configured to segment the time-frequency representation into K non-uniform sub-bands along the frequency axis. Each sub-band may be referred to herein as an "X sub-band". k ”, k=1,...,K.

[0020] The band segmentation submodel 106 can be configured to apply a separate multilayer perceptron (MLP) layer to each subband. MLP is a modern feedforward artificial neural network, including fully connected neurons with nonlinear type activation functions organized into at least three layers (e.g., MLP layers). MLP is able to distinguish data that cannot be linearly separated. Each MLP layer may include a root mean square normalization (RMSNorm) layer followed by a linear layer. Each RMSNorm layer can regularize the total input of the neurons in a layer according to the root mean square (RMS), thereby giving the model rescaling invariance and implicit learning rate adaptability. The output of each transformed subband can be fed into the transformer block 108.

[0021] The transformer block 108 can receive the output from the band segmentation sub-model 106. The transformer block 108 can perform self-attention calculations alternately along the time axis and the frequency axis. For example, the transformer block 108 can decompose the calculations along different dimensions. (Multiple) time domain transformers and (multiple) frequency domain transformers can be applied in an interlaced manner along the time (T) axis and the frequency axis (e.g., K axis). The transformer along the T axis and the transformer along the K axis can have the same architecture. Each transformer can include an attention module and a feedforward module.

[0022] In an embodiment, the transformer block 108 may include a stack of transformer blocks. The stack may include L transformer blocks. Each transformer block may include a time domain transformer and a frequency domain transformer. By applying the time domain transformer and the frequency domain transformer in an interleaved manner, the stack of L transformer blocks may perform self-attention calculations alternately along the time axis and the frequency axis. In other embodiments, the transformer block 108 may include a time domain transformer and a frequency domain transformer. The time domain transformer and the frequency domain transformer may be applied L times in an interleaved manner. The output of the transformer block 108 (hereinafter referred to as “H L ”) may have a shape of T×K×D and may be fed into the multi-band mask estimation sub-model 110.

[0023] The multi-band mask estimation sub-model 110 may receive H L The multi-band mask estimation sub-model 110 may be configured to project each sub-band representation into a corresponding sub-band frequency. If two sub-bands overlap, their resulting sub-band estimates may be averaged. The multi-band mask estimation sub-model 110 may apply K separate MLP layers to H L Each MLP layer may include an RMSNorm layer, followed by a fully connected layer with Tanh activation, and followed by a fully connected layer with a gated linear unit (GLU) layer. The output of the Kth MLP layer of the multi-band mask estimation sub-model may have a shape of (2×C)×T×F k . The number 2×C indicates that each channel contains real and imaginary values. All outputs can be cascaded along the frequency axis and combined to generate output 112. Output 112 can include a linear spectrum, an ideal binary mask (IBM), an ideal ratio mask (IRM), and / or a complex IRM (cIRM). For example, output 112 can include a cIRM with a shape of C×T×F Output 112 can separate audio 102 based on the sound source. Figure 2 The BS-RoFormer system 105 is discussed in more detail.

[0024] Figure 2An example system 200 for implementing audio source separation is shown. The system 200 includes the BS-RoFormer system 105. A piece of audio (e.g., music, an audio signal, etc.) can be represented by a waveform. The waveform of the input mixture may be referred to herein as "x", where C may represent the number of channels, and L may represent the length of the audio signal. If the audio is a mono recording (e.g., the audio is recorded using one microphone on one channel), C may be equal to 1. If the audio is a stereo recording (e.g., the audio is recorded using two microphones on two different channels), C may be equal to 2. The waveform x may be transformed into a time-frequency representation, such as a complex spectrum, by a short-time Fourier transform (STFT). As described above with respect to Figure 1 As explained, the time-frequency representation may be referred to herein as "X", where T and F may represent the number of frames and the number of frequency bins, respectively.

[0025] The time-frequency representation (eg, complex spectrum) X may be input into the BS-RoFormer system 105. The BS-RoFormer system 105 may include a frequency domain neural network. The BS-RoFormer system 105 may be referred to herein as f θ The BS-RoFormer system 105 can be modeled by a set of learnable parameters θ.

[0026] Unlike the existing MSS system that directly applies a neural network to the complex spectrum X, the BS-RoFormer system 105 can apply a band splitting operation to X. X is non-uniform in different frequency intervals. For example, the frequency pattern of a musical instrument in a low frequency interval is different from the frequency pattern of a musical instrument in a high frequency interval. The band splitting submodule 106 can split X into K non-uniform sub-bands along the frequency axis. As described above with respect to Figure 1 As described, the sub-band may be referred to herein as "X k ",in Among them, F k is the number of frequency bins in the Kth band. All subbands X k The entire complex spectrum X can be constructed, and More subbands may improve the performance of the BS-RoFormer system 105 .

[0027] The frequency band segmentation submodule 106 can divide each subband X into ySeparate MLPs are applied. Each MLP may include an RMSNorm layer. An RMSNorm layer may be followed by a linear layer. An RMSNorm layer may regularize the total input to the neurons in a layer according to the root mean square. RMSNorm is an effective improvement on LayerNorm normalization in that it estimates the normalization statistics directly from the total input to the neurons within the hidden layer, so the normalization does not introduce any new dependencies between training cases. The kth linear layer may include a learnable matrix. The learnable matrix may have (C×F k )×D and a learnable bias of shape D. The output of each transformed frequency band may be referred to herein as having a shape of The shape is T×D, where the superscript 0 indicates It is the input to the 0th layer of the following transformer block. All The stacking can be performed along the frequency dimension to obtain a stack H of shape T×K×F 0 . Stack H 0 can be input into the converter block 108 .

[0028] The transformer block 108 may receive an output representation from the frequency band segmentation submodule 106 The converter block 108 may receive as input. To construct the transformer, T×K can be used as the sequence length. However, this may lead to an intractable system complexity of T 2 ×K 2 To address this problem, the computation can be decomposed along different dimensions. To decompose the computation along different dimensions, two transformers can be applied: one can be applied along the T axis, and the other can be applied along the K axis. One advantage of decomposing the transformer operation is that global information can be captured while reducing the computational complexity from T to K. 2 ×K 2 Reduce to T 2 ×K+K 2 × T. In addition, the frequency band segmentation submodule 106 can reduce the time complexity by reducing the dimension from F to K along the frequency axis.

[0029] In an embodiment, the transformer block 108 may include a stack of transformer blocks. The stack may include L transformer blocks. Each transformer block may include a time domain transformer and a frequency domain transformer. Each transformer block may receive As input, l = 1, ..., L. A stack of L transformer blocks can perform self-attention calculations alternately along the time axis and the frequency axis by applying the time domain transformer and the frequency domain transformer to the corresponding input in an interleaved manner. In other embodiments, the transformer block 108 may include a time domain transformer and a frequency domain transformer. The time domain transformer and the frequency domain transformer can be applied L times in an interleaved manner.

[0030] The transformer applied along the time axis (ie, T axis) and the transformer applied along the frequency axis (eg, K axis) can have the same architecture. The architecture of the transformer applied along the T axis and the transformer applied along the K axis involves two-step sequence modeling, which alternates self-attention between the frequency axis and the time axis.

[0031] Figure 3 An example transformer architecture 300 is shown. The transformer architecture 300 may include an attention module 304 and a feedforward module 302. The attention module may be used to enable the CNN to learn and focus more on important information rather than learning useless background information. The attention module 304 may first input an input tensor of shape B×L×D into an RMSNorm module 305. The output of the RMSNorm module 305 may be fed into a query layer, a key layer, and a value layer. The query layer, the key layer, and the value layer may be respectively connected by a fully connected layer W q , W k , W v The outputs of the query layer, key layer, and value layer can be represented by Q, K, and V respectively. The fully connected layer W q , W k , W v They may all have shape D×H, where H is the number of hidden dimensions. Then, Q, K, V can be split into multiple heads.

[0032] When modeling spectrograms, the positions of frequency and time are important because different time positions and frequency positions contain different information. In order to integrate the position information into our system, rotation embedding can be used for Q and K. Rotation embedding is a position embedding that encodes the absolute position information using a rotation matrix and naturally incorporates explicit relative position dependencies into the self-attention formula. The advantage of rotation embedding is that both the absolute and relative information of the sequence are captured. Then, the Q, K, and V values ​​can be input into the attention function 307. The attention function 307 can be expressed as follows:

[0033]

[0034] Although the attention module 304 is a core part of the BS-RoFormer system 105, the calculation of the attention is slow. In this way, flash attention can be adopted in the BS-RoFormer system 105. Then, a fully connected layer 308 with dropout can be used. Dropout refers to the practice of randomly ignoring certain nodes in a layer during training. Dropout is a regularization method that prevents overfitting by ensuring that no units depend on each other. After establishing the attention module 304, residual summation can be applied between the input and output of the attention module 304.

[0035] The feed-forward module 302 may receive the output of the attention module 304 as input. Figure 3 As shown, the feedforward module 302 includes an RMS Norm layer 309, a first fully connected layer 311, and a second fully connected layer 313. The first fully connected layer 311 may include GeLU activation and dropout. GeLU is a function that multiplies its input by the cumulative density function of the normal distribution at the input. The second fully connected layer 313 may include dropout. Residual addition can be applied between the input and output of the feedforward module 302.

[0036] Figure 4 An example algorithm 400 for applying an interleaved transformer along the time axis and the frequency axis is shown. The first step of the algorithm 400 may include inputting an H having a shape of B×T×K×D. l The second step of algorithm 400 may include converting H l The rearranged H is (B×K)×T×D. Then, the rearranged H can be aligned along the time axis (ie, T axis). l Apply the transformer. The fourth step of algorithm 400 may include rearranging the input into (B×T)×K×D. The rearranged H may then be aligned along the K axis. l Apply the transformer. The sixth step of algorithm 400 may include rearranging the output into an H matrix of shape B×T×K×D. l+1 The final output of the converter block 108 (hereinafter referred to as “H L ”) may have a shape of T×K×D and may be fed into the multi-band mask estimation sub-model 110.

[0037] The multi-band mask estimation sub-module 110 may receive the output of the transformer block 108. Similar to the band segmentation operation performed by the band segmentation sub-module 106, the multi-band mask estimation sub-module 110 may output a K separate MLP layers are applied. Each MLP layer may include an RMSNorm layer, followed by a fully connected layer with Tanh activation, and followed by a fully connected layer with a gated linear unit (GLU) layer. The output of the Kth MLP layer may have a shape of (2×C)×T×F k The number 2×C indicates that each channel contains real and imaginary values. All outputs may be cascaded along the frequency axis and combined to generate the output of the BS-RoFormer system 105 .

[0038] The output of the BS-RoFormer system 105 may include one or more of a linear spectrum, an ideal binary mask (IBM), an ideal ratio mask (IRM), or a complex IRM (cIRM). For example, the output of the BS-RoFormer system 105 may include a prediction mask 203. The prediction mask may include a (For example, a shape of C×T×F ) of the cIRM. The output of the BS-RoFormer system 105 (e.g., ) can be multiplied by the input complex spectrum X to generate a separated complex spectrum in Can Apply inverse STFT (iSTFT) to recover the separated signals in the time domain

[0039] During the process of training the BS-RoFormer system 105, the time domain reference true signal y is compared with the predicted signal The mean absolute error (MAE) loss between can be calculated as follows:

[0040]

[0041] Where N is the number of multi-resolution spectrograms. Both time domain MAE and multi-resolution spectrogram MAE can be added as objectives for optimizing the BS-RoFormer system 105 .

[0042] In an embodiment, during the process of training the BS-RoFormer system 105, audio recordings from different sources may initially be divided into 8 second audio segments using instant random cropping. Subsequently, loudness detection may be applied to discard segments with loudness levels below -50 dB. Additionally, a random gain in the range of ±3 dB may be introduced for each segment. To create a single training sample, a segment may be randomly selected for each source from the processed segments described above. The segments from different sources are mixed by linear addition. This random mixing strategy may produce audio samples that are not musically consistent.

[0043] In an embodiment, a Hanning window of size 2048 and a jump of size 10ms can be applied to calculate the STFT of an 8-second-long input signal. For the band segmentation submodule 106, the following band segmentation scheme can be applied: for frequencies below 1000Hz, 2 intervals per band; between 1000Hz and 2000Hz, 4 intervals per band; between 2000Hz and 4000Hz, 12 intervals per band; between 4000Hz and 8000Hz, 24 intervals per band; between 8000Hz and 16000Hz, 48 intervals per band; and the remaining intervals exceeding 16000Hz are equally divided into two bands. This results in a total of 62 bands. These bands are non-overlapping.

[0044] In an embodiment, the BS-RoFormer system 105 uses a fixed feature dimension D=384 throughout its architecture. It consists of 12 transformer blocks, each of which includes 8 heads, with a dropout rate of 0.1. The mask estimation module can utilize an MLP with a hidden layer dimension of 4D. For training, an AdamW optimizer can be used with a learning rate of 4×10 -4 The number of multi-resolution STFTs N is set to 5 to calculate the time domain reference true signal y and the predicted signal The mean absolute error (MAE) loss between , where the window size is set to = [4096, 2048, 1024, 512, 256], with a hop size of 147.

[0045] In an embodiment, three models are trained for "Vocal", "Bass", and "Drums" using exactly the same settings. To obtain "Other", the predictions generated by the previous models can be subtracted from the original mixture. Training is performed using 16 A100 GPUs with a total batch size of 128. All models can be trained with mixed precision (FP16) to speed up training and optimize memory usage.

[0046] Figure 5 An example process 500 for implementing audio source separation is shown. Figure 5 Although depicted as a series of operations in the present invention, one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the depicted operations.

[0047] A time-frequency representation may be received. The time-frequency representation may be non-uniform in different frequency intervals. For example, the frequency pattern of an instrument in a low frequency interval may be different from the frequency pattern of an instrument in a high frequency interval. At 502, a complex spectrum X may be segmented. The complex spectrum X may be segmented into K frequency bands (e.g., K non-uniform sub-bands) along a frequency axis. The complex spectrum X may be segmented by applying a frequency band segmentation operation to the complex spectrum X. The complex spectrum may be a time-frequency representation of an audio signal. Each of the K frequency bands may be represented as X k , k = 1, ..., K. Each frequency band X k Includes one or more frequency bins.

[0048] At 504, each frequency band X may be k Apply a separate multi-layer perceptron (MLP). For each frequency band X k Apply MLP to extract latent representation and obtain output H k 0 , where 0 indicates the 0th layer input to the transformer block. Each MLP layer may include an RMSNorm layer followed by a linear layer. Each RMSNorm layer may regularize the total input to the neurons in a layer according to the root mean square (RMS), giving the model rescaling invariance and implicit learning rate adaptability. The output H k 0 can be fed into the converter block.

[0049] At 506, the stack representation H may be 0 Apply the time domain transformer and the frequency domain transformer. The time domain transformer and the frequency domain transformer can be repeatedly applied L times in an interleaved manner to obtain H output from the transformer block. L . The transformer block can be configured to capture global information of the audio signal in the time domain and the frequency domain. In an example, the transformer block may include a stack of transformer blocks. The stack may include L transformer blocks. Each transformer block may include a time domain transformer and a frequency domain transformer. By applying the time domain transformer and the frequency domain transformer in an interleaved manner, the stack of L transformer blocks may perform self-attention calculations alternately along the time axis and the frequency axis. In other examples, the transformer block may include a time domain transformer and a frequency domain transformer. The time domain transformer and the frequency domain transformer may be applied L times in an interleaved manner. The final output H of the transformer block may have a shape of T×K×D and may be fed into a multi-band mask estimation submodel.

[0050] At 508, H Lis input to the multi-band mask estimation sub-model. The multi-band mask estimation sub-model can be configured to project each sub-band representation into the corresponding sub-band frequency. If two sub-bands overlap, their resulting sub-band estimates can be averaged. The multi-band mask estimation sub-model can apply K separate MLP layers to H L The outputs from the K individual MLP layers may be cascaded along the frequency axis and combined to generate an output. The output may include a complex ideal ratio mask (cIRM). The cIRM may be generated based on the output from the multi-band mask estimation sub-model. The cIRM is capable of separating audio signals based on sound sources.

[0051] Figure 6 An example process 600 for implementing audio source separation is shown. Figure 6 Although depicted as a series of operations in the present invention, one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the depicted operations.

[0052] At 602, a complex ideal ratio mask (cIRM) may be generated. The cIRM may be generated based on outputs from a multi-band mask estimation submodel. The cIRM is capable of separating audio signals based on sound sources. The prediction mask may include a signal represented as (For example, a shape of C×T×F ). At 604, at least one separated spectrum may be generated. The at least one separated spectrum may be generated based on multiplying the cIRM by the input spectrum X. The at least one separated spectrum may include a separated complex spectrum. in At 606, at least one separated signal in the time domain may be recovered. The at least one separated signal may be recovered by applying an inverse short-time Fourier transform (iSTFT) to the at least one separated spectrum. Applying iSTFT to recover separated signals in the time domain

[0053] Figure 7 An example process 700 implemented in the transformer layer is shown. Figure 7 Although depicted as a series of operations in the present invention, one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the depicted operations.

[0054] The transformer architecture according to the present disclosure may include an attention module and a feedforward module. The attention module may first input an input tensor of shape B×L×D into the RMSNorm module. At 702, the output from the RMSNorm module may be fed into the query layer, the key layer, and the value layer. The query layer, the key layer, and the value layer may be respectively connected by a fully connected layer W q , Wk , W v The outputs of the query layer, key layer, and value layer can be represented by Q, K, and V respectively. The fully connected layer W q , W k , W v All may have shape D×H, where H is a hidden dimension. Then, Q, K, and V may be split into multiple heads. At 704, the query (Q), key (K), and value (V) output from the query layer, key layer, and value layer may be split into multiple heads.

[0055] When modeling a spectrogram, the frequency position and the time position are important because different time positions and frequency positions contain different information. In order to integrate the position information into the BS-RoFormer system, rotation embedding can be used for Q and K. At 706, rotation embedding can be applied to Q and K. Rotation embedding is a position embedding that uses a rotation matrix to encode absolute position information and naturally incorporates explicit relative position dependencies into the self-attention formula. The advantage of rotation embedding is that both the absolute information and the relative information of the sequence are captured. At 708, a flash attention function can be applied to Q, K, and V. The flash attention function can be expressed as follows:

[0056]

[0057] At 710, a fully connected layer with dropout may be applied. After establishing the attention module, at 712, a residual summation may be applied between the input and output of the attention module of a transformer including the attention module and the feedforward module. The output of the attention module may be input into the feedforward module of the transformer.

[0058] Figure 8 An example process 800 for implementing audio source separation is shown. Figure 8 Although depicted as a series of operations in the present invention, one of ordinary skill in the art will appreciate that various embodiments may add, remove, reorder, or modify the depicted operations.

[0059] A time-frequency representation may be received. The time-frequency representation may be non-uniform in different frequency intervals. For example, the frequency pattern of an instrument in a low frequency interval may be different from the frequency pattern of an instrument in a high frequency interval. At 802, a complex spectrum X may be segmented. The complex spectrum X may be segmented into K frequency bands (e.g., K non-uniform sub-bands) along a frequency axis. The complex spectrum X may be segmented by applying a frequency band segmentation operation to the complex spectrum X. The complex spectrum may be a time-frequency representation of an audio signal. Each of the K frequency bands may be represented as X k , k = 1, ..., K. Each frequency band X k Includes one or more frequency bins.

[0060] At 804, each frequency band X may be k Apply a separate multi-layer perceptron (MLP). For each frequency band X k Apply MLP to extract latent representation and obtain output H k 0 , where 0 indicates the 0th layer input to the transformer block. Each MLP layer may include an RMSNorm layer followed by a linear layer. Each RMSNorm layer may regularize the total input to the neurons in a layer according to the root mean square (RMS), giving the model rescaling invariance and implicit learning rate adaptability. The output of each transformed subband may be fed into the transformer block.

[0061] At 806, the temporal position encoded frequencies may be inserted into a transformer block so that the transformer block knows information of different frequency bins. A transformer block (eg, transformer block 108) may be configured to capture global information of an audio signal in the time-frequency domain.

[0062] At 808, the stack representation H may be 0 Apply the time domain transformer and the frequency domain transformer. The time domain transformer and the frequency domain transformer can be repeatedly applied L times in an interleaved manner to obtain H output from the transformer block. L . In an example, the transformer block may include a stack of transformer blocks. The stack may include L transformer blocks. Each transformer block may include a time domain transformer and a frequency domain transformer. By applying the time domain transformer and the frequency domain transformer in an interleaved manner, the stack of L transformer blocks may perform self-attention calculations alternately along the time axis and the frequency axis. In other examples, the transformer block may include a time domain transformer and a frequency domain transformer. The time domain transformer and the frequency domain transformer may be applied L times in an interleaved manner. At 810, the output H of the transformer block L may be fed into a multi-band mask estimation sub-model (eg, the multi-band mask estimation sub-model 110 ).

[0063] At 812, a complex ideal ratio mask (cIRM) may be generated. The cIRM may be generated by cascading outputs from a multi-band mask estimation submodel along a frequency axis. The multi-band mask estimation submodel includes K separate MLP layers. Each of the K separate MLP layers may include an RMS Norm layer, followed by a fully connected layer with a Tanh activation, and followed by a fully connected layer with a gated linear unit (GLU) layer. The cIRM is capable of separating audio signals based on sound sources.

[0064] In one embodiment, the BS-RoFormer system described herein is trained on the MUSDB18HQ dataset. The BS-RoFormer system is evaluated on the MUSDB18 dataset for comparison with existing MSS systems. The MUSDB18HQ dataset contains 100 songs for training. All recordings are in stereo and have a sampling rate of 48kHz. Each recording contains four separate tracks, including vocals, bass, drums, and others. A separate separation system can be trained for each track. In addition to the MUSDB18HQ dataset, 500 high-quality musical works (e.g., from the Internet) were collected. Such high-quality musical works may be referred to as "in-house" in this article. The in-house dataset is added to the MUSDB18HQ dataset to study the improvement of system performance with more training data.

[0065] During the evaluation of the BS-RoFormer system, the signal-to-distortion ratio (SDR) was used as an evaluation metric. The SDR score was calculated using the following equation:

[0066]

[0067] A higher SDR indicates a better separation quality. Fig. 9 900. Table 900 shows the SDR associated with the BS-RoFormer system described herein, as well as the SDR associated with various existing MSS systems. As shown in Table 900, the SDR of BS-RoFormer is higher than that of the existing MSS systems for vocals, bass, drums, other instruments, and on average. Because a higher SDR indicates better separation quality, Table 900 shows that BS-RoFormer is better than the existing MSS systems at audio source separation.

[0068] Fig.10 An example table 1000 is shown. Table 1000 shows that the BS-RoFormer system performs better than existing MSS systems. The BS-RoFormer system described herein (referred to as SAMI-ByteDance in table 1000) achieves an average SDR score of 9.965 dB. The SDR score of 9.965 dB is higher than the average SDR scores of all other MSS systems depicted in table 1000. The performance of the BS-RoFormer system is far better than the MSS system that performs next to it (referred to as ZFTurbo in table 1000). ZFTurbo achieves an average SDR score of only 9.259 dB.

[0069] The BS-RoFormer system described in this paper solves the music source separation (MSS) problem. The BS-RoFormer system described in this paper is a time-frequency domain MSS system and is an improvement on the band segmentation recursive neural network (BSRNN). The BS-RoFormer system described in this paper includes a band segmentation module, a transformer block, and a band combination module. The transformer block includes transformers interlaced along the time axis and the frequency axis. The BS-RoFormer system described in this paper achieves a high level of SDR on the MUSDB18 dataset. Time domain features can be additionally integrated into the BS-RoFormer system to further improve existing MSS systems.

[0070] Fig.11 A computing device is shown that can be used in various aspects, such as Figure 1-Figure 3 Models, modules and / or devices shown. Figure 1-Figure 3 , any or all of the components can be Fig.11 The system may be implemented by one or more instances of computing device 1100. Fig.11 The computer architecture shown illustrates a conventional server computer, workstation, desktop computer, laptop, tablet computer, network appliance, PDA, e-reader, digital cellular telephone, or other computing node and can be used to perform any aspects of the computer described herein, such as to implement the methods described herein.

[0071] The computing device 1100 may include a substrate or "motherboard" that is a printed circuit board to which a number of components or devices may be connected via a system bus or other electrical communication paths. One or more central processing units (CPUs) 1104 may operate in conjunction with a chipset 1106. The CPU(s) 1104 may be a standard programmable processor that performs the arithmetic and logic operations necessary for the operation of the computing device 1100.

[0072] (Multiple) CPUs 1104 can perform necessary operations by switching from one state to the next via manipulation of switch elements that distinguish and change discrete physical states. Switch elements can typically include electronic circuits that maintain one of two binary states (such as flip-flops), and electronic circuits that provide output states based on logical combinations of the states of one or more other switch elements (such as logic gates). These basic switch elements can be combined to create more complex logic circuits, including registers, adder-subtractors, arithmetic logic units, floating point units, etc.

[0073] CPU(s) 1104 may be augmented or replaced by other processing units, such as GPU(s) 1105. GPU(s) 1105 may include processing units dedicated to, but not necessarily limited to, highly parallel computing, such as graphics processing and other visualization-related processing.

[0074] The chipset 1106 may provide an interface between the CPU(s) 1104 and the rest of the components and devices on the substrate. The chipset 1106 may provide an interface to a random access memory (RAM) 1108 used as main memory in the computing device 1100. The chipset 1106 may also provide an interface to a computer-readable storage medium such as a read-only memory (ROM) 1120 or a non-volatile RAM (NVRAM) (not shown) for storing basic routines that may help boot the computing device 1100 and transfer information between various components and devices. The ROM 1120 or NVRAM may also store other software components necessary for the operation of the computing device 1100 according to various aspects described herein.

[0075] The computing device 1100 can operate in a networked environment using logical connections to remote computing nodes and computer systems through a local area network (LAN). Chipset 1106 may include functionality for providing network connectivity through a network interface controller (NIC) 1122, such as a Gigabit Ethernet adapter. NIC 1122 is capable of connecting the computing device 1100 to other computing nodes through network 1116. It should be understood that multiple NICs 1122 may be present in the computing device 1100 to connect the computing device to other types of networks and remote computer systems.

[0076] The computing device 1100 can be connected to a mass storage device 1128 that provides non-volatile storage for the computer. The mass storage device 1128 can store system programs, applications, other program modules, and data that have been described in more detail herein. The mass storage device 1128 can be connected to the computing device 1100 by a storage controller 1124 connected to the chipset 1106. The mass storage device 1128 can be composed of one or more physical storage units. The mass storage device 1128 may include a management component 1111. The storage controller 1124 can be connected to the physical storage unit through a serial attached SCSI (SAS) interface, a serial advanced technology attachment SATA) interface, a fiber channel (FC) interface, or other types of interfaces for physically connecting and transmitting data between a computer and a physical storage unit.

[0077] The computing device 1100 can store data on the mass storage device 1128 by transforming the physical state of the physical storage unit to reflect the stored information. The specific transformation of the physical state may depend on various factors and different implementations of this specification. Examples of these factors may include, but are not limited to, the technology used to implement the physical storage unit and whether the mass storage device 1128 is characterized as primary storage or secondary storage, etc.

[0078] For example, the computing device 1100 can store information to the mass storage device 1128 by issuing instructions via the storage controller 1124 to change the magnetic properties of a specific location in the disk drive unit, the reflective or refractive properties of a specific location in the optical storage unit, or the electrical properties of a specific capacitor, transistor, or other discrete component in the solid-state storage unit. Other transformations of the physical media are possible without departing from the scope and spirit of the present specification, wherein the foregoing examples are provided only to make the description more convenient. The computing device 1100 can also read information from the mass storage device 1128 by detecting the physical state or properties of one or more specific locations in the physical storage unit.

[0079] In addition to the mass storage device 1128 described above, the computing device 1100 may also access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. It should be understood by those skilled in the art that computer-readable storage media may be any available media that provides storage for non-transitory data and can be accessed by the computing device 1100.

[0080] By way of example and not limitation, computer-readable storage media may include volatile and non-volatile computer-readable storage media, temporary and non-temporary computer-readable storage media, and removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM ("EPROM"), electrically erasable programmable ROM ("EEPROM"), flash memory or other solid-state memory technology, compact disk ROM ("CD-ROM"), digital versatile disk ("DVD"), high-definition DVD ("HD-DVD"), Blu-ray or other optical storage devices, cassettes, magnetic tape, magnetic disk storage devices, other magnetic storage devices, or any other medium that can be used to store desired information in a non-temporary manner.

[0081] Mass storage devices (such as Fig.11The mass storage device 1128 shown can store an operating system for controlling the operation of the computing device 1100. The operating system can include a version of the LINUX operating system. The operating system can include a version of the WINDOWS server operating system from Microsoft. According to a further aspect, the operating system can include a version of the UNIX operating system. Various mobile phone operating systems, such as IOS and ANDROID, can also be utilized. It should be understood that other operating systems can also be utilized. The mass storage device 1128 can store other systems or applications and data utilized by the computing device 1100.

[0082] The mass storage device 1128 or other computer-readable storage medium may also be encoded with computer-executable instructions that, when loaded into the computing device 1100, transform the computing device from a general-purpose computing system into a special-purpose computer capable of implementing the aspects described herein. As described above, these computer-executable instructions transform the computing device 1100 by specifying how the CPU(s) 1104 transition between states. The computing device 1100 may access a computer-readable storage medium storing computer-executable instructions that, when executed by the computing device 1100, may perform the methods described herein.

[0083] Computing devices (such as Fig.11 The computing device 1100 shown in the figure may also include an input / output controller 1132 for receiving and processing input from a plurality of input devices, such as a keyboard, a mouse, a touch pad, a touch screen, an electronic stylus, or other types of input devices. Similarly, the input / output controller 1132 may provide output to a display, such as a computer monitor, a flat panel display, a digital projector, a printer, a plotter, or other types of output devices. It should be understood that the computing device 1100 may not include Fig.11 All components shown in the Fig.11 Other components not explicitly shown in the figure or may be used in conjunction with Fig.11 A completely different architecture is shown in .

[0084] As described herein, a computing device may be a physical computing device such as Fig.11 The computing device 1100 of the present invention may also include a virtual machine host process and one or more virtual machine instances. The computer executable instructions may be indirectly executed by the physical hardware of the computing device by interpreting and / or executing instructions stored and executed in the context of the virtual machine.

[0085] It should be understood that these methods and systems are not limited to specific methods, specific components or specific implementations. It should also be understood that the terminology used herein is for describing specific embodiments only and is not intended to be limiting.

[0086] As used in the specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" one particular value and / or to "about" another particular value. When such a range is expressed, another embodiment includes from a particular value and / or to another particular value. Similarly, when a value is expressed as being approximate by using the antecedent "about", it will be understood that the particular value forms another embodiment. It will also be understood that the endpoints of each range in a range are significant both relative to the other endpoint and independently of the other endpoint.

[0087] "Optional" or "optionally" means that the subsequently described event or circumstance may or may not occur, and that the description includes instances where said event or circumstance occurs and instances where it does not.

[0088] Throughout the description and claims of this specification, the word "include" and variations of the word (such as "comprising" and "including") mean "including but not limited to", and are not intended to exclude, for example, other components, integers or steps. "Exemplary" means "an example of..." and is not intended to convey an indication of a preferred embodiment or an ideal embodiment. "Such as" is not used in a limiting sense, but for the purpose of explanation.

[0089] Components that can be used to perform the described methods and systems are described. When describing combinations, subsets, interactions, groupings, etc. of these components, it should be understood that, although specific references to each of the various individual and common combinations and arrangements of these components may not be explicitly described, for all methods and systems, each is specifically contemplated and described herein. This applies to all aspects of the application, including but not limited to the operations in the described methods. Therefore, if there are various additional operations that can be performed, it should be understood that each of these additional operations can be performed using any specific embodiment or combination of embodiments of the described methods.

[0090] The presented methods and systems may be understood more readily by reference to the following detailed description of the preferred embodiments and the examples included therein and to the accompanying drawings and their descriptions.

[0091] As will be appreciated by those skilled in the art, these methods and systems may take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software aspects and hardware aspects. In addition, these methods and systems may take the form of a computer program product on a computer-readable storage medium in which computer-readable program instructions (e.g., computer software) are embodied. More particularly, the presented methods and systems may take the form of network-implemented computer software. Any suitable computer-readable storage medium may be utilized, including a hard disk, a CD-ROM, an optical storage device, or a magnetic storage device.

[0092] Embodiments of these methods and systems are described below with reference to block diagrams and flow charts of methods, systems, devices, and computer program products. It will be appreciated that each block in the block diagrams and flow charts and combinations of blocks in the block diagrams and flow charts can be implemented by computer program instructions, respectively. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed on the computer or other programmable data processing device create components for implementing the functions specified in one or more flow chart blocks.

[0093] These computer program instructions may also be stored in a computer-readable memory, which may direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including computer-readable instructions for implementing the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device, thereby producing a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flowchart blocks.

[0094] The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, in some embodiments, certain method or process blocks can be omitted. The methods and processes described herein are also not limited to any particular order, and the blocks or states associated therewith can be performed in other appropriate orders. For example, the blocks or states described can be performed in an order different from the specific description, or multiple blocks or states can be combined in a single block or state. The example blocks or states can be performed serially, in parallel, or in some other way. Blocks or states can be added to the described exemplary embodiments, or blocks or states can be deleted from the described exemplary embodiments. The example systems and components described herein can be configured in different ways than described. For example, compared with the described example embodiments, elements can be added to, removed from, or rearranged.

[0095] It will also be understood that various items are shown as being stored in memory or on storage devices while being used, and that these items or portions thereof may be transferred between memory and other storage devices for the purpose of memory management and data integrity. Alternatively, in other embodiments, some or all of the software modules and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. In addition, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways, such as at least partially in firmware and / or hardware, including but not limited to one or more application specific integrated circuits (“ASICs”), standard integrated circuits, controllers (e.g., by executing appropriate instructions and including microcontrollers and / or embedded controllers), field programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), etc. Some or all of the modules, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a computer-readable medium (e.g., a hard disk, memory, a network, or a portable media product) for reading by an appropriate device or via an appropriate connection. Systems, modules, and data structures may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless-based media and wired / cable-based media), and may take various forms (e.g., as part of a single analog signal or a multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products may also take other forms. Accordingly, the present invention may be practiced using other computer system configurations.

[0096] Although these methods and systems have been described in conjunction with preferred embodiments and specific examples, it is not intended to limit the scope to the specific embodiments set forth, as the embodiments herein are intended in all respects to be illustrative rather than restrictive.

[0097] Unless otherwise expressly stated, it is not intended that any method described herein be interpreted as requiring that its operations be performed in a specific order. Accordingly, in the event that a method claim does not actually recite the order in which its operations are to be followed, or in the event that the operations are not otherwise specifically stated in the claims or specification to be limited to a specific order, no order is intended to be inferred in any way. This applies to any possible non-explicit basis for interpretation, including: logical issues regarding the arrangement of steps or operational flow; plain meaning derived from grammatical organization or punctuation; and the number or type of embodiments described in the specification.

[0098] It will be apparent to those skilled in the art that various modifications and variations may be made without departing from the scope or spirit of the present disclosure. Other embodiments will be apparent to those skilled in the art in view of the specification and practice described herein. It is intended that the specification and exemplary drawings be considered merely exemplary, with the true scope and spirit being indicated by the appended claims.

Claims

1. A method for achieving improved audio source separation, comprising: The spectrum X is divided into K frequency bands along the frequency axis by applying a band division operation to the spectrum X, wherein the spectrum is a time-frequency representation of the audio signal, wherein each of the K frequency bands is denoted as X k , k = 1, ..., K, and each frequency band X k including one or more frequency intervals; For each frequency band X k Apply a single multi-layer perceptron MLP to extract the latent representation and obtain the output H k 0 , where 0 indicates the 0th layer input of the transformer block; For stacking, H 0 Apply the time domain transformer and the frequency domain transformer, and repeat the application of the time domain transformer and the frequency domain transformer L times in an interleaved manner to obtain H output from the transformer block L , wherein the transformer block is configured to capture global information of the audio signal in the time-frequency domain; as well as The H L The audio signals are input into a multi-band mask estimation sub-model, and a complex ideal ratio mask cIRM is generated based on the output from the multi-band mask estimation sub-model, wherein the cIRM is capable of separating the audio signals based on sound sources.

2. The method according to claim 1, further comprising: The time position coded frequencies are inserted into each transformer so that the transformer block has information about the different frequency intervals.

3. The method according to claim 1, further comprising: generating at least one separated spectrum based on multiplying the cIRM by the input spectrum X; as well as By applying an inverse short-time Fourier transform iSTFT to the at least one separated frequency spectrum, at least one separated signal is restored in the time domain.

4. The method of claim 1, wherein each MLP comprises a root mean square normalization (RMSNorm) layer and a linear layer.

5. The method of claim 1, wherein each transformer comprises an attention module and a feed-forward module.

6. The method of claim 5, wherein the attention module is configured to perform operations comprising: The output from the RMSNorm module is fed into the fully connected layers W q , W k , W v Modeling query layer, key layer, and value layer; Splitting the query Q, key K, and value V output from the query layer, the key layer, and the value layer into multiple heads; Applying a rotation embedding to the Q and the K; Applying a flash attention function to the Q, the K, and the V; Applying fully connected layers with dropout; and A residual summation is applied between the input and output of the attention module.

7. A method according to claim 5, wherein the output from the attention module is fed into the feed-forward module, wherein the feed-forward module includes an RMSNorm layer, a fully connected layer with Gaussian error linear unit GeLU activation and dropout, a fully connected layer with dropout, and wherein residual addition is applied between the input and output of the feed-forward module.

8. The method of claim 1, wherein the multi-band mask estimation submodel comprises K separate MLP layers, and wherein each of the K separate MLP layers comprises an RMSNorm layer, followed by a Tanh activated fully connected layer, and followed by a gated linear unit (GLU) layer.

9. The method according to claim 1, further comprising: The cIRM is generated by concatenating the outputs from the multi-band mask estimation sub-model along the frequency axis.

10. A system for achieving improved audio source separation, comprising: at least one processor; as well as at least one memory communicatively coupled to the at least one processor and comprising computer readable instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: The spectrum X is divided into K frequency bands along the frequency axis by applying a band division operation to the spectrum X, wherein the spectrum is a time-frequency representation of the audio signal, wherein each of the K frequency bands is denoted as X k , k = 1, ..., K, and each frequency band X k including one or more frequency intervals; For each frequency band X k Apply a single multi-layer perceptron MLP to extract the latent representation and obtain the output H k 0 , where 0 indicates the 0th layer input of the transformer block; For stacking, H 0 Apply the time domain transformer and the frequency domain transformer, and repeat the application of the time domain transformer and the frequency domain transformer L times in an interleaved manner to obtain H output from the transformer block L , wherein the transformer block is configured to capture global information of the audio signal in the time-frequency domain; as well as The H L The audio signals are input into a multi-band mask estimation sub-model, and a complex ideal ratio mask cIRM is generated based on the output from the multi-band mask estimation sub-model, wherein the cIRM is capable of separating the audio signals based on sound sources.

11. The system of claim 10, wherein the operations further comprise: generating at least one separated spectrum based on multiplying the cIRM by the input spectrum X; as well as By applying an inverse short-time Fourier transform iSTFT to the at least one separated frequency spectrum, at least one separated signal is restored in the time domain.

12. The system of claim 10, wherein the operations further comprise: The cIRM is generated by concatenating the outputs from the multi-band mask estimation sub-model along the frequency axis.

13. The system of claim 10, wherein each transformer comprises an attention module and a feed-forward module.

14. The system of claim 13, wherein the attention module is configured to perform operations comprising: The output from the RMSNorm module is fed into the fully connected layers W q , W k , W v Modeling query layer, key layer, and value layer; Splitting the query Q, key K, and value V output from the query layer, the key layer, and the value layer into multiple heads; Applying a rotation embedding to the Q and the K; Applying a flash attention function to the Q, the K, and the V; Applying fully connected layers with dropout; and A residual summation is applied between the input and output of the attention module.

15. A system according to claim 13, wherein the output from the attention module is fed into the feed-forward module, wherein the feed-forward module includes an RMSNorm layer, a fully connected layer with Gaussian error linear unit (GeLU) activation and dropout, a fully connected layer with dropout, and wherein residual addition is applied between the input and output of the feed-forward module.

16. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a processor, cause the processor to perform operations comprising: The spectrum X is divided into K frequency bands along the frequency axis by applying a band division operation to the spectrum X, wherein the spectrum is a time-frequency representation of the audio signal, wherein each of the K frequency bands is denoted as X k , k = 1, ..., K, and each frequency band X k including one or more frequency intervals; For each frequency band X k Apply a single multi-layer perceptron MLP to extract the latent representation and obtain the output H k 0 , where 0 indicates the 0th layer input of the transformer block; For stacking, H 0 Apply the time domain transformer and the frequency domain transformer, and repeat the application of the time domain transformer and the frequency domain transformer L times in an interleaved manner to obtain H output from the transformer block L , wherein the transformer block is configured to capture global information of the audio signal in the time-frequency domain; as well as The H L The audio signals are input into a multi-band mask estimation sub-model, and a complex ideal ratio mask cIRM is generated based on the output from the multi-band mask estimation sub-model, wherein the cIRM is capable of separating the audio signals based on sound sources.

17. The non-transitory computer-readable storage medium of claim 16, the operations further comprising: generating at least one separated spectrum based on multiplying the cIRM by the input spectrum X; as well as By applying an inverse short-time Fourier transform iSTFT to the at least one separated frequency spectrum, at least one separated signal is restored in the time domain.

18. The non-transitory computer-readable storage medium of claim 16, the operations further comprising: The cIRM is generated by concatenating the outputs from the multi-band mask estimation sub-model along the frequency axis.

19. The non-transitory computer-readable storage medium of claim 16, wherein each transformer comprises an attention module and a feed-forward module.

20. The non-transitory computer-readable storage medium of claim 19, wherein the attention module is configured to perform operations comprising: The output from the RMSNorm module is fed into the fully connected layers W q , W k , W v Modeling query layer, key layer, and value layer; Splitting the query Q, key K, and value V output from the query layer, the key layer, and the value layer into multiple heads; Applying a rotation embedding to the Q and the K; Applying a flash attention function to the Q, the K, and the V; Applying fully connected layers with dropout; and A residual summation is applied between the input and output of the attention module.

Images