A sound separation and target sound extraction method based on a unified architecture

Abstract

The application discloses a sound separation and target sound extraction method based on a unified architecture, and relates to the technical field of audio signal processing, and comprises the following steps: in a first training stage, an attractor network is used to estimate the number of sound sources in a mixed audio signal to generate an attractor embedding; the attractor embedding is input into a separation backbone network to generate separated sound; in a second training stage, a multi-modal clue processing network is used to process clue information of multiple modes to generate a clue embedding; the attractor embedding or the clue embedding is randomly selected as the input of the separation backbone network; in the training stage, an alignment loss function between the attractor embedding and the clue embedding is calculated; in the inference stage, if there is no clue information, a sound separation task is performed; and if there is 1 to 3 clue information, a target sound extraction task is performed. According to the application, the task is flexibly selected according to the input clue, and the application has better adaptability and flexibility and is suitable for a complex sound scene.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, and in particular to a method for sound separation and target sound extraction based on a unified architecture. Background Technology

[0002] In the field of audio signal processing, sound separation (SS) and target sound extraction (TSE) are two key technologies. Sound separation aims to separate different sound sources from a mixed audio signal, while target sound extraction focuses on extracting specific sounds that the user is interested in from the mixed audio.

[0003] In complex acoustic environments, universal sound separation provides high-quality input for subsequent audio analysis tasks. Universal Sound Separation (USS) has become a core task, aiming to separate any type of sound source (including speech, music, ambient sounds, and instrumental sounds) from complex audio mixtures, without relying on a predefined number or type of sound sources. This flexibility greatly expands the application scope of sound processing, making it indispensable in fields such as environmental monitoring and multimedia content analysis.

[0004] Despite the progress made in sound separation models in recent years, existing methods still face some limitations. For example, many models require the number of sound sources in the mixed audio to be predefined before the inference stage, which limits their applicability in real-world scenarios, since the number of sound sources is often uncertain in real-world scenarios.

[0005] To address these issues, target sound extraction leverages prior knowledge (i.e., cues) about the target sound from an unknown number of sound sources in mixed audio. For example, the AudioSep model demonstrates significant performance by using natural language descriptions as extraction cues; APT uses audio samples as cues for the TSE task; and PixelPlayer utilizes visual information to extract the target sound.

[0006] Two drawbacks still exist in practical applications: First, pre-existing clues may be of low quality or impossible to find in practical applications, which will reduce the performance of target sound extraction and even extract the wrong target; Second, when the number of sound sources is fixed, the target sound extraction method usually performs worse than the sound separation method.

[0007] Furthermore, existing sound separation and target sound extraction methods are usually designed independently for a single task, making it difficult to switch flexibly within the same model, which limits their application in complex sound scenarios.

[0008] Therefore, those skilled in the art are dedicated to developing a sound separation and target sound extraction method based on a unified architecture, which can flexibly choose to perform SS or TSE tasks according to the input cues, has better adaptability and flexibility, and is suitable for a variety of complex sound scenarios. Summary of the Invention

[0009] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to flexibly switch between sound separation and specific sound extraction in the same model.

[0010] To achieve the above objectives, this invention provides a sound separation and target sound extraction method based on a unified architecture, which is implemented based on a separation backbone network, an attractor network, and a multimodal cue processing network.

[0011] It includes a training phase and an inference phase; the training phase includes a first training phase and a second training phase.

[0012] In the first training phase, the attractor network is used to estimate the number of sound sources in the mixed audio signal, generate attractor embeddings, and calculate the sound source counting loss; the attractor embeddings are input into the separation backbone network to generate the separated sound, and the sound separation loss is calculated.

[0013] In the second training phase, the multimodal cue processing network is used to process cue information from multiple modalities to generate cue embeddings; the attractor embedding or the cue embedding is randomly selected as the input to the separating backbone network; if the cue embedding is selected as the input to the separating backbone network, all seven modal combinations are used for equal-probability training.

[0014] During the training phase, if the input of the separation backbone network is the attractor embedding, the matching order between the separated audio and the real audio is calculated when the sound separation loss function is minimized. Based on the matching order, the alignment loss function between the attractor embedding and the cue embedding is calculated to narrow the semantic space distance between the attractor and cue representations.

[0015] During the reasoning phase, if there are no clues, the sound separation task is performed; if there are 1 to 3 clues, the target sound extraction task is performed.

[0016] Furthermore, the separated backbone network includes:

[0017] An encoder is used to generate hidden features of mixed audio signals;

[0018] A separator is used to estimate a mask for each sound source and multiply the hidden features element-wise with the mask corresponding to each sound source to obtain a hidden representation;

[0019] The decoder, consisting of transposed convolutional layers symmetrical to the encoder, is used to estimate the separated sound sources.

[0020] Furthermore, the separator for estimating a mask for each sound source includes:

[0021] The hidden features are passed through a LayerNorm layer and a linear layer, and then divided into overlapping blocks of size K=250 with a 50% overlap rate.

[0022] A dual-path module is used to integrate Transformer blocks within and between blocks;

[0023] Sequence aggregation is performed, and learnable weights are used to represent time regions with different dominant sounds in different subspaces. Frame-level embeddings W are generated and input into the attractor network or the multimodal cue processing network.

[0024] The output V of the dual-path module is multiplied element-wise with the source sound representation of the attractor network or the multimodal cue processing network to form the input of the three-path module; the three-path module extends the dual-path design by adding a cross-channel Transformer block to capture the relationship between channels;

[0025] The output of the three-path module is sequentially passed through a parameterized ReLU layer, overlapped and added, a gated output layer with two linear layers, and a linear layer with a ReLU activation function to generate a mask.

[0026] Furthermore, the attractor network includes an LSTM encoder and an LSTM decoder;

[0027] The LSTM encoder updates its hidden state using the following formula. and unit state

[0028]

[0029] In this context, the hidden states and cell states of the LSTM encoder are initialized as zero vectors: and

[0030] The LSTM decoder estimates the attractor embedding using the following formula:

[0031]

[0032] In each step, the hidden state of the LSTM decoder As an attractor of sound category s, its dimension D is related to the frame-level embedding W. tThe dimensions are consistent; the hidden states and cell states of the LSTM decoder are initialized by the final state of the LSTM encoder: and

[0033] Furthermore, the attractor network also includes:

[0034] The probability of the presence of the attractor embedding is calculated using a fully connected layer with a sigmoid activation function, as shown in the following formula:

[0035]

[0036] in, and These are the trainable weights and bias parameters of the fully connected layer; a s The estimated characterization of the s-th sound in the mixed audio signal is represented by the attractor.

[0037] Each p exi Compare with a predefined threshold θ; if p exi If the value is greater than θ, then the attractor embedding is considered to exist.

[0038] Furthermore, the multimodal cue processing network encodes input cues from text, video, and audio into a unified D-dimensional space using a dedicated encoder; it employs a multimodal splicing strategy to form unified multimodal cues; and it uses an attention-based cue fusion module to align and fuse the multimodal cues in time, thereby extracting global cue information to obtain the cue embedding.

[0039] Furthermore, the dedicated encoder converts different modalities into D-dimensional embeddings, including:

[0040] The text encoder, using a pre-trained DistilBERT model, converts natural language descriptions into text embeddings. Among them, T t It is the number of word tags;

[0041] The video encoder processes video frames using a pre-trained Swing Transformer to generate video embedding vectors. Among them, T v It is the number of frames;

[0042] The sound encoder maps one-hot encoded sound event tags to sound embedding vectors through a linear layer.

[0043] Furthermore, through a multimodal concatenation strategy, the text embedding vector, video embedding vector, and audio embedding vector are concatenated to form a unified multimodal cue:

[0044]

[0045] Furthermore, the attention-based cue fusion module includes:

[0046] Using the output W of sequence aggregation as the query and U as the key and value, the clues are fused through a multi-head attention mechanism to obtain the fused clue embedding:

[0047]

[0048] MultiHeadAttention(Q,K,V) represents the multi-head attention mechanism, where Q, K, and V represent the query key and value, respectively.

[0049] Furthermore, the alignment loss function is defined as follows:

[0050]

[0051] in, This is achieved by finding the optimal permutation between attractor embeddings and cue embeddings, and calculating the mean squared error loss. Given N attractors and N cues, and the optimal permutation π obtained from the final separation PIT loss, the average InfoNCE loss for each pair of corresponding elements is calculated.

[0052] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0053] 1. This invention integrates sound separation (SS) and target sound extraction (TSE) tasks into a unified architecture, which can flexibly select to execute SS or TSE tasks according to the input cues, and has better adaptability and flexibility, making it suitable for a variety of complex sound scenarios;

[0054] 2. This invention can automatically estimate the number of sound sources in mixed audio through attractor networks, without manual intervention or predefined numbers, making it more practical in complex acoustic environments and capable of handling mixed audio signals with any number of sound sources. In sound separation tasks, this invention improves performance by 1.3dB compared to the existing single-task SS baseline model, significantly enhancing the separation effect in complex sound scenes. Furthermore, compared to the baseline model Sepformer, it does not require predefined numbers of sound sources and can separate mixed audio signals with uncertain numbers of sound sources.

[0055] 3. This invention supports multi-modal cue input and can fuse these cues into a unified feature space through an attention mechanism. This multi-modal processing capability significantly enhances robustness and adaptability in practical applications, especially when the cue quality is low or some modalities are missing. In the target sound extraction task, this invention improves the performance by 1.6dB compared to the existing single-task TSE baseline model and exhibits stronger robustness and adaptability under multi-modal cue fusion. Even when only a single-modal cue is provided, it can achieve good target sound extraction performance.

[0056] 4. This invention adopts a two-stage training strategy. The first stage focuses on the sound separation task, and the second stage combines the target sound extraction task for joint training. This strategy not only improves adaptability, but also further optimizes the performance of target sound extraction without compromising the separation performance by calculating the alignment loss function of attractor embedding and cue embedding.

[0057] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the overall architecture of a preferred embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of a separator structure according to a preferred embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of a multimodal cue processing network structure according to a preferred embodiment of the present invention;

[0061] Figure 4 This is a visual demonstration diagram of the attractor and cue in a preferred embodiment of the present invention;

[0062] Figure 5 This is a graph showing the inference speed evaluation under different numbers of sound sources, representing a preferred embodiment of the present invention. Detailed Implementation

[0063] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0064] This embodiment provides a method for sound separation and target sound extraction based on a unified architecture, such as... Figure 1 As shown, its architecture includes a separate backbone network, an attractor network, and a multimodal cue processing network.

[0065] The separation backbone network employs an encoder-splitter-decoder structure, responsible for feature extraction and sound separation of the mixed audio signal. The attractor network dynamically estimates the number of sound sources in the mixed audio signal and generates attractor embeddings. The multimodal cue processing network processes multimodal cues such as text, video, and audio tags, generating unified cue embeddings.

[0066] 1. Separate the backbone network

[0067] Given a single-channel mixed audio signal from J sound sources (where L is of arbitrary length), the goal of separating the backbone network is to estimate the number of sound sources based on the mixed audio signal x and the attractor network. Estimate the signal of each sound source

[0068] The encoder uses a 1D convolutional layer and a ReLU activation function to generate hidden features of the mixed audio signals. The separator estimates a mask for each sound source. By multiplying X element by m j The estimated hidden representation is obtained. The decoder consists of transposed convolutional layers, symmetrical to the encoder, that estimate the separated sound sources.

[0069] The separators employ both time-domain-based SepFormer and frequency-domain-based BSRNN models; this section primarily focuses on the SepFormer-based method. For example... Figure 2 As shown, a structure similar to SepFormer is used as the separator, where U, V, W, Y, and Z are intermediate features; N is the time dimension of the mixed audio features; K is a constant, representing the size of each chunk during the chunking process; J is the number of sound sources in the mixed audio; and C is the time dimension of the features after chunking.

[0070] First, the hidden feature X is segmented into overlapping blocks of size K=250 with a 50% overlap rate after passing through a LayerNorm layer and a linear layer.

[0071] Next, these overlapping blocks are fed into the dual-path module, which integrates Transformer blocks within and between blocks.

[0072] Subsequently, sequence aggregation is performed, and learnable weights are used to represent time regions with different dominant sounds in different subspaces. The output W is then fed into an attractor network or a multimodal cue processing network.

[0073] Furthermore, the output V of the dual-path module is compared with the source sound representation from the attractor network or multimodal cue processing network. Element-wise multiplication forms the input of the three-path module. The three-path module extends the two-path design by adding a cross-channel Transformer block to capture the relationships between channels.

[0074] Finally, the three-path module outputs The mask is generated by sequentially passing through a parameterized ReLU layer, an overlap-addition (OVA) layer, a gated output layer with two linear layers, and a linear layer with a ReLU activation function.

[0075] 2. Attractor Network

[0076] Attractor networks are used to estimate the number of different sound categories and attractor embeddings from mixed audio signals, thereby enabling sound separation without prior knowledge of the number of sound sources.

[0077] The attractor network employs an LSTM encoder-decoder framework to embed frames at the W level. t Convert to a global attractor.

[0078] The LSTM encoder updates its hidden state using the following formula. and unit state

[0079]

[0080] In this context, the hidden states and cell states of the LSTM encoder are initialized as zero vectors: and

[0081] The LSTM decoder estimates the attractor embedding using the following formula:

[0082]

[0083] The hidden state of the LSTM decoder at each step As an attractor embedding for sound category s, its dimension D is related to the frame-level embedding W. t The dimensions are consistent.

[0084] The hidden states and cell states of the LSTM decoder are initialized by the final states of the LSTM encoder: and

[0085] The probability of attractor embedding is calculated using a fully connected layer with a sigmoid activation function, as shown in the following formula:

[0086]

[0087] in, and These are the trainable weights and bias parameters of the fully connected layer, a s This represents the estimated characterization attractor for the s-th sound in the Mixture audio signal. Each p... exi Compare with a predefined threshold θ. If p exi If the value is greater than θ, then the attractor is considered to exist; otherwise, it is considered that there are no more sound sources in the mixed audio signal.

[0088] 3. Multimodal cue processing network

[0089] Multimodal cue processing networks take cues from multiple modalities (e.g., text, video, audio) as input and generate a comprehensive cue embedding. For example... Figure 3 As shown, firstly, each modality is encoded into a unified D-dimensional space using a dedicated encoder. Then, a multimodal concatenation strategy is employed to form unified multimodal cues. Finally, these cues are temporally aligned and fused using an attention-based cue fusion module to extract global cue information, resulting in the cue embedding.

[0090] A dedicated encoder is used to convert different modalities into D-dimensional embeddings for target sound extraction, including:

[0091] The text encoder, using a pre-trained DistilBERT model, converts natural language descriptions into text embedding vectors. Among them, T t It represents the number of word tags.

[0092] The video encoder processes video frames using a pre-trained Swing Transformer to generate video embedding vectors. Where T v It is the number of frames.

[0093] The sound encoder maps one-hot encoded sound event tags to sound embedding vectors through a linear layer.

[0094] The multimodal concatenation strategy concatenates text embedding vectors, video embedding vectors, and audio embedding vectors to form a unified multimodal cue.

[0095]

[0096] The attention-based clue fusion module uses the output W of sequence aggregation as the query and U as the key and value. It fuses clues through a multi-head attention mechanism to obtain the fused clue embedding.

[0097]

[0098] Here, MultiHeadAttention(Q,K,V) represents a multi-head attention mechanism, where Q, K, and V represent the query key and value, respectively. Clearly, the fused cue embedding C... u It has the same length T as the sound embedding vector Q. a .

[0099] 4. Training and Reasoning Strategies

[0100] This embodiment employs a unique training and inference strategy, including a training phase and an inference phase; the training phase includes a first training phase and a second training phase.

[0101] In the first training phase, an attractor network is used to estimate the number of sound sources in the mixed audio signal, generate attractor embeddings, and calculate the sound source counting loss. The attractor embeddings are then input into the separation backbone network to generate the separated sound and calculate the sound separation loss.

[0102] In the second training phase, a multimodal cue processing network is used to process cue information from multiple modalities (text, video, and audio) to generate cue embeddings; attractor embeddings (30% probability) or cue embeddings (70% probability) are randomly selected as inputs to the separation backbone network.

[0103] If we choose cue embeddings as input to the separate backbone network, we train it with equal probability using all seven modality combinations. The seven modality combinations include three unimodal combinations (text, video, and audio), three bimodal combinations (text + video, text + audio, and video + audio), and one trimodal combination (text + video + audio).

[0104] During training, when the input to the separation network is an attractor embedding, the matching order between the separated audio and the real audio is calculated to minimize the sound separation loss function. Based on the matching order, an alignment loss function is calculated between the attractor embedding and the cue embedding to narrow the semantic spatial distance between the attractor and cue representations. This strategy enhances the adaptability and robustness of this embodiment and significantly reduces the training time.

[0105] During the reasoning phase, if there are no clues, perform the sound separation task; if there are 1 to 3 clues, perform the target sound extraction task.

[0106] 5. Objective function

[0107] The objective function in this embodiment is a combination of three different loss functions: sound separation loss function. Source counting loss function and the alignment loss function between attractor embeddings and cue embeddings

[0108] In multi-source sound separation tasks, using signal-to-noise ratio (SNR) combined with permutation-invariant training (PIT) as the objective function, the sound separation loss function can be expressed as:

[0109]

[0110] Among them, Π K Representing all possible permutations, π is a permutation mapping, K is the number of target sound sources, and T is the signal length. It is the value of the k-th estimated sound source at time step t under the arrangement of π, while s k (t) is the value of the target sound source at time step t.

[0111] The source counting loss function uses cross-entropy to measure and evaluate the accuracy of the model's estimation of the number of sound sources in a mixed audio signal, as shown in the following formula:

[0112]

[0113] Among them, y i It's a real label, while p exi It is the predicted probability.

[0114] Alignment loss function Defined as follows:

[0115]

[0116] By finding the optimal permutation between attractor embeddings and cue embeddings, and calculating the mean squared error (MSE) loss, it can be expressed as:

[0117]

[0118] Where π represents a permutation, Π K It is the set of all possible permutations, and D is the embedding dimension. π(m),i Let represent the i-th value of the m-th attractor under the permutation π, and c m,i It is the i-th value of the m-th cue. M is the total number of cue embeddings or attractor embeddings.

[0119] Given N attractors and N clues And the optimal permutation π obtained from the final separation PIT loss, InfoNCE loss. This is obtained by calculating the average InfoNCE loss for each pair of corresponding elements:

[0120]

[0121] in, It is the embedding vector of the i-th attractor. is the embedding vector of the j-th cue, π(i) is the index of the cue that best matches the i-th attractor according to the separation PIT loss, and τ is the temperature adjustment coefficient.

[0122] This embodiment exhibits good performance indicators in practical applications, including:

[0123] 1. Sound separation performance

[0124] This embodiment uses the USE framework. Sepformer and BSRNN are baseline models for sound separation. USE-S and USE-B are models implemented within the USE framework using Sepformer and BSRNN as separators, respectively. Stage 1 trains only the separation task, while stage 2 trains both SS and TSE tasks based on stage 1, thus possessing both sound separation and target sound extraction capabilities. nMix refers to the fact that each Mixture in the dataset consists of n sounds. Traditional Sepformer and BSRNN models can only be trained on data with a fixed number of Mixes, while this embodiment can train on data with a variable number of Mixes (here, it can train on 2-Mix and 3-Mix data simultaneously), thanks to the attractor network's ability to predict the number of sound sources.

[0125] As shown in Table 1, the sound separation performance under the USE framework is significantly better than the existing Sepformer and BSRNN models. USE-S stage 1 achieved SNR improvements of 8.7 and 7.9 dB on the Seen (within the domain) and Unseen (outside the domain) datasets of 2Mix, respectively, and 6.4 and 5.2 dB on the Seen dataset of 3Mix, which is 1.3 dB higher than Sepformer. USE-S stage 2, trained simultaneously on SS and TSE, shows that the separation metric does not decrease but rather slightly increases while achieving the TSE task. USE-B stage 1 achieved SNR improvements of 8.7 and 7.9 dB on the Seen and Unseen datasets of 2Mix, respectively, and 6.4 and 5.2 dB on the Seen dataset of 3Mix. USE-B stage 2, trained simultaneously on SS and TSE, shows that the separation metric does not decrease but rather slightly increases while achieving the TSE task.

[0126] Table 1. Comparison of signal-to-noise ratio improvement in sound separation tasks (unit: dB)

[0127]

[0128] 2. Sound source counting accuracy

[0129] As shown in Table 2, the source counting accuracy of USE is approximately 85.24% in the 3-Mix scenario, but slightly lower in the 2-Mix scenario. This can be attributed to the fact that each audio track in the AudioSet dataset may still contain multiple sound events, even after using Sound Event Detection (SED) to isolate segments. This may lead to a decrease in the accuracy of source number estimation under the 2-Mix condition. However, this level of accuracy indicates that the model can still reliably estimate the number of sound sources in the mixed audio, providing strong support for sound separation tasks.

[0130] Table 2 Comparison of Sound Source Counting Accuracy

[0131] Within the domain 79.74％ 84.42％ outside the territory 74.68％ 85.24％

[0132] 3. Target sound extraction performance

[0133] As shown in Table 3, USE performs exceptionally well under multimodal cue conditions in the target sound extraction task. For example, on the 2Mix Seen datasets Seen and Unseen, USE-B achieves SNR improvements of 8.9 dB and 8.8 dB, respectively, outperforming existing DCCRN models by 29.0% and 35.4%. Even with only one modal cue, USE maintains high performance, demonstrating its strong robustness and adaptability.

[0134] Table 3 Comparison of signal-to-noise ratio improvement in target sound extraction under multimodal cue conditions (unit: dB)

[0135]

[0136] like Figure 4 As shown, during the inference process, four 3Mix samples were randomly selected, two from the Seen dataset and two from the Unseen dataset. t-SNE visualization was performed on the attractor and clue in each mix sample.

[0137] like Figure 4 As shown, different types of sounds have a certain spatial distance in the t-SNE diagram, indicating that they are separable to a certain extent in the feature space.

[0138] Furthermore, attractors and cues of the same sound type exhibit similarity in feature space, meaning that if one is missing, the other can serve as a valid substitute.

[0139] As shown in Table 4, the extraction performance of USE-S (based on Sepformer) and USE-B (based on BSRNN) was compared with that of models trained only for the target sound extraction task.

[0140] Table 4 Comparison of Signal-to-Noise Ratio Improvement in Target Sound Extraction (unit: dB)

[0141]

[0142] The results showed that the extraction performance of USE-S and USE-B did not decrease after joint training with sound separation and target sound extraction.

[0143] In fact, they also showed a slight improvement in performance on the target sound extraction task. USE-S achieved an SNR improvement of 8.5 and 8.1 dB on the 2-Mix Seen and Unseen datasets, respectively, and an SNR improvement of 6.5 and 5.9 dB on the 3-Mix Seen and Unseen datasets, respectively, showing a slight improvement over Sepformer. USE-B achieved an SNR improvement of 8.9 and 8.8 dB on the 2-Mix Seen and Unseen datasets, respectively, and an SNR improvement of 6.3 and 5.0 dB on the 3-Mix Seen and Unseen datasets, respectively, showing a significant improvement over BSRNN.

[0144] Furthermore, these models are still able to perform sound separation tasks using EDA (Attractor Network) modules even without cues.

[0145] As shown in Table 5, the performance of different general-purpose sound extraction models on the target sound extraction task was also compared. Most of these models used the same SED pruning strategy as the processing method in this embodiment to process the AudioSet dataset.

[0146] However, these models may differ in details, so these results should only be used as a reference. Nevertheless, it is clear from the table that the USE-B model proposed in this embodiment performs best on the TSE task, with performance comparable to the BSRNN+ClueNet model trained alone.

[0147] Table 5. Comparison of Signal-to-Noise Ratio Improvement of General Sound Extraction Models (in dB)

[0148] MAE (audio) 5.6 / USS (audio) 5.6 / LASS(text) 6.8 / AudioSep(text) 7.7 / DCCRN (text + tag + video) 6.9 / Sepformer(text+tag+video) 8.5 6.5 USE-S (text + tag + video) 8.5 6.5 BSRNN(text+tag+video) 8.4 5.9 USE-B (text + tag + video) 8.9 6.3

[0149] 4. General sound separation performance

[0150] As shown in Table 6, a BSRNN-based USE-B model was also trained on a large-scale general audio dataset, which includes a general dataset mainly remixed by VGGSound (2-6 mix sources), AudioSet (2-3 mix sources), FUSS (2-4 mix sources), and Musan (music) + Librispeech (speech) (2-5 mix sources).

[0151] Table 6. Comparison of SNR improvement in sound separation using USE-B on composite datasets (unit: dB)

[0152]

[0153] During the evaluation process, two scenarios were considered: one where the number of sound sources to be separated is known, and the other where the number of sound sources is unknown and requires estimation using an EDA module.

[0154] First, USE-B demonstrates its ability to adapt to mixtures with varying numbers of sound sources and performs exceptionally well across different datasets. Second, it reveals that methods using EDA modules to predict the number of sound sources are generally comparable to, and in some cases even better than, methods based on a pre-defined fixed number of sound sources. Finally, USE-B not only successfully completed the separation task but also excelled in the target sound extraction task.

[0155] As shown in Table 7, the USE-B model was also compared with other competitive models in terms of scale-invariant signal-to-noise ratio (SI-SNRi) on the FUSS test set without reverberation. It can be seen that the USE-B model achieves excellent separation performance in scenarios with 2, 3, and 4 mixed sources.

[0156] Table 7 Comparison of Scale-Invariant Signal-to-Noise Ratio Improvements (Unit: dB)

[0157] TDCN++ 11.2 11.6 7.4 USE-B 12.8 13.1 11.9

[0158] like Figure 5 As shown, the inference speed for extracting audio signals from 1 to 6 sound sources from mixed audio was also evaluated. Observations revealed that the computational complexity increases linearly with the number of sound sources during inference. Notably, even when inferring up to six sound sources, the computational requirement in GFLOPS remains below 30. This characteristic ensures the real-time performance and high efficiency of the USE-B model during inference.

[0159] This invention is applicable to a variety of practical application scenarios, including but not limited to:

[0160] 1. Environmental monitoring: Isolating specific sound sources (such as animal calls, traffic noise, etc.) from complex environmental sounds.

[0161] 2. Multimedia content analysis: Extract target sounds from video or audio content, and even from real-world scenarios such as conference rooms, to enhance user experience.

[0162] 3. Intelligent voice assistant: Extracts user voice commands in noisy environments, improving the accuracy of voice recognition.

[0163] 4. Audio editing and creation: Helps audio engineers quickly separate and edit different sound components in audio.

[0164] This invention is based on existing deep learning frameworks (such as ESPnet-SE) and is easily integrated with other audio processing systems. Furthermore, its multimodal cue processing capabilities allow for flexible expansion of input modalities to meet specific needs, further enhancing the applicability of this invention.

[0165] This invention prioritizes computational efficiency in its design, employing a highly efficient encoder-splitter-decoder structure and a lightweight multimodal processing module. It can run on resource-constrained devices, such as embedded systems or mobile devices, and thus possesses significant practical value.

[0166] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for sound separation and target sound extraction based on a unified architecture, characterized in that, The method is based on a separate backbone network, an attractor network, and a multimodal cue processing network. It includes a training phase and an inference phase; the training phase includes a first training phase and a second training phase. In the first training phase, the attractor network is used to estimate the number of sound sources in the mixed audio signal, generate attractor embeddings, and calculate the sound source counting loss; The attractor is embedded into the separation backbone network to generate the separated sound, and the sound separation loss is calculated. In the second training phase, the multimodal cue processing network is used to process cue information from multiple modalities to generate cue embeddings; the attractor embedding or the cue embedding is randomly selected as the input to the separating backbone network; if the cue embedding is selected as the input to the separating backbone network, all seven modal combinations are used for equal-probability training. During the training phase, if the input of the separation backbone network is the attractor embedding, the matching order between the separated audio and the real audio is calculated when the sound separation loss function is minimized. Based on the matching order, the alignment loss function between the attractor embedding and the cue embedding is calculated to narrow the semantic space distance between the attractor and cue representations. During the reasoning phase, if there are no clues, the sound separation task is performed; if there are 1 to 3 clues, the target sound extraction task is performed.

2. The sound separation and target sound extraction method based on a unified architecture as described in claim 1, characterized in that, The separated backbone network includes: An encoder is used to generate hidden features of mixed audio signals; A separator is used to estimate a mask for each sound source and multiply the hidden features element-wise with the mask corresponding to each sound source to obtain a hidden representation; The decoder, consisting of transposed convolutional layers symmetrical to the encoder, is used to estimate the separated sound sources.

3. The sound separation and target sound extraction method based on a unified architecture as described in claim 2, characterized in that, The separator is used to estimate a mask for each sound source, including: The hidden features are passed through a LayerNorm layer and a linear layer, and then divided into overlapping blocks of size K=250 with a 50% overlap rate. A dual-path module is used to integrate Transformer blocks within and between blocks; Sequence aggregation is performed, and learnable weights are used to represent time regions with different dominant sounds in different subspaces. Frame-level embeddings W are generated and input into the attractor network or the multimodal cue processing network. The output V of the dual-path module is multiplied element-wise with the source sound representation of the attractor network or the multimodal cue processing network to form the input of the three-path module; the three-path module extends the dual-path design by adding a cross-channel Transformer block to capture the relationship between channels; The output of the three-path module is sequentially passed through a parameterized ReLU layer, overlapped and added, a gated output layer with two linear layers, and a linear layer with a ReLU activation function to generate a mask.

4. The sound separation and target sound extraction method based on a unified architecture as described in claim 1, characterized in that, The attractor network includes an LSTM encoder and an LSTM decoder; The LSTM encoder updates its hidden state using the following formula. and unit state : In this context, the hidden states and cell states of the LSTM encoder are initialized as zero vectors: and ; C The time dimension of the features after block division; The LSTM decoder estimates the attractor embedding using the following formula: In each step, the hidden state of the LSTM decoder As an attractor of sound category s, its dimension D is related to frame-level embedding. The dimensions are consistent; the hidden states and cell states of the LSTM decoder are initialized by the final state of the LSTM encoder: and .

5. The sound separation and target sound extraction method based on a unified architecture as described in claim 4, characterized in that, The attractor network also includes: The probability of the presence of the attractor embedding is calculated using a fully connected layer with a sigmoid activation function, as shown in the following formula: in, and These are the trainable weights and bias parameters of the fully connected layer; The estimated characterization of the s-th sound in the mixed audio signal is represented by the attractor. Each With predefined threshold Compare; if If so, then the attractor embedding is considered to exist.

6. The sound separation and target sound extraction method based on a unified architecture as described in claim 1, characterized in that, The multimodal cue processing network encodes input cues from text, video, and audio into a unified D-dimensional space using a dedicated encoder; it then employs a multimodal concatenation strategy to form unified multimodal cues; and finally, a cue fusion module based on an attention mechanism aligns and fuses the multimodal cues in time to extract global cue information, thus obtaining the cue embedding.

7. The sound separation and target sound extraction method based on a unified architecture as described in claim 6, characterized in that, The dedicated encoder converts different modes into D-dimensional embeddings, including: The text encoder, using a pre-trained DistilBERT model, converts natural language descriptions into text embeddings. ,in, T t It is the number of word tags; The video encoder processes video frames using a pre-trained Swing Transformer to generate video embedding vectors. ,in, T v It is the number of frames; The sound encoder maps one-hot encoded sound event tags to sound embedding vectors through a linear layer. .

8. The sound separation and target sound extraction method based on a unified architecture as described in claim 7, characterized in that, By employing a multimodal concatenation strategy, the text embedding vector, video embedding vector, and audio embedding vector are concatenated to form a unified multimodal cue: 。 9. The sound separation and target sound extraction method based on a unified architecture as described in claim 8, characterized in that, The attention-based cue fusion module includes: Using the output W of sequence aggregation as the query and U as the key and value, the clues are fused through a multi-head attention mechanism to obtain the fused clue embedding: MultiHeadAttention(Q,K,V) represents a multi-head attention mechanism, where Q, K, and V represent the query, key, and value, respectively.

10. The sound separation and target sound extraction method based on a unified architecture as described in claim 1, characterized in that, The alignment loss function is defined as follows: in, This is obtained by finding the optimal permutation between the attractor embedding and the cue embedding, and calculating the mean squared error loss; By giving N an attractor and N One clue, and the optimal permutation obtained from the final separation PIT loss. π This is obtained by calculating the average InfoNCE loss for each pair of corresponding elements.

Images