Sound source separation method, sound source separation device, and computer device

Abstract

This application relates to a sound source separation method, a sound source separation device, and a computer device. The method includes: acquiring the STFT time-frequency spectrum of a mixed audio signal; processing the STFT time-frequency spectrum using a sound source separation model to obtain the separation result of the mixed audio signal; the sound source separation model includes an encoder, a feature modeling network, and a decoder; the encoder is used to extract features from the STFT time-frequency spectrum to obtain multi-level time-frequency features; the feature modeling network is used to alternately model the highest-level time-frequency feature in the multi-level time-frequency features in the time and frequency dimensions to obtain a first modeled time-frequency feature; the feature modeling network is also used to perform long-time-series modeling of the highest-level time-frequency feature in the time dimension to obtain a second modeled time-frequency feature, and then fuse the first and second modeled time-frequency features to obtain a fused feature; the decoder is used to determine the separation result of the mixed audio signal based on the multi-level time-frequency features and the fused feature. This method can improve the accuracy of sound source separation.

Description

Technical Field

[0001] This application relates to the field of audio signal processing, and in particular to sound source separation methods, sound source separation devices, and computer equipment. Background Technology

[0002] In the field of audio signal processing, audio source separation refers to the technique of recovering individual original sound sources from a mixed multi-source audio signal. With the rapid development of applications such as intelligent voice interaction, music production, hearing aids, and conference transcription, the industry has an urgent need for high-quality audio source separation technology.

[0003] In related technologies, audio source separation models typically employ deep learning model structures such as convolutional neural networks, recurrent neural networks, and Transformers to achieve audio source separation. In real-time audio source separation, deep learning models must satisfy causal constraints during time-domain processing, requiring them to utilize information from both current and past moments for modeling. This requirement makes long-sequence modeling in the time dimension one of the performance bottlenecks for causal real-time audio source separation models. Specifically, when processing long sequences, audio source separation models struggle to effectively capture the dependencies between time steps, resulting in low accuracy in actual audio separation.

[0004] There is currently no effective solution to the problem of low accuracy in audio separation in related technologies. Summary of the Invention

[0005] In view of this, this application provides a sound source separation method, a sound source separation device, and a computer device to improve the accuracy of audio separation.

[0006] Firstly, this embodiment provides a sound source separation method, the method comprising:

[0007] Obtain the STFT time spectrum of the mixed audio;

[0008] The source separation model is used to process the STFT time-frequency spectrum to obtain the separation result of the mixed audio. The source separation model includes an encoder, a feature modeling network, and a decoder. The encoder is used to extract features from the STFT time-frequency spectrum to obtain multi-level time-frequency features. The feature modeling network is used to alternately model the highest-level time-frequency feature in the multi-level time-frequency features in the time and frequency dimensions to obtain a first modeled time-frequency feature. The feature modeling network is also used to perform long-time series modeling of the highest-level time-frequency feature in the time dimension to obtain a second modeled time-frequency feature, and then fuse the first modeled time-frequency feature and the second modeled time-frequency feature to obtain a fused feature. The decoder is used to determine the separation result of the mixed audio based on the multi-level time-frequency features and the fused feature.

[0009] Secondly, this embodiment provides a sound source separation device, the device comprising:

[0010] The time-spectrum acquisition module is used to acquire the STFT time-spectrum of the mixed audio.

[0011] The audio source separation module is used to process the STFT time-frequency spectrum using an audio source separation model to obtain the separation result of the mixed audio. The audio source separation model includes an encoder, a feature modeling network, and a decoder. The encoder is used to extract features from the STFT time-frequency spectrum to obtain multi-level time-frequency features. The feature modeling network is used to alternately model the highest-level time-frequency feature in the multi-level time-frequency features in both time and frequency dimensions to obtain a first modeled time-frequency feature. The feature modeling network is also used to perform long-sequence modeling of the highest-level time-frequency feature in the time dimension to obtain a second modeled time-frequency feature, and then fuse the first and second modeled time-frequency features to obtain a fused feature. The decoder is used to determine the separation result of the mixed audio based on the multi-level time-frequency features and the fused feature.

[0012] Thirdly, this embodiment provides a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the sound source separation method described in the first aspect above.

[0013] The audio source separation method, device, and computer equipment provided in this application acquire the STFT time-frequency spectrum of the mixed audio and extract features from the STFT time-frequency spectrum using an encoder in the audio source separation model to obtain multi-level time-frequency features. Then, using a feature modeling network in the audio source separation model, through a dual-path alternating modeling approach in the time and frequency dimensions, the dependency relationship of the highest-level time-frequency features output by the encoder in both time and frequency dimensions is effectively captured, resulting in the first modeled time-frequency feature. Simultaneously, using the feature modeling network for long-term modeling, the long-distance time dependency of the highest-level time-frequency feature is effectively captured, resulting in the second modeled time-frequency feature. Furthermore, by fusing the first and second modeled time-frequency features to obtain a fused feature, the decoder in the audio source separation model recovers the separation result of the mixed audio based on the multi-level time-frequency features and the fused feature. Thus, the audio source separation model can not only jointly model the STFT time-frequency spectrum in the time and frequency dimensions but also effectively expand the receptive field in the time dimension by acquiring the second modeled time-frequency feature, thereby enhancing the audio source separation model's ability to model long-term time-frequency dependencies and improving the accuracy of audio separation. Attached Figure Description

[0014] Figure 1 A flowchart illustrating an embodiment of the sound source separation method provided in this application;

[0015] Figure 2 This is a schematic diagram of the sound source separation model provided in this application;

[0016] Figure 3 A schematic diagram of the feature modeling network provided in this application;

[0017] Figure 4 A schematic diagram of the temporal convolutional neural network provided in this application;

[0018] Figure 5 A schematic diagram of the structure of the temporal modeling subnetwork provided in this application;

[0019] Figure 6 A schematic diagram of the structure of the timing modeling unit provided in this application;

[0020] Figure 7 This is a schematic diagram of the structure of the gating fusion network provided in this application;

[0021] Figure 8 A hardware structure diagram of the computer device containing the audio source separation device provided in this application;

[0022] Figure 9 This is a schematic diagram of the structure of an embodiment of the sound source separation device provided in this application. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0024] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0025] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0026] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0027] Figure 1 This is a flowchart illustrating an embodiment of the audio source separation method provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0028] S101. Obtain the STFT time spectrum of the mixed audio.

[0029] Here, mixed audio refers to an audio signal obtained by superimposing at least two sound sources. The STFT (Short-Time Fourier Transform) time spectrum is obtained by performing a short-time Fourier transform on the mixed audio. Optionally, an STFT operation is performed on the mixed audio to obtain the STFT time spectrum.

[0030] S102. The source separation model is used to process the STFT time spectrum to obtain the separation result of the mixed audio. The source separation model includes an encoder, a feature modeling network, and a decoder. The encoder is used to extract features from the STFT time spectrum to obtain multi-level time-frequency features. The feature modeling network is used to alternately model the highest-level time-frequency features in the multi-level time-frequency features in the time and frequency dimensions to obtain the first modeled time-frequency features. The feature modeling network is also used to perform long-time-series modeling of the highest-level time-frequency features in the time dimension to obtain the second modeled time-frequency features, and then fuse the first modeled time-frequency features and the second modeled time-frequency features to obtain the fused features. The decoder is used to determine the separation result of the mixed audio based on the multi-level time-frequency features and the fused features.

[0031] Optionally, the STFT time spectrum can be input into a pre-trained audio source separation model to obtain the separation result of the mixed audio. Figure 2 Please refer to the structural diagram of the sound source separation model provided in this application. Figure 2 The following is an explanation of the sound source separation model:

[0032] In one possible implementation, the encoder includes multiple coding layers connected in sequence; wherein each coding layer is used to downsample the STFT time spectrum layer by layer to obtain multi-level time-frequency features corresponding to the multiple coding layers.

[0033] The encoder is a neural network module consisting of multiple cascaded coding layers. Each coding layer includes one or more stacked convolutional layers used to downsample the STFT time spectrum layer by layer to extract multi-scale time-frequency features. The time-frequency features are used to characterize the potential structural features of the STFT time spectrum in the time and frequency dimensions. Each coding layer outputs the time-frequency features corresponding to its level.

[0034] For example, the encoder first decomposes the STFT time-frequency spectrum into sub-bands of different frequency bands, and inputs each sub-band into the encoder in parallel. Specifically, in the encoder, each sub-band is configured with a convolutional kernel with independent parameters, and each sub-band is downsampled layer by layer through multiple stacked convolutional layers to extract the latent features of the sub-band, thereby obtaining multi-level time-frequency features corresponding to each sub-band signal. The above sub-band division can adopt a frequency band segmentation strategy with equal or non-equal width. The implementation principle of the frequency band segmentation strategy can be found in the relevant technical records, and will not be elaborated here.

[0035] In one possible implementation, the decoder includes multiple decoding layers corresponding to multiple coding layers; wherein, the first decoding layer in the multiple decoding layers is used as input to take the fused features and the hierarchical time-frequency features output by the encoder corresponding to the first decoding layer as input, and performs fusion and transposed convolution processing on the input features to obtain the upsampling result of the first decoding layer; the non-first decoding layers in the multiple decoding layers are used as input to take the upsampling result of the previous decoding layer and the hierarchical time-frequency features output by the encoder corresponding to the non-first decoding layer as input, and perform fusion and transposed convolution processing on the input features to obtain the upsampling result of the non-first decoding layer; the tail decoding layer in the multiple decoding layers is also used to determine the separation result based on the upsampling result of the tail decoding layer.

[0036] The decoder is a neural network module composed of multiple cascaded decoding layers. Each decoding layer includes one or more stacked transposed convolutional layers used to reconstruct the time-frequency representation of the input features, ultimately obtaining the complete, separated STFT time-frequency spectrum. Each decoding layer is opposite to the corresponding encoding layer in the encoder, forming a symmetrical structure. The first decoding layer is the layer that performs the upsampling operation first in the multi-layer decoding process, and its input is the highest-level time-frequency feature output by the encoder. The last decoding layer is the layer that performs the upsampling operation last in the multi-layer decoding process and is also responsible for outputting the final reconstructed complete STFT time-frequency spectrum.

[0037] Upsampling results refer to the features generated after each layer of the decoder performs deconvolution (i.e., transposed convolution) on the input features. Optionally, each layer of the decoder includes a fusion module and an upsampling module. The fusion module fuses the time-frequency features output from the corresponding layer in the encoder with the features output from the previous layer's decoder, and uses the fused features as the input to the upsampling module in the current decoding layer. The upsampling module is used to upsample the fused features through one or more stacked transposed convolutional layers to obtain the upsampling result of the current layer. The parameters of each encoding layer and the selected fusion method can be configured according to requirements and are not limited here.

[0038] Determining the separation result based on the upsampling results of the tail layer decoding layer refers to performing a sub-band merging operation on the upsampling results of each sub-band output after the tail layer decoding layer completes upsampling, thereby generating a complete and separated STFT complex spectrum. Further, performing an inverse short-time Fourier transform on the separated STFT complex spectrum converts it into a time-domain waveform, yielding the separation result of the mixed audio.

[0039] In one possible implementation, the feature modeling network can use a pre-trained separator, such as one based on LSTM (Long Short-Term Memory) or GRU (Gate Recurrent Unit), to alternately model the highest-level time-frequency features in the multi-level time-frequency features across the time and frequency dimensions. The highest-level time-frequency feature refers to the time-frequency feature output by the last coding layer in the encoder to perform a downsampling operation. The first modeled time-frequency feature is used to characterize the deep dependencies of the STFT time-frequency spectrum in both the temporal and frequency domains.

[0040] Specifically, the separator can be a multi-layered stacked dual-path recurrent neural network. For example, in a dual-path recurrent neural network constructed using LSTM or GRU, one branch expands along the time dimension for time-series modeling, while the other branch expands along the frequency dimension for spectral modeling. The two branches alternately perform one or more rounds of modeling. For example, after time-series modeling is performed in one branch, the intermediate features obtained from the current time-series modeling are input into the other branch for spectral modeling. Then, the intermediate features obtained from the current spectral modeling are input into the path where time-series modeling is performed, and this process is repeated to obtain the first modeled time-frequency features. The specific structure and training method of the aforementioned LSTM or GRU can be found in relevant technical documents and will not be elaborated upon here.

[0041] In one possible implementation, the feature modeling network can use neural networks with long-term temporal modeling capabilities, such as temporal convolutional neural networks or Transformer networks, to perform long-term temporal modeling of the highest-level time-frequency features. The second modeled time-frequency features are used to characterize the dependence of the STFT time-frequency spectrum over long distances.

[0042] In one possible implementation, the feature modeling network can fuse the first modeling time-frequency features and the second modeling time-frequency features by means of element-wise addition, weighted fusion, channel splicing, etc., to obtain fused features.

[0043] In this embodiment, after obtaining the STFT time-frequency spectrum of the mixed audio, the encoder in the source separation model extracts features from the STFT time-frequency spectrum of the mixed audio to obtain multi-level time-frequency features. Then, the feature modeling network in the source separation model effectively captures the time and frequency dependencies of the highest-level time-frequency features output by the encoder through a dual-path alternating modeling method in the time and frequency dimensions, obtaining the first modeled time-frequency features. Simultaneously, the feature modeling network effectively captures the long-distance time dependency of the highest-level time-frequency features output by the encoder through long-time-series modeling, obtaining the second modeled time-frequency features. After fusing the first and second modeled time-frequency features to obtain the fused features, the decoder in the source separation model recovers the separation result of the mixed audio based on the multi-level time-frequency features and the fused features. In this way, the source separation model can not only jointly model the STFT time-frequency spectrum in the time and frequency dimensions, but also effectively expand the receptive field in the time dimension by obtaining the second modeled time-frequency features, thereby enhancing the source separation model's ability to model long-time-series dependencies and improving the accuracy of audio separation.

[0044] Figure 3 A schematic diagram of the feature modeling network provided in this application is shown below. Figure 3 As shown, the feature modeling network includes a separator, a temporal convolutional neural network, and a gated fusion network. The gated fusion network is connected to the separator and the temporal convolutional neural network, respectively. The separator is used to alternately model the highest-level time-frequency features in the time and frequency dimensions to obtain the first modeled time-frequency features. The temporal convolutional neural network is used to perform long-term time-series modeling of the highest-level time-frequency features in the time dimension to obtain the second modeled time-frequency features. The gated fusion network is used to fuse the first and second modeled time-frequency features to output the fused features.

[0045] A separator is a neural network structure used to estimate and extract one or more latent features from the highest-level time-frequency features. The structure of a separator is described above and will not be repeated here. A temporal convolutional neural network is a convolutional neural network structure used to model the temporal dependencies in the highest-level time-frequency features. A gated fusion network is a neural network structure that dynamically adjusts the gate weights corresponding to the first and / or second modeled time-frequency features based on gate units to achieve weighted fusion.

[0046] Furthermore, in one embodiment, Figure 4 A schematic diagram of the temporal convolutional neural network provided in this application is shown below. Figure 4 As shown, the temporal convolutional neural network includes an input projection layer, an input reshaping layer, a temporal modeling subnetwork, an output reshaping layer, and an output projection layer.

[0047] (1) Input projection layer, used to map the highest level time-frequency features to the first preset dimension corresponding to the time convolutional neural network to obtain the mapped features.

[0048] Here, the first preset dimension is the number of input channels required by the temporal convolutional neural network. Optionally, the input projection layer uses a 1x1 two-dimensional convolutional kernel to project the number of channels of the highest-level time-frequency features output by the encoder onto the first preset dimension. Furthermore, to improve the stability of the feature representation, a normalization layer along the channel dimension can be introduced after performing the two-dimensional convolution operation to normalize the mapping result of the first preset dimension. Even further, to enhance the non-linear expressive power of the features, a PReLU (Parametric ReLU) activation function can be introduced after the normalization layer.

[0049] (2) Input reshaping layer, used to merge the frequency dimension and channel dimension of the mapped features into a merged channel dimension to obtain merged features.

[0050] The dimensions of the mapped features are represented as (B, C, F, T), where B represents the batch dimension, C represents the channel dimension, F represents the frequency dimension, and T represents the time dimension. Optionally, the input reshaping layer reshapes the mapped features (B, C, F, T) into the form (B, C×F, T), where C×F represents the merged channel dimension. By merging the channel dimension, features originally distributed in the channel and frequency spaces can be compressed into a single channel dimension, thus facilitating one-dimensional modeling along the time dimension by the temporal modeling subnetwork, reducing the computational complexity of the temporal convolutional neural network, and improving the time dimension modeling capability.

[0051] (3) Temporal modeling sub-network, used to perform long-term temporal modeling on the merged features in the time dimension to obtain the initial modeling features.

[0052] Optionally, the temporal modeling subnetwork includes multiple stacked temporal modeling modules, each of which performs convolution operations on the input features along the time dimension to model long-term temporal dependencies.

[0053] To improve modeling stability, the time series modeling module can use a residual connection structure. Figure 5 A schematic diagram of the structure of the temporal modeling subnetwork provided in this application is shown below. Figure 5 As shown, the temporal modeling subnetwork includes multiple sequentially connected temporal modeling modules; each temporal modeling module includes a temporal modeling unit and a fusion layer; wherein, the temporal modeling unit is used to perform convolution processing on the input features of the current temporal modeling module to obtain the convolution processing result; the fusion layer is used to fuse the input features and the convolution processing result to obtain the output features corresponding to the temporal modeling module.

[0054] Each temporal modeling unit can employ convolutional operations such as causal convolution or dilated causal convolution to achieve convolutional processing while preserving temporal causality. The fusion layer fuses the input features and the convolutional processing results, which means adding the input features and the convolutional processing results element-wise. Each temporal modeling module retains the low-level temporal information in the original input features while fusing it with the high-level features obtained from its own convolutional processing, enabling the output preliminary modeling results to more accurately extract the potential characteristics of the merged input features in the temporal dimension.

[0055] Furthermore, Figure 6 The schematic diagram of the temporal modeling unit provided in this application shows that the temporal modeling unit includes sequentially connected pointwise convolutional sub-units, dilated convolutional sub-units, regularization layers, and a first pointwise convolutional layer. The pointwise convolutional sub-unit includes a second pointwise convolutional layer, a first nonlinear activation function, and a first normalization layer, used to sequentially perform one-dimensional pointwise convolution, nonlinear transformation, and normalization on the input features. The dilated convolutional sub-unit includes a dilated convolutional layer, a second nonlinear activation function, and a second normalization layer, used to sequentially perform one-dimensional dilated convolution, nonlinear transformation, and normalization on the features output by the pointwise convolutional sub-units. The regularization layer is used to perform regularization on the features output by the dilated convolutional sub-units. The first pointwise convolutional layer is used to perform one-dimensional pointwise convolution on the features output by the regularization layer to obtain the convolution processing result.

[0056] In this unit, the second pointwise convolutional layer, the first nonlinear activation function, and the first normalization layer are sequentially connected. The first normalization layer is used to normalize the input features along the channel dimension (Channel-wise LayerNorm). One-dimensional pointwise convolution can fuse information from different channels, and nonlinear activation and normalization can stabilize the distribution of features and better fit complex temporal patterns.

[0057] In the dilated convolutional subunit, the dilated convolutional layer, the second nonlinear activation function, and the second normalization layer are connected sequentially. The second normalization layer is used to normalize the input features along the channel dimension. The dilated convolutional layer expands the receptive field without increasing the number of parameters, and the second normalization layer further stabilizes the distribution of intermediate features.

[0058] Regularization can further standardize the feature distribution. Finally, the first pointwise convolutional layer can integrate the feature information extracted by dilated convolution and adjust the number of feature channels.

[0059] For example, both the first and second nonlinear activation functions are PReLU. Optionally, the dilated convolutional layer can use dilated casual conv1d (dilated causal one-dimensional convolution). It is understood that the first and second nonlinear activation functions can also be different nonlinear activation functions. Other convolution methods can be used for the dilated convolutional layer, such as non-causal dilated one-dimensional convolution. Specific implementation methods for nonlinear activation, pointwise convolution, dilated convolution, normalization, and regularization can be found in the documentation on neural network-related techniques, and will not be elaborated upon here.

[0060] (4) Output reshaping layer, used to split the merged channel dimension of the initial modeling features into channel dimension and frequency dimension to obtain split features.

[0061] The initial modeling features are represented by the dimension (B, C×F, T). Optionally, the output reshaping layer reshapes the initial modeling features (B, C×F, T) into a four-dimensional form (B, C, F, T), thus obtaining the splitting process; the above-mentioned dimensional change process can be achieved through related operators such as reshape, compute, and transpose.

[0062] (5) Output projection layer, used to split the feature mapping to the second preset dimension corresponding to the decoder to obtain the second modeling time-frequency feature.

[0063] The second preset dimension is the number of input channels required by the decoder. Optionally, the output projection layer projects the number of channels of the split features output by the output reshaping layer onto the second preset dimension using a 1x1 two-dimensional convolution kernel. Furthermore, to improve the stability of the feature representation, a normalization layer along the channel dimension can be introduced after performing the 1x1 two-dimensional convolution operation to normalize the mapping result of the second preset dimension.

[0064] In this embodiment, the feature dimensions input to the temporal modeling subnetwork are adjusted by the input projection layer and the input reshaping layer, enabling the temporal modeling subnetwork to obtain accurate long-term temporal modeling based on the merged features of the input, thus obtaining the initial modeling features. Subsequently, by splitting the merged channel dimension in the initial modeling features, the original one-dimensional channel features can be decomposed back into two independent dimensions: channel and frequency. The dimension mapping is then achieved through the output projection layer, which facilitates the subsequent gating fusion network to fuse the first modeling time-frequency features with the second modeling time-frequency features.

[0065] In some of these embodiments, Figure 7 This is a schematic diagram of the gated fusion network provided in this application. Please refer to... Figure 7The gated fusion network comprises a splicing subnetwork, an update subnetwork, a gated subnetwork, and an output subnetwork. The splicing subnetwork splices the first and second modeling time-frequency features to generate spliced ​​features; the update subnetwork extracts features from the spliced ​​features to obtain updated features; the gated subnetwork generates gate weights for the updated features based on the spliced ​​features; and the output subnetwork adjusts the updated features according to the gate weights to obtain adjusted features, and then fuses the first modeling time-frequency features and the adjusted features to obtain fused features.

[0066] The update subnetwork extracts features through nonlinear transformations and convolutions, resulting in updated features. The gated subnetwork uses a gating mechanism that nonlinearly maps the input features or their transformed forms to obtain gate weights that adjust the importance of the updated features in the fused features.

[0067] For example, the separator, as the main branch, outputs the first modeling time-frequency feature (main), and the temporal convolutional neural network, as another branch, outputs the second modeling time-frequency feature. The concatenation subnetwork concatenates the first and second modeling time-frequency features along the channel dimension to generate the concatenated feature (fused). The fused feature output by the output subnetwork is obtained by element-wise addition of the first modeling time-frequency feature and the adjustment feature, specifically expressed as follows:

[0068] Fusion Feature = main + gate × update

[0069] In this embodiment, the contribution of the updated features to the final output fusion features is adaptively controlled based on the gating mechanism, thereby adjusting the degree to which the sound source separation model retains the first modeling time-frequency features and the second modeling time-frequency features, so as to improve the accuracy of sound source separation.

[0070] In one embodiment, the updated subnetwork includes a first convolutional layer, a third normalization layer, and a third nonlinear activation function; wherein, the first convolutional layer is used to perform two-dimensional convolution processing on the concatenated features; the third normalization layer is used to normalize the features output by the first convolutional layer along the channel dimension; and the third nonlinear activation function is used to perform nonlinear transformation on the features output by the third normalization layer.

[0071] For example, the first convolutional layer uses a 1x1 two-dimensional convolution; the non-linear activation function is the PReLU activation function. The fused data is fed into an update sub-network consisting of a two-dimensional convolutional layer, a normalization layer along the channel dimension, and PReLU, to obtain the updated features.

[0072] In this embodiment, the updating subnetwork performs nonlinear transformation and feature extraction on the splicing features, which can effectively capture the potential information in the splicing features, so that the updated features can accurately express the characteristics of time-frequency features in the time and frequency dimensions.

[0073] In one embodiment, the gated subnetwork includes: at least one second convolutional layer and a fourth nonlinear activation function; wherein the second convolutional layer is used to perform two-dimensional convolution processing on the updated features; and the fourth nonlinear activation function is used to perform nonlinear transformation on the features output by the second convolutional layer.

[0074] For example, the second convolutional layer uses a 1x1 two-dimensional convolution; the fourth non-linear activation function uses the sigmoid activation function. The fused is fed into a gated subnetwork consisting of a two-dimensional convolutional layer and a sigmoid activation function to obtain the gate weights.

[0075] In this embodiment, the gated subnetwork obtains the gate weights of the updated features based on the gate mechanism, which can realize adaptive correction of the updated features and enhance the robustness of the output features of the feature modeling network.

[0076] Corresponding to the aforementioned embodiment of the sound source separation method, this application also provides an embodiment of the sound source separation device.

[0077] An embodiment of the audio source separation device disclosed in this application can be applied to a computer device. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the computer device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 8 The diagram shown is a hardware structure diagram of a computer device containing the audio source separation device of this application. Except for... Figure 8 In addition to the processor, memory, network interface, and non-volatile memory shown, the computer device in which the device is located in the embodiment may also include other hardware depending on the actual function of the audio source separation device, which will not be described in detail here.

[0078] Figure 9 This is a schematic diagram of an embodiment of the sound source separation device provided in this application. Please refer to... Figure 9 The apparatus provided in this embodiment includes:

[0079] The time spectrum acquisition module 901 is used to acquire the STFT time spectrum of the mixed audio.

[0080] The audio source separation module 902 is used to process the STFT time-frequency spectrum using an audio source separation model to obtain the separation result of the mixed audio. The audio source separation model includes an encoder, a feature modeling network, and a decoder. The encoder is used to extract features from the STFT time-frequency spectrum to obtain multi-level time-frequency features. The feature modeling network is used to alternately model the highest-level time-frequency feature in the multi-level time-frequency features in the time and frequency dimensions to obtain the first modeled time-frequency feature. The feature modeling network is also used to perform long-time-series modeling of the highest-level time-frequency feature in the time dimension to obtain the second modeled time-frequency feature, and to fuse the first modeled time-frequency feature and the second modeled time-frequency feature to obtain the fused feature. The decoder is used to determine the separation result of the mixed audio based on the multi-level time-frequency features and the fused feature.

[0081] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0082] In some embodiments, the feature modeling network includes a separator, a temporal convolutional neural network, and a gated fusion network; the gated fusion network is connected to the separator and the temporal convolutional neural network respectively; wherein, the separator is used to alternately model the highest-level time-frequency features in the time dimension and the frequency dimension to obtain a first modeled time-frequency feature; the temporal convolutional neural network is used to perform long-time series modeling of the highest-level time-frequency features in the time dimension to obtain a second modeled time-frequency feature; the gated fusion network is used to fuse the first modeled time-frequency feature and the second modeled time-frequency feature to output a fused feature.

[0083] Further, in one embodiment, the temporal convolutional neural network includes an input projection layer, an input reshaping layer, a temporal modeling subnetwork, an output reshaping layer, and an output projection layer; wherein, the input projection layer is used to map the highest-level time-frequency features to a first preset dimension corresponding to the temporal convolutional neural network to obtain mapped features; the input reshaping layer is used to merge the frequency dimension and channel dimension of the mapped features into a merged channel dimension to obtain merged features; the temporal modeling subnetwork is used to perform long-term temporal modeling on the merged features in the time dimension to obtain initial modeling features; the output reshaping layer is used to split the merged channel dimension of the initial modeling features into a channel dimension and a frequency dimension to obtain split features; the output projection layer is used to map the split features to a second preset dimension corresponding to the decoder to obtain second modeled time-frequency features.

[0084] Optionally, the temporal modeling subnetwork includes multiple sequentially connected temporal modeling modules; each temporal modeling module includes a temporal modeling unit and a fusion layer; wherein, the temporal modeling unit is used to perform convolution processing on the input features of the current temporal modeling module to obtain the convolution processing result; the fusion layer is used to fuse the input features and the convolution processing result to obtain the output features corresponding to the temporal modeling module.

[0085] Optionally, the temporal modeling unit includes sequentially connected pointwise convolutional sub-units, dilated convolutional sub-units, a regularization layer, and a first pointwise convolutional layer; wherein, the pointwise convolutional sub-unit includes a second pointwise convolutional layer, a first nonlinear activation function, and a first normalization layer, used to sequentially perform one-dimensional pointwise convolution, nonlinear transformation, and normalization on the input features; the dilated convolutional sub-unit includes a one-dimensional dilated convolutional layer, a second nonlinear activation function, and a second normalization layer, used to sequentially perform dilated convolution, nonlinear transformation, and normalization on the features output by the pointwise convolutional sub-units; the regularization layer is used to perform regularization on the features output by the dilated convolutional sub-units; and the first pointwise convolutional layer is used to perform one-dimensional pointwise convolution on the features output by the regularization layer to obtain the convolution processing result.

[0086] Furthermore, in one embodiment, the gated fusion network includes a splicing subnetwork, an update subnetwork, a gated subnetwork, and an output subnetwork, wherein: the splicing subnetwork is used to splice the first modeling time-frequency feature and the second modeling time-frequency feature to generate a spliced ​​feature; the update subnetwork is used to extract features from the spliced ​​feature to obtain an updated feature; the gated subnetwork is used to generate gate weights for the updated feature based on the spliced ​​feature; and the output subnetwork is used to adjust the updated feature according to the gate weights to obtain an adjusted feature, and to fuse the first modeling time-frequency feature and the adjusted feature to obtain a fused feature.

[0087] Optionally, the updated subnetwork includes a first convolutional layer, a third normalization layer, and a third nonlinear activation function; wherein, the first convolutional layer is used to perform two-dimensional convolution processing on the concatenated features; the third normalization layer is used to normalize the features output by the first convolutional layer along the channel dimension; and the third nonlinear activation function is used to perform nonlinear transformation on the features output by the third normalization layer.

[0088] Please continue to refer to Figure 8 This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0089] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0090] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0091] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for separating sound sources, characterized in that, The method includes: Obtain the STFT time spectrum of the mixed audio; The source separation model is used to process the STFT time-frequency spectrum to obtain the separation result of the mixed audio. The source separation model includes an encoder, a feature modeling network, and a decoder. The encoder is used to extract features from the STFT time-frequency spectrum to obtain multi-level time-frequency features. The feature modeling network is used to alternately model the highest-level time-frequency feature in the multi-level time-frequency features in the time and frequency dimensions to obtain a first modeled time-frequency feature. The feature modeling network is also used to perform long-time series modeling of the highest-level time-frequency feature in the time dimension to obtain a second modeled time-frequency feature, and then fuse the first modeled time-frequency feature and the second modeled time-frequency feature to obtain a fused feature. The decoder is used to determine the separation result of the mixed audio based on the multi-level time-frequency features and the fused feature.

2. The method according to claim 1, characterized in that, The feature modeling network includes a separator, a temporal convolutional neural network, and a gated fusion network; the gated fusion network is connected to the separator and the temporal convolutional neural network, respectively; wherein... The separator is used to alternately model the highest-level time-frequency features in the time dimension and the frequency dimension to obtain the first modeled time-frequency features. The temporal convolutional neural network is used to perform long-term time-series modeling on the highest-level time-frequency features in the time dimension to obtain the second modeled time-frequency features. The gated fusion network is used to fuse the first modeling time-frequency features and the second modeling time-frequency features, and output the fused features.

3. The method according to claim 2, characterized in that, The temporal convolutional neural network includes an input projection layer, an input reshaping layer, a temporal modeling subnetwork, an output reshaping layer, and an output projection layer; wherein, The input projection layer is used to map the highest-level time-frequency features to the first preset dimension corresponding to the time convolutional neural network to obtain the mapped features; The input reshaping layer is used to merge the frequency dimension and channel dimension of the mapped feature into a merged channel dimension to obtain the merged feature. The temporal modeling sub-network is used to perform long-term temporal modeling on the merged features in the time dimension to obtain initial modeling features; The output reshaping layer is used to split the merged channel dimension of the initial modeling features into a channel dimension and a frequency dimension to obtain split features; The output projection layer is used to map the split features to the second preset dimension corresponding to the decoder to obtain the second modeling time-frequency features.

4. The method according to claim 3, characterized in that, The temporal modeling subnetwork includes multiple sequentially connected temporal modeling modules; each temporal modeling module includes a temporal modeling unit and a fusion layer; wherein... The temporal modeling unit is used to perform convolution processing on the input features of the current temporal modeling module to obtain the convolution processing result; The fusion layer is used to fuse the input features and the convolution processing results to obtain the output features corresponding to the temporal modeling module.

5. The method according to claim 4, characterized in that, The temporal modeling unit includes sequentially connected pointwise convolutional sub-units, dilated convolutional sub-units, a regularization layer, and a first pointwise convolutional layer; wherein... The pointwise convolutional subunit includes a second pointwise convolutional layer, a first nonlinear activation function, and a first normalization layer, used to sequentially perform one-dimensional pointwise convolution, nonlinear transformation, and normalization processing on the input features. The dilated convolution subunit includes a one-dimensional dilated convolution layer, a second nonlinear activation function, and a second normalization layer, which are used to sequentially perform dilated convolution, nonlinear transformation, and normalization processing on the features output by the pointwise convolution subunit. The regularization layer is used to perform regularization processing on the features output by the dilated convolutional subunit; The first pointwise convolutional layer is used to perform one-dimensional pointwise convolution processing on the features output by the regularization layer to obtain the convolution processing result.

6. The method according to claim 2, characterized in that, The gated fusion network includes a splicing subnetwork, an update subnetwork, a gated subnetwork, and an output subnetwork, wherein; The splicing sub-network is used to splice the first modeling time-frequency feature and the second modeling time-frequency feature to generate spliced ​​features; The updating sub-network is used to extract features from the spliced ​​features to obtain updated features; The gated subnetwork is used to generate the gate weights of the updated features based on the splicing features; The output sub-network is used to adjust the updated features according to the gating weights to obtain the adjusted features, and to fuse the first modeling time-frequency features and the adjusted features to obtain the fused features.

7. The method according to claim 6, characterized in that, The updated subnetwork includes a first convolutional layer, a third normalization layer, and a third nonlinear activation function; wherein, The first convolutional layer is used to perform two-dimensional convolution processing on the spliced ​​features; The third normalization layer is used to normalize the features output by the first convolutional layer along the channel dimension; The third nonlinear activation function is used to perform nonlinear transformation on the features output by the third normalization layer.

8. The method according to claim 6 or 7, characterized in that, The gated subnetwork includes: at least one second convolutional layer and a fourth nonlinear activation function; wherein... The second convolutional layer is used to perform two-dimensional convolution processing on the updated features; The fourth nonlinear activation function is used to perform nonlinear transformations on the features output by the second convolutional layer.

9. A sound source separation device, characterized in that, The device includes: The time-spectrum acquisition module is used to acquire the STFT time-spectrum of the mixed audio. The audio source separation module is used to process the STFT time-frequency spectrum using an audio source separation model to obtain the separation result of the mixed audio. The audio source separation model includes an encoder, a feature modeling network, and a decoder. The encoder is used to extract features from the STFT time-frequency spectrum to obtain multi-level time-frequency features. The feature modeling network is used to alternately model the highest-level time-frequency feature in the multi-level time-frequency features in both time and frequency dimensions to obtain a first modeled time-frequency feature. The feature modeling network is also used to perform long-sequence modeling of the highest-level time-frequency feature in the time dimension to obtain a second modeled time-frequency feature, and then fuse the first and second modeled time-frequency features to obtain a fused feature. The decoder is used to determine the separation result of the mixed audio based on the multi-level time-frequency features and the fused feature.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images