Sound source separation method and device

Abstract

The embodiment of the invention provides a sound source separation method and device, and relates to the technical field of data processing. The method comprises the following steps: converting a to-be-separated audio signal from a time domain signal into a time-frequency domain signal; frequency band segmentation is carried out on the time-frequency domain signal, so that the time-frequency domain signal is segmented into a plurality of sub-band signals, and frequency bands of the plurality of sub-band signals are not overlapped; respectively acquiring spectrum characteristics of the plurality of sub-band signals; acquiring a frequency spectrum mask of at least one sound source of the audio signal to be separated according to the frequency spectrum characteristics of the plurality of sub-band signals; and acquiring an audio signal of the at least one sound source according to the spectrum mask of the at least one sound source and the time-frequency domain signal. The embodiment of the invention is used for improving the robustness of a sound source separation algorithm.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a method and apparatus for separating audio sources. Background Technology

[0002] Audio source separation, also known as audio signal source separation or audio source separation, is an audio processing technique that aims to separate individual audio source components from a mixed audio signal.

[0003] Traditional audio source separation schemes primarily rely on signal processing techniques, such as independent component analysis (ICM). These schemes extract features from one or more audio sources by performing frequency domain or time-frequency domain analysis on the audio signal, thereby achieving source separation. However, these schemes often fail to achieve ideal separation results when dealing with complex audio signals. With the development of machine learning and deep learning technologies, data-driven schemes have made significant progress in the field of audio source separation. By training deep neural networks (DNNs), features and patterns for audio source separation can be learned from large amounts of audio data, resulting in more accurate and efficient source separation. However, these algorithms exhibit insufficient robustness when performing source separation on different types of audio signals. For example, if a certain audio signal is not included in the training dataset, the algorithm's source separation performance for that type of audio signal may decrease significantly. Summary of the Invention

[0004] In view of this, embodiments of this application provide a sound source separation method and apparatus to improve the robustness of sound source separation algorithms.

[0005] To achieve the above objectives, the technical solutions provided in this application are as follows:

[0006] In a first aspect, embodiments of this application provide a sound source separation method, including:

[0007] The audio signal to be separated is transformed from a time-domain signal to a time-frequency domain signal;

[0008] The time-frequency domain signal is divided into multiple sub-band signals, and the frequency bands of the multiple sub-band signals do not overlap.

[0009] The spectral characteristics of the multiple sub-band signals are obtained respectively;

[0010] Based on the spectral characteristics of the multiple sub-band signals, obtain the spectral mask of at least one sound source of the audio signal to be separated;

[0011] The audio signal of at least one sound source is obtained based on the spectral mask of the at least one sound source and the time-frequency domain signal.

[0012] As an optional implementation of this application, the step of transforming the audio signal to be separated from a time-domain signal to a time-frequency domain signal includes:

[0013] The audio signal to be separated is subjected to a short-time Fourier transform to transform it from a time-domain signal to a time-frequency domain signal.

[0014] As an optional implementation of this application, the step of performing frequency band segmentation on the time-frequency domain signal to divide the time-frequency domain signal into multiple sub-band signals includes:

[0015] The first frequency band of the time-frequency domain signal is divided into frequency bands based on a first frequency band interval, and the second frequency band of the time-frequency domain signal is divided into frequency bands based on a second frequency band interval;

[0016] Wherein, the maximum frequency of the first frequency band is less than the minimum frequency of the second frequency band, and the first frequency band spacing is less than the second frequency band spacing.

[0017] As an optional implementation of this application, the step of obtaining the spectral characteristics of the plurality of sub-band signals includes:

[0018] Feature extraction is performed on the multiple sub-band signals respectively to obtain the sub-band features of each sub-band signal;

[0019] Stack the sub-band features of each sub-band signal to obtain the stacked features;

[0020] The spectral characteristics of each sub-band signal are obtained based on the stacking characteristics.

[0021] As an optional implementation of this application, the step of extracting features from the plurality of sub-band signals to obtain the sub-band features of each sub-band signal includes:

[0022] Each sub-band signal is feature extracted by a multilayer perceptron corresponding to each sub-band signal to obtain the sub-band features of each sub-band signal;

[0023] Each subband signal corresponds to a multilayer perceptron consisting of a root mean square normalization layer and a linear layer connected in series.

[0024] As an optional implementation of this application, obtaining the spectral characteristics of each sub-band signal based on the stacking characteristics includes:

[0025] The local feature extraction module models the features of each sub-band of the stacked features in the time dimension to obtain local features composed of the temporal features of each sub-band signal.

[0026] The local features are transposed to obtain the first transposed feature;

[0027] The global feature extraction module models each temporal feature of the first transposed feature in the feature stacking dimension to obtain global features composed of the frequency band features of each sub-band signal.

[0028] Perform a transpose operation on the global feature to obtain a second transpose feature;

[0029] The second transposed feature is fused using a multi-head self-attention mechanism to obtain the fused feature.

[0030] The fusion features are split to obtain the spectral features of each sub-band signal.

[0031] As an optional implementation of this application, the local feature extraction module and / or the global feature extraction module are dual-path Mamba modules; the dual-path Mamba module includes:

[0032] The first path includes: a first Mamba block, a first root mean square normalization layer, and a first adder. The input of the first Mamba block is the input of the dual-path Mamba module. The input of the first root mean square normalization layer is the output of the first Mamba block. The input of the first adder is the output of the first Mamba block and the output of the first root mean square normalization layer. The output of the first adder is the output of the first path.

[0033] The second path includes: a flip layer, a second Mamba block, a second root mean square normalization layer, and a second adder. The flip layer is used to flip the input of the dual-path Mamba module. The input of the second Mamba block is the output of the flip layer. The input of the second root mean square normalization layer is the output of the second Mamba block. The input of the second adder is the output of the second Mamba block and the output of the second root mean square normalization layer. The output of the second adder is the output of the second path.

[0034] A splicing layer, used to splice the output of the first path and the output of the second path;

[0035] A linear layer, the input of which is the output of the splicing layer, and the output of which is the output of the dual-path Mamba module.

[0036] As an optional implementation of this application, obtaining the spectral mask of at least one sound source of the audio signal to be separated based on the spectral characteristics of each sub-band signal includes:

[0037] For each sound source, the spectral characteristics of each sub-band signal are processed by the corresponding mask estimation module to obtain the spectral mask of each sub-band signal corresponding to the sound source, and the spectral masks of each sub-band signal corresponding to the sound source are spliced ​​together to obtain the spectral mask of the sound source.

[0038] Each sub-band's corresponding mask estimation module consists of a multilayer perceptron and a gated linear unit connected in series. The multilayer perceptron consists of a root mean square normalization layer and a linear layer connected in series.

[0039] As an optional implementation of this application, the step of obtaining the audio signal of at least one sound source based on the spectral mask of the at least one sound source and the time-frequency domain signal includes:

[0040] Calculate the product of the time-frequency domain signal and the spectral mask of each sound source to obtain the time-frequency domain signal of each sound source;

[0041] The time-frequency domain signals of each sound source are transformed into time-domain signals to obtain the audio signals of each sound source.

[0042] As an optional implementation of this application, the step of transforming the time-frequency domain signals of each sound source into time-domain signals includes:

[0043] Inverse short-time Fourier transform is performed on the time-frequency domain signals of each sound source to transform the time-frequency domain signals of each sound source into time-domain signals.

[0044] As an optional implementation of this application, the audio separation method is based on a sound source separation model, which includes:

[0045] The transformation module is used to perform the step of transforming the audio signal to be separated from a time-domain signal to a time-frequency domain signal;

[0046] The segmentation module is used to perform the step of segmenting the time-frequency domain signal into multiple sub-band signals.

[0047] The acquisition module is used to perform the steps of acquiring the spectral characteristics of the plurality of sub-band signals respectively;

[0048] A mask estimation module is used to perform the step of obtaining a spectral mask of at least one sound source of the audio signal to be separated based on the spectral characteristics of the plurality of sub-band signals;

[0049] An output module is configured to perform the step of acquiring an audio signal of at least one sound source based on the spectral mask of the at least one sound source and the time-frequency domain signal.

[0050] As an optional implementation of this application, before implementing the audio separation method based on the sound source separation model, the method further includes:

[0051] Obtain a training data set, which includes: multiple sets of training data, each set of training data including: sample audio signals and reference audio signals of at least one sound source corresponding to the sample audio signals;

[0052] The sound source separation model is trained based on the training dataset.

[0053] As an optional implementation of this application, training the sound source separation model based on the training dataset includes:

[0054] The sample audio signal is input into the sound source separation model, and the predicted audio signal of at least one sound source corresponding to the sample audio signal and the time-frequency domain signal of the predicted audio signal of at least one sound source corresponding to the sample audio signal are obtained from the output of the sound source separation model.

[0055] Obtain the time-frequency domain signal of the reference audio signal of at least one sound source corresponding to the sample audio signal;

[0056] The loss value corresponding to the sample audio signal is calculated based on the reference audio signal of at least one sound source corresponding to the sample audio signal, the predicted audio signal of at least one sound source corresponding to the sample audio signal, the time-frequency domain signal of the predicted audio signal of at least one sound source corresponding to the sample audio signal, and the time-frequency domain signal of the reference audio signal of at least one sound source corresponding to the sample audio signal.

[0057] The model parameters of the sound source separation model are updated based on the loss value.

[0058] Secondly, this application provides a sound source separation device, comprising:

[0059] The transformation unit is used to transform the audio signal to be separated from a time-domain signal to a time-frequency domain signal;

[0060] A segmentation unit is used to perform frequency band segmentation on the time-frequency domain signal to divide the time-frequency domain signal into multiple sub-band signals, wherein the frequency bands of the multiple sub-band signals do not overlap.

[0061] An acquisition unit is used to acquire the spectral characteristics of the plurality of sub-band signals respectively;

[0062] The processing unit is used to obtain the spectral mask of at least one sound source of the audio signal to be separated based on the spectral characteristics of the plurality of sub-band signals.

[0063] An output unit is used to acquire the audio signal of at least one sound source based on the spectral mask of the at least one sound source and the time-frequency domain signal.

[0064] As an optional implementation of this application, the transformation unit is specifically used to perform a short-time Fourier transform on the audio signal to be separated, so as to transform the audio signal to be separated from a time-domain signal into a time-frequency domain signal.

[0065] As an optional implementation of this application, the segmentation unit is specifically used to perform frequency band segmentation on the first frequency band of the time-frequency domain signal based on a first frequency band interval, and to perform frequency band segmentation on the second frequency band of the time-frequency domain signal based on a second frequency band interval;

[0066] Wherein, the maximum frequency of the first frequency band is less than the minimum frequency of the second frequency band, and the first frequency band spacing is less than the second frequency band spacing.

[0067] As an optional implementation of this application, the acquisition unit is specifically used to extract features from the plurality of sub-band signals respectively to obtain sub-band features of each sub-band signal; stack the sub-band features of each sub-band signal to obtain stacked features; and obtain the spectral features of each sub-band signal based on the stacked features.

[0068] As an optional implementation of this application, the acquisition unit is specifically used to extract features from each sub-band signal through a multilayer perceptron corresponding to each sub-band signal, so as to obtain the sub-band features of each sub-band signal.

[0069] Each subband signal corresponds to a multilayer perceptron consisting of a root mean square normalization layer and a linear layer connected in series.

[0070] As an optional implementation of this application, the acquisition unit is specifically used to model the sub-band features of the stacked features in the time dimension using a local feature extraction module to obtain local features composed of the temporal features of each sub-band signal; to transpose the local features to obtain a first transposed feature; to model the temporal features of the first transposed feature in the feature stack dimension using a global feature extraction module to obtain global features composed of the frequency band features of each sub-band signal; to transpose the global features to obtain a second transposed feature; to fuse the second transposed feature using a multi-head self-attention mechanism to obtain a fused feature; and to split the fused feature to obtain the spectral features of each sub-band signal.

[0071] As an optional implementation of this application, the local feature extraction module and / or the global feature extraction module are dual-path Mamba modules; the dual-path Mamba module includes:

[0072] The first path includes: a first Mamba block, a first root mean square normalization layer, and a first adder. The input of the first Mamba block is the input of the dual-path Mamba module. The input of the first root mean square normalization layer is the output of the first Mamba block. The input of the first adder is the output of the first Mamba block and the output of the first root mean square normalization layer. The output of the first adder is the output of the first path.

[0073] The second path includes: a flip layer, a second Mamba block, a second root mean square normalization layer, and a second adder. The flip layer is used to flip the input of the dual-path Mamba module. The input of the second Mamba block is the output of the flip layer. The input of the second root mean square normalization layer is the output of the second Mamba block. The input of the second adder is the output of the second Mamba block and the output of the second root mean square normalization layer. The output of the second adder is the output of the second path.

[0074] A splicing layer, used to splice the output of the first path and the output of the second path;

[0075] A linear layer, the input of which is the output of the splicing layer, and the output of which is the output of the dual-path Mamba module.

[0076] As an optional implementation of this application, the processing unit is specifically used to process the spectral features of each sub-band signal for each sound source through the corresponding mask estimation module to obtain the spectral mask of each sub-band signal corresponding to the sound source, and to stitch together the spectral masks of each sub-band signal corresponding to the sound source to obtain the spectral mask of the sound source.

[0077] Each sub-band's corresponding mask estimation module consists of a multilayer perceptron and a gated linear unit connected in series. The multilayer perceptron consists of a root mean square normalization layer and a linear layer connected in series.

[0078] As an optional implementation of this application, the output unit is specifically used to calculate the product of the time-frequency domain signal and the spectral mask of each sound source to obtain the time-frequency domain signal of each sound source; and to transform the time-frequency domain signal of each sound source into a time domain signal to obtain the audio signal of each sound source.

[0079] As an optional implementation of this application, the output unit is specifically used to perform inverse short-time Fourier transform on the time-frequency domain signals of each sound source, so as to transform the time-frequency domain signals of each sound source into time-domain signals.

[0080] As an optional implementation of this application, the device includes a sound source separation model, the sound source separation model comprising:

[0081] The transformation module is used to transform the audio signal to be separated from a time-domain signal to a time-frequency domain signal;

[0082] The segmentation module is used to perform frequency band segmentation on the time-frequency domain signal;

[0083] The acquisition module is used to acquire the spectral characteristics of the multiple sub-band signals respectively;

[0084] The mask estimation module is used to obtain the spectral mask of at least one sound source of the audio signal to be separated based on the spectral characteristics of the plurality of sub-band signals.

[0085] The output module is used to acquire the audio signal of at least one sound source based on the spectral mask of the at least one sound source and the time-frequency domain signal.

[0086] As an optional implementation of this application, the acquisition unit is further configured to, before implementing the audio separation method based on the audio source separation model, further include: acquiring a training data set, the training data set including: multiple sets of training data, any set of training data including: sample audio signals and reference audio signals of at least one audio source corresponding to the sample audio signals; and training the audio source separation model based on the training data set.

[0087] As an optional implementation of this application, the acquisition unit is specifically used to input the sample audio signal into the sound source separation model, and acquire the predicted audio signal of at least one sound source corresponding to the sample audio signal and the time-frequency domain signal of the predicted audio signal of at least one sound source corresponding to the sample audio signal output by the sound source separation model; acquire the time-frequency domain signal of the reference audio signal of at least one sound source corresponding to the sample audio signal; calculate the loss value corresponding to the sample audio signal based on the reference audio signal of at least one sound source corresponding to the sample audio signal, the predicted audio signal of at least one sound source corresponding to the sample audio signal, the time-frequency domain signal of the predicted audio signal of at least one sound source corresponding to the sample audio signal, and the time-frequency domain signal of the reference audio signal of at least one sound source corresponding to the sample audio signal; and update the model parameters of the sound source separation model based on the loss value.

[0088] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program and the processor is used to cause the electronic device to implement the sound source separation method described in any of the above embodiments when executing the computer program.

[0089] Fourthly, embodiments of this application provide a computer-readable storage medium that, when executed by a computing device, causes the computing device to implement the audio source separation method described in any of the above embodiments.

[0090] Fifthly, embodiments of this application provide a computer program product that, when run on a computer, enables the computer to implement the audio source separation method described in any of the above embodiments.

[0091] The audio source separation method provided in this application first transforms the audio signal to be separated from a time-domain signal to a time-frequency domain signal using an audio source separation model. Then, it performs frequency band segmentation on the time-frequency domain signal to divide it into multiple non-overlapping sub-band signals. Next, it obtains the spectral characteristics of each sub-band signal. Based on the spectral characteristics of each sub-band signal, it obtains the spectral mask of at least one audio source of the audio signal to be separated, and obtains the audio signal of at least one audio source based on the spectral mask of the at least one audio source and the time-frequency domain signal. Because the audio source separation method provided in this application segments the time-frequency domain signal of the audio signal to be separated into multiple non-overlapping sub-band signals and then obtains the spectral characteristics of each sub-band signal, this application allows the audio source separation model to obtain the spectral characteristics of the frequency bands more precisely, thus improving the robustness of the audio source separation algorithm. Attached Figure Description

[0092] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0093] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings that need to be called in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0094] Figure 1 This is one of the flowcharts of the sound source separation method provided in the embodiments of this application;

[0095] Figure 2 This is a schematic diagram of the sound source separation model provided in the embodiments of this application;

[0096] Figure 3 This is a schematic diagram of the feature extraction module of the sound source separation model provided in the embodiments of this application;

[0097] Figure 4 This is a schematic diagram of the mask estimation module of the sound source separation model provided in the embodiments of this application;

[0098] Figure 5 This is a schematic diagram of the output module of the sound source separation model provided in the embodiments of this application;

[0099] Figure 6 The second flowchart of the sound source separation method provided in the embodiments of this application;

[0100] Figure 7 This is a schematic diagram of the structure of the dual-path Mamba module provided in the embodiments of this application;

[0101] Figure 8 This is a schematic diagram of the structure of a Mamba block provided in an embodiment of this application;

[0102] Figure 9 This is a schematic diagram of the feature processing module of the sound source separation model provided in the embodiments of this application;

[0103] Figure 10 This is a schematic diagram of the sound source separation device provided in the embodiments of this application;

[0104] Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0105] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0106] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein. Obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0107] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Furthermore, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0108] This application provides a method for separating audio sources. The subject executing this method can be an electronic device such as a mobile phone, personal computer, handheld computer, vehicle-mounted device, or server, or an audio source separation device integrated into an electronic device.

[0109] Reference Figure 1 As shown, the sound source separation method includes the following steps S11 to S15:

[0110] S11. Transform the audio signal to be separated from the time domain signal into the time-frequency domain signal.

[0111] The audio signal to be separated in this application embodiment can be a two-channel multitrack mixed audio signal, a mono multitrack mixed audio signal, or a multichannel multitrack mixed audio signal. Here, a channel refers to an independent path for sound during recording or playback; it is the path used by the audio system to transmit and process sound information. Each channel can carry an independent audio signal, which may differ in spatial location, timbre, volume, etc. For example, in common stereo audio, there are two channels: a left channel and a right channel. These two channels work together to create a sense of space and stereo effect. A track refers to a channel used to record, edit, and process a single audio element or a group of related audio elements. An important function of a sound track is to separate different audio elements so that they can be edited and processed individually. For example, the audio signal to be separated is music.

[0112] Time-domain signals are a way of representing how a signal changes over time. Audio signals are signals whose frequency, amplitude, and phase change over time; therefore, audio signals are essentially time-domain signals. Time-frequency domain signals are a way of describing signals that considers both time and frequency dimensions simultaneously, and can show the frequency components of a signal at different times and how they change.

[0113] In some embodiments, the audio signal to be separated can be transformed from a time-domain signal to a time-frequency domain signal by performing a Short-Time Fourier Transform (STFT) or a Wavelet Transform (WT).

[0114] In some embodiments, the audio separation method is implemented based on a sound source separation model. (See reference...) Figure 2 As shown, the audio source separation model includes: a transformation module 21, which is used to transform the audio signal x to be separated from a time-domain signal to a time-frequency domain signal X∈R. C×F×T The steps.

[0115] S12. Perform frequency band segmentation on the time-frequency domain signal to divide the time-frequency domain signal into multiple sub-band signals.

[0116] The frequency bands of the multiple sub-band signals do not overlap.

[0117] That is, the time-frequency domain signal is segmented from the frequency dimension, and the segmented signal is defined as a sub-band signal.

[0118] In some embodiments, the step of performing frequency band segmentation on the time-frequency domain signal to divide the time-frequency domain signal into multiple sub-band signals includes:

[0119] The first frequency band of the time-frequency domain signal is divided into frequency bands based on a first frequency band interval, and the second frequency band of the time-frequency domain signal is divided into frequency bands based on a second frequency band interval;

[0120] Wherein, the maximum frequency of the first frequency band is less than the minimum frequency of the second frequency band, and the first frequency band spacing is less than the second frequency band spacing.

[0121] That is, a smaller frequency band spacing is used for frequency band division in the low frequency band, while a larger frequency band spacing is used for frequency band division in the high frequency band.

[0122] Since the human ear is more sensitive to low frequencies, using smaller frequency band intervals for low frequencies and larger frequency band intervals for high frequencies can ensure the quality of audio data processing while avoiding excessive data processing.

[0123] Reference Figure 2 As shown, in some embodiments, the sound source separation model further includes a segmentation module 22. The segmentation module 22 is used to segment the frequency domain signal X∈R... C×F×T It is divided into multiple sub-band signals X(1), X(2)...X(N).

[0124] S13. Obtain the spectral characteristics of the multiple sub-band signals respectively.

[0125] In some embodiments, the audio separation method is implemented based on a sound source separation model. (See reference...) Figure 2As shown, the sound source separation model includes an acquisition module 23. The acquisition module 23 is used to perform the steps of acquiring the spectral characteristics f(1), f(2), ... f(N) of the plurality of sub-band signals X(1), X(2) ... X(N) respectively.

[0126] In some embodiments, the above step S13 (obtaining the spectral characteristics of the plurality of sub-band signals respectively) is implemented by including the following steps a to c:

[0127] Step a: Extract features from the multiple sub-band signals to obtain the sub-band features of each sub-band signal.

[0128] In some embodiments, the sound source separation model includes a feature extraction module corresponding to each sub-band signal, which performs feature extraction on the plurality of sub-band signals to obtain sub-band features of each sub-band signal, including: inputting each sub-band signal into the corresponding feature extraction module and obtaining the sub-band features of each sub-band signal output by the feature extraction module corresponding to each sub-band signal.

[0129] In some embodiments, the feature extraction module corresponding to each sub-band signal is a Multi-Layer Perceptron (MLP). That is, feature extraction is performed on the multiple sub-band signals to obtain the sub-band features of each sub-band signal, including: extracting features from each sub-band signal using the corresponding MLP to obtain the sub-band features of each sub-band signal. Each MLP corresponding to a sub-band signal consists of a root-mean-square normalization layer (RMSNorm) and a linear layer connected in series. The root-mean-square normalization layer is a model structure that performs normalization operations based on the root-mean-square value of the input data, used to normalize the features of the input data to a specific range, making the data distribution more reasonable and facilitating subsequent processing and analysis. The linear layer, also known as a fully-connected layer, is a basic layer structure in neural networks. Its main function is to map the input vector to the output vector through a set of learnable weights and biases, thereby performing a linear transformation on the input data.

[0130] Reference Figure 2 As shown, in some embodiments, the acquisition module 23 includes a feature extraction module 231. The feature extraction module 231 is used to extract features from the plurality of sub-band signals respectively to obtain the sub-band features F(1), F(2)...F(N) of each sub-band signal.

[0131] Reference Figure 3As shown, in some embodiments, the feature extraction module 231 includes a feature extraction unit 300 corresponding to each sub-band signal. That is, the feature extraction module 300 includes N feature extraction units 300, where N is the number of sub-band signals obtained by frequency band segmentation of the time-frequency domain signal. Each feature extraction unit 300 consists of a root mean square normalization layer 301 and a linear layer 302 connected in series.

[0132] Step b: Stack the sub-band features of each sub-band signal to obtain the stacked features.

[0133] In some embodiments, stacking the sub-band features of each sub-band signal to obtain stacked features includes: processing the sub-band features of the sub-band signal into features of the same size, and then stacking the sub-band features of each sub-band signal in ascending order of frequency.

[0134] For example, if the sub-band features of a sub-band signal are processed to a size of D×T, then the size of the stacked features obtained by stacking the sub-band features of each sub-band signal is N×D×T. Here, D is the number of feature points, T is the number of audio frames in the audio signal to be separated, and N is the number of sub-band signals obtained by frequency band segmentation of the time-frequency domain signal.

[0135] Reference Figure 2 As shown, in some embodiments, the acquisition module 23 further includes a stacking module 232. The stacking module 232 is used to stack the sub-band features of each sub-band signal to acquire the stacked features F∈R. N×D×T .

[0136] Step c: Obtain the spectral characteristics of each sub-band signal based on the stacking characteristics.

[0137] Spectral features refer to parameters or attributes extracted from the spectrum (frequency domain representation) of a signal to describe its frequency characteristics. Spectral features can include important information such as the distribution of the signal across different frequency components and the degree of energy concentration.

[0138] In some embodiments, the stacked features can be input into the state space model to obtain the spectral features of each sub-band signal output by the state space model.

[0139] Reference Figure 2 As shown, in some embodiments, the acquisition module 23 further includes a feature processing module 233. The feature processing module is used to process the stacked features F∈R. N×D×T Processing is performed to obtain the spectral characteristics f(1), f(2)...f(N) of each sub-band signal.

[0140] S14. Obtain the spectral mask of at least one sound source of the audio signal to be separated based on the spectral characteristics of each sub-band signal.

[0141] In some embodiments, the audio signal to be separated is music, and at least one sound source may include: human voice and musical instruments such as bass, drums, and guitar.

[0142] In some embodiments, the audio source separation model includes a mask estimation module corresponding to each sub-band signal. This module obtains the spectral mask of at least one audio source of the audio signal to be separated based on the spectral characteristics of each sub-band signal. The process includes: processing the spectral characteristics of each sub-band signal using the corresponding mask estimation module for each audio source to obtain the spectral mask of each sub-band signal corresponding to the audio source; and concatenating the spectral masks of each sub-band signal corresponding to the audio source to obtain the spectral mask of the audio source. Each sub-band mask estimation module consists of an MLP and a gated linear unit (GLU). The MLP is composed of a root mean square normalization layer and a linear layer connected in series.

[0143] In some embodiments, the audio separation method is implemented based on a sound source separation model. (See reference...) Figure 2 As shown, the audio source separation model further includes a mask estimation module 24. The mask estimation module 24 is used to obtain the spectral mask of at least one audio source of the audio signal to be separated based on the spectral characteristics of each sub-band signal. n represents the number of sound sources.

[0144] Reference Figure 4 As shown, in some embodiments, the mask estimation module 24 includes a mask estimation structure 240 corresponding to each sound source. The mask estimation structure 240 corresponding to any sound source includes N mask estimation units 41 and a splicing layer 42. Each mask estimation unit 41 consists of an MLP 411 and a GLU 412 connected in series. The splicing layer 42 is used to splice the spectral masks of each sub-band output by the N mask estimation units 41 to obtain the spectral mask of the sound source. Here, N is the number of sub-band signals obtained by frequency band segmentation of the time-frequency domain signal.

[0145] S15. Obtain the audio signal of at least one sound source based on the spectral mask of the at least one sound source and the time-frequency domain signal.

[0146] Reference Figure 2 As shown, in some embodiments, the audio source separation model further includes an output module 25. The output module 25 is used to obtain the audio signal of at least one audio source based on the spectral mask of the at least one audio source and the time-frequency domain signal.

[0147] In some embodiments, obtaining the audio signal of at least one sound source based on the spectral mask of the at least one sound source and the time-frequency domain signal includes: calculating the product of the time-frequency domain signal and the spectral mask of each sound source to obtain the time-frequency domain signal of each sound source; and transforming the time-frequency domain signal of each sound source into a time-domain signal to obtain the audio signal of each sound source.

[0148] Reference Figure 5 As shown, in some embodiments, the output module 25 includes a multiplier 501 and an inverse transform module 502. The multiplier 501 is used to calculate the product of the time-frequency domain signal and the spectral mask of the sound source, so as to output the time-frequency domain signal of the sound source. The inverse transformation module 502 is used to inversely transform the time-frequency domain signal of the sound source into a time-domain signal to obtain the audio signal of each sound source.

[0149] The audio source separation method provided in this application first transforms the audio signal to be separated from a time-domain signal to a time-frequency domain signal using an audio source separation model. Then, it performs frequency band segmentation on the time-frequency domain signal to divide it into multiple non-overlapping sub-band signals. Next, it obtains the spectral characteristics of each sub-band signal. Based on the spectral characteristics of each sub-band signal, it obtains the spectral mask of at least one audio source of the audio signal to be separated, and obtains the audio signal of at least one audio source based on the spectral mask of the at least one audio source and the time-frequency domain signal. Because the audio source separation method provided in this application segments the time-frequency domain signal of the audio signal to be separated into multiple non-overlapping sub-band signals and then obtains the spectral characteristics of each sub-band signal, this application allows the audio source separation model to obtain the spectral characteristics of the frequency bands more precisely, thus improving the robustness of the audio source separation algorithm.

[0150] As an extension and refinement of the above embodiments, this application also provides another sound source separation method, referring to... Figure 6 As shown, the sound source separation method includes the following steps:

[0151] S601. Perform a short-time Fourier transform on the audio signal to be separated to transform the audio signal to be separated from a time-domain signal to a time-frequency domain signal.

[0152] Short-Time Fourier Transform (SFT) is a signal processing technique that performs windowing on a signal. When performing SFT on an audio signal to be separated, a suitable window function is first set for the audio signal. The length of the window function determines the sampling rate of the audio signal to be separated. Then, the window function is slid along the time axis, and a Fourier transform is performed on the audio signal to be separated within each window. This yields the spectrum of the audio signal to be separated in different time segments (determined by the window function), effectively transforming the audio signal from a time-domain signal to a time-frequency-domain signal.

[0153] S602. Perform frequency band segmentation on the time-frequency domain signal to divide the time-frequency domain signal into multiple sub-band signals.

[0154] In some embodiments, frequency band segmentation is performed on the time-frequency domain signal to divide the time-frequency domain signal into multiple sub-band signals, including: selecting a preset number of frequency band segmentation points on the time-frequency domain signal, dividing the time-frequency domain signal into multiple frequency bands based on the selected frequency band segmentation points, and then segmenting each frequency band based on the frequency band interval corresponding to each frequency band to divide the time-frequency domain signal into multiple sub-band signals.

[0155] For example, select 5 frequency band division points on the time-frequency domain signal, and divide the time-frequency domain signal into 6 frequency bands based on the selected 5 frequency band division points, and then divide each frequency band based on the frequency band interval corresponding to each frequency band.

[0156] For example, the five frequency band division points can be 1000kHz, 2000kHz, 4000kHz, 8000kHz, and 16000kHz. The six frequency bands obtained by dividing the time-frequency domain signal based on 1000kHz, 2000kHz, 4000kHz, 8000kHz, and 16000kHz are as follows: the first frequency band less than 1000Hz, the second frequency band [1000Hz, 2000Hz); the third frequency band [2000Hz, 4000Hz); the fourth frequency band [4000Hz, 8000Hz); the fifth frequency band [8000Hz, 16000Hz); and the sixth frequency band greater than or equal to 16000Hz.

[0157] For example, the five frequency band division points can also be 2000kHz, 4000kHz, 6000kHz, 8000kHz, and 10000kHz. The six frequency bands obtained by dividing the time-frequency domain signal based on 2000kHz, 4000kHz, 6000kHz, 8000kHz, and 10000kHz are as follows: a first frequency band less than 2000Hz, a second frequency band [2000Hz, 4000Hz); a third frequency band [4000Hz, 6000Hz); a fourth frequency band [6000Hz, 8000Hz); a fifth frequency band [8000Hz, 10000Hz); and a sixth frequency band greater than or equal to 10000Hz.

[0158] In some embodiments, the frequency band spacing corresponding to each frequency band increases sequentially.

[0159] For example, the first frequency band of the time-frequency domain signal can be uniformly divided into 24 sub-band signals based on the frequency band spacing corresponding to the first frequency band; the second frequency band of the time-frequency domain signal can be uniformly divided into 12 sub-band signals based on the frequency band spacing corresponding to the second frequency band; the third frequency band of the time-frequency domain signal can be uniformly divided into 8 sub-band signals based on the frequency band spacing corresponding to the third frequency band; the fourth frequency band of the time-frequency domain signal can be uniformly divided into 8 sub-band signals based on the frequency band spacing corresponding to the fourth frequency band; the fifth frequency band of the time-frequency domain signal can be uniformly divided into 8 sub-band signals based on the frequency band spacing corresponding to the first frequency band; and the sixth frequency band of the time-frequency domain signal can be uniformly divided into 2 sub-band signals based on the frequency band spacing corresponding to the first frequency band. That is, the time-frequency domain signal can be divided into 62 sub-band signals through the above segmentation scheme.

[0160] S603. Extract features from each sub-band signal using the MLP corresponding to each sub-band signal to obtain the sub-band features of each sub-band signal.

[0161] Each subband signal's corresponding MLP consists of a root mean square normalization layer and a linear layer connected in series.

[0162] S604. Stack the sub-band characteristics of each sub-band signal to obtain the stacked characteristics.

[0163] S605. The local feature extraction module models the features of each sub-band of the stacked features in the time dimension to obtain local features composed of the temporal features of each sub-band signal.

[0164] In some embodiments, the local feature extraction module is a dual-path Mamba module, as shown below. Figure 7 As shown, the dual-path Mamba module includes:

[0165] The first path 71 includes: a first Mamba block 711, a first root mean square normalization layer 712, and a first adder 713. The input of the first Mamba block 711 is the input In of the dual-path Mamba module. The input of the first root mean square normalization layer 712 is the output of the first Mamba block 711. The input of the first adder 713 is the output of the first Mamba block and the output of the first root mean square normalization layer 712. The output of the first adder is the output out1 of the first path.

[0166] The second path 72 includes: a flip layer 721, a second Mamba block 722, a second root mean square normalization layer 723, and a second adder 724. The flip layer 721 is used to flip the input In of the dual-path Mamba module. The input of the second Mamba block 722 is the output of the flip layer 721. The input of the second root mean square normalization layer 723 is the output of the second Mamba block. The input of the second adder 724 is the output of the second Mamba block and the output of the second root mean square normalization layer 723. The output of the second adder 724 is the output out2 of the second path.

[0167] A splicing layer 73 is used to splice the output out1 of the first path and the output out2 of the second path;

[0168] Linear layer 74, the input of which is the output of splicing layer 73, and the output of the linear layer is the output of the dual-path Mamba module.

[0169] The Mamba block is a neural network architecture built around a State-Space Model (SSM) as its core module. In the Mamba block, sequential data is treated as a dynamic system, where each element in the sequence corresponds to the system's input or output at a certain moment.

[0170] Reference Figure 8As shown, in some embodiments, the Mamba block includes: a first linear layer 81, a convolutional layer 82, a first activation function layer 83, a state space model 84, a second linear layer 85, a second activation function layer 86, a multiplier 87, and a third linear layer 88. The first linear layer 81, the convolutional layer 82, the first activation function layer 83, and the state space model 84 are connected in series to process the input of the Mamba block. The second linear layer 85 and the second activation function layer 86 are connected in series to process the input of the Mamba block. The multiplier 87 is used to calculate the product of the output o1 of the state space model 84 and the output o2 of the second activation function layer 86. The third linear layer 88 is used to linearly map the output of the multiplier 87 to obtain the output of the Mamba block.

[0171] For a continuous system, the state-space model 84 uses system input, system output, and state variables to describe it. It maps the system input x(t) to the system output y(t) through the hidden state h(t). The mapping process can be expressed by the following formula:

[0172] h′(t)=Δh(t)+Bx(t)

[0173] y(t)=Ch′(t)

[0174] Where A is the state transition matrix of state-space model 84, B is the mapping matrix from input to state variables of state-space model 84, and C is the mapping matrix from state variables to output of state-space model 84.

[0175] In the Mamba block, the state-space model is input-selective, meaning that A, B, and C are functions of the system input x(t). A, B, and C are updated based on the input at each time step, thereby enabling the state-space model to selectively propagate or forget information based on the system input, thus enhancing the model's expressive power.

[0176] In the above embodiments, the local feature extraction module is a dual-path Mamba module. Therefore, the above embodiments can make full use of historical and future information to improve the accuracy of sound source separation.

[0177] S606. Perform a transpose operation on the local feature to obtain a first transpose feature.

[0178] For example, if the feature tensor of the local feature is N×D×T, then the feature tensor of the first transposed feature is T×d×N.

[0179] S607. The global feature extraction module models each temporal feature of the first transposed feature in the feature stacking dimension to obtain a global feature composed of the frequency band features of each sub-band signal.

[0180] In some embodiments, the global feature extraction module is a dual-path Mamba module. The structure and working principle of the dual-path Mamba module can be referred to the local feature extraction module, and will not be repeated here to avoid redundancy.

[0181] S608. Perform a transpose operation on the global feature to obtain a second transpose feature.

[0182] For example, if the feature tensor of the global feature is T×D×N, then the feature tensor of the second transpose feature is N×D×T.

[0183] S609. The second transposed feature is fused using a multi-head self-attention (HMSA) mechanism to obtain fused features.

[0184] Multi-head self-attention is a type of self-attention mechanism that works as follows: In the self-attention calculation of each head, an attention score is obtained by calculating the dot product of the query vector and the key vector. This score reflects the importance of each element to the current element, and the value vector is the object of weighted summation based on the attention scores. Multiple heads work in parallel, and each head may focus on different aspects. The outputs of these different heads are integrated through concatenation or other fusion methods, so that the final output contains information from multiple perspectives, thus more comprehensively representing the input sequence.

[0185] S610. The fusion feature is split to obtain the spectral features of each sub-band signal.

[0186] Reference Figure 9 As shown, the feature processing modules used to implement steps S605 to S610 in the sound source separation model include: a local feature extraction module 91, a first transpose module 92, a global feature extraction module 93, a second transpose module 94, a multi-head autonomous capability module 95, and a splitting module 96. The local feature extraction module 91 is used to model the features of each sub-band of the stacked feature F in the time dimension to obtain local features F composed of the temporal features of each sub-band signal. intra The first transpose module 92 is used for local features F intra Perform a transpose operation to obtain the first transpose feature F tran1 The global feature extraction module 93 is used to extract the first transposed feature F along the feature stacking dimension. tran1Modeling the various time-series characteristics, we obtain the global feature F composed of the frequency band characteristics of each sub-band signal. inter The second transpose module 94 is used to process the global feature F. inter Perform a transpose operation to obtain the second transpose feature F tran2 The multi-head autonomy module 95 is used for the second transposed feature F based on the multi-head self-attention mechanism. tran2 Process to obtain fusion features F fusi The splitting module 96 is used to split the fused feature F. fusi To obtain the spectral characteristics f(1), f(2)...f(N) of the subband signal.

[0187] S611. For each sound source, the spectral characteristics of each sub-band signal are processed by the corresponding mask estimation module to obtain the spectral mask of each sub-band signal corresponding to the sound source, and the spectral masks of each sub-band signal corresponding to the sound source are spliced ​​together to obtain the spectral mask of the sound source.

[0188] Each sub-band's corresponding mask estimation module consists of an MLP and a GLU connected in series. The MLP consists of a root mean square normalization layer and a linear layer connected in series.

[0189] S612. Calculate the product of the time-frequency domain signal and the spectral mask of each sound source to obtain the time-frequency domain signal of each sound source.

[0190] S613. Transform the time-frequency domain signals of each sound source into time-domain signals respectively to obtain the audio signals of each sound source.

[0191] In some embodiments, the time-frequency domain signals of each sound source are transformed into time-domain signals to obtain the audio signals of each sound source, including: performing Inverse Short-Time Fourier Transform (ISTFT) on the time-frequency domain signals of each sound source to transform the time-frequency domain signals of each sound source into time-domain signals.

[0192] The audio source separation method provided in this application further includes training an audio source separation model before transforming the audio signal to be separated from a time-domain signal to a time-frequency domain signal (i.e., before starting audio source separation of the audio signal to be separated). In some embodiments, training the audio source separation model includes: acquiring a training data set, and training the audio source separation model based on the training data set. The training data set includes multiple sets of training data, each set of training data including: a sample audio signal and a reference audio signal of at least one audio source corresponding to the sample audio signal.

[0193] In some embodiments, training the sound source separation model based on the training dataset includes the following steps a and d:

[0194] Step a: Input the sample audio signal into the sound source separation model, and obtain the predicted audio signal of at least one sound source corresponding to the sample audio signal output by the sound source separation model.

[0195] That is, firstly, the audio source separation module performs steps such as time-domain signal to time-frequency domain signal transformation and frequency band segmentation to obtain the spectral mask of at least one audio source. Then, the spectral mask of the audio source is applied to the time-frequency domain signal of the sample audio to obtain the audio time-frequency domain signal of the audio source. Finally, the inverse short-time Fourier transform is performed on the time-frequency domain signal of the audio source to obtain the predicted audio signal of the audio source.

[0196] The implementation process of step a can also be summarized by the following formula:

[0197]

[0198] Where X(t) is the time-frequency domain signal of the sample audio. Let be the spectral mask of the sound source t. Given the time-frequency domain signal of the sound source t, ISTFT() is the short-time Fourier transform operation. Let be the predicted audio signal for sound source t.

[0199] Step b: Obtain the time-frequency domain signal of the reference audio signal of at least one sound source corresponding to the sample audio signal.

[0200] In some embodiments, a short-time Fourier transform can be performed on the reference audio signals of each sound source corresponding to the sample audio signal to obtain the time-frequency domain signal of the reference audio signal of at least one sound source corresponding to the sample audio signal.

[0201] Step c: Calculate the loss value corresponding to the sample audio signal based on the reference audio signal of at least one sound source corresponding to the sample audio signal, the predicted audio signal of at least one sound source corresponding to the sample audio signal, the time-frequency domain signal of the predicted audio signal of at least one sound source corresponding to the sample audio signal, and the time-frequency domain signal of the reference audio signal of at least one sound source corresponding to the sample audio signal.

[0202] In some embodiments, the loss function used to calculate the loss value corresponding to the sample audio signal is:

[0203]

[0204] L ALL =L MAE +L STFT

[0205] Among them, s and These are the reference audio signal and the predicted audio signal of the sound source, S and... These are the time-frequency domain signals of the reference audio signal and the predicted audio signal, respectively, where t and f are the time and frequency indices, respectively; |S(t,f)| represents the modulus value of the reference audio signal at the frequency point, i.e., the amplitude value of the reference audio signal. p is a hyperparameter, which can be 0.3; that is, the amplitude spectrum value is raised to the power of 0.3; e is the natural logarithm base, j is the imaginary sign, and ψ is the complex number operator used to calculate the angular frequency; e jψS(t,f) Indicates phase.

[0206] Step d: Update the model parameters of the sound source separation model based on the loss value.

[0207] In some embodiments, backpropagation can be performed based on the loss value to update the model parameters of the sound source separation model.

[0208] Based on the same inventive concept, as an implementation of the above method, this application embodiment also provides a sound source separation device. This embodiment corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the sound source separation device in this embodiment can correspondingly implement all the contents of the aforementioned method embodiment.

[0209] This application provides a sound source separation device. Figure 10 This is a schematic diagram of the sound source separation device, as shown below. Figure 10 As shown, the sound source separation device 100 includes:

[0210] The transformation unit 101 is used to transform the audio signal to be separated from a time-domain signal into a time-frequency domain signal;

[0211] The segmentation unit 102 is used to perform frequency band segmentation on the time-frequency domain signal to divide the time-frequency domain signal into multiple sub-band signals, wherein the frequency bands of the multiple sub-band signals do not overlap.

[0212] The acquisition unit 103 is used to acquire the spectral characteristics of the plurality of sub-band signals respectively;

[0213] Processing unit 104 is used to obtain a spectral mask of at least one sound source of the audio signal to be separated based on the spectral characteristics of the plurality of sub-band signals.

[0214] The output unit 105 is used to acquire the audio signal of at least one sound source based on the spectral mask of the at least one sound source and the time-frequency domain signal.

[0215] As an optional implementation of this application, the transformation unit 101 is specifically used to perform a short-time Fourier transform on the audio signal to be separated, so as to transform the audio signal to be separated from a time-domain signal into a time-frequency domain signal.

[0216] As an optional implementation of this application, the segmentation unit 102 is specifically used to perform frequency band segmentation on the first frequency band of the time-frequency domain signal based on the first frequency band interval, and to perform frequency band segmentation on the second frequency band of the time-frequency domain signal based on the second frequency band interval;

[0217] Wherein, the maximum frequency of the first frequency band is less than the minimum frequency of the second frequency band, and the first frequency band spacing is less than the second frequency band spacing.

[0218] As an optional implementation of this application, the segmentation unit 102 is specifically used to uniformly divide the first frequency band of the time-frequency domain signal into 24 sub-band signals; uniformly divide the second frequency band of the time-frequency domain signal into 12 sub-band signals; uniformly divide the third frequency band of the time-frequency domain signal into 8 sub-band signals; uniformly divide the fourth frequency band of the time-frequency domain signal into 8 sub-band signals; uniformly divide the fifth frequency band of the time-frequency domain signal into 8 sub-band signals; and uniformly divide the sixth frequency band of the time-frequency domain signal into 2 sub-band signals.

[0219] The first frequency band, the second frequency band, the third frequency band, the fourth frequency band, the fifth frequency band, and the sixth frequency band do not overlap and their combination constitutes the frequency domain range of the time-frequency domain signal.

[0220] As an optional implementation of this application, the acquisition unit 103 is specifically used to extract features from the plurality of sub-band signals respectively to obtain sub-band features of each sub-band signal; stack the sub-band features of each sub-band signal to obtain stacked features; and obtain the spectral features of each sub-band signal based on the stacked features.

[0221] As an optional implementation of this application, the acquisition unit 103 is specifically used to extract features from each sub-band signal through the multilayer perceptron corresponding to each sub-band signal, so as to obtain the sub-band features of each sub-band signal.

[0222] Each subband signal corresponds to a multilayer perceptron consisting of a root mean square normalization layer and a linear layer connected in series.

[0223] As an optional implementation of this application, the acquisition unit 103 is specifically used to model the sub-band features of the stacked features in the time dimension using a local feature extraction module to obtain local features composed of the temporal features of each sub-band signal; to transpose the local features to obtain a first transposed feature; to model the temporal features of the first transposed feature in the feature stack dimension using a global feature extraction module to obtain global features composed of the frequency band features of each sub-band signal; to transpose the global features to obtain a second transposed feature; to fuse the second transposed feature using a multi-head self-attention mechanism to obtain a fused feature; and to split the fused feature to obtain the spectral features of each sub-band signal.

[0224] As an optional implementation of this application, the local feature extraction module and / or the global feature extraction module are dual-path Mamba modules; the dual-path Mamba module includes:

[0225] The first path includes: a first Mamba block, a first root mean square normalization layer, and a first adder. The input of the first Mamba block is the input of the dual-path Mamba module. The input of the first root mean square normalization layer is the output of the first Mamba block. The input of the first adder is the output of the first Mamba block and the output of the first root mean square normalization layer. The output of the first adder is the output of the first path.

[0226] The second path includes: a flip layer, a second Mamba block, a second root mean square normalization layer, and a second adder. The flip layer is used to flip the input of the dual-path Mamba module. The input of the second Mamba block is the output of the flip layer. The input of the second root mean square normalization layer is the output of the second Mamba block. The input of the second adder is the output of the second Mamba block and the output of the second root mean square normalization layer. The output of the second adder is the output of the second path.

[0227] A splicing layer, used to splice the output of the first path and the output of the second path;

[0228] A linear layer, the input of which is the output of the splicing layer, and the output of which is the output of the dual-path Mamba module.

[0229] As an optional implementation of this application, the processing unit 104 is specifically used to process the spectral features of each sub-band signal for each sound source through the corresponding mask estimation module to obtain the spectral mask of each sub-band signal corresponding to the sound source, and to stitch together the spectral masks of each sub-band signal corresponding to the sound source to obtain the spectral mask of the sound source.

[0230] Each sub-band's corresponding mask estimation module consists of a multilayer perceptron and a gated linear unit connected in series. The multilayer perceptron consists of a root mean square normalization layer and a linear layer connected in series.

[0231] As an optional implementation of this application, the output unit 105 is specifically used to calculate the product of the time-frequency domain signal and the spectral mask of each sound source to obtain the time-frequency domain signal of each sound source; and to transform the time-frequency domain signal of each sound source into a time domain signal to obtain the audio signal of each sound source.

[0232] As an optional implementation of this application, the output unit 105 is specifically used to perform inverse short-time Fourier transform on the time-frequency domain signals of each sound source, so as to transform the time-frequency domain signals of each sound source into time-domain signals.

[0233] As an optional implementation of this application, the device includes a sound source separation model, the sound source separation model comprising:

[0234] The transformation module is used to transform the audio signal to be separated from a time-domain signal to a time-frequency domain signal;

[0235] The segmentation module is used to perform frequency band segmentation on the time-frequency domain signal;

[0236] The acquisition module is used to acquire the spectral characteristics of the multiple sub-band signals respectively;

[0237] The mask estimation module is used to obtain the spectral mask of at least one sound source of the audio signal to be separated based on the spectral characteristics of the plurality of sub-band signals.

[0238] The output module is used to acquire the audio signal of at least one sound source based on the spectral mask of the at least one sound source and the time-frequency domain signal.

[0239] As an optional implementation of this application, the acquisition unit 104 is further configured to, before implementing the audio separation method based on the audio source separation model, further include: acquiring a training data set, the training data set including: multiple sets of training data, any set of training data including: sample audio signals and reference audio signals of at least one audio source corresponding to the sample audio signals; and training the audio source separation model based on the training data set.

[0240] As an optional implementation of this application, the acquisition unit 104 is specifically used to input the sample audio signal into the sound source separation model, and acquire the predicted audio signal of at least one sound source corresponding to the sample audio signal and the time-frequency domain signal of the predicted audio signal of at least one sound source corresponding to the sample audio signal output by the sound source separation model; acquire the time-frequency domain signal of the reference audio signal of at least one sound source corresponding to the sample audio signal; calculate the loss value corresponding to the sample audio signal based on the reference audio signal of at least one sound source corresponding to the sample audio signal, the predicted audio signal of at least one sound source corresponding to the sample audio signal, the time-frequency domain signal of the predicted audio signal of at least one sound source corresponding to the sample audio signal, and the time-frequency domain signal of the reference audio signal of at least one sound source corresponding to the sample audio signal; and update the model parameters of the sound source separation model based on the loss value.

[0241] The audio source separation device provided in this application embodiment can execute the audio source separation method provided in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0242] Based on the same inventive concept, embodiments of this application also provide an electronic device. Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 11 As shown, the electronic device provided in this embodiment includes a memory 111 and a processor 112. The memory 111 is used to store a computer program, and the processor 112 is used to execute the audio source separation method provided in the above embodiment when executing the computer program.

[0243] Based on the same inventive concept, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the computing device to implement the sound source separation method provided in the above embodiments.

[0244] Based on the same inventive concept, this application also provides a computer program product that, when run on a computer, enables the computing device to implement the sound source separation method provided in the above embodiments.

[0245] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0246] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0247] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0248] Computer-readable media include both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0249] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An audio separation method, characterized in that, include: The audio signal to be separated is transformed from a time-domain signal to a time-frequency domain signal; The time-frequency domain signal is divided into multiple sub-band signals, and the frequency bands of the multiple sub-band signals do not overlap. The spectral characteristics of the multiple sub-band signals are obtained respectively; Based on the spectral characteristics of the multiple sub-band signals, obtain the spectral mask of at least one sound source of the audio signal to be separated; The audio signal of at least one sound source is obtained based on the spectral mask of the at least one sound source and the time-frequency domain signal.

2. The method according to claim 1, characterized in that, The step of transforming the audio signal to be separated from a time-domain signal to a time-frequency domain signal includes: The audio signal to be separated is subjected to a short-time Fourier transform (STFT) to transform it from a time-domain signal to a time-frequency domain signal.

3. The method according to claim 1, characterized in that, The step of performing frequency band segmentation on the time-frequency domain signal to divide the time-frequency domain signal into multiple sub-band signals includes: The first frequency band of the time-frequency domain signal is divided into frequency bands based on a first frequency band interval, and the second frequency band of the time-frequency domain signal is divided into frequency bands based on a second frequency band interval; Wherein, the maximum frequency of the first frequency band is less than the minimum frequency of the second frequency band, and the first frequency band spacing is less than the second frequency band spacing.

4. The method according to claim 1, characterized in that, The step of acquiring the spectral characteristics of the plurality of sub-band signals includes: Feature extraction is performed on the multiple sub-band signals respectively to obtain the sub-band features of each sub-band signal; Stack the sub-band features of each sub-band signal to obtain the stacked features; The spectral characteristics of each sub-band signal are obtained based on the stacking characteristics.

5. The method according to claim 4, characterized in that, The step of extracting features from the plurality of sub-band signals to obtain the sub-band features of each sub-band signal includes: The sub-band features of each sub-band signal are obtained by extracting features from each sub-band signal using a multilayer perceptron (MLP) corresponding to each sub-band signal. Each subband signal's corresponding MLP consists of a root mean square normalization layer and a linear layer connected in series.

6. The method according to claim 4, characterized in that, The step of obtaining the spectral characteristics of each sub-band signal based on the stacking characteristics includes: The local feature extraction module models the features of each sub-band of the stacked features in the time dimension to obtain local features composed of the temporal features of each sub-band signal. The local features are transposed to obtain the first transposed feature; The global feature extraction module models each temporal feature of the first transposed feature in the feature stacking dimension to obtain global features composed of the frequency band features of each sub-band signal. Perform a transpose operation on the global feature to obtain a second transpose feature; The second transposed feature is fused using a multi-head self-attention mechanism to obtain the fused feature. The fusion features are split to obtain the spectral features of each sub-band signal.

7. The method according to claim 6, characterized in that, The local feature extraction module and / or the global feature extraction module are dual-path Mamba modules; The dual-path Mamba module includes: The first path includes: a first Mamba block, a first root mean square normalization layer, and a first adder. The input of the first Mamba block is the input of the dual-path Mamba module. The input of the first root mean square normalization layer is the output of the first Mamba block. The input of the first adder is the output of the first Mamba block and the output of the first root mean square normalization layer. The output of the first adder is the output of the first path. The second path includes: a flip layer, a second Mamba block, a second root mean square normalization layer, and a second adder. The flip layer is used to flip the input of the dual-path Mamba module. The input of the second Mamba block is the output of the flip layer. The input of the second root mean square normalization layer is the output of the second Mamba block. The input of the second adder is the output of the second Mamba block and the output of the second root mean square normalization layer. The output of the second adder is the output of the second path. A splicing layer, used to splice the output of the first path and the output of the second path; A linear layer, the input of which is the output of the splicing layer, and the output of which is the output of the dual-path Mamba module.

8. The method according to claim 1, characterized in that, The step of obtaining the spectral mask of at least one sound source of the audio signal to be separated based on the spectral characteristics of each sub-band signal includes: For each sound source, the spectral features of each sub-band signal are processed by the corresponding mask estimation module to obtain the spectral mask of each sub-band signal corresponding to the sound source, and the spectral masks of each sub-band signal corresponding to the sound source are spliced ​​together to obtain the spectral mask of the sound source. Each sub-band's corresponding mask estimation module consists of a series of MLPs and gated linear units (GLUs), with the MLP consisting of a series of root mean square normalization layers and linear layers.

9. The method according to claim 1, characterized in that, The step of obtaining the audio signal of at least one sound source based on the spectral mask of the at least one sound source and the time-frequency domain signal includes: Calculate the product of the time-frequency domain signal and the spectral mask of each sound source to obtain the time-frequency domain signal of each sound source; The time-frequency domain signals of each sound source are transformed into time-domain signals to obtain the audio signals of each sound source.

10. The method according to claim 9, characterized in that, The process of transforming the time-frequency domain signals of each sound source into time-domain signals includes: The inverse short-time Fourier transform (ISTFT) is performed on the time-frequency domain signals of each sound source to transform the time-frequency domain signals of each sound source into time-domain signals.

11. The method according to any one of claims 1-10, characterized in that, The audio separation method is implemented based on a sound source separation model, which includes: The transformation module is used to perform the step of transforming the audio signal to be separated from a time-domain signal to a time-frequency domain signal; The segmentation module is used to perform the step of segmenting the time-frequency domain signal into multiple sub-band signals. The acquisition module is used to perform the steps of acquiring the spectral characteristics of the plurality of sub-band signals respectively; A mask estimation module is used to perform the step of obtaining a spectral mask of at least one sound source of the audio signal to be separated based on the spectral characteristics of the plurality of sub-band signals; An output module is configured to perform the step of acquiring an audio signal of at least one sound source based on the spectral mask of the at least one sound source and the time-frequency domain signal.

12. The method according to claim 11, characterized in that, Before implementing the audio separation method based on the sound source separation model, the method further includes: Obtain a training data set, which includes: multiple sets of training data, each set of training data including: sample audio signals and reference audio signals of at least one sound source corresponding to the sample audio signals; The sound source separation model is trained based on the training dataset.

13. The method according to claim 12, characterized in that, The training of the sound source separation model based on the training dataset includes: The sample audio signal is input into the sound source separation model, and the predicted audio signal of at least one sound source corresponding to the sample audio signal and the time-frequency domain signal of the predicted audio signal of at least one sound source corresponding to the sample audio signal are obtained from the output of the sound source separation model. Obtain the time-frequency domain signal of the reference audio signal of at least one sound source corresponding to the sample audio signal; The loss value corresponding to the sample audio signal is calculated based on the reference audio signal of at least one sound source corresponding to the sample audio signal, the predicted audio signal of at least one sound source corresponding to the sample audio signal, the time-frequency domain signal of the predicted audio signal of at least one sound source corresponding to the sample audio signal, and the time-frequency domain signal of the reference audio signal of at least one sound source corresponding to the sample audio signal. The model parameters of the sound source separation model are updated based on the loss value.

14. A sound source separation device, characterized in that, include: The transformation unit is used to transform the audio signal to be separated from a time-domain signal to a time-frequency domain signal; A segmentation unit is used to perform frequency band segmentation on the time-frequency domain signal to divide the time-frequency domain signal into multiple sub-band signals, wherein the frequency bands of the multiple sub-band signals do not overlap. An acquisition unit is used to acquire the spectral characteristics of the plurality of sub-band signals respectively; The processing unit is used to obtain the spectral mask of at least one sound source of the audio signal to be separated based on the spectral characteristics of the plurality of sub-band signals. An output unit is used to acquire the audio signal of at least one sound source based on the spectral mask of the at least one sound source and the time-frequency domain signal.

15. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is used to store a computer program and the processor is used to cause the electronic device to implement the sound source separation method according to any one of claims 1-13 when executing the computer program.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to implement the sound source separation method according to any one of claims 1-13.

17. A computer program product, characterized in that, When the computer program product is run on a computer, the computer implements the sound source separation method according to any one of claims 1-13.

Images