A two-stage vocal accompaniment separation method based on pre-separation and post-compensation

Abstract

The application discloses a two-stage vocal accompaniment separation method based on pre-separation and post-compensation. In the first stage, the amplitude spectrum features and phase features of the mixed audio signal are extracted first, and the amplitude spectrum features of the vocal and accompaniment are separated from the amplitude spectrum features of the mixed audio signal by using a stacked hourglass architecture. An additional term is added to the loss function to constrain the separation source, so that the predicted vocal signal contains as little accompaniment signal as possible, and the predicted accompaniment signal contains as little vocal signal as possible. In the second stage, a signal compensation model based on a gated TCN and an asymmetric convolution network in series is used to compensate the separation result of the first stage, the long-time dependence in the music signal is captured through dilated convolution, irrelevant information is effectively filtered, fine-grained features are extracted by using the parallel structure of the asymmetric convolution, and more pure vocal and accompaniment tracks are obtained.

Description

Technical Field

[0001] This invention belongs to the field of audio processing technology, and relates to the direction of sound signal separation, and in particular to a two-stage method for separating vocal accompaniment based on separation followed by compensation. Background Technology

[0002] Vocal-accompaniment separation technology is an audio processing technique used to separate the vocals and accompaniment from a mixed audio track. This process typically involves sophisticated algorithms and machine learning techniques to ensure the separated audio track maintains high quality. This technology has many important applications in virtual music education, significantly enhancing the student learning experience and teaching effectiveness. It not only improves the flexibility and convenience of education but also enriches teaching content and resources, and enhances interactivity and personalized experiences.

[0003] Early vocal accompaniment separation techniques primarily relied on signal processing methods and basic mathematical models. While these methods could achieve certain results in specific situations, they generally had many limitations. In recent years, neural network-based methods have made significant progress and development in this field, especially the U-Net and stacked hourglass architecture models, which are highly competitive. However, they also have shortcomings, such as the quality of the separated audio signal being inferior to the original audio signal, and the problem of incomplete separation, resulting in the separated audio track containing sound components that do not belong to that track, affecting the separation quality. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a two-stage vocal and accompaniment separation method based on separation followed by compensation. In the first stage, an improved stacked hourglass architecture model and an optimized constraint loss function are used to initially separate the mixed audio signals, resulting in separated vocal and accompaniment signals. In the second stage, further compensation is applied to the separated signals to obtain purer vocal and accompaniment tracks, thereby improving the quality of the separated signals and alleviating the problem of incomplete signal separation.

[0005] A two-stage vocal accompaniment separation method based on separation followed by compensation includes the following steps:

[0006] Step 1: Preprocess the audio signal to obtain the amplitude spectrum features and phase features of the mixed audio signal, as well as the amplitude spectrum features S1 of the human voice and the amplitude spectrum features S2 of the accompaniment.

[0007] Step 1.1: Sample the original audio and convert it into a discrete digital signal that the computer system can process. The original audio sampling rate is typically 44.1kHz or 48kHz, which can be downsampled to 22.05kHz or 16kHz. Audio downsampling will lose high-frequency information, but in many vocal and accompaniment separation tasks, the audio information of vocals and accompaniment is mainly concentrated in the lower frequency range. Therefore, downsampling can be performed without significantly affecting the separation effect, in order to reduce the amount of data and reduce computational complexity. In addition, the anti-aliasing filter in the downsampling process can remove high-frequency noise in the original signal, which helps to improve the quality of the audio signal.

[0008] Step 1.2: Process the audio signal x[n] STFT transformation is performed to obtain complex spectrum characteristics This represents the spectral information of an audio signal at different time points and frequency components, and then the complex spectral characteristics are analyzed. Amplitude spectrum characteristics are obtained by taking the absolute value. :

[0009] X t ,f = ∑ n=0 N−1 x[n]∙ω[n−t]∙ e −j2πfn (1)

[0010]

[0011] in, It's a window function. It is a time frame index. It is a frequency bin index. It is the length of each frame of the signal. It is the sample index of the audio signal.

[0012] Step 2: Use the amplitude spectrum features of the mixed signal as input to the first-stage stacked hourglass architecture model to obtain the mask estimates of the vocals and accompaniment separated in the first stage. and .

[0013] Step 2.1: Construct a stacked hourglass network model framework, consisting of four cascaded hourglass modules. To alleviate the diminishing marginal utility problem caused by stacking too many identical modules, a Real Fast Fourier Transform (RFFT) is applied after the odd-numbered hourglass modules, and an Inverse Real Fast Fourier Transform (IRFFT) is applied after the even-numbered hourglass modules. This maps the output features to a new feature space, improving model performance. RFFT and IRFFT are two variants of the Discrete Fourier Transform (DFT), specifically designed for processing real-valued signals. RFFT converts real-valued signals in the time domain to signals in the frequency domain. Because the input signal is real, the RFFT result is symmetric, requiring only half of the frequency components to be computed, thus reducing computational complexity. IRFFT is the inverse of RFFT, used to recover real-valued signals from the frequency domain back to the time domain. Their formulas are as follows:

[0014] X k =RFFT(x[n])= ∑ n=0 N−1 x[n]∙ e −j2πkn / N (3)

[0015] x n =IRFFT(X k )= 1 N ∑ k=0 N 2 X[k]∙ e j2πkn / N (4)

[0016] in, X[k] The k-th component of the frequency domain signal x[n] Let N be the nth component of the real signal in the time domain, where N is the total number of samples of the real signal in the time domain, k is the frequency index, and n is the time index.

[0017] Step 2.2: To alleviate the problem of incomplete signal separation, a loss function is set. for:

[0018]

[0019]

[0020]

[0021]

[0022]

[0023] in, An index representing the number of separate sources. An index representing the number of hourglass modules. Indicates the first The separation source in the first The time-frequency mask output by each hourglass module. Represents element-wise product. express Norm, Indicates mean square error. and For adaptively adjusted loss function weights, This is a hyperparameter.

[0024] Improved loss function Through the additional items This constrains the separation source, ensuring that the predicted vocal signal contains as little accompaniment signal as possible, and the predicted accompaniment signal contains as little vocal signal as possible. Furthermore, it adaptively adjusts the weights. and To handle unbalanced sound sources and optimize the quality of sound source separation.

[0025] Step 3: Mask the voices estimated in the first stage. and accompaniment mask Element-wise multiplication with the amplitude spectrum features of the mixed signal yields preliminary estimates of the amplitude spectrum features of the vocals and accompaniment. and .

[0026] Step 4: Combine the amplitude spectrum characteristics of the vocals and accompaniment estimated in the first stage. and After processing and transformation, the signal is input into the second-stage signal compensation module to obtain the final predicted amplitude spectrum features of the vocals and accompaniment. and In the second stage, a signal compensation model based on a gated TCN and an asymmetric convolutional network was selected. The gated TCN network effectively expands the receptive field through dilated convolution, enabling it to capture long-term dependencies in the music signal. Simultaneously, its gating mechanism selectively transmits information, allowing the model to better capture complex temporal dependencies and effectively filter out irrelevant information, enhancing the model's expressive power and enabling more accurate separation of vocal parts and accompaniment, resulting in a clearer and purer signal. The parallel structure of the asymmetric convolutional network can extract fine-grained features and also capture temporal and frequency correlations. Similarly, to suppress the problem of diminishing marginal utility, RFFT and IRFFT transforms were introduced between the models.

[0027] Step 4.1: First, analyze the amplitude spectrum characteristics of the mixed signal. The amplitude spectrum characteristics of vocals and accompaniment predicted in the first stage, respectively. and Subtraction yields the residual signal , The signals are then input into the vocal signal compensation module and the accompaniment signal compensation module, respectively.

[0028]

[0029]

[0030] Step 4.2: The residual signal is first input into a single information compensation module, which consists of a gated TCN and an asymmetric convolutional network connected in series. It first passes through a gated linear unit (GLU), then through a one-dimensional convolution and GRU module, and is added to the original residual signal. This signal is then input into the asymmetric convolutional network, passing through an SE block and PreLU, and then through 1... 3 and 3 The output of the two parallel branches of the asymmetric convolution is obtained by adding the outputs of the two branches and then performing a series of regularization methods and convolution operations.

[0031] The gated linear unit (GLU) first performs one-dimensional convolution to extract features from the input data. The extracted features are then computed in parallel and input into two parallel branches, where convolution, normalization, and activation operations are performed respectively. One of the branches also undergoes sigmoid processing. The outputs of the two branches are multiplied together. Through this gating mechanism, the information flow can be flexibly controlled, important information can be selectively transmitted, and irrelevant or noisy information can be suppressed, thereby enhancing the model's expressiveness and generalization ability.

[0032] Step 4.3: Input the output result into the next information compensation module, and repeat this operation for a total of n information compensation modules. An RFFT transform is applied after the information compensation module for odd-numbered sequences, and an IRFFT transform is applied after the information compensation module for even-numbered sequences. Here, n is an even number.

[0033] Step 4.4: Combine the residual signals of vocals and accompaniment predicted in Step 4.3. and The amplitude spectrum characteristics of vocals and accompaniment predicted in the first stage, respectively. and By adding them together, we obtain the final predicted amplitude spectrum characteristics of the vocals and accompaniment. and :

[0034]

[0035]

[0036] Step 5: Extract amplitude spectrum characteristics and Phase characteristics of mixed audio signals The vocal and accompaniment signals are then combined and reconstructed using a short-time inverse Fourier transform.

[0037] The present invention has the following beneficial effects:

[0038] The stacked hourglass model is improved by using RFFT and IRFFT transforms to alleviate the problem of diminishing marginal utility. The improved loss function in the first stage and the signal compensation module in the second stage are used to obtain purer vocal and accompaniment tracks, thereby alleviating the problem of incomplete signal separation and improving the quality of the separated audio. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of a two-stage vocal accompaniment separation model based on separation followed by compensation.

[0040] Figure 2 This is a schematic diagram of the vocal / accompaniment signal compensation module;

[0041] Figure 3 This is a structural diagram of a single information compensation module. Detailed Implementation

[0042] The present invention will be further explained below with reference to the accompanying drawings;

[0043] like Figure 1 As shown, a two-stage vocal accompaniment separation method based on separation followed by compensation specifically includes the following steps:

[0044] Step 1: Preprocess the audio signal to obtain the amplitude spectrum features and phase features of the mixed audio signal, as well as the amplitude spectrum features S1 of the human voice and the amplitude spectrum features S2 of the accompaniment.

[0045] Step 1.1: First, the audio file is split. Here, the librosa library is used as a tool to read the audio signal file and perform downsampling.

[0046] Step 1.2: Obtain the complex spectrum characteristics of the sampled audio signal through STFT transformation. The amplitude spectrum characteristics of the audio signal are obtained by taking the absolute value and calculating the complex matrix radians, respectively. and phase characteristics .

[0047] Step 1.3: Segment the amplitude spectrum features of the audio signal along the time dimension to obtain a value of 512. Amplitude spectrum characteristics of 64.

[0048] Step 2: Obtain the mask estimates for the vocals and accompaniment separated in the first stage. and .

[0049] Step 2.1: Construct a stacked hourglass network model framework, which includes four hourglass modules consisting of several convolutional layers, pooling layers, and upsampling layers. RFFT transformation is introduced after the first and third hourglass modules, and IRFFT transformation is introduced in the second and fourth hourglass modules.

[0050] Step 2.2: In an hourglass module, a series of downsampling operations are first performed to condense information and retain key features. Then, upsampling operations are used to gradually refine the reconstructed feature map, with skip connections introduced in between to help recover spatial details. The human voice signal mask is obtained in the b-th hourglass module. Accompaniment signal mask .

[0051] Step 2.3: To alleviate the problem of incomplete signal separation, a loss function is set. for:

[0052]

[0053]

[0054]

[0055]

[0056]

[0057] in, This represents the amplitude spectrum characteristics of the mixed signal. An index representing the number of separate sources. An index representing the number of hourglass modules. Indicates the first The separation source in the first The time-frequency mask output by each hourglass module. Represents element-wise product. express Norm, Indicates mean square error. and For adaptively adjusted loss function weights, This is a hyperparameter.

[0058] Step 3: Use the estimated time-frequency mask to obtain the amplitude spectrum features of the predicted vocals and accompaniment.

[0059] The voice mask estimated in the first stage and accompaniment mask The amplitude spectrum characteristics of the mixed signal respectively Element-wise multiplication yields a preliminary estimate of the amplitude spectrum characteristics of the vocals and accompaniment. and :

[0060]

[0061]

[0062] Step 4: As Figure 2 As shown, a signal compensation module based on a gated TCN architecture and an asymmetric convolutional network architecture is constructed.

[0063] Step 4.1: First, analyze the amplitude spectrum characteristics of the mixed signal. The amplitude spectrum characteristics of vocals and accompaniment predicted in the first stage, respectively. and Subtracting them, we get a value of 512. 64 residual signal , The signals are then input into the vocal signal compensation module and the accompaniment signal compensation module, respectively.

[0064] Step 4.2: As Figure 3 As shown, the residual signal is first input into a single information compensation module, which consists of a gated TCN and an asymmetric convolutional network connected in series. It first passes through a gated linear unit (GLU), then through a one-dimensional convolution and GRU module, and is added to the original residual signal. This signal is then input into the asymmetric convolutional network, passing through an SE block and PreLU, and finally through a 1... 3 and 3 The output of the two parallel branches of the asymmetric convolution is obtained by adding the outputs of the two branches and then performing a series of regularization methods and convolution operations.

[0065] Step 4.3: Input the output result into the next information compensation module, and repeat this operation for a total of 16 information compensation modules. An RFFT transform is applied after the information compensation module for odd-numbered sequences, and an IRFFT transform is applied after the information compensation module for even-numbered sequences.

[0066] Step 4.4: Combine the residual signals of vocals and accompaniment predicted in Step 4.3. and The amplitude spectrum characteristics of vocals and accompaniment predicted in the first stage, respectively. and By adding them together, we obtain the final predicted amplitude spectrum characteristics of the vocals and accompaniment. and .

[0067] Step 5: Reconstruct the vocal and accompaniment signals.

[0068] Step 5.1: Analyze the amplitude spectrum characteristics predicted in the second stage. and Phase characteristics of mixed audio signals By combining the two signals, the vocal and accompaniment signals are reconstructed using short-time inverse Fourier transform.

[0069] Step 5.2: The segmented reconstructed signals are spliced ​​together along the time dimension to obtain the complete reconstructed audio signal.

Claims

1. A two-stage vocal accompaniment separation method based on pre-separation and post-compensation, characterized in that: Specifically comprising the following steps: Step 1: preprocessing the mixed audio signal to obtain the amplitude spectrum feature and phase feature of the mixed audio signal, and the corresponding amplitude spectrum feature S1 of the vocal and the amplitude spectrum feature S2 of the accompaniment; specifically: Step 1.1: sampling the mixed audio signal into a discrete digital signal, and then performing a downsampling operation; Step 1.2: Processing the audio signal Performing STFT transform to get complex spectrum feature Taking absolute value to get amplitude spectrum feature : ； ； wherein, is a window function, is a time frame index, is a frequency bin index, , is a sample index of the mixed audio signal; Step 2: Estimate the first stage of the vocal mask using the amplitude spectrum features of the mixed audio signal as input to the first stage of the stacked hourglass architecture model with the accompaniment mask ; Step 3: The first stage estimate of the vocals mask and the accompaniment mask are respectively element-wise multiplied with the amplitude spectrum features of the mixed audio signal to obtain a preliminary estimate of the amplitude spectrum features of the vocals and the amplitude spectrum features of the accompaniment ; Step 4: the amplitude spectrum features of the human voice are processed and transformed into the signal compensation module of the second stage to obtain the final predicted amplitude spectrum features of the human voice are processed and transformed into the signal compensation module of the second stage to obtain the final predicted amplitude spectrum features of the human voice are processed and transformed into the signal compensation module of the second stage to obtain the final predicted amplitude spectrum features of the human voice The specific steps are as follows: Step 4.1: subtracting the amplitude spectrum features of the mixed audio signal respectively from the amplitude spectrum features of the estimated vocal signal in the first stage and the amplitude spectrum features of the accompaniment to obtain residual signals , , and input into the vocal signal compensation module and the accompaniment signal compensation module respectively; Step 4.2: inputting the residual signal into a model of a single gated TCN and an asymmetric convolution network in series to obtain an output result; Step 4.3: repeating step 4.2 to perform multiple residual signal compensation operations; Step 4.4: Adding the residual signal of the voice compensated in step 4.3 to the residual signal of the accompaniment to the amplitude spectrum features of the voice estimated in the first stage to the amplitude spectrum features of the accompaniment to obtain the final predicted amplitude spectrum features of the voice to the amplitude spectrum features of the accompaniment ; Step 5: the amplitude spectrum characteristics of the final predicted vocal of the accompaniment of the mixed audio signal In combination, the vocal and accompaniment signals are then reconstructed by inverse short-time Fourier transform.

2. The two-stage a cappella separation method based on pre-separation and post-compensation of claim 1, wherein: The stacked hourglass architecture model comprises 4 cascaded hourglass modules, and is connected with an RFFT transformation after the odd sequence hourglass module and an IRFFT transformation after the even sequence hourglass module: ； ； wherein is the kth component of the frequency domain signal, is the nth component of the real signal in the time domain.

3. The two-stage a cappella separation method based on pre-separation and post-compensation of claim 1, wherein: Loss function of stacked hourglass architecture model is: ； ； ； ； ； wherein, denotes an index representing the number of separation sources, denotes an index representing the number of sandglass modules, denotes the time-frequency mask output by the th separation source at the th sandglass module, denotes an element-wise product, denotes a norm, denotes a mean squared error, and is a loss function weight for adaptive adjustment, is a hyperparameter.

4. The two-stage a cappella separation method based on separation first and compensation second of claim 1, wherein: 16 times of signal compensation operation are performed.

5. A computer readable storage medium having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the two-stage vocal-accompaniment separation method of any one of claims 1-4.

Images