Method, apparatus, terminal and medium for separating human voice from accompaniment
By using feature extraction and decoupling processing in a multi-branch dynamic logic design, combined with a multi-head self-attention layer and a residual connection layer, the problems of noise estimation accuracy and overlapping feature processing in the separation of vocals and accompaniment in the prior art are solved, and high-precision separation of vocals and accompaniment is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHENGDU XIAOCHANG TECH CO LTD
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-09
AI Technical Summary
Existing vocal and accompaniment separation technologies are highly dependent on noise estimation accuracy, prone to generating musical noise, lack effective evaluation and adaptability processing logic, and have simple processing methods for overlapping feature components, which cannot meet the separation requirements of highly mixed audio.
A multi-branch dynamic logic design is adopted. Through feature extraction, decoupling processing and feature quality level evaluation, combined with short-time Fourier transform, convolutional network, batch normalization, multi-head self-attention layer and residual connection layer, the separation scheme is dynamically adjusted to achieve high-precision separation.
It achieves high-precision separation of vocals and accompaniment, adapts to different separation requirements, improves the ability to process overlapping features, and enhances the separation effect.
Smart Images
Figure CN122177149A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, terminal and medium for separating human voice and accompaniment. Background Technology
[0002] Vocal and accompaniment separation is one of the core research directions in the field of audio signal processing, widely used in music production, karaoke systems, audio noise reduction, and speech enhancement. Traditional separation techniques are based on the energy distribution differences between vocals and accompaniment in the time-frequency domain. They filter the time-frequency features of the mixed audio by calculating masking functions such as ideal binary masking (IBM) and ideal ratio masking (IRM), preserving the time-frequency components of the target signal. The core assumption is that the energy of vocals and accompaniment does not overlap in some time-frequency units; or, by estimating the spectral characteristics of the accompaniment noise, the noise spectrum is subtracted from the mixed audio spectrum to achieve vocal extraction. With the development of deep learning technology, neural network-based separation methods have gradually become mainstream. Their core is to utilize networks to automatically learn the feature differences between vocals and accompaniment, replacing the manual design rules of traditional methods. Representative techniques include: Recurrent Neural Network (RNN) / Long Short-Term Memory (LSTM) methods: targeting the temporal characteristics of audio signals, RNNs / LSTMs capture the temporal dependencies of long sequences, improving the ability to model the temporal features of vocals and accompaniment. Transformer-based methods: Introducing a self-attention mechanism to model the global dependencies of audio time-frequency features, solving the problem of vanishing gradients in long RNN sequences and local receptive fields of CNNs, and further improving feature decoupling effects; Hybrid model methods: Combining the local feature extraction capabilities of CNNs with the global modeling capabilities of Transformers to construct end-to-end separation models, which has become the current mainstream research direction.
[0003] However, existing techniques for separating vocals and accompaniment have the following shortcomings: they are highly dependent on the accuracy of noise estimation and are prone to generating "musical noise"; most existing deep learning methods adopt a "model output is the final result" approach, where the neural network obtains the separated audio through simple inverse time-frequency transformation after feature decoupling, lacking effective evaluation and adaptation logic for the model output features; existing methods often use single feature extraction or decoupling structures in their neural network layers, and handle overlapping feature components of vocals and accompaniment in a simple way (such as directly discarding or evenly distributing them), without considering the amplitude and energy distribution characteristics of the overlapping components; when faced with highly mixed and difficult-to-separate audio (such as musical segments where vocals and accompaniment frequencies highly overlap), existing methods only perform a single model inference, and there are no effective remedial measures when the separation degree cannot meet the actual needs. Summary of the Invention
[0004] The main purpose of this application is to provide a method, device, terminal and medium for separating vocals and accompaniment, which aims to achieve high-precision separation of vocals and accompaniment through multi-branch dynamic logic design.
[0005] To achieve the above objectives, this application provides a method for separating vocals and accompaniment, the method comprising: Obtain the raw waveform data of the mixed audio; The original waveform data is subjected to feature extraction processing to obtain the feature extraction tensor corresponding to the mixed audio. The feature extraction tensor is decoupled to obtain the vocal feature tensor and the accompaniment feature tensor; Based on the vocal feature tensor and the accompaniment feature tensor, a feature quality level is determined, wherein the feature quality level is used to characterize the degree of separation between the vocal feature tensor and the accompaniment feature tensor. Based on the aforementioned feature quality level, the vocal feature tensor and the accompaniment feature tensor are processed using a preset differentiated separation execution scheme to obtain target vocal waveform data and target accompaniment waveform data.
[0006] Specifically, the step of performing feature extraction processing on the original waveform data to obtain the feature extraction tensor corresponding to the mixed audio includes: The original waveform data is subjected to short-time Fourier transform processing to obtain a spectral feature map; The spectral feature map is convolved by a pre-defined one-dimensional convolutional network to obtain an intermediate convolutional feature map. The intermediate convolutional feature maps are batch normalized by a preset batch normalization network to obtain batch normalized feature maps. Based on the preset element activation rules, the activation feature map is obtained according to the batch normalized feature map; The activated feature map is processed based on a preset pooling kernel to obtain the feature extraction tensor.
[0007] Specifically, the step of obtaining the activation feature map based on the preset element activation rules and the batch normalized feature map includes: The elements greater than 0 in the batch normalized feature map are retained, and the elements less than 0 in the batch normalized feature map are set to 0 to obtain the activation feature map.
[0008] Specifically, the step of processing the activation feature map based on a preset pooling kernel to obtain the feature extraction tensor includes: Calculate the average value of all elements in the activated feature map that are covered by the receptive field corresponding to the preset pooling kernel to obtain the pooled feature value. Based on the pooled feature values, the feature extraction tensor is obtained.
[0009] Specifically, the decoupling process of the feature extraction tensor to obtain the vocal feature tensor and the accompaniment feature tensor includes: The feature extraction tensor is input into a preset multi-head self-attention layer to obtain multi-head self-attention features with the same dimension as the feature extraction tensor. The multi-head self-attention features and the feature extraction tensor are input into a preset residual connection layer to obtain a residual feature tensor with the same dimension as the feature extraction tensor. The residual feature tensor is subjected to depth convolution to obtain a depth convolution feature tensor; The depthwise convolutional features are subjected to pointwise convolution processing to obtain the vocal feature tensor and the accompaniment feature tensor.
[0010] Specifically, determining the feature quality level based on the vocal feature tensor and the accompaniment feature tensor includes: Based on the human voice feature tensor and the accompaniment feature tensor, the first similarity matrix corresponding to the human voice feature tensor and the second similarity matrix corresponding to the accompaniment feature tensor are calculated respectively. Based on the first similarity matrix and the second similarity matrix, the first global mean corresponding to the first similarity matrix and the second global mean corresponding to the second similarity matrix are calculated respectively. Based on the preset frequency dimension weight, the preset time dimension weight, the first global mean, and the second global mean, the fusion separation index corresponding to the mixed audio is calculated. The feature quality level is determined by comparing the fusion separation index with the preset separation threshold range.
[0011] Specifically, the characteristic quality level is a high resolution level, a medium resolution level, or a low resolution level; Based on the feature quality level, the process of processing the vocal feature tensor and the accompaniment feature tensor using a preset differentiated separation execution scheme to obtain target vocal waveform data and target accompaniment waveform data includes: If the feature quality level is the high separation level, then the inverse short-time Fourier transform is performed on the human voice feature tensor and the accompaniment feature tensor respectively to obtain the target human voice waveform data and the target accompaniment waveform data. If the feature quality level is the medium separation level, then the human voice feature tensor is subjected to high frequency enhancement processing and inverse short-time Fourier transform processing in sequence, and the accompaniment feature tensor is subjected to noise suppression processing and inverse short-time Fourier transform processing in sequence to obtain the target human voice waveform data and the target accompaniment waveform data. If the feature quality level is the low separation level, then based on the amplitude ratio of the overlapping feature components between the vocal feature tensor and the accompaniment feature tensor, the overlapping feature components are allocated to the vocal feature tensor and the accompaniment feature tensor to obtain the allocated vocal feature tensor and the allocated accompaniment feature tensor; the preset multi-head self-attention layer is adjusted, and based on the adjusted preset multi-head self-attention layer, the allocated vocal feature tensor and the allocated accompaniment feature tensor are decoupled to redetermine the feature quality level between the allocated vocal feature tensor and the allocated accompaniment feature tensor.
[0012] To achieve the above objectives, this application also provides a vocal and accompaniment separation device, the device comprising: The first unit is used to acquire the raw waveform data of the mixed audio; The second unit is used to perform feature extraction processing on the original waveform data to obtain the feature extraction tensor corresponding to the mixed audio. The third unit is used to decouple the feature extraction tensor to obtain the human voice feature tensor and the accompaniment feature tensor. The fourth unit is used to determine the feature quality level based on the vocal feature tensor and the accompaniment feature tensor, wherein the feature quality level is used to characterize the degree of separation between the vocal feature tensor and the accompaniment feature tensor. The fifth unit is used to process the vocal feature tensor and the accompaniment feature tensor based on the feature quality level and through a preset differentiated separation execution scheme to obtain target vocal waveform data and target accompaniment waveform data.
[0013] To achieve the above objectives, this application also provides a terminal, including a memory storing multiple instructions; the processor loads instructions from the memory to execute the steps in any of the methods provided in this application.
[0014] To achieve the above objectives, this application also provides a medium storing a plurality of instructions adapted for loading by a processor to execute the steps in any of the methods provided in this application.
[0015] This application provides a method, apparatus, terminal, and medium for separating vocals and accompaniment. The method first acquires the original waveform data of the mixed audio; performs feature extraction processing on the original waveform data to obtain a feature extraction tensor corresponding to the mixed audio; decouples the feature extraction tensor to obtain a vocal feature tensor and an accompaniment feature tensor; determines a feature quality level based on the vocal and accompaniment feature tensors; and processes the vocal and accompaniment feature tensors using a preset differentiated separation execution scheme based on the feature quality level to obtain target vocal waveform data and target accompaniment waveform data, thereby achieving high-precision separation of vocals and accompaniment. Attached Figure Description
[0016] Figure 1 A flowchart illustrating the method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the device provided in the embodiments of this application; Figure 3 This is a schematic diagram of the terminal structure provided in an embodiment of this application. Detailed Implementation
[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0018] Existing techniques for separating vocals and accompaniment have the following shortcomings: they are highly dependent on the accuracy of noise estimation and are prone to generating "musical noise"; most existing deep learning methods adopt a "model output is the final result" approach, where the neural network completes feature decoupling and only obtains the separated audio through simple inverse time-frequency transformation, lacking effective evaluation and adaptation logic for the model output features; existing methods often use single feature extraction or decoupling structures in their neural network layers, and the handling of overlapping feature components between vocals and accompaniment is simple (such as direct discarding or even distribution), without considering the amplitude and energy distribution characteristics of the overlapping components; when faced with highly mixed and difficult-to-separate audio (such as musical segments where vocals and accompaniment frequencies highly overlap), existing methods only perform a single model inference, and there are no effective remedial measures when the separation degree cannot meet the actual needs.
[0019] Therefore, this application provides a method, apparatus, terminal, and medium for separating vocals and accompaniment to solve practical technical problems.
[0020] In some embodiments, the device may be integrated into an electronic device, such as a terminal or server.
[0021] In some embodiments, the server may also be implemented as a terminal.
[0022] The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.
[0023] The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited herein.
[0024] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.
[0025] This application provides a method for separating vocals and accompaniment, which can achieve high-precision separation of vocals and accompaniment through multi-branch dynamic logic design.
[0026] In some embodiments, the method is as follows Figure 1 The specific process can be as follows: S110: Obtain the raw waveform data of the mixed audio.
[0027] In some embodiments, the original waveform data Xwave of the mixed audio, with dimensions R, can be obtained through mixing software or other technical means. T×1 , where T represents the audio time step and 1 represents a single channel.
[0028] S120. Perform feature extraction processing on the original waveform data to obtain the feature extraction tensor corresponding to the mixed audio.
[0029] In some embodiments, the feature extraction processing of the original waveform data to obtain the feature extraction tensor corresponding to the mixed audio includes the steps A1 to A5 shown below: A1. Perform short-time Fourier transform processing on the original waveform data to obtain the spectral feature map.
[0030] In some embodiments, a short-time Fourier transform (STFT) is performed on the Xwave to obtain a spectral feature map Xspec of the mixed audio, with dimension R. F×T′×1, where F is the frequency dimension and T′ is the number of time frames after STFT.
[0031] A2. The spectral feature map is processed by convolution through a preset one-dimensional convolutional network to obtain an intermediate convolutional feature map.
[0032] In some embodiments, a weighted summation operation is performed on each local receptive field of the spectral feature map Xspec, specifically by selecting a convolutional kernel weight matrix K with dimension R. kf×kt×1×Cout Where kf is the kernel size in the frequency direction, kt is the kernel size in the time direction, and Cout is the number of output channels; the feature elements within the receptive field are multiplied by the corresponding weight elements of the convolution kernel, summed, and then superimposed with the bias vector b (of dimension R). Cout The intermediate convolutional feature map Xconv_mid is obtained by calculating the intermediate feature value as follows: intermediate feature value = Σ(receptive field feature element × corresponding convolutional kernel weight) + corresponding channel bias value. The intermediate convolutional feature map Xconv_mid has dimension R. F′×T′′×Cout , where F′=F kf+1, T′′=T′ kt+1.
[0033] A3. The intermediate convolutional feature maps are batch normalized using a preset batch normalization network to obtain batch normalized feature maps.
[0034] In some embodiments, a preset batch normalization network is used to calculate the mean μc and variance σc for all feature elements in each channel. 2 Then, perform standardization on each feature element of the channel, i.e., the standardized element value = (original element value - μc) / (σc). 2 + () To prevent extremely small values with a denominator of 0, a linear transformation is finally performed using a scaling factor γc and an offset factor βc to obtain the batch-normalized feature map. The calculation relationship is: batch-normalized element value = standardized element value × γc + βc. The batch-normalized feature map Xbn has the same dimension as Xconv_mid.
[0035] A4. Based on the preset element activation rules, obtain the activation feature map according to the batch normalized feature map.
[0036] In some embodiments, obtaining the activation feature map based on the batch normalized feature map according to the preset element activation rule includes the following specific implementation process: The elements greater than 0 in the batch normalized feature map are retained, and the elements less than 0 in the batch normalized feature map are set to 0 to obtain the activation feature map.
[0037] A5. Based on a preset pooling kernel, the activated feature map is processed to obtain the feature extraction tensor.
[0038] In some embodiments, the process of processing the activation feature map based on a preset pooling kernel to obtain the feature extraction tensor includes the steps A51 to A52 shown below: A51. Calculate the average value of all elements in the activated feature map that are covered by the receptive field corresponding to the preset pooling kernel to obtain the pooled feature value.
[0039] A52. Based on the pooled feature values, the feature extraction tensor is obtained.
[0040] Specifically, the preset pooling kernel size is pf×pt (frequency×time), and the stride is sf×st. The average value of all elements within the receptive field covered by each pooling kernel is calculated as the pooled feature value. Finally, based on the pooled feature values, the feature extraction tensor Xfeat is obtained, with dimension R. F′′×T′′′×Cout , where F′′=(F′ pf) / (sf+1).
[0041] S130. Decouple the feature extraction tensor to obtain the vocal feature tensor and the accompaniment feature tensor.
[0042] In some embodiments, the decoupling process of the feature extraction tensor to obtain the vocal feature tensor and the accompaniment feature tensor includes the steps B1 to B4 shown below: B1. Input the feature extraction tensor into a preset multi-head self-attention layer to obtain multi-head self-attention features with the same dimension as the feature extraction tensor.
[0043] In some embodiments, through the preset multi-head self-attention layer: Xfeat is transformed by three independent linear transformation matrices WQ, WK, and WV (each with dimension R). Cout×Cout The functions are mapped to query tensor Q, key tensor K, and value tensor V, respectively, with dimensions consistent with Xfeat. Q, K, and V are divided into N headers according to channel dimension, resulting in subquery Qi, subkey Ki, and subvalue Vi (i=1,2,...,N) for each header. Attention weights are calculated for each header: weight matrix Ai = softmax((Qi×KiT) / dk), where dk is the channel dimension of subkey Ki, × represents matrix multiplication, T represents matrix transpose, and the softmax function normalizes the weights to the 0-1 interval. The weight matrix Ai is multiplied by the subvalue Vi to obtain the attention feature Xattn_i = Ai×Vi for each header. The attention features of the N headers are concatenated according to channel dimension, and then transformed by a linear transformation matrix WO (dimension R).N×Cout×Cout The multi-head self-attention feature Xattn is obtained by fusing the features.
[0044] B2. Input the multi-head self-attention features and the feature extraction tensor into a preset residual connection layer to obtain a residual feature tensor with the same dimension as the feature extraction tensor.
[0045] In some embodiments, the corresponding elements of the multi-head self-attention features and the feature extraction tensor are directly added together to obtain the residual feature element values. The calculation relationship is: residual feature element value = corresponding element value of Xattn + corresponding element value of Xfeat. Finally, through the preset residual connection layer, the residual feature tensor Xres is output, with the same dimension as Xfeat.
[0046] B3. Perform depth convolution on the residual feature tensor to obtain a depth convolution feature tensor.
[0047] In some embodiments, convolution operations are performed independently on each input channel of the residual feature tensor Xres, and the convolution kernel weight matrix Kd has dimension R. kf×kt×1×Cout The calculation process is the same as that of the convolutional layer in step 1, resulting in the depthwise convolutional feature tensor Xdepth.
[0048] B4. Perform pointwise convolution processing on the depth convolution features to obtain the vocal feature tensor and the accompaniment feature tensor.
[0049] In some embodiments, a 1×1 convolution operation is performed on the depthwise convolutional feature tensor, and the convolution kernel weight matrix Kp has dimension R. 1×1×Cout×2×Cout The number of feature channels is doubled, corresponding to vocal feature channels and accompaniment feature channels respectively. After decoupling, we obtain the vocal feature tensor Xvoc and the accompaniment feature tensor Xacc, both with dimension R. F′′×T′′′×Cout .
[0050] S140. Based on the vocal feature tensor and the accompaniment feature tensor, determine the feature quality level, wherein the feature quality level is used to characterize the degree of separation between the vocal feature tensor and the accompaniment feature tensor.
[0051] In some embodiments, determining the feature quality level based on the vocal feature tensor and the accompaniment feature tensor includes the steps C1 to C4 as shown below: C1. Based on the vocal feature tensor and the accompaniment feature tensor, calculate the first similarity matrix corresponding to the vocal feature tensor and the second similarity matrix corresponding to the accompaniment feature tensor, respectively.
[0052] In some embodiments, a cosine similarity calculation function is constructed to calculate the cosine similarity matrices Sf and St of Xvoc and Xacc in the frequency dimension and time dimension, respectively, that is, the first similarity matrix corresponding to the vocal feature tensor and the second similarity matrix corresponding to the accompaniment feature tensor.
[0053] C2. Based on the first similarity matrix and the second similarity matrix, calculate the first global mean corresponding to the first similarity matrix and the second global mean corresponding to the second similarity matrix.
[0054] In some embodiments, the global means Sfˉ and Stˉ corresponding to the two similarity matrices are calculated respectively, wherein the lower the global mean, the higher the feature separation degree of the corresponding dimension.
[0055] C3. Based on the preset frequency dimension weight, the preset time dimension weight, the first global mean, and the second global mean, the fusion separation index corresponding to the mixed audio is calculated.
[0056] In some embodiments, the frequency dimension weight ωf = 0.6 and the time dimension weight ωt = 0.4 are preset, and the weighted fusion separation index D = 1 is calculated. (ωf×Sfˉ+ωt×Stˉ), where D ranges from 0 to 1, and a larger D value indicates a higher degree of separation.
[0057] C4. Compare the fusion separation index with the preset separation threshold range to determine the feature quality level.
[0058] In some embodiments, if the fusion separation index is higher than the upper limit of the preset separation threshold range, it is determined to be a high separation level; if the fusion separation index is within the preset separation threshold range, it is determined to be a medium separation level; if the fusion separation index is lower than the lower limit of the preset separation threshold range, it is determined to be a low separation level. The higher the level, the higher the separation.
[0059] S150. Based on the feature quality level, the vocal feature tensor and the accompaniment feature tensor are processed by a preset differentiated separation execution scheme to obtain target vocal waveform data and target accompaniment waveform data.
[0060] In some embodiments, the characteristic quality level is a high resolution level, a medium resolution level, or a low resolution level.
[0061] Specifically, the step of processing the vocal feature tensor and the accompaniment feature tensor based on the feature quality level using a preset differentiated separation execution scheme to obtain target vocal waveform data and target accompaniment waveform data includes the following specific implementation process: If the feature quality level is the high separation level, then the inverse short-time Fourier transform is performed on the human voice feature tensor and the accompaniment feature tensor respectively to obtain the target human voice waveform data and the target accompaniment waveform data. If the feature quality level is the medium separation level, then the human voice feature tensor is subjected to high frequency enhancement processing and inverse short-time Fourier transform processing in sequence, and the accompaniment feature tensor is subjected to noise suppression processing and inverse short-time Fourier transform processing in sequence to obtain the target human voice waveform data and the target accompaniment waveform data. If the feature quality level is the low separation level, then based on the amplitude ratio of the overlapping feature components between the vocal feature tensor and the accompaniment feature tensor, the overlapping feature components are allocated to the vocal feature tensor and the accompaniment feature tensor to obtain the allocated vocal feature tensor and the allocated accompaniment feature tensor; the preset multi-head self-attention layer is adjusted, and based on the adjusted preset multi-head self-attention layer, the allocated vocal feature tensor and the allocated accompaniment feature tensor are decoupled to redetermine the feature quality level between the allocated vocal feature tensor and the allocated accompaniment feature tensor.
[0062] Specifically, if the feature has a high separation level, the feature-waveform mapping process is directly enabled: Xvoc and Xacc are input into the Inverse Short Time Fourier Transform (ISTFT) module respectively to restore the initial human voice waveform Wvoc_init and accompaniment waveform Wacc_init, without the need to perform subsequent enhancement and iterative optimization steps.
[0063] Specifically, if the feature has a medium separation level, enable the feature enhancement-waveform mapping process: Perform high-frequency enhancement on Xvoc: extract the feature components with frequencies higher than 2kHz from Xvoc, multiply them by the enhancement coefficient α=1.2, and leave the low-frequency components unchanged; To perform noise suppression on Xacc: construct a noise feature library, identify noise feature components in Xacc through template matching, and attenuate their amplitude to 0.3 times the original amplitude; The enhanced vocal feature Xvoc_enh and the suppressed accompaniment feature Xacc_sup are input into the ISTFT module to obtain the enhanced preliminary waveforms Wvoc_enh and Wacc_sup.
[0064] Specifically, if the feature has a low separation level, enable the iterative optimization-feature re-decoupling process: Extract the overlapping feature components Xoverlap = Xvoc ∩ Xacc from Xvoc and Xacc, and distribute the overlapping components to the two vocal and accompaniment feature tensors according to their amplitude ratios. Feed the distributed feature tensors Xvoc_dis and Xacc_dis back to the preset multi-head self-attention layer, adjust the scaling factor dk of the attention weight matrix in the preset multi-head self-attention layer to 0.7 times the original value, and re-execute the feature decoupling operation. Repeat the fusion separation evaluation process in step S140. If the feature quality level after iteration reaches the medium separation level or above, execute the feature enhancement process at the medium separation level. If the medium separation level or above is not reached after more than 5 iterations, output a separation failure prompt, or restart the corresponding process of the method.
[0065] In summary, this application provides a method for separating vocals and accompaniment, which can achieve high-precision separation of vocals and accompaniment.
[0066] To better implement the above methods, this application also provides a vocal and accompaniment separation device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.
[0067] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the specific integration of the vocal and accompaniment separation device into the terminal as an example.
[0068] For example, such as Figure 2 As shown, the vocal and accompaniment separation device 200 may include a first unit 201, a second unit 202, a third unit 203, a fourth unit 204, and a fifth unit 205. The device includes: The first unit is used to acquire the raw waveform data of the mixed audio; The second unit is used to perform feature extraction processing on the original waveform data to obtain the feature extraction tensor corresponding to the mixed audio. The third unit is used to decouple the feature extraction tensor to obtain the human voice feature tensor and the accompaniment feature tensor. The fourth unit is used to determine the feature quality level based on the vocal feature tensor and the accompaniment feature tensor, wherein the feature quality level is used to characterize the degree of separation between the vocal feature tensor and the accompaniment feature tensor. The fifth unit is used to process the vocal feature tensor and the accompaniment feature tensor based on the feature quality level and through a preset differentiated separation execution scheme to obtain target vocal waveform data and target accompaniment waveform data.
[0069] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.
[0070] As can be seen from the above, the embodiments of this application can achieve high-precision separation of vocals and accompaniment.
[0071] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.
[0072] In some embodiments, the product processing device may also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the vocal and accompaniment separation method of this application.
[0073] In this embodiment, the electronic device will be described in detail as a terminal, for example, such as... Figure 3 As shown, it illustrates a structural schematic diagram of the terminal 300 involved in an embodiment of this application. Specifically: The terminal 300 may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more media, a power supply 303, an input module 304, and a communication module 305. Those skilled in the art will understand that... Figure 3 The terminal 300 structure shown does not constitute a limitation on the terminal 300, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 301 is the control center of the terminal 300. It connects various parts of the terminal 300 via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 302, and by calling data stored in the memory 302, thereby providing overall monitoring of the terminal 300. In some embodiments, the processor 301 may include one or more processing cores; in some embodiments, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 301.
[0074] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal 300, etc. In addition, the memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.
[0075] The terminal 300 also includes a power supply 303 that supplies power to the various components. In some embodiments, the power supply 303 can be logically connected to the processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 303 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.
[0076] The terminal 300 may also include an input module 304, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
[0077] The terminal 300 may also include a communication module 305. In some embodiments, the communication module 305 may include a wireless module. The terminal 300 can perform short-range wireless transmission through the wireless module of the communication module 305, thereby providing users with wireless broadband Internet access. For example, the communication module 305 can be used to help users send and receive emails, browse web pages, and access streaming media.
[0078] Although not shown, terminal 300 may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, processor 301 in terminal 300 loads the executable files corresponding to the processes of one or more applications into memory 302 according to the following instructions, and processor 301 runs the applications stored in memory 302 to realize various functions, as follows: Obtain the raw waveform data of the mixed audio; The original waveform data is subjected to feature extraction processing to obtain the feature extraction tensor corresponding to the mixed audio. The feature extraction tensor is decoupled to obtain the vocal feature tensor and the accompaniment feature tensor; Based on the vocal feature tensor and the accompaniment feature tensor, a feature quality level is determined, wherein the feature quality level is used to characterize the degree of separation between the vocal feature tensor and the accompaniment feature tensor. Based on the aforementioned feature quality level, the vocal feature tensor and the accompaniment feature tensor are processed using a preset differentiated separation execution scheme to obtain target vocal waveform data and target accompaniment waveform data.
[0079] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.
[0080] As can be seen from the above, the embodiments of this application can achieve high-precision separation of vocals and accompaniment through multi-branch dynamic logic design.
[0081] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be accomplished by instructions, or by instructions controlling related hardware. These instructions can be stored in a medium and loaded and executed by a processor.
[0082] Therefore, embodiments of this application provide a medium storing multiple instructions that can be loaded by a processor to execute steps in any of the vocal and accompaniment separation methods provided in this application. For example, the instructions can execute the following steps: Obtain the raw waveform data of the mixed audio; The original waveform data is subjected to feature extraction processing to obtain the feature extraction tensor corresponding to the mixed audio. The feature extraction tensor is decoupled to obtain the vocal feature tensor and the accompaniment feature tensor; Based on the vocal feature tensor and the accompaniment feature tensor, a feature quality level is determined, wherein the feature quality level is used to characterize the degree of separation between the vocal feature tensor and the accompaniment feature tensor. Based on the aforementioned feature quality level, the vocal feature tensor and the accompaniment feature tensor are processed using a preset differentiated separation execution scheme to obtain target vocal waveform data and target accompaniment waveform data.
[0083] The medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0084] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a medium. A processor of a computer device reads the computer instructions from the medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.
[0085] Since the instructions stored in the medium can execute the steps in any of the vocal and accompaniment separation methods provided in the embodiments of this application, the beneficial effects that any of the vocal and accompaniment separation methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.
[0086] The foregoing has provided a detailed description of a method, apparatus, terminal, and medium for separating vocals and accompaniment according to embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for separating vocals and accompaniment, characterized in that, The method includes: Obtain the raw waveform data of the mixed audio; The original waveform data is subjected to feature extraction processing to obtain the feature extraction tensor corresponding to the mixed audio. The feature extraction tensor is decoupled to obtain the vocal feature tensor and the accompaniment feature tensor; Based on the vocal feature tensor and the accompaniment feature tensor, a feature quality level is determined, wherein the feature quality level is used to characterize the degree of separation between the vocal feature tensor and the accompaniment feature tensor. Based on the aforementioned feature quality level, the vocal feature tensor and the accompaniment feature tensor are processed using a preset differentiated separation execution scheme to obtain target vocal waveform data and target accompaniment waveform data.
2. The method as described in claim 1, characterized in that, The step of performing feature extraction processing on the original waveform data to obtain the feature extraction tensor corresponding to the mixed audio includes: The original waveform data is subjected to short-time Fourier transform processing to obtain a spectral feature map; The spectral feature map is convolved by a pre-defined one-dimensional convolutional network to obtain an intermediate convolutional feature map. The intermediate convolutional feature maps are batch normalized by a preset batch normalization network to obtain batch normalized feature maps. Based on the preset element activation rules, the activation feature map is obtained according to the batch normalized feature map; The activated feature map is processed based on a preset pooling kernel to obtain the feature extraction tensor.
3. The method as described in claim 2, characterized in that, The step of obtaining the activation feature map based on the preset element activation rules and the batch normalized feature map includes: The elements greater than 0 in the batch normalized feature map are retained, and the elements less than 0 in the batch normalized feature map are set to 0 to obtain the activation feature map.
4. The method as described in claim 2, characterized in that, The process of processing the activation feature map based on a preset pooling kernel to obtain the feature extraction tensor includes: Calculate the average value of all elements in the activated feature map that are covered by the receptive field corresponding to the preset pooling kernel to obtain the pooled feature value. Based on the pooled feature values, the feature extraction tensor is obtained.
5. The method as described in claim 1, characterized in that, The process of decoupling the feature extraction tensor to obtain the vocal feature tensor and the accompaniment feature tensor includes: The feature extraction tensor is input into a preset multi-head self-attention layer to obtain multi-head self-attention features with the same dimension as the feature extraction tensor. The multi-head self-attention features and the feature extraction tensor are input into a preset residual connection layer to obtain a residual feature tensor with the same dimension as the feature extraction tensor. The residual feature tensor is subjected to depth convolution to obtain a depth convolution feature tensor; The depthwise convolutional features are subjected to pointwise convolution processing to obtain the vocal feature tensor and the accompaniment feature tensor.
6. The method as described in claim 1, characterized in that, The determination of feature quality level based on the vocal feature tensor and the accompaniment feature tensor includes: Based on the human voice feature tensor and the accompaniment feature tensor, the first similarity matrix corresponding to the human voice feature tensor and the second similarity matrix corresponding to the accompaniment feature tensor are calculated respectively. Based on the first similarity matrix and the second similarity matrix, the first global mean corresponding to the first similarity matrix and the second global mean corresponding to the second similarity matrix are calculated respectively. Based on the preset frequency dimension weight, the preset time dimension weight, the first global mean, and the second global mean, the fusion separation index corresponding to the mixed audio is calculated. The feature quality level is determined by comparing the fusion separation index with the preset separation threshold range.
7. The method as described in claim 5, characterized in that, The characteristic quality level is a high resolution level, a medium resolution level, or a low resolution level; Based on the feature quality level, the process of processing the vocal feature tensor and the accompaniment feature tensor using a preset differentiated separation execution scheme to obtain target vocal waveform data and target accompaniment waveform data includes: If the feature quality level is the high separation level, then the inverse short-time Fourier transform is performed on the human voice feature tensor and the accompaniment feature tensor respectively to obtain the target human voice waveform data and the target accompaniment waveform data. If the feature quality level is the medium separation level, then the human voice feature tensor is subjected to high frequency enhancement processing and inverse short-time Fourier transform processing in sequence, and the accompaniment feature tensor is subjected to noise suppression processing and inverse short-time Fourier transform processing in sequence to obtain the target human voice waveform data and the target accompaniment waveform data. If the feature quality level is the low separation level, then based on the amplitude ratio of the overlapping feature components between the vocal feature tensor and the accompaniment feature tensor, the overlapping feature components are allocated to the vocal feature tensor and the accompaniment feature tensor to obtain the allocated vocal feature tensor and the allocated accompaniment feature tensor; the preset multi-head self-attention layer is adjusted, and based on the adjusted preset multi-head self-attention layer, the allocated vocal feature tensor and the allocated accompaniment feature tensor are decoupled to redetermine the feature quality level between the allocated vocal feature tensor and the allocated accompaniment feature tensor.
8. A device for separating vocals from accompaniment, characterized in that, The device includes: The first unit is used to acquire the raw waveform data of the mixed audio; The second unit is used to perform feature extraction processing on the original waveform data to obtain the feature extraction tensor corresponding to the mixed audio. The third unit is used to decouple the feature extraction tensor to obtain the human voice feature tensor and the accompaniment feature tensor. The fourth unit is used to determine the feature quality level based on the vocal feature tensor and the accompaniment feature tensor, wherein the feature quality level is used to characterize the degree of separation between the vocal feature tensor and the accompaniment feature tensor. The fifth unit is used to process the vocal feature tensor and the accompaniment feature tensor based on the feature quality level and through a preset differentiated separation execution scheme to obtain target vocal waveform data and target accompaniment waveform data.
9. A terminal, characterized in that, The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps of the method as described in any one of claims 1 to 7.
10. A medium, characterized in that, The medium stores a plurality of instructions adapted for loading by a processor to execute the steps of the method according to any one of claims 1 to 7.