Music source separation method and wearable device

Abstract

This application discloses a music source separation method and wearable device, relating to the field of signal processing technology. It employs a UNet architecture neural network model for music source separation. The neural network model includes at least an encoder and a decoder. The method includes: acquiring the original signal of the music source to be separated; converting the original signal to the frequency domain to obtain a frequency domain signal; performing feature encoding processing on the frequency domain signal through the encoder to obtain encoded feature data, wherein the feature encoding processing includes feature extraction, and the encoder performs feature extraction processing at least by using causal convolutional TFC-TDF blocks; and performing feature decoding processing on the encoded feature data through the decoder to output the music source separation result. This application achieves low-latency music source separation in resource-constrained edge devices.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of signal processing, in particular to a music source separation method and a wearable device. BACKGROUND

[0002] In recent years, with the wide popularity of intelligent terminal devices, end-side music source separation (MSS) technology has gradually become a key entry to improve human-computer interaction experience. Music source separation refers to the technical process of extracting different component audio tracks from a mixed music signal, which has important value in audio enhancement, content creation, interactive entertainment and other applications.

[0003] Music source separation technology has mainly experienced the evolution from traditional methods to deep learning-based technology routes. Early methods are mostly based on traditional machine learning algorithms such as principal component analysis and independent component analysis, which basically assume that different source signals are statistically independent of each other, and try to achieve audio track separation using this property. However, this kind of method has limited separation effect when dealing with real complex music scenes. With the development of deep learning technology, models such as feedforward neural networks, convolutional neural networks and recurrent neural networks are widely used in MSS tasks, and the separation performance is improved through end-to-end learning.

[0004] However, the music source separation method based on deep learning usually has a very high parameter quantity and computational complexity, resulting in a large inference delay, which is difficult to meet the strict requirements of terminal devices for low latency. Therefore, in the resource-limited end-side environment, how to realize low-latency music source separation has become a core bottleneck problem faced by the current technology in the actual landing process. SUMMARY

[0005] The main purpose of the present application is to provide a music source separation method and a wearable device, aiming to solve the technical problem of how to realize low-latency music source separation in a resource-limited situation.

[0006] To achieve the above-mentioned purpose, the present application provides a music source separation method, which uses a neural network model with UNet architecture to perform music source separation. The neural network model at least includes an encoder and a decoder. The music source separation method comprises the following steps:

[0007] Obtaining an original signal of a music source to be separated, and converting the original signal to a frequency domain to obtain a frequency domain signal;

[0008] Performing feature encoding processing on the frequency domain signal through the encoder to obtain encoded feature data, wherein the feature encoding processing includes feature extraction, and the encoder at least performs feature extraction processing by using a TFC-TDF block using causal convolution;

[0009] performing feature decoding processing on the encoded feature data by the decoder, to output a music source separation result.

[0010] In an embodiment, the encoder comprises a first convolutional layer and a feature encoding network connected in sequence, and the feature extraction processing on the frequency domain signal by the encoder is performed to obtain encoded feature data.

[0011] The frequency domain signal is convoluted by the first convolutional layer to obtain low-level feature data.

[0012] The low-level feature data is encoded by the feature encoding network to obtain encoded feature data.

[0013] In an embodiment, the feature encoding processing further comprises downsampling, and the feature encoding network comprises at least one set of a first TFC-TDF block, a first DPRNN block and a downsampling layer connected in sequence, and the feature extraction processing on the low-level feature data by the feature encoding network to obtain encoded feature data comprises:

[0014] In each set of the first TFC-TDF block, the first DPRNN block and the downsampling layer connected in sequence, the first input data is processed by the first TFC-TDF block to obtain first intermediate feature data, wherein when the first TFC-TDF block is the first layer TFC-TDF block of the encoder, the first input data is the low-level feature data, and when the first TFC-TDF block is the intermediate TFC-TDF block of the encoder, the first input data is the processing result output by the downsampling layer of the previous set.

[0015] The first intermediate feature data is processed by the DPRNN block to obtain second intermediate feature data.

[0016] The second intermediate feature data is downsampled by the downsampling layer to obtain downsampled feature data.

[0017] After the downsampling processing of the downsampling layer at the tail layer is completed, the downsampled feature data obtained by the downsampling layer at the tail layer is determined as the encoded feature data.

[0018] In an embodiment, the feature extraction processing on the first intermediate feature data by the first DPRNN block to obtain second intermediate feature data comprises:

[0019] convert the first intermediate feature data into first tensor data, wherein feature dimensions of the first tensor data are respectively a channel number multiplied by a batch number dimension, a time frame number dimension, and a frequency component number dimension;

[0020] perform time series modeling processing on the first tensor data through a unidirectional gated recurrent unit to obtain time series feature data;

[0021] convert the time series feature data into second tensor data, wherein feature dimensions of the second tensor data are respectively a batch number multiplied by a time frame number dimension, a frequency component number dimension, and a channel number dimension;

[0022] perform frequency domain structure modeling processing on the time series feature data through a bidirectional gated recurrent unit to obtain global feature data;

[0023] generate second intermediate feature data based on the global feature data, wherein feature dimensions of the second intermediate feature data are consistent with those of the first intermediate feature data.

[0024] In an embodiment, the first TFC-TDF block comprises a first causal convolution two-dimensional layer, a first full connection layer, a second full connection layer, and a second causal convolution two-dimensional layer connected in sequence, and an input residual of the first full connection layer is connected to an output of the second full connection layer, and an input of the first causal convolution two-dimensional layer is connected to an output of the second causal convolution two-dimensional layer through a two-dimensional convolution layer residual.

[0025] In an embodiment, the decoder comprises a feature decoding network and a second convolution layer, and the step of performing feature decoding processing on the encoded feature data through the decoder to output a music source separation result comprises:

[0026] performing feature decoding processing on the encoded feature data through each group of the feature decoding network in sequence to obtain decoded feature data;

[0027] performing convolution processing on the decoded feature data through the second convolution layer to output the music source separation result.

[0028] In an embodiment, the feature decoding processing comprises feature extraction and up-sampling, and the feature decoding network comprises at least one group of up-sampling layers, a second TFC-TDF block, and a second DPRNN block connected in sequence, and the step of performing feature decoding processing on the encoded feature data through each group of the feature decoding network in sequence to obtain decoded feature data comprises:

[0029] In each group of sequentially connected the up-sampling layer, the second TFC-TDF block and the second DPRNN block, the second input data is up-sampled by the up-sampling layer to obtain up-sampled feature data, wherein when the up-sampling layer is the first layer of the decoder, the second input data is the encoded feature data, and when the up-sampling layer is the intermediate layer of the decoder, the second input data is the processing result output by the second DPRNN block of the previous group;

[0030] The up-sampled feature data is processed by the second TFC-TDF block to obtain third intermediate feature data;

[0031] The third intermediate feature data is processed by the second DPRNN block to obtain fourth intermediate feature data;

[0032] After the second DPRNN block at the tail layer completes the feature extraction processing, the fourth intermediate feature data obtained by the second DPRNN block at the tail layer is determined as the decoded feature data.

[0033] In an embodiment, the neural network model further comprises a DPRNN intermediate layer, and the step of processing the encoded feature data by the decoder to output the music source separation result comprises:

[0034] The encoded feature data is processed by the DPRNN intermediate layer to obtain high-dimensional feature data;

[0035] The high-dimensional feature data is processed by the decoder to output the music source separation result.

[0036] In an embodiment, after the step of processing the encoded feature data by the decoder to output the music source separation result, the method further comprises:

[0037] The music source separation result is processed by inverse Fourier transform to obtain a separated time-domain audio track signal, wherein the time-domain audio track signal comprises a vocal track signal, a drum track signal, a bass track signal and other audio track signals.

[0038] In addition, to achieve the above-mentioned purposes, the present application also provides a wearable device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the music source separation method as described above.

[0039] In addition, to achieve the above object, the present application also provides a readable storage medium, which is a computer readable storage medium, and a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the steps of the music source separation method.

[0040] The present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the music source separation method as described above.

[0041] The one or more technical solutions provided by the present application have at least the following technical effects:

[0042] The present application effectively solves the problem of high computational complexity and long inference delay of the music source separation model based on deep learning by adopting a neural network model based on the UNet architecture and introducing a TFC-TDF (Temporal Frequency Convolution - Time Domain Filtering) block with causal convolution as a feature extraction unit of the encoder. Specifically, after obtaining the original signal of the music source to be separated, the original signal is converted to the frequency domain, and the sparse characteristics and structured features of the audio signal in the frequency domain are fully utilized to reduce the computational burden for subsequent processing. Subsequently, the encoder uses at least a TFC-TDF block integrating causal convolution to extract features from the frequency domain signal. The application of causal convolution ensures that the model only relies on current and historical information, eliminating the dependence on future frames, so that a streaming processing mode of one frame in and one frame out can be realized, thereby greatly reducing the algorithm latency and meeting the requirements of real-time processing. At the same time, the TFC-TDF block can efficiently model the time-frequency feature relationship in the frequency domain, while maintaining the feature expression ability while compressing the computational amount. Finally, the decoder reconstructs and recovers the encoded features. The symmetrical encoder-decoder design of the UNet architecture ensures the integrity of information transmission, and the UNet architecture itself has the characteristics of high parameter sharing degree and clear calculation path. Its symmetrical encoder-decoder structure can efficiently extract and reconstruct multi-level features from the frequency domain signal, significantly reducing the parameter amount and computational burden of the whole model, and is more conducive to deployment on terminal devices. In this way, low-delay music source separation can be achieved in a resource-limited situation. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, the other drawings can be obtained based on these drawings without any creative effort.

[0045] Figure 1 A flowchart of a first embodiment of the music source separation method of the present application;

[0046] Figure 2 A music source separation flowchart related to an embodiment of the music source separation method of the present application;

[0047] Figure 3 A neural network model structure based on UNet architecture related to an embodiment of the music source separation method of the present application;

[0048] Figure 4 A block structure of the TFC-TDF block related to an embodiment of the music source separation method of the present application;

[0049] Figure 5 A feature extraction flowchart of the DPRNN block related to an embodiment of the music source separation method of the present application;

[0050] Figure 6 A device structure of the hardware running environment related to the music source separation method in the embodiments of the present application.

[0051] The purposes, functional features and advantages of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0052] In order to make the above purposes, features and advantages of the present application more apparent and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those of ordinary skill in the art without any creative effort fall within the scope of protection of the present application.

[0053] In recent years, with the popularity of intelligent terminal devices such as smart phones, smart tablets, AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, and the like, end-side music source separation technology has become a key entry for human-computer interaction. Music source separation refers to a technology of separating each component track (such as vocals, drums, bass, and others) from a mixed piece of music. Traditional music source separation systems usually rely on cloud processing, but due to network delay, privacy and security, and power consumption limitations, end-side music source separation technology has gradually become a research hotspot. Early music source separation technology is based on traditional machine learning algorithms, which assume that different source signals are statistically independent, and thus use principal component analysis or independent component analysis to separate different tracks in music. This algorithm does not perform well in real music separation tasks. Deep learning-based methods use feedforward neural networks, convolutional neural networks, and recurrent neural networks to enhance music source separation performance, achieving excellent performance. However, these models have high parameter quantity and computational complexity, and have large algorithm latency, making it difficult to run in real time on end-side devices.

[0054] In practical applications, especially on intelligent terminals, the deployment of music source separation systems is usually limited by computing resources and algorithm latency. Although deep learning-based music source separation technology has excellent performance, it is still necessary to explore its applicability under low latency and low power consumption requirements. Therefore, under limited resources, how to achieve efficient and real-time music source separation remains a challenge that needs to be addressed.

[0055] Based on this, the main solution of the present application is to use a neural network model with UNet architecture for music source separation, the neural network model at least including an encoder and a decoder, obtaining an original signal of a music source to be separated, and converting the original signal to a frequency domain to obtain a frequency domain signal; performing feature encoding processing on the frequency domain signal through the encoder to obtain encoded feature data, wherein the feature encoding processing includes feature extraction, and the encoder at least performs feature extraction processing by using a TFC-TDF block using causal convolution; performing feature decoding processing on the encoded feature data through the decoder to output a music source separation result.

[0056] After obtaining the original signal of the music source to be separated, the original signal is converted to the frequency domain, the sparse characteristics and structured features of the audio signal in the frequency domain are fully utilized, and the computational burden for subsequent processing is reduced. Subsequently, the encoder uses at least a TFC-TDF block integrating causal convolution to extract features from the frequency domain signal. The application of causal convolution ensures that the model only relies on current and historical information, eliminating the dependence on future frames, so that a streaming processing mode of one frame in and one frame out can be realized, thereby greatly reducing the algorithm delay and meeting the real-time processing requirements. At the same time, the TFC-TDF block can efficiently model the time-frequency feature relationship in the frequency domain, while maintaining the feature expression ability while compressing the computational amount. Finally, the decoder reconstructs and recovers the encoded features. The symmetrical encoder-decoder design of the UNet architecture ensures the integrity of information transmission, and the UNet architecture itself has the characteristics of high parameter sharing degree and clear calculation path. The symmetrical encoder-decoder structure can efficiently extract and reconstruct multi-level features from the frequency domain signal, significantly reducing the parameter amount and computational burden of the whole model, and is more conducive to deployment on terminal devices. In this way, low-delay music source separation can be achieved in a resource-limited situation.

[0057] It should be noted that the execution subject of each embodiment of the music source separation method of the present application can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a wearable device capable of realizing the above functions, such as AR glasses, VR glasses, earphones, etc. The music source separation method of each embodiment of the present application does not make specific limitations on this.

[0058] Based on this, the music source separation method of the first embodiment is proposed, which uses a neural network model with a UNet architecture for music source separation. The neural network model at least includes an encoder and a decoder. As shown in Figure 1 The music source separation method includes the following steps S10-S30:

[0059] Step S10, obtaining the original signal of the music source to be separated, and converting the original signal to the frequency domain to obtain a frequency domain signal;

[0060] The original signal of the music source to be separated can be obtained by real-time acquisition through an audio acquisition device or reading from a storage medium. The original signal is usually a time-domain waveform signal and can be in single-channel or multi-channel format. For ease of subsequent description, each embodiment of the present application is described by taking a two-channel signal as an example.

[0061] After obtaining the original signal, the time-domain signal is converted to a frequency-domain representation for subsequent neural network processing to obtain a frequency-domain signal.

[0062] Referring to Figure 2As shown, the embodiment preferably adopts a short-time Fourier transform (STFT) to complete the conversion. The STFT divides the continuous time-domain signal into multiple short-time overlapping time frames by sliding a fixed-length window function (such as a Hamming window) on the time axis, and then performs a discrete Fourier transform (DFT) on each frame of signal to obtain a time-frequency spectrum containing time and frequency information. This conversion represents the non-stationary music signal as a time-varying spectral feature, which is more conducive to separating the structural features of different audio tracks in the frequency domain.

[0063] After STFT conversion, the frequency-domain signal can be represented as a tensor with dimensions C x T x F. Here, C represents the number of channels, T represents the number of time frames, and F represents the number of frequency components. For a two-channel input signal, to fully preserve the phase information, the embodiment decomposes the complex spectrum of each channel into real and imaginary components, resulting in a total of 4 channels: left channel real part, left channel imaginary part, right channel real part, and right channel imaginary part.

[0064] Referring to Figure 2 As shown, the embodiment inputs the obtained 4 x T x F dimensional frequency-domain tensor into a neural network model based on the UNet architecture for subsequent encoding and decoding processing. By explicitly utilizing both the real and imaginary parts of the spectrum, i.e., modeling both the amplitude and phase simultaneously, the time-frequency structure of the audio signal can be more fully preserved, which helps to improve the restoration degree and auditory quality of the final separated audio tracks.

[0065] In step S20, the frequency-domain signal is processed by an encoder to obtain encoded feature data, wherein the feature encoding process includes feature extraction, and the encoder at least uses a TFC-TDF block with causal convolution to perform feature extraction processing;

[0066] The encoder includes one or more TFC-TDF blocks with causal convolution, and each TFC-TDF block includes a time-frequency convolution (TFC) module and a time-frequency-depth (TDF) module. The TFC module uses causal convolution to perform convolution operations on the frequency-domain signal to extract the time-frequency features of the signal. Causal convolution ensures that the signal of the current time frame is processed only by relying on the information of the previous time frames, avoiding the reliance on future frame information. The TDF module further processes the extracted time-frequency features to extract more abstract features and enhance the distinguishability of the features.

[0067] In step S30, the encoded feature data is processed by a decoder to output a music source separation result.

[0068] The encoded feature data is processed by a symmetric decoder structure. The decoder usually contains multiple cascaded feature decoding blocks, each decoding block corresponds to the encoding block in the encoder in structure. Each decoding block receives two inputs: one is the up-sampled output feature from the previous decoding block, and the other is the feature output from the same level in the encoder path through a skip connection. The skip connection mechanism directly transmits the rich detailed information captured in the encoding stage to the decoding process, effectively making up for the fine structure and high frequency components that may be lost in the feature compression process, thereby significantly improving the quality and clarity of the final reconstructed signal.

[0069] The decoder can be up-sampled by transposed convolution or interpolation, etc. to gradually restore the spatial resolution of the feature map and reduce the number of channels. Finally, the output layer of the decoder (usually a convolution layer) maps the reconstructed features to the time-frequency domain estimates of each target sound source (such as vocals, drums, bass and other instruments). For each sound source, its output is a 4xTxF-dimensional tensor, consistent with the dimension of the complex spectrum representation of the input signal, which constitutes the complex spectrum estimate of the model for that sound source. The output of this convolution layer is the music source separation result generated by the neural network model.

[0070] Further, as shown in Figure 2 To obtain playable time-domain audio waveforms, the frequency-domain tensors of each track output by the model (i.e. the music source separation result) can be converted back to time-domain signals via inverse short-time Fourier transform (ISTFT). Figure 2 The process of separating the original mixed signal into four independent tracks of "vocals", "drums", "bass" and "others" is exemplarily shown. In actual applications, the type and number of target sound sources can be flexibly set according to specific needs, and this embodiment does not limit this.

[0071] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and will not be repeated hereinafter. On this basis, the encoder includes a first convolutional layer and a feature encoding network connected in sequence, and the feature extraction processing of the frequency domain signal is performed by the encoder to obtain encoded feature data;

[0072] Step A10, performing convolution processing on the frequency domain signal by the first convolutional layer to obtain low-level feature data;

[0073] The first convolutional layer refers to a convolutional layer in the encoder, and the first convolutional layer performs a convolution operation on the input frequency domain signal using a convolution kernel of a predetermined size (for example, 1*1). This operation aims to preliminarily extract local time-frequency features in the input frequency spectrum, such as edges, textures, and other basic patterns. By adjusting the number of convolution kernels, the dimension transformation and feature compression of the input channel are realized, forming a feature map containing rich low-level semantic information, i.e., the low-level feature data.

[0074] Step A20, performing feature encoding processing on the low-level feature data through the feature encoding network to obtain encoded feature data.

[0075] The feature encoding network includes at least one or more TFC-TDF blocks. Each TFC-TDF block uses causal convolution (Causal Convolution) for feature extraction to capture the harmonic structure in the frequency domain and the time sequence dependence in the time domain, respectively, while ensuring that the model only depends on historical and current information, meeting the low-latency requirement. The entire encoding process reduces the resolution of the feature map while increasing the number of channels through stepwise downsampling (such as using stride convolution or pooling operation), and finally outputs the encoded feature data, providing a strong feature representation for subsequent decoder reconstruction.

[0076] In one possible implementation, the feature encoding processing further includes downsampling, and the feature encoding network includes at least one set of sequentially connected first TFC-TDF block, first DPRNN block, and downsampling layer. The step of performing feature extraction processing on the low-level feature data through the feature encoding network to obtain encoded feature data includes:

[0077] Step B10, in each set of sequentially connected first TFC-TDF block, first DPRNN block, and downsampling layer, performing feature extraction processing on the first input data through the first TFC-TDF block to obtain first intermediate feature data, wherein when the first TFC-TDF block is the first layer TFC-TDF block of the encoder, the first input data is the low-level feature data, and when the first TFC-TDF block is the intermediate TFC-TDF block of the encoder, the first input data is the processing result output by the downsampling layer of the previous set;

[0078] The first TFC-TDF block and the first DPRNN block respectively refer to a TFC-TDF block and a DPRNN (Dual-Path Recurrent Neural Network) block in the encoder.

[0079] Specifically, referring to Figure 4As shown, the TFC-TDF block includes a first causal convolutional 2D layer (CasualConv2D), a first fully connected layer (Fully Connected Layer, FC), a second fully connected layer, and a second causal convolutional 2D layer connected in sequence. The input residual of the first fully connected layer is connected to the output of the second fully connected layer, and the input of the first causal convolutional 2D layer passes through a 2D convolutional layer (…). Figure 4 The diagram shows the output of the 1x1 Conv2D residual connected to the second causal convolutional 2D layer. That is, the first input data is first processed through a causal convolutional 2D layer, then through two fully connected layers for feature transformation, and then through another causal convolutional 2D layer. Simultaneously, the input data is also processed through a 1x1 convolutional 2D layer. Finally, the outputs of these two paths are merged to form the final output.

[0080] The TFC-TDF block uses causal convolution to extract features from the input data and generate first intermediate feature data, which contains richer time-frequency information.

[0081] Step B20: The first intermediate feature data is processed by the DPRNN block to obtain the second intermediate feature data;

[0082] The DPRNN block is a deep recurrent neural network module that can further process the first intermediate feature data, capture the time-series dependencies in the feature data, and thus generate second intermediate feature data. This feature data can better represent the dynamic characteristics of the input signal.

[0083] Step B30: The second intermediate feature data is downsampled through the downsampling layer to obtain downsampled feature data;

[0084] The downsampling layer can employ pooling or other downsampling techniques to downsample the second intermediate feature data, reducing the spatial dimensionality of the feature data while retaining important feature information, thus generating downsampled feature data. Downsampling helps reduce computational complexity and improves the level of feature abstraction.

[0085] Step B40: After the downsampling process is completed in the downsampling layer of the tail layer, the downsampled feature data obtained by the downsampling layer of the tail layer is determined to be coded feature data.

[0086] Through repeated stacking and processing of multiple sets (first TFC-TDF block + first DPRNN block + downsampling layer), the feature map is gradually compressed and abstracted. The downsampled feature data output by the last set of downsampling layers (i.e., the tail layer) is the highly abstract, semantically rich encoded feature data finally produced by the entire feature encoding network. This data will be passed to the decoder for subsequent sound source separation and reconstruction.

[0087] In a possible implementation, the step of performing feature extraction processing on the first intermediate feature data by the first DPRNN block to obtain second intermediate feature data comprises:

[0088] Step C10, converting the first intermediate feature data into first tensor data, wherein the feature dimensions of the first tensor data are respectively a channel number multiplied by a batch number dimension, a time frame number dimension, and a frequency component number dimension;

[0089] Taking the channel number as C, the batch number as B, the time frame number as T, and the frequency component number as F, the first intermediate feature data can be converted into first tensor data with a BC*T*F dimension distribution through a reshaping operation.

[0090] Step C20, performing time series modeling processing on the first tensor data by a unidirectional gated recurrent unit to obtain time series feature data;

[0091] A unidirectional gated recurrent unit (GRU) layer is used to perform forward sequence scanning on the first tensor data along the time step dimension. The unidirectional GRU ensures that its output only depends on the current and past input information, strictly maintaining the causality of the model to meet the low delay constraint required by the end-side real-time processing. The GRU layer captures the dynamic changes and dependencies of the signal in the time dimension, and outputs feature data containing time series context information.

[0092] Step C30, converting the time series feature data into second tensor data, wherein the feature dimensions of the second tensor data are respectively a batch number multiplied by a time frame number dimension, a frequency component number dimension, and a channel number dimension;

[0093] After completing the time dimension modeling, the dimensions of the time series feature data are converted and reshaped to organize them into second tensor data with a BT*F*C dimension distribution. This conversion takes the frequency component number dimension as a new sequence dimension, preparing for subsequent recursive modeling in the frequency domain dimension.

[0094] A bidirectional gated recurrent unit (GRU) layer is used to perform sequence modeling on the second tensor data along the frequency component number dimension. The bidirectional GRU can simultaneously utilize the global frequency context information before and after each frequency component, thereby efficiently modeling the harmonic structure and global frequency spectrum dependency between different frequency components, and outputting feature data containing global frequency domain context information.

[0095] Step C40, performing frequency domain structure modeling processing on the time series feature data by a bidirectional gated recurrent unit to obtain global feature data;

[0096] Step C50, generating second intermediate feature data based on the global feature data, wherein the second intermediate feature data is consistent with the feature dimension of the first intermediate feature data.

[0097] The global feature data processed by the bidirectional GRU is subjected to a final dimension transformation and reshaping operation to restore it to the feature dimension consistent with the input first intermediate feature data, ensuring that the data can seamlessly flow into the subsequent network layer.

[0098] In one possible implementation, the step of generating second intermediate feature data based on the global feature data comprises:

[0099] Step D10, converting the feature dimension of the global feature data to be consistent with the first intermediate feature data to obtain target data;

[0100] Step D20, returning to execute the step of converting the first intermediate feature data into first tensor data by taking the target data as new first intermediate feature data;

[0101] Step D30, until a preset repetition threshold is reached, determining the target data obtained after the preset repetition threshold is reached as the second intermediate feature data.

[0102] Let the preset repetition threshold be L, and this loop processing mechanism encapsulates the above steps C10 to C50 into a block that can be repeatedly executed. By repeating multiple times (for example, 4 or 6 times), the model can perform deeper time series and frequency domain joint optimization on the input features, thereby gradually enhancing its feature expression capability. Finally, the target data obtained after L times of loop processing is determined as the final output of the first DPRNN block, i.e., the second intermediate feature data.

[0103] For example, in a specific implementation, the processing flow of the DPRNN block is as follows Figure 5As shown, first reshape the first intermediate feature data of B*C*T* F dimension distribution into the first tensor data of BC*T*F dimension distribution. In order to ensure that the model does not depend on future frame information, a unidirectional GRU model is used for modeling at this time, followed by a full connection layer (FC) and a layer normalization (Layernorm) process. This module mainly improves the time sequence modeling capability of the music signal. Then, the first tensor data of BC*T*F dimension distribution is subjected to reshape operation, permute operation and reshape operation, so as to obtain the second tensor data of BT*F*C dimension distribution. Since no time sequence information is depended on at this time, a bidirectional GRU model is used for modeling, followed by a full connection layer (FC) and a layer normalization (Layernorm) process. This module mainly improves the local modeling capability of the music signal. Finally, the second tensor data of BT*F*C dimension distribution is subjected to reshape operation and permute operation, so as to restore to the dimension distribution of B*C*T*F. The above process is repeated L times (not shown in the figure), and the target data of B*C*T*F dimension distribution finally obtained is determined as the second intermediate feature data. Figure 5

[0104] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as the above-mentioned embodiments one and two can refer to the above introduction, and the subsequent will not be described in detail. On this basis, the decoder includes a feature decoding network and a second convolutional layer, and the step of performing feature decoding processing on the encoded feature data through the decoder to output the music source separation result includes:

[0105] Step E10, performing feature decoding processing on the encoded feature data through each group of the feature decoding network in turn to obtain decoded feature data;

[0106] The feature decoding network is composed of a plurality of cascaded decoding blocks, and each decoding block corresponds to the processing unit of the corresponding level in the encoder in structure. Each decoding block receives two inputs: one from the output feature of the previous decoding layer, and the other from the feature map of the same level in the encoder path through the skip connection. Through the layer-by-layer processing of the multi-level decoding block, the resolution of the feature map is gradually restored, and finally the decoded feature data containing detailed information is output.

[0107] Step E20, performing convolution processing on the decoded feature data through the second convolutional layer to output the music source separation result.

[0108] ​The second convolutional layer refers to a convolutional layer in the decoder, which maps the high-dimensional decoding feature data output by the decoder to the number of target sound sources. The convolutional layer does not change the spatial resolution (i.e., the number of time frames T and the number of frequency components F) of the feature map, but adjusts the number of channels thereof. For each target sound source (such as vocals, drums, bass, etc.), the output is a tensor consistent with the dimension of the original input frequency domain signal (for example, 4xTxF). The tensor can be directly used as the complex spectrum of the target sound source estimated by the model (i.e., the result of the point multiplication of the mask and the mixed spectrum), and the output of the second convolutional layer is the final music source separation result.

[0109] In a possible implementation, the feature decoding process includes feature extraction and up-sampling, the feature encoding network includes at least one group of sequentially connected up-sampling layers, second TFC-TDF blocks and second DPRNN blocks, and the step of sequentially performing feature decoding on the encoded feature data by each group of the feature decoding network to obtain decoding feature data includes:

[0110] In step F10, the up-sampling layer is used to perform up-sampling processing on the second input data to obtain up-sampled feature data, wherein when the up-sampling layer is the first layer of the decoder, the second input data is the encoded feature data, and when the up-sampling layer is an intermediate layer of the decoder, the second input data is the processing result output by the second DPRNN block of the previous group.

[0111] The second TFC-TDF block and the second DPRNN block respectively refer to a TFC-TDF block and a DPRNN block in the decoder.

[0112] The up-sampling layer can perform a spatial resolution enhancement operation on the input feature through transposed convolution or interpolation algorithm to gradually restore the size of the feature map in the time dimension and the frequency dimension. The operation is the inverse process of the down-sampling process of the encoder, and is designed to expand the highly compressed encoded feature map to a higher resolution to lay a foundation for subsequent detail reconstruction.

[0113] In step F20, the second TFC-TDF block is used to perform feature extraction processing on the up-sampled feature data to obtain third intermediate feature data.

[0114] The structure of the second TFC-TDF block is consistent with that of the first TFC-TDF block, which will not be described here.

[0115] In step F30, the second DPRNN block is used to perform feature extraction processing on the third intermediate feature data to obtain fourth intermediate feature data.

[0116] Similarly, the feature extraction process of the second DPRNN block is consistent with that of the first DPRNN block. The second DPRNN block also adopts a double-path recurrent neural network structure to sequentially model the sequence in the time dimension and the frequency dimension for the input third intermediate feature data. This module further enhances the discriminability of feature expression, effectively integrates global context information, and reconstructs the fine structure of the sound source signal, outputting fourth intermediate feature data containing semantic information.

[0117] Step F40, after the second DPRNN block of the tail layer completes the feature extraction process, the fourth intermediate feature data obtained by the second DPRNN block of the tail layer is determined as the decoding feature data.

[0118] After repeated stacking and processing of multiple groups (up-sampling layer + second TFC-TDF block + second DPRNN block), the resolution of the feature map is gradually restored to a size close to that of the original input frequency domain signal. The fourth intermediate feature data output by the second DPRNN block of the last group (i.e., the tail layer) of the network is the decoding feature data finally generated by the entire feature decoding network, which will be passed to the second convolutional layer to generate the final sound source separation result.

[0119] Based on the first, second and / or third embodiments of the present application, in the fourth embodiment of the present application, the same or similar contents as the above-mentioned embodiments one, two and three can be referred to the above introduction, and will not be repeated hereinafter. On this basis, the neural network model further includes a DPRNN intermediate layer, and the step of performing feature decoding processing on the encoded feature data by the decoder to output a music source separation result, comprising:

[0120] Step G10, performing feature extraction processing on the encoded feature data by the DPRNN intermediate layer to obtain high-dimensional feature data;

[0121] The DPRNN intermediate layer adopts a double-path recurrent neural network structure to receive the encoded feature data output from the encoder and perform feature extraction processing thereon. The feature extraction process thereof can be consistent with that of the first DPRNN block, which will not be repeated here.

[0122] Step G20, performing feature decoding processing on the high-dimensional feature data by the decoder to output a music source separation result.

[0123] The decoder receives high-dimensional feature data enhanced by the intermediate layers of the DPRNN and performs feature decoding and reconstruction through a series of symmetrical decoding networks (such as an upsampling layer + a second TFC-TDF block + a second DPRNN block). Each decoding network gradually recovers the spatial resolution of the feature map through upsampling operations and uses skip connections to fuse feature details from corresponding layers in the encoder path to accurately reconstruct the time-frequency structure of the audio source signal. Finally, the decoder output is mapped to the time-frequency representation of each target audio source through a convolutional layer, which is the final music source separation result of the model.

[0124] For example, to aid in understanding the neural network model based on the U-NET architecture after combining this embodiment with the first, second, and third embodiments described above, a specific embodiment is now provided. In this specific embodiment, refer to... Figure 3 As shown, the neural network model based on the U-NET architecture consists of three modules: encoder, middle block, and decoder. The skip connections between the encoder and decoder are primarily responsible for passing low-level feature information from the encoder to the decoder to ensure reconstruction quality.

[0125] The encoder initially used 1x1 2D convolutional layers (1*1 Conv2D) to increase the number of channels from 4 to C1. Then, stacked D groups ( Figure 3 The diagram shows three groups of TFC-TDF blocks, DPRNN blocks, and downsampled layers. The number of channels is increased from C1 to C2, and then from C2 to C3, while the number of frequency components is decreased from F to F / 2, and then from F / 2 to F / 4, ultimately resulting in encoded feature data with a B*C3*T*F / 4 dimensional distribution.

[0126] The intermediate layer is implemented through DPRNN blocks. The encoded feature data with a B*C3*T*F / 4 dimension distribution is extracted through the DPRNN blocks of the intermediate layer to obtain high-level feature data, which is then input into the decoder.

[0127] The decoder has a similar structure to the encoder, consisting of stacked D groups ( Figure 3 The diagram shows a three-part structure consisting of an upsample layer, a TFC-TDF block, and a DPRNN block. The upsample block is implemented using a two-dimensional transposed convolution, as shown below. Figure 2 As shown, each upsampling increases the F dimension of the features to twice its original value, 2F. Specifically, for B*C*T*F features, the kernel of the 2D transposed convolution is set to (1, 2), and the stride is set to (1, 2). This ensures that the time dimension remains unaffected, thus preserving the causality of the model. Both the TFC-TDF block and the DPRNN block are consistent with the encoder.

[0128] The TFC-TDF block in the first group of the encoder is connected to the DPRNN block in the third group of the decoder by a skip connection, the TFC-TDF block in the second group of the encoder is connected to the DPRNN block in the second group of the decoder by a skip connection, and the TFC-TDF block in the third group of the encoder is connected to the DPRNN block in the first group of the decoder by a skip connection.

[0129] It should be noted that the above examples are only used to assist in understanding the present embodiment, and do not constitute a limitation on the neural network model based on the U-NET architecture of the present embodiment. Further forms of simple transformation based on this technical concept are within the scope of protection of the present application.

[0130] In addition, the present embodiment further proposes a wearable device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the music source separation method as described above.

[0131] Reference Figure 6 which shows a structural schematic diagram of a wearable device suitable for being used to implement the present embodiment. The wearable device in the present embodiment can also include, but is not limited to, AR glasses, VR glasses, earphones, AR headsets, VR headsets, and the like. Figure 6 The wearable device shown is only an example, and should not bring any limitation to the function and use range of the present embodiment.

[0132] As Figure 6As shown, the wearable device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of the wearable device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the wearable device to communicate wirelessly or wired with other devices to exchange data. Although the wearable device with various systems is shown in the figure, it should be understood that all of the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0133] In particular, the processes described above with reference to the flowcharts can be implemented as computer software programs according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0134] The wearable device provided by the embodiments of the present disclosure adopts the music source separation method in the above embodiments, and can solve the technical problem of how to realize low-delay music source separation in a resource-limited case. Compared with the prior art, the wearable device provided by the present disclosure has the same beneficial effects as the music source separation method provided by the above embodiments, and other technical features in the wearable device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0135] It should be understood that portions of the application disclosed can be implemented in hardware, software, firmware, or combinations thereof. In the description of the embodiments above, specific features, structures, materials or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0136] The above description is merely illustrative of the application, and the scope of the application should not be limited thereto. Any variations and modifications of the application that fall within the scope of the claims are to be considered to be within the scope and spirit of the application. Therefore, the scope of the application should be determined not by the embodiments disclosed above, but by the appended claims and their equivalents.

[0137] In addition, to achieve the above object, the embodiment of the application further provides a readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the music source separation method in the above embodiment.

[0138] The computer readable storage medium provided by the embodiment of the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency: radio frequency), etc., or any suitable combination of the above.

[0139] The above computer readable storage medium can be contained in a wearable device; or can exist separately without being assembled into a wearable device.

[0140] The above computer readable storage medium carries one or more programs, which, when executed by the wearable device, cause the wearable device to implement the flow steps of any of the above embodiments.

[0141] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0142] The flow diagrams and the block diagrams in the drawings are meant as methodological and functional description of implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0143] The modules involved in the embodiments of the present application can be implemented in software or hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0144] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above music source separation method, and can solve the technical problem of how to realize low-delay music source separation in a resource-limited situation. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the music source separation method provided by the above embodiments, which will not be described here.

[0145] In addition, the embodiment of the present application further provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the music source separation method as described above.

[0146] The computer program product embodiment of the present application is basically the same as the above-mentioned music source separation method embodiments, and will not be described here.

[0147] It should be noted that in this paper, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or system. Without more limitations, the element defined by the sentence "comprises a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0148] The above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by software plus the necessary general hardware platform, of course, also can be realized by hardware, but in many cases, the former is the better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software sensors, which are stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc) as described above, including a number of instructions to make a wearable device (which can be a mobile phone, computer, server or network device, etc.) execute the methods described in various embodiments of the present application.

[0150] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method of musical source separation, characterized by, The music source separation method adopts a neural network model with a UNet architecture to separate music sources, and the neural network model at least includes an encoder and a decoder. The music source separation method comprises the following steps: An original signal of a music source to be separated is obtained, and the original signal is converted into a frequency domain to obtain a frequency domain signal; The frequency domain signal is processed by the encoder for feature coding to obtain coded feature data, wherein the feature coding process includes feature extraction, and the encoder at least uses a TFC-TDF block with causal convolution to perform feature extraction processing; The coded feature data is processed by the decoder for feature decoding to output a music source separation result; The encoder includes a first convolution layer and a feature coding network connected in sequence, and the frequency domain signal is processed by the encoder for feature extraction to obtain coded feature data; The frequency domain signal is processed by the first convolution layer for convolution to obtain low-level feature data; The low-level feature data is processed by the feature coding network for feature coding to obtain coded feature data; The feature coding process further includes downsampling, and the feature coding network includes at least one set of a first TFC-TDF block, a first DPRNN block and a downsampling layer connected in sequence, and the step of processing the low-level feature data by the feature coding network for feature extraction to obtain coded feature data comprises: In each set of the first TFC-TDF block, the first DPRNN block and the downsampling layer connected in sequence, the first TFC-TDF block is used to process first input data for feature extraction to obtain first intermediate feature data, wherein when the first TFC-TDF block is the first layer TFC-TDF block of the encoder, the first input data is the low-level feature data, and when the first TFC-TDF block is the intermediate TFC-TDF block of the encoder, the first input data is the processing result output by the downsampling layer of the previous set; The first intermediate feature data is processed by the DPRNN block for feature extraction to obtain second intermediate feature data; The second intermediate feature data is processed by the downsampling layer for downsampling to obtain downsampled feature data; After the downsampling layer at the tail layer completes the downsampling processing, the downsampled feature data obtained by the downsampling layer at the tail layer is determined as the coded feature data.

2. The musical source separation method of claim 1, wherein, The step of processing the first intermediate feature data by the first DPRNN block for feature extraction to obtain second intermediate feature data comprises: The first intermediate feature data is converted into first tensor data, wherein the feature dimensions of the first tensor data are respectively the channel number multiplied by the batch number dimension, the time frame number dimension and the frequency component number dimension; The first tensor data is processed by a unidirectional gated recurrent unit for time series modeling to obtain time series feature data; The time series feature data is converted into second tensor data, wherein the feature dimensions of the second tensor data are respectively the batch number multiplied by the time frame number dimension, the frequency component number dimension and the channel number dimension; The time sequence feature data is subjected to frequency domain structure modeling processing by a bidirectional gated recurrent unit to obtain global feature data. Second intermediate feature data is generated based on the global feature data, wherein the second intermediate feature data is consistent with the feature dimension of the first intermediate feature data.

3. The musical source separation method of claim 1, wherein, The first TFC-TDF block comprises a first causal convolution two-dimensional layer, a first full connection layer, a second full connection layer and a second causal convolution two-dimensional layer connected in sequence, and the input residual of the first full connection layer is connected to the output of the second full connection layer, and the input of the first causal convolution two-dimensional layer is connected to the output of the second causal convolution two-dimensional layer through a two-dimensional convolution layer residual.

4. The musical source separation method of claim 1, wherein, The decoder comprises a feature decoding network and a second convolution layer, and the feature decoding processing of the encoded feature data by the decoder to output the music source separation result comprises: The encoded feature data is subjected to feature decoding processing by each group of the feature decoding network in sequence to obtain decoded feature data; The decoded feature data is subjected to convolution processing by the second convolution layer to output the music source separation result.

5. The musical source separation method of claim 4, wherein, The feature decoding processing comprises feature extraction and up-sampling, and the feature encoding network comprises at least one group of up-sampling layers, second TFC-TDF blocks and second DPRNN blocks connected in sequence, and the feature decoding processing of the encoded feature data by each group of the feature decoding network in sequence to obtain decoded feature data comprises: In each group of the up-sampling layers, the second TFC-TDF blocks and the second DPRNN blocks connected in sequence, the second input data is subjected to up-sampling processing by the up-sampling layer to obtain up-sampling feature data, wherein when the up-sampling layer is the first layer of the decoder, the second input data is the encoded feature data, and when the up-sampling layer is the middle layer of the decoder, the second input data is the processing result output by the second DPRNN block of the previous group; The up-sampling feature data is subjected to feature extraction processing by the second TFC-TDF block to obtain third intermediate feature data; The third intermediate feature data is subjected to feature extraction processing by the second DPRNN block to obtain fourth intermediate feature data; After the feature extraction processing of the second DPRNN block of the tail layer is completed, the fourth intermediate feature data obtained by the second DPRNN block of the tail layer is determined as the decoded feature data.

6. The musical source separation method according to any one of claims 1 to 5, wherein, The neural network model further comprises a DPRNN middle layer, and the feature decoding processing of the encoded feature data by the decoder to output the music source separation result comprises: The encoded feature data is subjected to feature extraction processing by the DPRNN middle layer to obtain high-dimensional feature data; The high-dimensional feature data is subjected to feature decoding processing by the decoder to output the music source separation result.

7. The musical source separation method according to any one of claims 1 to 5, wherein, After the feature decoding processing of the encoded feature data by the decoder to output the music source separation result, the method further comprises: The music source separation result is subjected to inverse Fourier transform processing to obtain a separated time-domain audio track signal, wherein the time-domain audio track signal comprises a human voice audio track signal, a drum audio track signal, a bass audio track signal and other audio track signals.

8. A wearable device, comprising: The application also provides a computer readable storage medium storing a computer program product. The application also provides a computer readable storage medium storing a computer program product.

Images