Music signal separation method and device, storage medium and program product

Abstract

The embodiment of the invention discloses a music signal separation method and device, a storage medium and a program product, and is applied to the technical field of audio processing. Obtaining a frequency domain signal corresponding to the music signal and a music separation model; inputting the frequency domain signal into a music separation model, and executing the steps through the music separation model: segmenting the frequency domain signal to obtain a first feature sequence corresponding to the frequency domain signal; performing sequence modeling on the first feature sequence by using convolution operation to obtain a second feature sequence; the features in the second feature sequence are obtained by capturing dependency of local features in the first feature sequence through convolution operation; based on the second feature sequence, predicting a masking value of a music track division signal corresponding to the music signal; outputting a music track division signal in the music signal based on the frequency domain signal and the masking value; the method does not need to calculate a large number of signal features, and can be applied to a mobile terminal.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio processing technology, and in particular to a music signal separation method, device, storage medium and program product. Background Art

[0002] In existing music software, music signals are often separated to obtain music sub-track signals; for example, music signals are separated into vocal signals and accompaniment signals, or accompaniment signals are separated into instrument signals such as drums, bass, and piano. Common music separation methods are based on the frequency domain: after converting the music signal into a frequency domain signal, it is separated into frequency domain sub-track signals, and then converted into time domain sub-track signals.

[0003] Music signals contain a large number of audio features, and the music separation process needs to calculate and process a large number of signal features at the same time, which easily leads to the need for a large amount of computing resources and high power consumption in the music separation process, making it difficult to apply to mobile terminals with limited resources and low power consumption requirements. Summary of the invention

[0004] The embodiments of the present application provide a music signal separation method, device, storage medium and program product. The music signal separation method does not require calculation and processing of a large number of signal features, can reduce computing resources and power consumption, and can be applied to mobile terminals.

[0005] The present application embodiment provides a music signal separation method, comprising:

[0006] Obtaining a frequency domain signal corresponding to a music signal and a music separation model;

[0007] The frequency domain signal is input into the music separation model, and the music separation model performs the following steps:

[0008] Segmenting the frequency domain signal to obtain a first characteristic sequence corresponding to the frequency domain signal;

[0009] Using a convolution operation to perform sequence modeling on the first feature sequence to obtain a second feature sequence; the features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through the convolution operation;

[0010] Based on the second feature sequence, predicting a masking value of a music sub-track signal corresponding to the music signal;

[0011] Based on the frequency domain signal and the masking value, a music track signal in the music signal is output.

[0012] Furthermore, the method further comprises:

[0013] Acquire an initial music separation model and a preset music separation model, wherein the preset music separation model captures the dependency of global features in the sequence when modeling the sequence;

[0014] The music track signal label is superimposed on the music signal and then inputted into the initial music separation model and the preset music separation model respectively, so as to obtain the first music track signal output by the initial music separation model and the second music track signal output by the preset music separation model;

[0015] Determine a first loss value based on the music track signal label and the first music track signal, and determine a second loss value based on the first music track signal and the second music track signal;

[0016] determining a target loss value based on the first loss value and the second loss value;

[0017] The model parameters of the initial music separation model are updated based on the target loss value until the initial music separation model converges to obtain a trained music separation model.

[0018] Furthermore, the segmenting of the frequency domain signal to obtain a first characteristic sequence corresponding to the frequency domain signal includes:

[0019] Frame the frequency domain signal based on a preset time window to obtain a time dimension sequence composed of multiple audio frame features;

[0020] Alternatively, the frequency domain signal is divided into sub-bands based on frequency points of a preset bandwidth to obtain a frequency domain dimension sequence composed of a plurality of sub-band features.

[0021] Further, the first feature sequence includes: a time dimension sequence;

[0022] The using a convolution operation to perform sequence modeling on the first feature sequence includes:

[0023] Using a convolution operation to perform sequence modeling on the time dimension sequence in the frequency domain dimension, and performing sequence modeling on the modeled time dimension sequence in the time dimension;

[0024] Alternatively, a convolution operation is used to perform sequence modeling on the time dimension sequence, and sequence modeling is performed on the modeled time dimension sequence in the frequency domain dimension.

[0025] Further, the first feature sequence includes: a frequency domain dimension sequence;

[0026] The using a convolution operation to perform sequence modeling on the first feature sequence includes:

[0027] Using a convolution operation to perform sequence modeling on the frequency domain dimension sequence in the time dimension, and performing sequence modeling on the modeled frequency domain dimension sequence in the frequency domain dimension;

[0028] Alternatively, a convolution operation is used to perform sequence modeling on the frequency domain dimension sequence in the frequency domain dimension, and sequence modeling is performed on the modeled frequency domain dimension sequence in the time dimension.

[0029] Furthermore, the predicting, based on the second feature sequence, of a masking value of a music track signal corresponding to the music signal includes:

[0030] Input each feature in the second feature sequence into a normalization layer and a multi-layer perceptron to obtain a masking value corresponding to each feature;

[0031] The masking values ​​corresponding to the multiple features in the second feature sequence are combined to obtain the masking value of the music track signal.

[0032] Further, outputting a music track signal in the music signal based on the frequency domain signal and the masking value includes:

[0033] Multiplying the frequency domain signal by the masking value to obtain a music track frequency domain signal;

[0034] The music sub-track frequency domain signal is subjected to inverse Fourier transform, and the music sub-track signal in the music signal is output.

[0035] The present application also provides a music signal separation device, comprising:

[0036] An acquisition unit, used to acquire a frequency domain signal corresponding to the music signal and a music separation model;

[0037] An input unit is used to input the frequency domain signal into the music separation model, and the music separation model performs the following steps:

[0038] Segmenting the frequency domain signal to obtain a first characteristic sequence corresponding to the frequency domain signal;

[0039] Using a convolution operation to perform sequence modeling on the first feature sequence to obtain a second feature sequence; the features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through the convolution operation;

[0040] Based on the second feature sequence, predicting a masking value of a music sub-track signal corresponding to the music signal;

[0041] Based on the frequency domain signal and the masking value, a music track signal in the music signal is output.

[0042] The present application also provides a music signal separation device, comprising:

[0043] Processor, memory, and input and output interfaces;

[0044] The memory is a short-term storage memory or a persistent storage memory;

[0045] The processor is configured to communicate with the memory and execute instructions in the memory to perform the above method.

[0046] An embodiment of the present application also provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the method described above.

[0047] The embodiment of the present application also provides a computer program product including instructions or a computer program. When the computer program product is run on a computer, the computer executes the method described above.

[0048] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0049] It can be seen that in the embodiment of the present application, the frequency domain signal corresponding to the music signal and the music separation model are obtained, and the music separation model is used to execute: the first feature sequence corresponding to the frequency domain signal is sequence modeled using a convolution operation to obtain a second feature sequence; wherein the features in the second feature sequence are obtained by capturing the dependency of local features in the first feature sequence through a convolution operation; based on the masking value predicted by the second feature sequence, the music track signal in the music signal is obtained; that is, in the first feature sequence, only the local features need to be calculated and processed, which reduces the features of the calculation and processing, and there is no need to calculate and process a large number of signal features, which can reduce computing resources and reduce power consumption, and can be applied to mobile terminals. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0051] Figure 1 A schematic diagram of a communication architecture for separating music signals disclosed in an embodiment of the present application;

[0052] Figure 2 A flowchart of music signal separation disclosed in an embodiment of the present application;

[0053] Figure 3A music signal separation flow chart based on frequency domain dimension sequence disclosed in an embodiment of the present application;

[0054] Figure 4 A music signal separation flow chart based on a time dimension sequence disclosed in an embodiment of the present application;

[0055] Figure 5 A network structure diagram of a BSRNN model disclosed in an embodiment of the present application;

[0056] Figure 6 A network architecture diagram of a music separation model disclosed in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of training a music separation model disclosed in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of a music signal separation device disclosed in an embodiment of the present application;

[0059] Fig. 9 This is a schematic diagram of another music signal separation device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] In the description of the embodiments of the present application, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the embodiments of the present application.

[0062] In the description of the embodiments of the present application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present application can be understood according to specific circumstances.

[0063] Existing,music separation communication architectures such as Figure 1 As shown, it includes: a music signal separation device 101 and a music software 102, wherein the music signal separation device 101 can be connected to multiple music software 102, and the music signal separation device 101 can obtain the music signal through the music software 102 and perform music separation on the music signal; wherein, the music signal stored in the music software 102 can be obtained, or the music signal can be obtained from the server through the music software 102, and the specific details are not limited here.

[0064] In the existing music signal separation device, a music separation model is often used to separate music signals. The corresponding music signal separation process is as follows: after converting the time domain music signal into a frequency domain signal, the frequency domain signal is divided into sub-bands according to the frequency points to form a frequency domain dimension sequence composed of multiple sub-band features; then, the frequency domain dimension sequence is sequence modeled by a sequence modeling operator to separate the music track signal in the frequency domain signal, and then the music track signal in the music signal is obtained. Commonly used music separation models are: BSRNN model based on sub-band segmentation and BSRopeFormer model, and the corresponding sequence modeling operators are LSTM (Long Short-Term Memory Network) and Transformer (Self-Attention Network); wherein, the LSTM operator involves multiple steps in calculation, including input gate, forget gate and output gate, etc., and the global features of the frequency domain dimension sequence need to be used to frequently access and update its internal memory state during calculation; and the Transformer operator also needs to pay attention to the global features of the frequency domain dimension sequence during calculation, and the amount of calculation will increase quadratically with the increase of the context length of the global features. That is, in the process of music separation, LSTM and Transformer need to calculate and process a large number of signal features at the same time when modeling the sequence, which easily leads to the need for a large amount of computing resources and generates high power consumption, making it difficult to apply to mobile terminals with limited resources and low power consumption requirements, that is, the mobile terminal has poor support for LSTM and Transformer terminals. The mobile terminal can be a mobile phone with Android or iOS system, or a smart speaker with Linux system, etc., which is not limited here.

[0065] Therefore, the embodiment of the present application provides a music signal separation method, which does not require calculation and processing of a large number of signal features, can reduce computing resources and power consumption, and can be applied to mobile terminals; Figure 2 As shown, the specific steps include:

[0066] 201. Obtain a frequency domain signal corresponding to the music signal and a music separation model trained based on the music track signal.

[0067] In the embodiment of the present application, the music signal separation device can obtain the frequency domain signal corresponding to the music signal and the music separation model. The music signal is a time domain signal, that is, a signal with time as the independent variable and a signal feature (audio feature) that changes with time. The signal feature can be a feature such as audio amplitude or audio phase. The music signal can be converted into a corresponding frequency domain signal by Fourier transform, that is, a signal with frequency as the independent variable and a signal feature that changes with frequency.

[0068] The music separation model is a separation model trained based on music track signals, and is used to separate music track signals from music signals, wherein the music track signal refers to the audio signal of a single track, which can be: a vocal signal or an accompaniment signal, or a drum set, bass, piano and other instrument signals in the accompaniment signal. Based on the different music track signals used in training, the music separation model separates different music track signals. For example, if a vocal signal is used in training, the trained music separation model can separate the vocal signal from the music signal.

[0069] 202. Input the frequency domain signal into a music separation model, and execute steps 203 to 206 through the music separation model.

[0070] That is, the frequency domain signal corresponding to the music signal can be input into the music separation model, and the music sub-track signal in the music signal can be separated by the music separation model, as follows:

[0071] 203. Divide the frequency domain signal to obtain a first characteristic sequence corresponding to the frequency domain signal.

[0072] In the music separation model, the frequency domain signal can be segmented to obtain a first feature sequence corresponding to the frequency domain signal. The frequency domain signal can be segmented into signals of multiple audio frames, and the features of multiple audio frames form a corresponding frequency domain dimension sequence, or the frequency domain signal can be segmented into signals of multiple sub-bands, and the features of multiple sub-bands form a corresponding time dimension sequence.

[0073] 204. Perform sequence modeling on the first feature sequence using a convolution operation to obtain a second feature sequence.

[0074] Next, the first feature sequence corresponding to the frequency domain signal of the convolution operation can be used to perform sequence modeling to obtain a second feature sequence. Among them, the features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through convolution operation. It can be understood that sequence modeling refers to modeling and updating each feature in the first feature sequence to obtain an updated second feature sequence. When the convolution operation is used for sequence modeling, local features can be used to model each feature; for example, if the first feature sequence after segmentation contains N features, for a certain feature, M features around the feature can be used, and the M features are given corresponding weights and then updated to obtain a new feature after modeling; wherein M is less than N. By using local features to model each feature through convolution operation, the corresponding amount of calculation is greatly reduced compared to modeling using global features in the sequence.

[0075] 205. Based on the second feature sequence, predict a masking value of a music track signal corresponding to the music signal.

[0076] Next, based on the second feature sequence, the masking value of the music track signal corresponding to the music signal can be predicted. Specifically, each feature corresponding to the second feature sequence can be separated, and the masking value of each feature in the second feature sequence can be calculated. By combining the masking values ​​of multiple features in the second feature sequence, the masking value of the music track signal can be obtained. It can be understood that when the frequency domain signal is input into the music separation model trained based on the music track signal, the predicted masking value is the masking value of the music track signal, and the masking value is used to indicate the frequency of the music track signal.

[0077] 205. Output a music track signal in the music signal based on the frequency domain signal and the masking value.

[0078] After predicting the masking value of the music track signal, the music track signal in the music signal can be obtained and output based on the frequency domain signal and the masking value. That is, by masking the frequency corresponding to the masking value in the frequency domain signal, the music track frequency domain signal corresponding to the frequency of the music track signal can be obtained, and the music track frequency domain signal is inverse Fourier transformed to obtain the music track signal (time domain signal) in the music signal.

[0079] Specifically, the frequency domain signal corresponding to the music signal can be multiplied by the masking value to obtain the music track frequency domain signal; the music track frequency domain signal can be inverse Fourier transformed to obtain the music track signal in the music signal, that is, the signal characteristics (amplitude, phase, etc.) of each frequency point in the music track frequency domain signal can be inverse Fourier transformed to obtain the music track signal in the music signal, and the music track signal can be separated from the music signal.

[0080] It can be seen that in the embodiment of the present application, the frequency domain signal corresponding to the music signal and the music separation model are obtained, and the music separation model is used to execute: the first feature sequence corresponding to the frequency domain signal is sequence modeled using a convolution operation to obtain a second feature sequence; wherein the features in the second feature sequence are obtained by capturing the dependency of local features in the first feature sequence through a convolution operation; based on the masking value predicted by the second feature sequence, the music track signal in the music signal is obtained; that is, in the first feature sequence, only the local features need to be calculated and processed, which reduces the features of the calculation and processing, and there is no need to calculate and process a large number of signal features, which can reduce computing resources and reduce power consumption, and can be applied to mobile terminals.

[0081] In an operative manner, the music separation model includes: a segmentation module, a sequence modeling module and a masking value estimation module, and the sequence modeling module includes at least: a convolution module, which is mainly used for sequence modeling through convolution operation, and the convolution module can be a transposed convolution, a grouped convolution, a hole convolution or a depth separation convolution, which is not specifically limited here. By inputting the frequency domain signal corresponding to the music signal into the trained music separation module, the music track signal in the music signal can be separated. That is, the frequency domain signal can be input into the segmentation module, and the above step 203 is executed by the segmentation module to obtain the first feature sequence corresponding to the frequency domain signal; then, the first feature sequence is input into the sequence modeling module, and the above step 204 is executed by the convolution module in the sequence modeling module, and the second feature sequence is obtained by sequence modeling; the second feature sequence is input into the masking value estimation module, and the above step 205 is executed by the masking value estimation module to predict the masking value; the above step 206 is executed based on the frequency domain signal and the masking value, and the music track signal in the music signal is output.

[0082] Furthermore, in the embodiment of the present application, when performing sequence modeling, the frequency domain dimension sequence corresponding to the frequency domain signal can be modeled to process the dependency of the frequency domain signal on the frequency point, such as Figure 3 As shown, the specific steps are as follows:

[0083] 301. Obtain a music separation model trained based on music track separation signals.

[0084] In an embodiment of the present application, the process of training a music separation model based on a music track signal is as follows: obtaining an initial music separation model and a preset music separation model, wherein the preset music separation model captures the dependency of global features in a sequence during sequence modeling. That is, the sequence modeling module of the preset music separation model at least includes: a global sequence modeling operator, which is used to capture the dependency of global features in a sequence; wherein, the preset music separation model may be: a BSRNN model or a BSRopeFormer model, and the corresponding global sequence modeling operator may be: an LSTM or a Transformer, and the global sequence modeling operator in the preset music separation model may be replaced with a convolutional module to obtain an initial music separation model.

[0085] like Figure 5 The figure shows the network structure diagram of the BSRNN model, where A is the overall pipeline, B is the band split module, C is the band and sequence modeling module (i.e., sequence modeling module), and D is the mask estimation module.

[0086] In the frequency band segmentation module, the input is the complex spectrum X∈C of the frequency domain signal generated by Fourier transform of the music signal F×T , F and T are frequency and time dimensions respectively, using a predefined bandwidth Divide the complex spectrum into K sub-band spectra i = 1…K. The real and imaginary parts of each sub-band spectrum are concatenated and sent to the normalization layer (Norm) and the fully connected layer (FC) to generate a sub-band feature Z composed entirely of real numbers. i ∈R N×T , all K sub-band features are merged into the full-band feature vector Z∈R N×K×T , is a three-dimensional matrix vector, that is, a frequency domain dimension sequence.

[0087] In the frequency band and time series modeling module, the feature vector Z of the full frequency band is modeled in the time dimension T through the sequence-level RNN, and then passes through the normalization layer (Norm) in the sequence-level RNN, and then passes through the BLSTM (i.e. LSTM) and the fully connected layer (FC) for sequence modeling. Then, the modeled Z is sequence modeled in the frequency domain dimension K through the band-level RNN, that is, it passes through the normalization layer (Norm) in the band-level RNN, and then passes through the BLSTM (i.e. LSTM) and the fully connected layer (FC) for sequence modeling. The frequency domain dimension sequence finally output by the frequency band and time series modeling module is defined as Q∈R N×K×T .

[0088] In the masking value estimation module, the frequency domain dimension sequence Q outputted at the end is first divided into K groups of sub-band features, and then each group of sub-band features is input into the normalization layer (Norm) and a multi-layer perceptron (MLP) with one hidden layer to obtain the corresponding masking value The real and imaginary parts of i=1…K. The masking values ​​of all sub-band features are combined to obtain the full-band masking value M∈C F×T Finally, multiply the full-band mask value by the complex spectrum X to obtain the target spectrum S∈C F×T .

[0089] In the embodiment of the present application, the LSTM operator in the BSRNN model can be replaced with a convolution module (Conv Block) to obtain an initial music separation model (BSConv), such as Figure 6 As shown, at this time, sequence modeling is performed through a convolution module in the sequence modeling module.

[0090] Next, the music track signal label is superimposed on the music signal and then input into the initial music separation model and the preset music separation model respectively, so as to obtain the first music track signal output by the initial music separation model and the second music track signal output by the preset music separation model; it can be understood that the music track signal label and the output first music track signal and the second music track signal belong to the same track signal. If the vocal signal is used as the label, the output first music track signal and the second music track signal are both vocal signals. A first loss value is determined based on the music track signal label and the first music track signal, and a second loss value is determined based on the first music track signal and the second music track signal; wherein the mean square error formula can be used as the loss function to obtain the loss value. A target loss value is determined based on the first loss value and the second loss value, wherein the first loss value can be multiplied by the weight and then added to the second loss value to obtain the target loss value. The model parameters of the initial music separation model are updated based on the target loss value until the initial music separation model converges to obtain a trained music separation model. Specifically, when the target loss value is less than a preset threshold, it can be determined that the initial music separation model converges. The preset threshold can be understood as, when the target loss value is expressed using JS divergence, the preset threshold can be 0.4 or 0.5, which is not specifically limited here.

[0091] Among them, the training of the music separation model is as follows Figure 7 As shown in the figure, the BSConv small model is the initial music separation model, which is obtained by replacing the LSTM operator with the convolution module in the BSRNN large model. The track data (music track signal labels) are superimposed on the song data (original music signal) and then input into the BSConv small model and the BSRNN large model respectively. Based on the track data and the model output of the BSConv small model, the first loss value is determined. Based on the model output of the BSConv small model and the output of the BSRNN large model, determine the second loss value Get the target loss value L loss The model parameters of the masked value prediction in the BSConv small model are updated through back propagation of the target loss value until the BSConv small model converges to obtain a trained music separation model.

[0092] It can be seen that by fitting the music track signal label with the output of the initial music separation model to construct the first loss value, when updating the model parameters of the initial music separation model, the model parameters of the masking value prediction can be updated, and the masking value estimation module in the trained music separation module predicts the masking value of the music track signal label.

[0093] By fitting the output of the initial music separation model with the output of the preset music separation model to construct a second loss value, the performance of the trained music separation model can be brought close to the level of the preset music separation model. The preset music separation model captures the dependency of global features and can output music track signals more accurately, thereby improving the performance of the music separation model.

[0094] Among them, the trained music separation model can be deployed on a mobile terminal, that is, the music separation device of the embodiment of the present application can be a mobile terminal, which greatly improves the music coverage rate and can perform music signal separation offline on the mobile terminal, reducing the storage cost of music separation.

[0095] 302. Divide the frequency domain signal into sub-bands based on frequency points of a preset bandwidth to obtain a frequency domain dimension sequence composed of multiple sub-band features.

[0096] In an embodiment of the present application, the frequency domain signal can be divided into sub-bands based on the frequency points of the preset bandwidth to obtain a frequency domain dimension sequence composed of multiple sub-band features. The frequency points of the preset bandwidth are multiple adjacent frequency points as one sub-band. For example, if the frequency domain signal contains 1024 frequency points, three adjacent frequency points can be used as one sub-band, and multiple sub-band features can be obtained by dividing, and the corresponding frequency domain dimension sequence is obtained.

[0097] 303. Use convolution operation to perform sequence modeling on the frequency domain dimension sequence in the time dimension and the frequency domain dimension respectively to obtain a second feature sequence.

[0098] That is, in the sequence modeling module of the music separation model, the convolution module can be used to perform sequence modeling on the frequency domain dimension sequence in the time dimension and the frequency domain dimension respectively to obtain a second feature sequence. At this time, the second feature sequence is the frequency domain dimension sequence after sequence modeling.

[0099] Specifically, the convolution operation can be used to perform sequence modeling on the frequency domain dimension sequence in the time dimension, and the modeled frequency domain dimension sequence can be modeled on the frequency domain dimension; that is, first use the convolution module to calculate the local dependency of each sub-band feature in the frequency domain dimension sequence on adjacent audio frames, and then use the convolution module to calculate the local dependency of adjacent sub-band features.

[0100] Alternatively, a convolution operation is used to perform sequence modeling on the frequency domain dimension sequence in the frequency domain dimension, and the modeled frequency domain dimension sequence is then sequence modeled on the time dimension. That is, the convolution module is first used to calculate the local dependency of adjacent sub-band features, and then the convolution module is used to calculate the local dependency of adjacent audio frames for each sub-band feature in the frequency domain dimension sequence.

[0101] 304. Based on the second feature sequence, a masking value of the music track signal is predicted.

[0102] In the embodiment of the present application, the first feature sequence for sequence modeling is a frequency domain dimension sequence, and the second feature sequence obtained after modeling is a frequency domain dimension sequence, and each feature in the corresponding second feature sequence is a sub-band feature. Among them, the masking value estimation module of the music separation model includes: a normalization layer and a multi-layer perceptron, that is, each sub-band feature can be input into the normalization layer and the multi-layer perceptron to obtain the masking value corresponding to each sub-band feature; the masking values ​​corresponding to multiple sub-band features in the second feature sequence are merged to obtain the masking value of the music track signal.

[0103] 305. Output a music track signal in the music signal based on the frequency domain signal and the masking value.

[0104] It is understandable that step 305 is similar to the above step 206, and the details are not repeated here.

[0105] Furthermore, in the embodiment of the present application, when performing sequence modeling, the time dimension sequence (time domain dimension sequence) corresponding to the frequency domain signal can also be modeled to process the time dependency of the frequency domain signal; Figure 4 As shown, the specific steps are as follows:

[0106] 401. Obtain a music separation model trained based on music track separation signals.

[0107] It is understandable that step 401 is similar to the above step 301 and will not be described in detail here.

[0108] 402. Frame the frequency domain signal based on a preset time window to obtain a time dimension sequence composed of multiple audio frame features.

[0109] In an embodiment of the present application, the frequency domain signal can be framed based on a preset time window to obtain a time dimension sequence composed of multiple audio frame features.

[0110] For example, the sampling point n corresponding to the time window can be 1024, the window shift hop_length can be 256, and the window length win_length can be 1024. Therefore, after the frequency domain signal is framed using the window function, the duration of each audio frame is 256 / 16000=16ms. Therefore, when the music signal is 4 minutes, 15,000 audio frames can be divided, and a time dimension sequence composed of 15,000 audio frame features can be obtained.

[0111] 403. Use convolution operation to perform sequence modeling on the time dimension sequence in the time dimension and the frequency domain dimension respectively to obtain a second feature sequence.

[0112] That is, in the sequence modeling module of the music separation model, the convolution module can be used to perform sequence modeling on the time dimension sequence in the time dimension and the frequency domain dimension respectively to obtain a second feature sequence; at this time, the second feature sequence is the time dimension sequence after sequence modeling.

[0113] Specifically, the convolution operation can be used to perform sequence modeling on the time dimension sequence in the frequency domain dimension, and the modeled time dimension sequence can be modeled on the time dimension; that is, first use the convolution module to calculate the local dependency of each audio frame feature in the time dimension sequence on adjacent sub-bands, and then use the convolution module to calculate the local dependency of adjacent audio frame features.

[0114] Alternatively, a convolution operation is used to perform sequence modeling on the time dimension sequence in the time dimension, and the modeled time dimension sequence is then sequence modeled on the frequency domain dimension. That is, the convolution module is first used to calculate the local dependency of adjacent audio frame features, and then the convolution module is used to calculate the local dependency of adjacent sub-bands for each audio frame feature in the time dimension sequence.

[0115] 404. Based on the second feature sequence, a masking value of the music track signal is predicted.

[0116] In the embodiment of the present application, the first feature sequence for sequence modeling is a time dimension sequence, and the second feature sequence obtained after modeling is a time dimension sequence, and each feature in the corresponding second feature sequence is a feature of each audio frame. That is, each audio frame feature can be input into the normalization layer and the multi-layer perceptron to obtain the masking value corresponding to each audio frame feature; the masking values ​​corresponding to multiple audio frame features in the second feature sequence are merged to obtain the masking value of the music track signal.

[0117] 405. Output a music track signal in the music signal based on the frequency domain signal and the masking value.

[0118] It is understandable that step 405 is similar to the above step 206, and the details are not repeated here.

[0119] The present application also provides a music signal separation device. Figure 8 As shown, including:

[0120] An acquisition unit 801 is used to acquire a frequency domain signal corresponding to a music signal and a music separation model;

[0121] The input unit 802 is used to input the frequency domain signal into the music separation model, and the music separation model performs the following steps:

[0122] Segmenting the frequency domain signal to obtain a first characteristic sequence corresponding to the frequency domain signal;

[0123] Using a convolution operation to perform sequence modeling on the first feature sequence to obtain a second feature sequence; the features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through the convolution operation;

[0124] Based on the second feature sequence, predicting a masking value of a music sub-track signal corresponding to the music signal;

[0125] Based on the frequency domain signal and the masking value, a music track signal in the music signal is output.

[0126] The present application embodiment also provides a music signal separation device 900, such as Fig. 9 As shown, the music signal separation device 900 of the embodiment of the present application may include one or more processors CPU (CPU, central processing units) 901 and a memory 902, and the memory 902 stores one or more application programs or data.

[0127] The memory 902 may be a volatile storage or a persistent storage. The program stored in the memory 902 may include one or more modules, each of which may include a series of instruction operations in the electronic device. Furthermore, the processor 901 may be configured to communicate with the memory 902 and execute a series of instruction operations in the memory 902 on the music signal separation device 900.

[0128] The music signal separation device 900 may also include one or more power supplies 905, one or more wired or wireless network interfaces 904, one or more input and output interfaces 903, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0129] The processor 901 can execute the operations performed by the aforementioned specific method embodiment, and the details will not be repeated here.

[0130] An embodiment of the present application also provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the method described above.

[0131] The embodiment of the present application also provides a computer program product including instructions or a computer program. When the computer program product is run on a computer, the computer executes the method described above.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0136] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A music signal separation method, characterized in that: include: Obtaining a frequency domain signal corresponding to a music signal and a music separation model; The frequency domain signal is input into the music separation model, and the music separation model performs the following steps: Segmenting the frequency domain signal to obtain a first characteristic sequence corresponding to the frequency domain signal; Using a convolution operation to perform sequence modeling on the first feature sequence to obtain a second feature sequence; the features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through the convolution operation; Based on the second feature sequence, predicting a masking value of a music sub-track signal corresponding to the music signal; Based on the frequency domain signal and the masking value, a music track signal in the music signal is output.

2. The music signal separation method according to claim 1, characterized in that: The method further comprises: Acquire an initial music separation model and a preset music separation model, wherein the preset music separation model captures the dependency of global features in the sequence when modeling the sequence; The music track signal label is superimposed on the music signal and then inputted into the initial music separation model and the preset music separation model respectively, so as to obtain the first music track signal output by the initial music separation model and the second music track signal output by the preset music separation model; Determine a first loss value based on the music track signal label and the first music track signal, and determine a second loss value based on the first music track signal and the second music track signal; determining a target loss value based on the first loss value and the second loss value; The model parameters of the initial music separation model are updated based on the target loss value until the initial music separation model converges to obtain a trained music separation model.

3. The music signal separation method according to claim 1, characterized in that: The step of dividing the frequency domain signal to obtain a first characteristic sequence corresponding to the frequency domain signal includes: Frame the frequency domain signal based on a preset time window to obtain a time dimension sequence composed of multiple audio frame features; Alternatively, the frequency domain signal is divided into sub-bands based on frequency points of a preset bandwidth to obtain a frequency domain dimension sequence composed of a plurality of sub-band features.

4. The music signal separation method according to claim 1, characterized in that: The first feature sequence includes: a time dimension sequence; The using a convolution operation to perform sequence modeling on the first feature sequence includes: Using a convolution operation to perform sequence modeling on the time dimension sequence in the frequency domain dimension, and performing sequence modeling on the modeled time dimension sequence in the time dimension; Alternatively, a convolution operation is used to perform sequence modeling on the time dimension sequence, and sequence modeling is performed on the modeled time dimension sequence in the frequency domain dimension.

5. The music signal separation method according to claim 1, characterized in that: The first feature sequence includes: a frequency domain dimension sequence; The using a convolution operation to perform sequence modeling on the first feature sequence includes: Using a convolution operation to perform sequence modeling on the frequency domain dimension sequence in the time dimension, and performing sequence modeling on the modeled frequency domain dimension sequence in the frequency domain dimension; Alternatively, a convolution operation is used to perform sequence modeling on the frequency domain dimension sequence in the frequency domain dimension, and sequence modeling is performed on the modeled frequency domain dimension sequence in the time dimension.

6. The music signal separation method according to claim 1, characterized in that: The predicting, based on the second feature sequence, of a masking value of a music sub-track signal corresponding to the music signal comprises: Input each feature in the second feature sequence into a normalization layer and a multi-layer perceptron to obtain a masking value corresponding to each feature; The masking values ​​corresponding to the multiple features in the second feature sequence are combined to obtain the masking value of the music track signal.

7. The music signal separation method according to claim 1, characterized in that: The step of outputting a music track signal in the music signal based on the frequency domain signal and the masking value comprises: Multiplying the frequency domain signal by the masking value to obtain a music track frequency domain signal; The music sub-track frequency domain signal is subjected to inverse Fourier transform, and the music sub-track signal in the music signal is output.

8. A music signal separation device, characterized in that: include: An acquisition unit, used to acquire a frequency domain signal corresponding to the music signal and a music separation model; An input unit is used to input the frequency domain signal into the music separation model, and the music separation model performs the following steps: Segmenting the frequency domain signal to obtain a first characteristic sequence corresponding to the frequency domain signal; Using a convolution operation to perform sequence modeling on the first feature sequence to obtain a second feature sequence; the features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through the convolution operation; Based on the second feature sequence, predicting a masking value of a music sub-track signal corresponding to the music signal; Based on the frequency domain signal and the masking value, a music track signal in the music signal is output.

9. A music signal separation device, characterized in that: include: Processor, memory, and input and output interfaces; The memory is a transient storage memory or a persistent storage memory; The processor is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

11. A computer program product comprising instructions or a computer program, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images