Music signal separation methods, devices, storage media and software products

Abstract

This application discloses a music signal separation method, apparatus, storage medium, and program product for use in the field of audio processing technology. It obtains the frequency domain signal corresponding to the music signal and a music separation model; inputs the frequency domain signal into the music separation model, and executes the following steps through the music separation model: segmenting the frequency domain signal to obtain a first feature sequence corresponding to the frequency domain signal; performing sequence modeling on the first feature sequence using convolution operations to obtain a second feature sequence; the features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through convolution operations; based on the second feature sequence, predicting the masking value of the music track signal corresponding to the music signal; and outputting the music track signal in the music signal based on the frequency domain signal and the masking value. This method eliminates the need for computational processing of a large number of signal features and can be applied to mobile devices.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to music signal separation methods, apparatus, storage media, and program products. Background Technology

[0002] In existing music software, music signals are often separated into individual tracks; for example, the music signal is separated into vocal and accompaniment signals, or the accompaniment signal is separated into instrument signals such as drums, bass, and piano. A common music separation method is based on the frequency domain: after converting the music signal into a frequency domain signal, it is separated into frequency domain tracks, and then converted into time domain tracks.

[0003] Music signals contain a large number of audio features. The music separation process requires simultaneous calculation and processing of a large number of signal features, which can easily lead to a large amount of computing resources and high power consumption during the music separation process, making it difficult to apply to mobile devices with limited resources and low power consumption requirements. Summary of the Invention

[0004] This application provides a music signal separation method, apparatus, storage medium, and program product. The music signal separation method does not require calculation and processing of a large number of signal features, which can reduce computing resources and power consumption, and can be applied to mobile devices.

[0005] This application provides a method for separating music signals, including:

[0006] Obtain the frequency domain signal corresponding to the music signal, as well as the music separation model;

[0007] The frequency domain signal is input into the music separation model, and the music separation model performs the following steps:

[0008] The frequency domain signal is segmented to obtain the first feature sequence corresponding to the frequency domain signal;

[0009] The first feature sequence is modeled using convolution operations to obtain the second feature sequence; the features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through convolution operations.

[0010] Based on the second feature sequence, the masking value of the music track signal corresponding to the music signal is predicted;

[0011] Based on the frequency domain signal and the masking value, the music track signals in the music signal are output.

[0012] Furthermore, the method also includes:

[0013] Obtain an initial music separation model and a preset music separation model, wherein the preset music separation model captures the dependencies of global features in the sequence during sequence modeling;

[0014] After superimposing the music track segmentation signal label onto the music signal, input the initial music separation model and the preset music separation model respectively to obtain the first music track segmentation signal output by the initial music separation model and the second music track segmentation signal output by the preset music separation model;

[0015] A first loss value is determined based on the music track segmentation signal tag and the first music track segmentation signal, and a second loss value is determined based on the first music track segmentation signal and the second music track segmentation signal.

[0016] Determine the target loss value based on the first loss value and the second loss value;

[0017] The model parameters of the initial music separation model are updated based on the target loss value until the initial music separation model converges, thus obtaining the trained music separation model.

[0018] Furthermore, the step of segmenting the frequency domain signal to obtain the first feature sequence corresponding to the frequency domain signal includes:

[0019] The frequency domain signal is framed based on a preset time window to obtain a time-dimensional sequence composed of multiple audio frame features.

[0020] Alternatively, the frequency domain signal can be divided into sub-bands based on a preset bandwidth frequency point to obtain a frequency domain dimension sequence composed of multiple sub-band features.

[0021] Furthermore, the first feature sequence includes: a time-dimensional sequence;

[0022] The step of performing sequence modeling on the first feature sequence using convolution operations includes:

[0023] The time-dimensional sequence is modeled in the frequency domain using convolution operations, and the modeled time-dimensional sequence is modeled in the time dimension.

[0024] Alternatively, convolution operations can be used to perform sequence modeling on the time-dimensional sequence in the time dimension, and the modeled time-dimensional sequence can be performed on the frequency-domain sequence.

[0025] Furthermore, the first feature sequence includes: a frequency domain dimension sequence;

[0026] The step of performing sequence modeling on the first feature sequence using convolution operations includes:

[0027] The frequency domain sequence is modeled in the time dimension using convolution operations, and the modeled frequency domain sequence is modeled in the frequency domain dimension.

[0028] Alternatively, convolution operations can be used to perform sequence modeling on the frequency domain dimension of the frequency domain sequence, and then the modeled frequency domain dimension sequence can be performed on the time dimension.

[0029] Furthermore, the step of predicting the masking value of the music track signal corresponding to the music signal based on the second feature sequence includes:

[0030] Each feature in the second feature sequence is input into a normalization layer and a multilayer perceptron to obtain the masking value corresponding to each feature.

[0031] The masking values ​​corresponding to multiple features in the second feature sequence are merged to obtain the masking value of the music track-by-track signal.

[0032] Furthermore, the step of outputting the music track segmentation signal in the music signal based on the frequency domain signal and the masking value includes:

[0033] Multiplying the frequency domain signal by the masking value yields the music track frequency domain signal;

[0034] Perform an inverse Fourier transform on the frequency domain signal of the music track segment to output the music track segment signal in the music signal.

[0035] This application also provides a music signal separation device, including:

[0036] The acquisition unit is used to acquire the frequency domain signal corresponding to the music signal, as well as the music separation model;

[0037] The input unit is used to input the frequency domain signal into the music separation model, and the music separation model performs the following steps:

[0038] The frequency domain signal is segmented to obtain the first feature sequence corresponding to the frequency domain signal;

[0039] The first feature sequence is modeled using convolution operations to obtain the second feature sequence; the features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through convolution operations.

[0040] Based on the second feature sequence, the masking value of the music track signal corresponding to the music signal is predicted;

[0041] Based on the frequency domain signal and the masking value, the music track signals in the music signal are output.

[0042] This application also provides a music signal separation device, including:

[0043] Processor, memory, and input / output interfaces;

[0044] The memory is either a short-term storage memory or a persistent storage memory;

[0045] The processor is configured to communicate with the memory and execute instructions in the memory to perform the methods described above.

[0046] This application also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described above.

[0047] This application also provides a computer program product containing instructions or computer programs, which, when run on a computer, causes the computer to perform the method described above.

[0048] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0049] As can be seen, in this embodiment, the frequency domain signal corresponding to the music signal and the music separation model are obtained. The music separation model is then used to perform sequence modeling on the first feature sequence corresponding to the frequency domain signal using convolution operations to obtain the second feature sequence. The features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through convolution operations. Based on the masking value predicted by the second feature sequence, the music track-separated signal in the music signal is obtained. That is, only local features need to be calculated in the first feature sequence, reducing the number of features to be calculated. There is no need to calculate a large number of signal features, which can reduce computing resources and power consumption, and can be applied to mobile devices. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0051] Figure 1 This is a schematic diagram of a communication architecture for music signal separation disclosed in an embodiment of this application;

[0052] Figure 2 This is a flowchart of a music signal separation process disclosed in an embodiment of this application;

[0053] Figure 3This is a flowchart of a music signal separation process based on a frequency domain dimension sequence disclosed in an embodiment of this application;

[0054] Figure 4 This is a flowchart of a music signal separation process based on a time-dimension sequence, as disclosed in an embodiment of this application.

[0055] Figure 5 This is a network structure diagram of a BSRNN model disclosed in an embodiment of this application;

[0056] Figure 6 This is a network architecture diagram of a music separation model disclosed in an embodiment of this application;

[0057] Figure 7 This is a training diagram of a music separation model disclosed in an embodiment of this application;

[0058] Figure 8 This is a schematic diagram of a music signal separation device disclosed in an embodiment of this application;

[0059] Figure 9 This is a schematic diagram of another music signal separation device disclosed in an embodiment of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] In the description of the embodiments of this application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0062] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application based on the specific circumstances.

[0063] Existing music-separated communication architectures, such as Figure 1 As shown, it includes: a music signal separation device 101 and music software 102. The music signal separation device 101 can be connected to multiple music software programs 102. The music signal separation device 101 can obtain music signals through the music software 102 and perform music separation on the music signals. Specifically, it can obtain music signals stored in the music software 102, or it can obtain music signals from a server through the music software 102. The specific method is not limited here.

[0064] Existing music signal separation devices often employ music separation models for music signal separation. The corresponding music signal separation process is as follows: after converting the time-domain music signal into a frequency-domain signal, the frequency-domain signal is segmented into sub-bands according to frequency points, forming a frequency-domain dimensional sequence composed of multiple sub-band features. Then, sequence modeling operators are used to perform sequence modeling on the frequency-domain dimensional sequence to separate the music track signals in the frequency-domain signal, thereby obtaining the music track signals in the music signal. Commonly used music separation models are the BSRNN model based on sub-band segmentation and the BSROPeFormer model, with corresponding sequence modeling operators being LSTM (Long Short-Term Memory) network and Transformer (Self-Attention Network). Among them, the LSTM operator involves multiple steps during computation, including input gate, forget gate, and output gate, and requires frequent access to and updating of its internal memory state using the global features of the frequency-domain dimensional sequence during computation. The Transformer operator also requires applying attention to the global features of the frequency-domain dimensional sequence during computation, and the computational load increases quadratically with the increase of the context length of the global features. In music separation, LSTM and Transformer require simultaneous computation and processing of numerous signal features during sequence modeling, leading to high computational resource requirements and significant power consumption. This makes them unsuitable for mobile devices with limited resources and low power consumption needs; in other words, mobile devices have poor support for LSTM and Transformer interfaces. These mobile devices can be Android or iOS phones, or Linux-based smart speakers, etc., but are not specifically limited here.

[0065] Therefore, this application provides a music signal separation method that does not require computational processing of a large number of signal features, thereby reducing computational resources and power consumption, and can be applied to mobile devices; such as Figure 2 As shown, the specific steps include the following:

[0066] 201. Obtain the frequency domain signal corresponding to the music signal and the music separation model trained based on the music track signal.

[0067] In this embodiment, the music signal separation device can acquire the frequency domain signal corresponding to the music signal, as well as the music separation model. The music signal is a time-domain signal, meaning it is a signal whose signal characteristics (audio characteristics) change with time, with time as the independent variable. These signal characteristics can be audio amplitude or audio phase, etc. The music signal can be converted into a corresponding frequency domain signal, meaning it is a signal whose signal characteristics change with frequency, with frequency as the independent variable, through Fourier transform.

[0068] This music separation model is trained based on music track signals and is used to separate music track signals from a music signal. These music track signals refer to the audio signals of a single track, which can be vocal signals, accompaniment signals, or instrument signals such as drums, bass, and piano within the accompaniment signal. Depending on the music track signals used during training, the music separation model will separate different music track signals. For example, if vocal signals are used during training, the trained music separation model can separate the vocal signals from the music signal.

[0069] 202. Input the frequency domain signal into the music separation model, and execute steps 203 to 206 through the music separation model.

[0070] The frequency domain signal corresponding to the music signal can be input into the music separation model, and the music separation model can separate the music track signals from the music signal, as follows:

[0071] 203. Segment the frequency domain signal to obtain the first feature sequence corresponding to the frequency domain signal.

[0072] In the music separation model, the frequency domain signal can be segmented to obtain the corresponding first feature sequence. Specifically, the frequency domain signal can be segmented into signals of multiple audio frames, with the features of these multiple audio frames forming the corresponding frequency domain dimension sequence; alternatively, the frequency domain signal can be segmented into signals of multiple sub-bands, with the features of these multiple sub-bands forming the corresponding time dimension sequence.

[0073] 204. Use convolution operations to perform sequence modeling on the first feature sequence to obtain the second feature sequence.

[0074] Next, sequence modeling can be performed using the first feature sequence corresponding to the frequency domain signal obtained through convolution operations to obtain the second feature sequence. The features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through convolution operations. It can be understood that sequence modeling refers to modeling and updating each feature in the first feature sequence to obtain the updated second feature sequence. In sequence modeling, convolution operations can use local features for each feature; for example, if the segmented first feature sequence contains N features, for a certain feature, the surrounding M features can be used, and the feature can be updated by assigning corresponding weights to the M features, resulting in a new modeled feature; where M is less than N. Modeling each feature using local features through convolution operations significantly reduces the computational cost compared to modeling using global features in the sequence.

[0075] 205. Based on the second feature sequence, the masking value of the music track signal corresponding to the music signal is predicted.

[0076] Next, based on the second feature sequence, the masking value of the music track signal corresponding to the music signal can be predicted. Specifically, each feature corresponding to the second feature sequence can be identified, and the masking value of each feature in the second feature sequence can be calculated. By merging the masking values ​​of multiple features in the second feature sequence, the masking value of the music track signal can be obtained. It can be understood that inputting the frequency domain signal into the music separation model trained based on the music track signal, the predicted masking value is the masking value indicating the frequency at which the music track signal exists.

[0077] 205. Based on the frequency domain signal and the masking value, output the music track signals in the music signal.

[0078] After predicting the masking value of the music track segment signal, the music track segment signal in the music signal can be obtained and output based on the frequency domain signal and the masking value. That is, by masking the frequency corresponding to the masking value in the frequency domain signal, the music track segment frequency domain signal corresponding to the frequency of the music track segment signal can be obtained. The music track segment frequency domain signal is then subjected to inverse Fourier transform to obtain the music track segment signal (time domain signal) in the music signal.

[0079] Specifically, the frequency domain signal corresponding to the music signal can be multiplied by the masking value to obtain the music track frequency domain signal; the music track frequency domain signal can be obtained by performing an inverse Fourier transform on the music track frequency domain signal, that is, by performing an inverse Fourier transform on the signal characteristics (amplitude, phase, etc.) of each frequency point in the music track frequency domain signal to obtain the music track signal in the music signal, thus separating the music track signal from the music signal.

[0080] As can be seen, in this embodiment, the frequency domain signal corresponding to the music signal and the music separation model are obtained. The music separation model is then used to perform sequence modeling on the first feature sequence corresponding to the frequency domain signal using convolution operations to obtain the second feature sequence. The features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through convolution operations. Based on the masking value predicted by the second feature sequence, the music track-separated signal in the music signal is obtained. That is, only local features need to be calculated in the first feature sequence, reducing the number of features to be calculated. There is no need to calculate a large number of signal features, which can reduce computing resources and power consumption, and can be applied to mobile devices.

[0081] In one feasible approach, the music separation model includes a segmentation module, a sequence modeling module, and a masking value estimation module. The sequence modeling module includes at least a convolution module, which is mainly used for sequence modeling through convolution operations. This convolution module can be a transposed convolution, grouped convolution, dilated convolution, or depthwise segregating convolution, and the specific type is not limited here. By inputting the frequency domain signal corresponding to the music signal into the trained music separation module, the music track signals in the music signal can be separated. Specifically, the frequency domain signal can be input into the segmentation module, and the segmentation module performs step 203 to obtain the first feature sequence corresponding to the frequency domain signal. Then, the first feature sequence is input into the sequence modeling module, and the convolution module in the sequence modeling module performs step 204 to obtain the second feature sequence. The second feature sequence is input into the masking value estimation module, and the masking value is predicted by performing step 205. Based on the frequency domain signal and the masking value, step 206 is performed to output the music track signals in the music signal.

[0082] Furthermore, in the embodiments of this application, when performing sequence modeling, the frequency domain dimension sequence corresponding to the frequency domain signal can be modeled to handle the frequency domain signal's dependence on frequency points, such as... Figure 3 As shown, the specific steps are as follows:

[0083] 301. Obtain the music separation model trained based on the music track signals.

[0084] In this embodiment, the process of training a music separation model based on music track-segmentation signals is as follows: An initial music separation model and a preset music separation model are obtained. The preset music separation model captures the dependencies of global features in the sequence during sequence modeling. Specifically, the sequence modeling module of the preset music separation model includes at least a global sequence modeling operator, which is used to capture the dependencies of global features in the sequence. The preset music separation model can be a BSRNN model or a BSRNN-Former model, and the corresponding global sequence modeling operator can be an LSTM or a Transformer. The global sequence modeling operator in the preset music separation model can be replaced with a convolutional module to obtain the initial music separation model.

[0085] like Figure 5 The diagram shows the network structure of the BSRNN model, where A is the overall pipeline, B is the band split module, C is the band and sequence modeling module (i.e., the sequence modeling module), and D is the mask estimation module.

[0086] In the frequency band segmentation module, the input is the complex spectrum X∈C of the frequency domain signal generated by the Fourier transform of the music signal. F×T F and T are the frequency and time dimensions, respectively, using a predefined bandwidth. Divide the complex spectrum into K sub-band spectra. i = 1…K. The real and imaginary parts of the spectrum of each subband are concatenated and fed into a normalization layer (Norm) and a fully connected layer (FC) to produce a subband feature Z consisting entirely of real numbers. i ∈R N×T All K sub-band features are merged into a full-band feature vector Z∈R. N×K×T , is a three-dimensional matrix vector, that is, a frequency domain dimension sequence.

[0087] In the frequency band and time series modeling module, a sequence-level RNN is used to model the feature vector Z of the entire frequency band in the time dimension T. This sequence-level RNN passes through a normalization layer (Norm), followed by a BLSTM (i.e., LSTM) and a fully connected layer (FC) for sequence modeling. Then, a band-level RNN is used to model the modeled Z in the frequency domain dimension K, again passing through a normalization layer (Norm), followed by a BLSTM (i.e., LSTM) and a fully connected layer (FC) for sequence modeling. The final frequency domain sequence output by the frequency band and time series modeling module is defined as Q∈R. N×K×T .

[0088] In the masking value estimation module, the final output frequency domain sequence Q is first divided into K groups of sub-band features. Then, each group of sub-band features is input into a normalization layer (Norm) and a multilayer perceptron (MLP) with one hidden layer to obtain the corresponding masking value. The real and imaginary parts of i = 1…K. The masking values ​​of all sub-band features are combined to obtain the full-band masking value M∈C. F×T Finally, multiplying this full-band masking value by the complex spectrum X yields the target spectrum S∈C. F×T .

[0089] In this embodiment, the LSTM operator in the BSRNN model can be replaced with a convolutional block to obtain the initial music separation model (BSConv), as follows: Figure 6 As shown, at this point, sequence modeling is performed using a convolution module in the sequence modeling module.

[0090] Next, the music track segmentation signal labels are superimposed on the music signals and input into the initial music separation model and the preset music separation model, respectively, to obtain the first music track segmentation signal output by the initial music separation model and the second music track segmentation signal output by the preset music separation model. It can be understood that the music track segmentation signal label belongs to the same track segmentation signal as the output first and second music track segmentation signals. If the human voice signal is used as the label, then both the output first and second music track segmentation signals are human voice signals. A first loss value is determined based on the music track segmentation signal label and the first music track segmentation signal, and a second loss value is determined based on the first and second music track segmentation signals; the mean squared error formula can be used as the loss function to obtain the loss value. A target loss value is determined based on the first and second loss values, which can be obtained by multiplying the first loss value by the weight and then adding the second loss value. The model parameters of the initial music separation model are updated based on the target loss value until the initial music separation model converges, resulting in a trained music separation model. Specifically, when the target loss value is less than a preset threshold, the initial music separation model can be considered to have converged. This preset threshold can be understood as 0.4 or 0.5 when the target loss value is represented by JS divergence, and is not limited here.

[0091] The training of the music separation model is as follows: Figure 7 As shown, the BSConv small model is the initial music separation model, obtained by replacing the LSTM operator with a convolutional module in the BSRNN large model. The track-separated data (music track signal tags) are superimposed on the song data (original music signal) and then input into the BSConv small model and the BSRNN large model respectively. Based on the track-separated data and the model output of the BSConv small model, the first loss value is determined. The second loss value is determined based on the output of the BSConv small model and the output of the BSRNN large model. Obtain the target loss value L loss The model parameters for masking value prediction in the BSConv mini-model are updated by backpropagation using the target loss value until the BSConv mini-model converges, resulting in a trained music separation model.

[0092] As can be seen, by fitting the music track segmentation signal labels with the output of the initial music separation model to construct the first loss value, the masking value prediction model parameters can be updated when updating the model parameters of the initial music separation model. The masking value estimation module in the trained music separation module predicts the masking value of the music track segmentation signal labels.

[0093] By fitting the output of the initial music separation model to the output of the preset music separation model to construct a second loss value, the performance of the trained music separation model can be made close to that of the preset music separation model. The preset music separation model captures the dependence of global features and can output the music track separation signal more accurately, thus improving the performance of the music separation model.

[0094] The trained music separation model can be deployed on a mobile device, meaning the music separation device in this application can be a mobile device. This greatly improves music coverage and allows for offline music signal separation on the mobile device, reducing the storage cost of music separation.

[0095] 302. Divide the frequency domain signal into sub-bands based on the frequency points of the preset bandwidth to obtain a frequency domain dimension sequence composed of multiple sub-band features.

[0096] In this embodiment, the frequency domain signal can be divided into sub-bands based on a preset bandwidth frequency point to obtain a frequency domain dimension sequence composed of multiple sub-band features. The preset bandwidth frequency point is defined as multiple adjacent frequency points forming a sub-band. For example, if the frequency domain signal contains 1024 frequency points, three adjacent frequency points can be divided into a sub-band to obtain multiple sub-band features and the corresponding frequency domain dimension sequence.

[0097] 303. Use convolution operations to model the frequency domain sequence in both the time and frequency domain dimensions to obtain the second feature sequence.

[0098] In the sequence modeling module of the music separation model, the convolution module can be used to perform sequence modeling on the frequency domain dimension sequence in both the time and frequency domain dimensions to obtain the second feature sequence. At this time, the second feature sequence is the frequency domain dimension sequence after sequence modeling.

[0099] Specifically, convolution operations can be used to perform sequence modeling on the frequency domain dimension sequence in the time dimension, and then to perform sequence modeling on the frequency domain dimension sequence after modeling. That is, firstly, the convolution module is used to calculate the local dependencies of each sub-band feature in the frequency domain dimension sequence on the adjacent audio frames, and then the convolution module is used to calculate the local dependencies of the adjacent sub-band features.

[0100] Alternatively, convolution operations can be used to model the frequency-domain sequence in the frequency domain dimension, and then the modeled frequency-domain sequence can be modeled in the time dimension. That is, first use the convolution module to calculate the local dependencies of adjacent sub-band features, and then use the convolution module to calculate the local dependencies of each sub-band feature in the frequency-domain sequence on adjacent audio frames.

[0101] 304. Based on the second feature sequence, the masking value of the music track signal is predicted.

[0102] In this embodiment, the first feature sequence for sequence modeling is a frequency domain sequence, and the second feature sequence obtained after modeling is also a frequency domain sequence. Each feature in the second feature sequence is a sub-band feature. The masking value estimation module of the music separation model includes a normalization layer and a multilayer perceptron. Each sub-band feature can be input into the normalization layer and the multilayer perceptron to obtain the masking value corresponding to each sub-band feature. The masking values ​​corresponding to multiple sub-band features in the second feature sequence are merged to obtain the masking value of the music track-separated signal.

[0103] 305. Based on the frequency domain signal and the masking value, output the music track signals in the music signal.

[0104] It is understandable that step 305 is similar to step 206 above, and will not be described in detail here.

[0105] Furthermore, in the embodiments of this application, when performing sequence modeling, the time-dimensional sequence (time-domain sequence) corresponding to the frequency-domain signal can also be modeled to handle the time dependency of the frequency-domain signal; such as Figure 4 As shown, the specific steps are as follows:

[0106] 401. Obtain the music separation model trained based on the music track signals.

[0107] It is understandable that step 401 is similar to step 301 above, and the details will not be repeated here.

[0108] 402. The frequency domain signal is divided into frames based on a preset time window to obtain a time-dimensional sequence composed of multiple audio frame features.

[0109] In this embodiment of the application, the frequency domain signal can be divided into frames based on a preset time window to obtain a time-dimensional sequence composed of multiple audio frame features.

[0110] For example, the sampling points n corresponding to this time window can be 1024, the window shift hop_length can be 256, and the window length win_length can be 1024. Therefore, after using this window function to divide the frequency domain signal into frames, the duration of each audio frame is 256 / 16000 = 16ms. Thus, when the music signal is 4 minutes long, it can be divided into 15000 audio frames, resulting in a time-dimensional sequence composed of the features of 15000 audio frames.

[0111] 403. Use convolution operations to model the time-dimensional sequence in both the time and frequency domains to obtain the second feature sequence.

[0112] In the sequence modeling module of the music separation model, the convolution module can be used to perform sequence modeling on the time dimension sequence in both the time dimension and the frequency domain dimension to obtain the second feature sequence; at this time, the second feature sequence is the time dimension sequence after sequence modeling.

[0113] Specifically, convolution operations can be used to model the time-dimensional sequence in the frequency domain, and then the modeled time-dimensional sequence can be modeled in the time dimension. That is, firstly, the convolution module is used to calculate the local dependencies of adjacent sub-bands for each audio frame feature in the time-dimensional sequence, and then the convolution module is used to calculate the local dependencies of adjacent audio frame features.

[0114] Alternatively, convolution operations can be used to model the time-dimension sequence in the time dimension, and then the modeled time-dimension sequence can be modeled in the frequency domain. That is, first, the convolution module is used to calculate the local dependencies of adjacent audio frame features, and then the convolution module is used to calculate the local dependencies of adjacent subbands for each audio frame feature in the time-dimension sequence.

[0115] 404. Based on the second feature sequence, the masking value of the music track signal is predicted.

[0116] In this embodiment, the first feature sequence for sequence modeling is a time-dimensional sequence, and the second feature sequence obtained after modeling is also a time-dimensional sequence. Each feature in the second feature sequence is a feature of each audio frame. That is, each audio frame feature can be input into a normalization layer and a multilayer perceptron to obtain the masking value corresponding to each audio frame feature; the masking values ​​corresponding to multiple audio frame features in the second feature sequence are then merged to obtain the masking value of the music track-by-track signal.

[0117] 405. Based on the frequency domain signal and the masking value, output the music track signals in the music signal.

[0118] It is understandable that step 405 is similar to step 206 above, and the details will not be repeated here.

[0119] This application also provides a music signal separation device, such as... Figure 8 As shown, it includes:

[0120] The acquisition unit 801 is used to acquire the frequency domain signal corresponding to the music signal and the music separation model;

[0121] Input unit 802 is used to input the frequency domain signal into the music separation model, and the music separation model performs the following steps:

[0122] The frequency domain signal is segmented to obtain the first feature sequence corresponding to the frequency domain signal;

[0123] The first feature sequence is modeled using convolution operations to obtain the second feature sequence; the features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through convolution operations.

[0124] Based on the second feature sequence, the masking value of the music track signal corresponding to the music signal is predicted;

[0125] Based on the frequency domain signal and the masking value, the music track signals in the music signal are output.

[0126] This application also provides a music signal separation device 900, such as... Figure 9 As shown, the music signal separation device 900 of this application embodiment may include one or more processors CPU (CPU, central processing unit) 901 and memory 902, wherein the memory 902 stores one or more applications or data.

[0127] The memory 902 can be volatile or persistent storage. The program stored in the memory 902 can include one or more modules, each module including a series of instruction operations on the electronic device. Furthermore, the processor 901 can be configured to communicate with the memory 902 and execute the series of instruction operations in the memory 902 on the music signal separation device 900.

[0128] The music signal separation device 900 may also include one or more power supplies 905, one or more wired or wireless network interfaces 904, one or more input / output interfaces 903, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0129] The processor 901 can perform the operations performed in the aforementioned specific method embodiments, which will not be described in detail here.

[0130] This application also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described above.

[0131] This application also provides a computer program product containing instructions or computer programs, which, when run on a computer, causes the computer to perform the method described above.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for separating music signals, characterized in that, include: Obtain the frequency domain signal corresponding to the music signal, as well as the music separation model; The frequency domain signal is input into the music separation model, and the music separation model performs the following steps: The frequency domain signal is segmented to obtain the first feature sequence corresponding to the frequency domain signal; The first feature sequence is modeled using convolution operations to obtain the second feature sequence; the features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through convolution operations. Based on the second feature sequence, the masking value of the music track signal corresponding to the music signal is predicted; Based on the frequency domain signal and the masking value, the music track signals in the music signal are output.

2. The music signal separation method according to claim 1, characterized in that, The method further includes: Obtain an initial music separation model and a preset music separation model, wherein the preset music separation model captures the dependencies of global features in the sequence during sequence modeling; After superimposing the music track segmentation signal label onto the music signal, input the initial music separation model and the preset music separation model respectively to obtain the first music track segmentation signal output by the initial music separation model and the second music track segmentation signal output by the preset music separation model; A first loss value is determined based on the music track segmentation signal tag and the first music track segmentation signal, and a second loss value is determined based on the first music track segmentation signal and the second music track segmentation signal. Determine the target loss value based on the first loss value and the second loss value; The model parameters of the initial music separation model are updated based on the target loss value until the initial music separation model converges, thus obtaining the trained music separation model.

3. The music signal separation method according to claim 1, characterized in that, The step of segmenting the frequency domain signal to obtain the first feature sequence corresponding to the frequency domain signal includes: The frequency domain signal is framed based on a preset time window to obtain a time-dimensional sequence composed of multiple audio frame features. Alternatively, the frequency domain signal can be divided into sub-bands based on a preset bandwidth frequency point to obtain a frequency domain dimension sequence composed of multiple sub-band features.

4. The music signal separation method according to claim 1, characterized in that, The first feature sequence includes: a time-dimensional sequence; The step of performing sequence modeling on the first feature sequence using convolution operations includes: The time-dimensional sequence is modeled in the frequency domain using convolution operations, and the modeled time-dimensional sequence is modeled in the time dimension. Alternatively, convolution operations can be used to perform sequence modeling on the time-dimensional sequence in the time dimension, and the modeled time-dimensional sequence can be performed on the frequency-domain sequence.

5. The music signal separation method according to claim 1, characterized in that, The first feature sequence includes: a frequency domain dimension sequence; The step of performing sequence modeling on the first feature sequence using convolution operations includes: The frequency domain sequence is modeled in the time dimension using convolution operations, and the modeled frequency domain sequence is modeled in the frequency domain dimension. Alternatively, convolution operations can be used to perform sequence modeling on the frequency domain dimension of the frequency domain sequence, and then the modeled frequency domain dimension sequence can be performed on the time dimension.

6. The music signal separation method according to claim 1, characterized in that, The step of predicting the masking value of the music track signal corresponding to the music signal based on the second feature sequence includes: Each feature in the second feature sequence is input into a normalization layer and a multilayer perceptron to obtain the masking value corresponding to each feature. The masking values ​​corresponding to multiple features in the second feature sequence are merged to obtain the masking value of the music track-by-track signal.

7. The music signal separation method according to claim 1, characterized in that, The step of outputting the music track segmentation signal in the music signal based on the frequency domain signal and the masking value includes: Multiplying the frequency domain signal by the masking value yields the music track frequency domain signal; Perform an inverse Fourier transform on the frequency domain signal of the music track segment to output the music track segment signal in the music signal.

8. A music signal separation device, characterized in that, include: The acquisition unit is used to acquire the frequency domain signal corresponding to the music signal, as well as the music separation model; The input unit is used to input the frequency domain signal into the music separation model, and the music separation model performs the following steps: The frequency domain signal is segmented to obtain the first feature sequence corresponding to the frequency domain signal; The first feature sequence is modeled using convolution operations to obtain the second feature sequence; the features in the second feature sequence are obtained by capturing the dependencies of local features in the first feature sequence through convolution operations. Based on the second feature sequence, the masking value of the music track signal corresponding to the music signal is predicted; Based on the frequency domain signal and the masking value, the music track signals in the music signal are output.

9. A music signal separation device, characterized in that, include: Processor, memory, and input / output interfaces; The memory is either a short-term storage memory or a persistent storage memory; The processor is configured to communicate with the memory and execute instructions in the memory to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 7.

11. A computer program product comprising instructions or a computer program, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 7.

Images