Main melody extraction method, system and computer storage medium based on attention

Abstract

The present invention proposes a main melody extraction method, system and computer storage medium based on attention. The main melody extraction method processes the audio signal of the main melody to be extracted to form a spectrogram; the spectrogram is subjected to multiple Tattention operations, first down-samples the output of at least one Tattention, and then up-samples the output of at least one Tattention, the number of up-sampling is consistent with the number of down-sampling, each Tattention takes the output or sampling result of the last Tattention as input, searches for the corresponding frequency in the spectrogram according to the output of the last Tattention, and extracts the main melody of the audio signal; the Tattention fuses the input information in the frequency domain and then passes through the time domain attention, and then performs the time domain information fusion and then passes through the channel domain attention and then outputs. The main melody extraction method improves the accuracy of main melody extraction by combining two types of attention.

Description

Technical Field

[0001] The present invention relates to the field of audio signal processing, and in particular to a main melody extraction method and system based on attention and a computer storage medium. Background Art

[0002] Main melody extraction is an important research task in the field of music signal processing. The specific content is that given a piece of music, the computer automatically analyzes the music content and extracts the main melody of the music. Main melody extraction has a wide range of applications in music structure analysis, music retrieval, music style classification, cover recognition and other fields.

[0003] Traditional main melody extraction methods are mainly divided into three categories: main melody extraction based on pitch significance, main melody extraction based on source separation, and machine learning-based methods. With the development of deep learning and its great achievements in the field of speech signal processing, many people use deep learning to extract the main melody of music and have achieved good results. Summary of the invention

[0004] In order to overcome the defects in the above-mentioned prior art, the purpose of the present invention is to provide a main melody extraction method, system and computer storage medium based on attention.

[0005] In order to achieve the above-mentioned object of the present invention, the present invention provides a main melody extraction method based on attention, comprising the following steps:

[0006] Processing the audio signal of the main melody to be extracted to form a spectrogram;

[0007] Perform multiple Tattention operations on the spectrogram, downsample the output of at least one Tattention, and then upsample the output of at least one Tattention, the number of upsampling is the same as the number of downsampling, each Tattention takes the output or sampling result of the last Tattention as input, and finds the corresponding frequency point and time in the spectrogram according to the output of the last Tattention, and extracts the main melody of the audio signal;

[0008] The Tattention fuses the frequency domain information of the input information through the time domain attention, and then fuses the time domain information through the channel domain attention and outputs it.

[0009] This main melody extraction method improves the accuracy of main melody extraction by combining two types of attention.

[0010] The preferred solution of the main melody extraction method is: when using time domain attention, the frequency domain information is fused to obtain F*T time domain attention, where F is frequency and T is time. At this time, F=1.

[0011] The preferred solution of the main melody extraction method is: when using channel domain attention, the time domain is fused to obtain C*T channel domain attention, T is time, C is channel, and at this time, T=1.

[0012] The preferred embodiment of the main melody extraction method is:

[0013] The steps of Tattention are:

[0014] For the spectrogram or input information X∈R C*F*T First, perform a two-dimensional convolution f1 operation to obtain S1, and then take the mean of the F dimension of S1, denoted as U∈R C*T , C is the channel, F is the frequency, T is the time, where S 1ij represents the element in row i and column j in S1;

[0015] Perform a one-dimensional convolution on U, and the result is recorded as V∈R C*T , V = f(U), and at the same time perform two-dimensional convolution f2 on S1 to obtain S2;

[0016] Take V as the attention map and multiply it by S2. The result is recorded as Z, Z = S2*softmax(V);

[0017] Take the average value of the T dimensions of V and record the result as D. Perform a two-dimensional convolution f3 on S2 to obtain S3, and then perform two fully connected layers on D to obtain D', D' = σ(W2Φ(W1D));

[0018] Multiply D' and S3 and get the result of channel domain attention. The formula for multiplying D' and S3 is S3*softmax(D').

[0019] The preferred solution of the main melody extraction method is as follows: the Tattention adds the output of the channel domain attention to the output shortcut of the skip-connections as the output of the Tattention, that is, output = S3*softmax(D') + shortcut.

[0020] The preferred solution of the main melody extraction method is: the main melody extracted from the audio signal is recorded as O∈R F*T , perform a one-dimensional convolution on O to reduce F to 1, and the result is recorded as M∈R T, then combine O and M to get the final result output'∈R (F +1)*T .

[0021] The preferred solution adds a tail specifically used for detecting the part without melody at the end of the entire network, so that there is no need to distinguish between the part with melody and the part without melody in the result, and all of them can be processed uniformly.

[0022] The preferred solution of the main melody extraction method is as follows: when the audio signal of the main melody to be extracted is processed to form a spectrogram, it is processed into a three-channel matrix consisting of the power-scaled spectrogram, generalized cepstrum and generalized cepstrum of spectrum by Combining Spectral and Temporal Representations.

[0023] The present invention also proposes a main melody extraction system, including a processor and a memory, wherein the processor and the memory are communicatively connected, and the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to execute operations corresponding to the main melody extraction method as described above.

[0024] The present invention also provides a computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the executable instruction enables a processor to execute operations corresponding to the main melody extraction method as described above.

[0025] The beneficial effects of the present invention are as follows: the present invention makes the overall structure U-shaped through downsampling and upsampling, extracts the main melody through time domain attention and channel domain attention, and the extraction result is effective and has high accuracy.

[0026] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0028] Figure 1 It is a schematic diagram of the network structure of the main melody extraction method;

[0029] Figure 2 This is a schematic diagram of the network structure of Tattention. DETAILED DESCRIPTION

[0030] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0031] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0032] Attention Mechanism is a data processing method in machine learning. The Attention Mechanism imitates the mechanism of human observation of things, focusing on some important local information, and then combining them to form an overall impression of the observed object.

[0033] like Figure 1 As shown, the present invention provides an embodiment of a main melody extraction method based on attention.

[0034] The overall network structure adopted by this method is as follows Figure 1 As shown, it is a U-shaped structure, the main body of which is a TAttention structure with downsampling and upsampling.

[0035] The method comprises the following steps:

[0036] The audio signal of the main melody to be extracted is preprocessed to form a spectrogram. In this embodiment, the audio signal of the main melody to be extracted is first processed to form a spectrogram of C×F×T (i.e., channel domain, frequency domain, and time domain). In this embodiment, C of the spectrogram is 3. When the spectrogram is preprocessed to form a spectrogram, a frequency array of the audio signal of the main melody to be extracted can be obtained.

[0037] It is preferred but not limited to Su L, Yang Y H. Combining Spectral and Temporal Representations for Multipitch Estimation of Polyphonic Music [J]. IEEE / ACM Transactions on Audio Speech & Language Processing, 2015, 23 (10): 1600-1612. The proposed method processes the audio signal of the main melody to be extracted through Combining Spectral and Temporal Representations (CFP) into a three-channel matrix consisting of the power-scaled spectrogram, generalized cepstrum (GC) and generalized cepstrum of spectrum (GCoS), that is, the three channels are the power-scaled spectrogram, generalized cepstrum and generalized cepstrum of spectrum. During the processing, a sampling frequency of 44100 Hz, a window size of 2048, and a frame shift of 256 (when the labeling interval is 6 ms) / 441 (when the labeling interval is 10 ms) are used to perform short-time Fourier transform (STFT).

[0038] Then, the processed spectrogram is subjected to multiple Tattention operations, the output of at least one Tattention is downsampled, and then the output of at least one Tattention is upsampled, the number of upsampling is consistent with the number of downsampling, and each Tattention is input with the output or sampling result of the last Tattention, and the output of the last Tattention is the main melody of the audio signal. In this embodiment, the spectrogram is subjected to 6 Tattention operations, the outputs of the first two Tattentions are downsampled, and the outputs of the second to last and third Tattentions are upsampled, and each sampling result is used as the input of the next Tattention.

[0039] Let me introduce Tattention in detail:

[0040] The TAttention structure is as follows Figure 2As shown in the figure, the entire process above the first × from top to bottom is the time domain attention, and the process between the first × and the second × is the channel attention. That is, the first half is a time domain attention structure, and the second half is the channel domain attention structure. The entire TAttention structure also uses skip-connections to solve problems such as gradient disappearance.

[0041] The TAttention operation is to fuse the frequency domain information of the input information through the time domain attention, and then add the output of the time domain information fused through the channel domain attention to the output shortcut of the skip-connections as the output of Tattention, that is, output = S3*softmax(D')+shortcut.

[0042] When using time domain attention, the time domain attention is obtained by fusing the frequency domain information, where F is the frequency and T is the time. At this time, F = 1. When using channel domain attention, the channel domain attention is obtained by fusing the time domain information, where T is the time and C is the channel. At this time, T = 1.

[0043] In this embodiment, let the spectrogram or input information be X∈R C*F*T , where C is the channel, F is the frequency, and T is the time.

[0044] For the spectrogram X∈R C*F*T First, perform a two-dimensional convolution f1 operation to obtain S1, and then take the mean of the F dimension of S1, denoted as U∈R C*T , Where S 1ij Represents the element in row i and column j in S1.

[0045] Next, we perform a one-dimensional convolution on U, and the result is recorded as V∈R C*T , V = f(U), and at the same time perform two-dimensional convolution f2 on S1 to obtain S2.

[0046] Then use V as the attention map and multiply it with S2. The result is recorded as Z, Z = S2*softmax(V).

[0047] The second half first takes the average value of the T dimensions of V, and the result is recorded as D. Perform a two-dimensional convolution f3 on S2 to obtain S3, and then perform two fully connected layers on D to obtain D', D'=σ(W2Φ(W1D)), W1 represents the fully connected layer one of the two fully connected layers, W2 represents the fully connected layer two of the two fully connected layers, σ represents the activation function of the fully connected layer one, and Φ represents the activation function of the fully connected layer two.

[0048] Finally, multiply D' by S3 and add the output of skip-connections, recorded as output, output = S3*softmax(D') + shortcut, where shortcut is the output of skip-connections, that is, the original input of attention.

[0049] A preferred solution of this embodiment is to add a tail structure specifically used for detecting the non-melody part at the end of the entire U-shaped network. Figure 1 The conve1d of the rightmost (1,1,T) of the U-shaped network in the figure does not need to distinguish between the melody part and the non-melody part in the result, and all of them can be processed uniformly. Specific implementation: Set the structure obtained by the U-shaped structure to O∈R F*T , that is, the output of the last TAttention is set to O∈R F*T , perform a one-dimensional convolution on O to reduce F to 1, and the result is recorded as M∈R T , then combine O and M to get the final result output'∈R (F+1)*T , and use this as the main theme for final extraction.

[0050] O and M are combined: the dimension of O is (F, T), the dimension of M is (1, T), and the dimension of the output output' is (F+1, T). The meaning of this output' is the number of frequency groups corresponding to the time of the main melody of the music. The value of output' has a one-to-one correspondence with the value of the frequency array, that is, the frequency array obtained by the previous data preprocessing is combined, and the corresponding frequency group in the frequency array is found through this frequency group number. The frequency corresponding to this frequency group is the frequency of the main frequency law, and finally the time and the corresponding frequency, that is, the main melody, are obtained.

[0051] illustrate:

[0052] 1. The (3, F, T) data obtained by preprocessing, namely the spectrogram, is used to input the network, and the time array and frequency array of the music can also be obtained. This frequency array is the array used for the final query.

[0053] 2. This last M is used to determine whether there is a main melody in a certain time period. Because there may be multiple situations for a certain time t: there is no main melody, there is a main melody; if there is a main melody at time t, it is also necessary to determine the frequency of the main melody. Therefore, in order to simplify the final judgment, this specific M is used to represent that there is no main melody at a certain time. Specific use: The final result output dimension is (F+1, T). Assuming that the first row is M, if the main melody frequency predicted by the neural network at a certain time t is in the first row, it is directly considered that there is no main melody at t. If it is predicted in other rows, it is considered that there is a main melody at time t, and then the frequency array is searched for the corresponding main melody frequency. The advantage of this is that the result processing of the neural network is simple. It only needs to determine the frequency of the main melody at t, and no longer needs to determine whether there is a main melody at t.

[0054] Specific Experiment Introduction

[0055] The training set of the experiment is 66 music clips selected from the Medley DB dataset, and the validation set is also 15 music clips selected from it. There are three test sets and they are selected from three datasets, namely ADC2004, MIREX-05 and Medley DB, with 12 in ADC2004, 9 in MIREX-05 and 12 in Medley DB.

[0056] All selected data sets are processed in the same way, each segment is cut into T = 256 frames in length to facilitate neural network processing. The code is implemented using pytorch, the loss function is BCELoss, Adam optimization, and the learning rate is 0.001.

[0057] The indicators used to evaluate the results are overall accuracy (OA), raw pitch accuracy (RPA), raw chroma accuracy (RCA), voicing recall (VR) and voicing false alarm (VFA). The python package used is mir-eval to obtain the results of the above values.

[0058] Ablation experiment

[0059] Table 1

[0060]

[0061] Table 1: Comparison of ablation experiment results on the ADC2004 and MIREX-05 test sets, where W / O co represents the result after removing the time domain attention and channel attention in TAttention, and W / O SE represents the result after removing the second half of TAttention.

[0062] The results of the ablation experiment are shown in Table 1. Overall, the complete network has better results. For the results after removing the attetnion, the VR is slightly higher on the ADC2004 and Medley-DB test sets, but overall the results are slightly worse than the complete network from the perspective of OA and VFA. Secondly, the results after removing the second half are generally slightly worse than the complete TAttention results. This also shows that the attention structure can indeed improve the effect.

[0063] Comparison of experimental results

[0064] The effect of the method proposed in this application on the test set and the effect of the other two methods, SegNet and FTANet, on the test set are shown in Table 2. All three methods use the training set in this article to train the model under the same conditions, and then test the results on the same test set. From the data comparison in Table 2, it can be seen that it is indeed effective and the results are relatively good. Compared with other networks, the network proposed in this article is higher than other networks on OA. On the MIREX-05 and ADC2004 test sets, VR is slightly lower than SegNet, but VFA is better than it; for the Medley-DB test set, VR is slightly lower than FTANet, but VFA is also better than FTANet.

[0065] Table 2

[0066]

[0067]

[0068] Table 2: Comparison of the results of this method with other methods on the test sets ADC2004, MIREC-05 and Medley-DB.

[0069] The present invention also proposes a main melody extraction system, which includes a processor and a memory, wherein the processor and the memory are communicatively connected, and the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to execute operations corresponding to the main melody extraction method as described above.

[0070] The present invention also provides a computer storage medium, which stores at least one executable instruction, and the executable instruction enables a processor to execute operations corresponding to the main melody extraction method as described above.

[0071] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0072] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A main melody extraction method based on attention, characterized in that: The following steps are involved: Processing the audio signal of the main melody to be extracted to form a spectrogram; The spectrogram is subjected to multiple Tattention operations, first downsampling the output of at least one Tattention, and then upsampling the output of at least one Tattention, the number of upsampling is the same as the number of downsampling, each Tattention takes the output or sampling result of the last Tattention as input, and the corresponding frequency is searched in the spectrogram according to the output of the last Tattention, and the main melody of the audio signal is extracted; The Tattention performs frequency domain information fusion on the input information and then passes it through time domain attention, and then performs time domain information fusion and then passes it through channel domain attention before outputting it; The steps of Tattention are: For the spectrogram or input information X∈R C*F*T First, perform a two-dimensional convolution f1 operation to obtain S1, and then take the mean of the F dimension of S1, denoted as U∈R C*T , C is the channel, F is the frequency, T is the time, where S 1ij represents the element in row i and column j in S1; Perform a one-dimensional convolution on U, and the result is recorded as V∈R C*T , V = f(U), and at the same time perform two-dimensional convolution f2 on S1 to obtain S2; Take V as the attention map and multiply it by S2. The result is recorded as Z, Z = S2*softmax(V); Take the average value of the T dimensions of V and record the result as D. Perform a two-dimensional convolution f3 on S2 to obtain S3, and then perform two fully connected layers on D to obtain D', D' = σ(W2Φ(W1D)); Multiply D' and S3 and get the result of channel domain attention. The formula for multiplying D' and S3 is S3*softmax(D').

2. The main melody extraction method based on attention according to claim 1 is characterized in that: The spectrogram is subjected to 6 Tattention operations, the outputs of the first two Tattention operations are downsampled, and the outputs of the penultimate and third Tattention operations are upsampled, and each sampling result is used as the input of the next Tattention operation.

3. The main melody extraction method based on attention according to claim 1 is characterized in that: When using time domain attention, the frequency domain information is fused to obtain the F*T time domain attention, where F is frequency and T is time. At this time, F=1.

4. The main melody extraction method based on attention according to claim 1 is characterized in that: When using channel domain attention, the time domain is fused to obtain the C*T channel domain attention, where T is time and C is channel. At this time, T=1.

5. The main melody extraction method based on attention according to claim 1 is characterized in that: The Tattention adds the output of the channel domain attention to the output shortcut of the skip-connections as the output of the Tattention, that is, output = S3*softmax(D') + shortcut.

6. The main melody extraction method based on attention according to claim 1 is characterized in that: The main melody extracted from the audio signal is recorded as O∈R F*T , perform a one-dimensional convolution on O to reduce F to 1, and the result is recorded as M∈R T , then combine O and M to get the final result output'∈R (F+1)*T , F is frequency, T is time.

7. The main melody extraction method based on attention according to claim 1 is characterized in that: When the audio signal of the main melody to be extracted is processed to form a spectrogram, it is processed into a three-channel matrix consisting of the power-scaled spectrogram, generalized cepstrum and generalized cepstrum of spectrum by Combining Spectral and Temporal Representations.

8. A main melody extraction system, characterized in that: The invention comprises a processor and a memory, wherein the processor and the memory are communicatively connected with each other, and the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to execute an operation corresponding to the main melody extraction method according to any one of claims 1 to 7.

9. A computer storage medium, characterized in that The computer storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the main melody extraction method according to any one of claims 1 to 7.

Images