Piano music segmentation and note recognition algorithm based on deep learning network

Abstract

The application discloses a piano music segmentation and note recognition algorithm based on a deep learning network, and comprises the following steps: based on a recurrent neural network, obtaining the starting point of each single tone from a continuous piano audio; according to the starting point of each single tone, segmenting a complete piano audio signal into several single tone audios; constructing an audio data set for pitch recognition; pre-processing signals in the audio data set, drawing a transformed spectrogram, and simultaneously segmenting the spectrogram according to the frequency band of each fundamental frequency to obtain a sub-spectrogram; taking the multiple sub-spectrograms segmented from the spectrogram as input, and recognizing the pitch through multiple convolutional neural networks. Compared with a traditional time domain method, the application no longer depends on the selection of a threshold, improves the generalization ability, compared with a traditional frequency domain method, simplifies the calculation process, and improves the running speed.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of piano music recognition, and particularly relates to a piano music segmentation and note recognition algorithm based on a deep learning network. BACKGROUND

[0002] Piano is called the king of musical instruments, and most people choose it as the first choice for learning musical instruments. However, the learning difficulty of piano is so great that a large amount of systematic training is required, and a professional piano teacher should be provided with guidance. Due to the realistic pressure, piano learners often do not have a large amount of time to practice piano, and the cost of one-on-one guidance by a professional piano teacher is too high. Therefore, a large number of piano accompaniment software has emerged, which indirectly promotes the research and development of music recognition in this field.

[0003] Music recognition is a basic problem in music information acquisition, and its core task is to convert a music signal into a symbolic language. The symbolic language is the music score, which records when each note starts and ends and the pitch of each note. It can be seen that the key of music recognition is note endpoint detection and pitch recognition.

[0004] Endpoint detection technology can be divided into time domain and frequency domain analysis technology. The time domain method is to realize endpoint segmentation through the amplitude difference, short-time energy difference and zero-crossing rate of music, which has the advantages of simplicity and rapidity, and good endpoint detection effect for percussion instruments. However, the disadvantage is that the detection effect is not ideal for music with weak starting sound and fast rhythm. At the same time, the time domain method often needs to set a threshold, which is greatly affected by human factors and has poor generalization ability. The frequency domain method is to convert the music signal into the frequency domain through short-time Fourier transform, constant Q transform and wavelet transform, and then complete segmentation. This method is better than the time domain method, but the cost is large calculation.

[0005] The traditional method in the field of pitch recognition is template matching and hidden Markov model. Among them, the template matching method needs to construct a fixed chord template as a dictionary in advance, and the series of chords are obtained by calculating the distance between the chord template and the chroma vector. The advantage is that it does not require professional music knowledge and training data of the model. When using this method, median filtering or smoothing technology is usually added to the implicit model to reflect the natural change process. The classic one is to use hidden Markov model.

[0006] The defect of traditional pitch recognition is the accuracy of recognition. For correct chord performance, it can be easily recognized, but for incorrect chord performance and arbitrary irregular chord, it cannot be well distinguished. SUMMARY

[0007] The purpose of the present application is to provide a piano music segmentation and note recognition algorithm based on a deep learning network, which no longer relies on the selection of a threshold, improves the generalization ability, improves the running speed, and improves the recognition accuracy of played errors, arbitrary irregular chords, and chords with multiple fundamental frequencies.

[0008] The technical scheme of the present application is as follows: a piano music segmentation and note recognition algorithm based on a deep learning network, comprising the following steps:

[0009] Step 1: based on a recurrent neural network, obtaining the starting point of each single tone from a continuous piano audio;

[0010] Step 2: according to the starting point of each single tone, the complete piano audio signal is segmented into several single tone audios;

[0011] Step 3: constructing an audio data set for pitch recognition, according to the easy-to-write characteristics of MIDI files, constructing a data set for each fundamental frequency, and the piano has 88 keys, i.e. 88 fundamental frequencies, and the data set has 88;

[0012] Step 4: pre-processing the signal in the audio data set;

[0013] Step 5: using multiple convolutional neural networks to complete the recognition of the pitch by taking the multiple sub-spectrograms obtained by segmenting the spectrogram as input.

[0014] Further, the step 1 comprises:

[0015] (1) calculating the short-time energy difference of adjacent two frames of the audio signal;

[0016] (2) constructing a network structure, using a bidirectional recurrent neural network, and the network adopts a seven-layer network model containing three recurrent units, three forgetting layers, and one fully connected layer, so that the recurrent units and the forgetting layers are distributed alternately, each recurrent unit and the forgetting layer form a local unit, and the input and output of each local unit are processed by residual error;

[0017] (3) designing a loss function as loss = -∑β·y i ·log(P i )+(1-y i )log(1-P i ), wherein y i represents the label of the end point frame, P i represents the probability value of predicting the end point frame, and the weight coefficient β value is 11;

[0018] (4) selecting a training strategy, adjusting various parameters, training and saving the model and weight coefficient;

[0019] (5) Load the model and weight coefficient, take the short-time energy difference of continuous frames as input, and take whether the middle frame is the starting point as the result to input into the recurrent neural network for recognition, so as to obtain the starting point of each single tone.

[0020] Further, the step 3 comprises:

[0021] (1) For each pitch, a set of fundamental frequencies corresponding to the chords in the data set is constructed, including adjacent tones, half frequencies and multiple frequencies;

[0022] (2) Select several fundamental frequencies in the above set, set different numbers of fundamental frequencies and different volumes to construct chords;

[0023] (3) Remove the overlapping tones to avoid data set imbalance.

[0024] The step 4 comprises:

[0025] (1) Perform constant Q transform on the audio segment containing only single tones, and the frequency transform range is the pitch range of the piano;

[0026] (2) Draw the transformed spectrogram, take the energy at the fundamental frequency as the maximum value, perform energy normalization, and reduce the interference of multiple frequencies or half frequencies;

[0027] (3) According to the different target fundamental frequencies to be detected, the spectrogram is divided to obtain the corresponding sub-spectrogram.

[0028] Further, the step 5 comprises:

[0029] (1) Construct the network structure, the convolutional neural network used has 50 layers, including one 3x3 convolutional layer, one fully connected layer, and four groups of residual blocks, each group containing 3, 4, 6, and 3 residual blocks, respectively, wherein each residual block contains three convolutional layers with convolutional kernels of 1x1, 3x3, and 1x1. The activation function used in the network is Relu, and the maximum pooling layer is set to 2x2;

[0030] (2) Design the loss function as loss = -∑α·y i ·log(P i )+(1-y i )log(1-P i ), wherein y i represents the label of the positive sample, P i represents the probability value of predicting a positive sample, and the value of the weight α is set to 2;

[0031] (3) For each fundamental frequency, select a training strategy, adjust the parameters, use the same network structure, and perform training respectively to obtain and save the model and weight coefficient;

[0032] (4) Load 88 models and weight coefficients respectively, input the multiple sub-spectrograms obtained by dividing the spectrogram into the corresponding convolutional neural network respectively for detection, and obtain all the fundamental frequencies contained.

[0033] Compared with the prior art, the present application has the following advantages:

[0034] (1) The present application adopts a recurrent neural network to obtain the starting point of each single tone from a continuous piano audio, which, compared with a traditional time domain method, no longer relies on the selection of a threshold, improves the generalization ability, and, compared with a traditional frequency domain method, simplifies the calculation process and improves the running speed.

[0035] (2) The present application adopts a convolutional neural network to obtain the fundamental frequency contained in each single tone from a single tone audio, which, compared with a traditional pitch recognition method, improves the recognition accuracy. The method can achieve good recognition accuracy for a chord with a playing error, an arbitrary irregular chord and an arbitrary chord with multiple fundamental frequencies. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 It is a flowchart of the piano music segmentation and note recognition algorithm based on the deep learning network of the present application.

[0037] Figure 2 It is a recurrent neural network structure diagram of the present application.

[0038] Figure 3 It is a convolutional neural network structure diagram of the present application.

[0039] Figure 4 It is a residual block structure diagram in the convolutional neural network of the present application. DETAILED DESCRIPTION

[0040] The specific embodiments of the present application will be described in detail below in combination with the drawings and implementation examples.

[0041] According to the drawings Figure 1 , the present application is a piano music segmentation and note recognition algorithm based on a deep learning network, characterized in that the method comprises the following steps:

[0042] Step 1, read a complete audio file, and perform framing and windowing processing.

[0043] Read the piano music signal, and perform framing and windowing to realize the conversion from the research on a non-stationary signal varying with time to the research on a short-time stationary signal. This step is the basis for the subsequent calculation of music-related features, realization of music segmentation and pitch recognition.

[0044] The sampling frequency used for reading the audio file is 44.1 kHz, the frame length is set to 2% of the sampling frequency, which contains 882 sampling points and 20 ms, the frame shift is half of the frame length, which contains 441 sampling points, the window function is selected as the Hanning window, and the formula is:

[0045]

[0046] Step 2, calculate the short-time energy difference of adjacent two frames, wherein the formula for calculating the short-time energy of each frame is:

[0047]

[0048] The formula for calculating the short-time energy difference is:

[0049]

[0050] The short-time energy difference of adjacent frames is calculated, and since the general noise is additive noise, the short-time energy difference of adjacent frames can be calculated. This can play a role in noise reduction. In addition, the short-time energy difference can overcome the characteristics of the short-time energy smoothing tone mutation, and can maximize the differentiation of each frame around the tone mutation frame, and is very suitable as an input feature of the recurrent neural network.

[0051] Step 3, take the short-time energy difference of 11 consecutive frames as input, and take whether the 6th frame is the starting point as the label to construct a sample. According to this method, several whole section of piano tone data sets are converted into data sets that can be used for training of recurrent neural networks, and the data sets are divided into training set, validation set and test set three parts, and the approximate allocation ratio is 7:2:1.

[0052] When constructing the training data set, the first 5 frames and the last 5 frames of the whole section of audio cannot be used as labels, and there is no corresponding complete input corresponding to them, which are directly discarded.

[0053] Step 4, send the data set into the recurrent neural network for training to obtain the model of audio segmentation. The network structure is as shown in Figure 2 The network adopts a seven-layer network model, including three recurrent units and three forget layers, the number of memories of each recurrent unit is 100, the output of each time step of the first two recurrent units is returned, return_sequences is set to True, only the output of the last time step of the last recurrent unit is returned, return_sequences is set to False, the Dropout of each forget layer is set to 0.5, and the last layer is a full connection layer. The input and output of each recurrent unit and forget layer in front are subjected to residual operation, which is beneficial to alleviate overfitting and gradient disappearance phenomenon, and speed up the training speed.

[0054] During training, the input dimension of data is 11x1 and the output dimension is 1. Since the number of audio endpoint frames in the musical signal is much smaller than that of non-endpoint frames, in order to alleviate the problem of sample imbalance, the loss function is set as:

[0055] loss = -∑β·y i ·log(P i )+(1-y i )log(1-P i )

[0056] where y i represents the label of the endpoint frame, P i represents the probability value of predicting the endpoint frame, since the energy difference is close to 0 in the 11-frame input data at the silent or note duration, it is easy to identify, but when the 11-frame input data contains an endpoint frame, it is difficult to accurately identify the position of the endpoint frame, so among the 11 difficult samples, 10 are negative samples and 1 is positive sample, so the value of β is 11. In order to minimize the loss function during training, the optimization strategy is set to Adam, the learning step is set to e -3 , and a total of 100 epochs are iterated. Since the training cost is large, the early stopping method is used to terminate the training in advance, and the validation is performed once every epoch, and when the accuracy of the validation set does not improve by 0.001 for 5 consecutive times, it is considered that the best model has been obtained or overfitting has occurred.

[0057] Step 5, using the obtained model, identify the starting point of each note in the whole piano musical signal, and segment the audio. The starting and ending points of two adjacent starting points are taken as the starting and ending points of a single note, and if the duration of the last single note is less than 4s, the ending point of the audio is taken as the termination point of the single note, otherwise it is defaulted to 4s.

[0058] Step 6, build a data set for pitch recognition. According to the easy-to-write characteristics of MIDI files, a data set is built for each fundamental frequency. There are 88 keys on the piano, i.e. 88 fundamental frequencies, and there are 88 data sets. Select other keys to form chords with the original sound of each piano key. The selection of other sounds can be divided into two categories:

[0059] (1) adjacent sound. The first difficulty in chord recognition is the differentiation of adjacent sounds. The closer the two single sounds, the smaller the frequency difference, and the higher the recognition difficulty. Therefore, in the implementation, the six sounds adjacent to the original sound of the data set are selected to participate in the construction. Taking central C (note name C4) as an example, the corresponding MIDI pitch value is 60, so MIDI pitch values 57, 58, 59, 61, 62, and 63 are all involved in the construction of this data set.

[0060] (2) Half frequency and octave frequency sound, the most difficult problem in chord recognition is the octave coincidence problem, on the sound with integer multiple relationship of fundamental frequency, their fundamental and harmonic overlap together, for difficult sound, difficult sample, all the sound with multiple relationship is considered, for example, central C, MIDI pitch value is 24, 36, 48, 72, 84, 96, 108, all participate in the construction of the data set. At this point, the MIDI values of adjacent sound and half frequency and octave frequency sound form a set, which is called the original set

[0061] In addition to determining the required pitch, it is also necessary to consider the number of fundamental frequencies contained in a chord. In the implementation of the data set, mainly contains three chords and seven chords, specifically, the generation steps of the chord can be divided into the following stages:

[0062] (1) Derive two sets from the original set, each number in the first set is 1 less than the original set, and each number in the second set is 1 more than the original set. It should be noted that the generated MIDI value should be between 21 and 108, and those exceeding the range will be automatically discarded. For example, central C, the first set is (23, 35, 47, 56, 58, 57, 61, 62, 71, 83, 107), and the second set is (25, 37, 49, 58, 59, 64, 62, 63, 73, 85)

[0063] (2) Write a MIDI file to generate a chord. For a three chord, the three sounds that make up the chord are the original sound, one sound in the original set, and one sound in the first set. For a seven chord, the four sounds that make up the chord are the original sound, one sound in the original set, one sound in the first set, and one sound in the second set. Each type of chord, whether it is a three chord or a seven chord, will have five volumes, including 25, 50, 75, 100, and 127.

[0064] (3) The duration of each chord is set to 4s to be consistent with the samples in The NSynth Dataset. When generating, overlapping sounds are removed to avoid data imbalance. Thus, the chord data set is constructed.

[0065] Step 7, perform constant Q transformation on the audio signals in the data set, and the frequency transformation range is the pitch range of the piano. Implementation:

[0066] (1) Read the existing wav file with a sampling frequency f s of 44.1kHz;

[0067] (2) The number of frequency line bands B within an octave is set to 12x5 = 60, and each semitone has 5 frequency line bands;

[0068] (3) In the sampling frequency f s Frame shift takes 512 sampling points, and the time length is 11.64 ms under the premise that the sampling frequency f is 44.1 kHz;

[0069] (4) The lowest frequency f min Set to 27.5 Hz

[0070] Step 8, draw the transformed spectrogram, and perform energy normalization to maximize the energy at the fundamental frequency, and reduce the interference of the frequency doubling or half frequency;

[0071] Step 9, perform gray processing on the energy-normalized spectrogram to reduce the difficulty of network training;

[0072] Step 10, divide the spectrogram according to the different target fundamental frequencies to be detected to obtain the corresponding sub-spectrogram.

[0073] Step 11, use a convolutional neural network to determine whether a single-tone audio contains a fundamental frequency for each fundamental frequency, and loop to detect all fundamental frequencies from MIDI pitch 21 to 108 to obtain all fundamental frequencies in the single-tone audio. The convolutional neural network used has 88 networks, each with the same structure, except for the weight coefficients after training. The network structure is shown in Figure 3 , which has 50 layers, including a 3x3 convolution layer, a fully connected layer, and 4 groups of residual blocks, each containing 3, 4, 6, and 3 residual blocks, respectively. The residual block structure is shown in Figure 4 , where each residual block contains three convolution layers with kernel sizes of 1x1, 3x3, and 1x1. The activation function used in the network is Relu, and the maximum pooling layer is set to 2x2.

[0074] During training, the input of the data set, the size of a batch is set to 4, the input signal is 4x5x400x1, because the first dimension of the image in the data set is 5, the value is small, so it is not expected to change this dimension in the process, so the maximum pooling layer and the padding of the convolution layer are set to same in each part, and the step is 1. In the first part of the network, the number of filters in the convolution layer is set to 64, and the output is 4x5x400x64. In the first residual operation, the number of filters is also set to 64, and the output is 4x5x400x64. In the second residual operation, the number of filters is set to 128, and the output is 4x5x400x128. In the third residual operation, the number of filters is set to 256, and the output is changed to 4x5x400x256. In the fourth residual operation, the number of filters is set to 512, and the output is changed to 4x5x400x512. In the average pooling layer, the entire dimension is flattened, and the output dimension is changed to 4x1024000. Finally, the fully connected layer has one neuron, and the final output dimension is 4x1.

[0075] Regarding the loss function, since the number of positive and negative samples in the data set is not balanced, the number of negative samples is greater than that of positive samples. In order to alleviate the problem of sample imbalance, the loss function is set as:

[0076] loss=-∑α·y i ·log(P i )+(1-y i )log(1-P i )

[0077] Where y i represents the label of the positive sample, P i represents the probability value of predicting a positive sample. Since the ratio of the number of positive samples to the number of negative samples is approximately 1:2, the value of the weight a is set to 2. During training, in order to find the minimum value of the loss function, the optimization strategy is set to Adam, the learning step is set to e -3 , and a total of 100 epochs are iterated. Since the training cost is large, the early stopping method is used to terminate the training in advance. Once an epoch is trained, the validation is performed once. When the accuracy of the validation set does not improve by 0.001 for 5 consecutive times, it is considered that the best model has been obtained or overfitting has occurred.

[0078] Step 12, the audio segment containing only single tone obtained in step 5 is sent to the model trained in step 11 to complete the pitch recognition.

[0079] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. The embodiments should therefore be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the above description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims with respect to the figures of the application.

Claims

1. A piano sound segmentation and note recognition algorithm based on a deep learning network, characterized in that, The method comprises the following steps: obtaining starting points of each single tone from a continuous piano audio based on a recurrent neural network; segmenting the complete piano audio signal into single tone audios according to the starting points of each single tone; constructing an audio data set for pitch recognition; preprocessing signals in the audio data set, drawing a transformed spectrogram, and segmenting the spectrogram according to frequency bands of each fundamental frequency to obtain a sub-spectrogram; inputting the sub-spectrograms segmented from the spectrogram into multiple convolutional neural networks for pitch recognition; the recurrent neural network comprises the following steps: calculating a short-time energy difference between two adjacent frames of the audio signal; inputting the short-time energy difference of a plurality of continuous frames into the recurrent neural network, taking whether the middle frame is a starting point as the result, and obtaining the starting point of each single tone; the network structure of the recurrent neural network is as follows: the recurrent neural network is bidirectional, the network adopts a seven-layer network model, includes three recurrent units, three forgetting layers and one fully connected layer, the recurrent units and the forgetting layers are arranged alternately, each recurrent unit and forgetting layer constitutes a local unit, the input and output of each local unit are subjected to residual processing, the number of memories of each recurrent unit is 100, the first two recurrent units return outputs at each time step, return_sequences is set to True, and the last recurrent unit returns an output only at the last time step, return_sequences is set to False, and the Dropout of each forgetting layer is set to 0.5; the preprocessing of the signals in the audio data set, the drawing of the transformed spectrogram, and the segmentation of the spectrogram according to the frequency bands of each fundamental frequency to obtain a sub-spectrogram comprise the following steps: performing constant Q transformation on the audio segment containing only a single tone, and the frequency transformation range is the pitch range of the piano; drawing the transformed spectrogram, taking the energy at the fundamental frequency as the maximum value, performing energy normalization, and reducing the interference of the multiple frequency or half frequency; dividing the spectrogram according to different target fundamental frequencies to obtain corresponding sub-spectrograms; the convolutional neural network comprises 88 convolutional neural networks, each convolutional neural network has the same structure but different weight coefficients, and the structure of the convolutional neural network is as follows: the convolutional neural network comprises 50 layers, one 3x3 convolutional layer, one fully connected layer, and four groups of residual blocks, each group of residual blocks comprises 3, 4, 6 and 3 residual blocks, each residual block comprises three convolutional layers with convolutional kernels of 1x1, 3x3 and 1x1, each convolutional layer in the network is connected with a Relu activation function and a BN layer, the maximum pooling layer is set to 2x2, and the padding of the maximum pooling layer and the convolutional layer in each part is set to same. 2.The piano sound segmentation and note recognition algorithm based on deep learning network according to claim 1, wherein, The recurrent neural network is trained with an input dimension of 11x1 and an output dimension of 1, and the loss function of the recurrent neural network is loss = -∑β·y i ·log(P i )+(1-y i )log(1-P i ), wherein y i represents the label of the endpoint frame, P i indicates the probability value of predicting the endpoint frame, and the weight coefficient β is 11. 3.The piano sound segmentation and note recognition algorithm based on deep learning network according to claim 1, wherein, when the complete piano audio signal is segmented into single tone audios, two adjacent starting points are taken as the start and end points of a single tone, if the time length of the last single tone is less than 4s, the end point of the audio is taken as the termination point of the single tone, otherwise, 4s is taken as the default. 4.The piano music segmentation and note recognition algorithm based on deep learning network according to claim 1, wherein, The audio data set is constructed separately for each base frequency, and 88 data sets are generated in total. 5.The piano sound segmentation and note recognition algorithm based on deep learning network according to claim 1, wherein, In the pitch recognition through the multiple convolutional neural networks: a convolutional neural network is used to determine whether a single audio contains a base frequency, and all base frequencies from 21 to 108 of the MIDI pitch are detected in a loop to obtain all base frequencies in the single audio.

6. The piano sound segmentation and note recognition algorithm based on a deep learning network according to claim 5, characterized in that, The convolutional neural network adopts a loss function: loss = -∑α·y i ·log(P i )+(1-y i )log(1-P i ), wherein y i represents a label of a positive sample, P i indicates a probability value of predicting a positive sample, and the value of the weight a is set to 2.

Images