A two-stage piano sound and chord recognition method based on deep learning

Abstract

The present invention relates to the technical field of piano sound and chord recognition, and discloses a two-stage piano sound and chord recognition method based on deep learning, which includes the following steps: the first stage: step 1, preparing training data: preparing clean single-note audio data for training. In order to improve the recognition effect under noise, data enhancement is also required. Here, the single-note audio data and different types of noise are processed with different signal-to-noise ratios to obtain enhanced audio data with noise. These data sets are mixed together and divided into a training set, a validation set and a test set according to 8:1:1. The two-stage piano sound and chord recognition method based on deep learning uses a neural network model to replace human decision-making, takes the fractional segment output by the monosyllable recognition model as input, directly outputs the result of chord recognition, simplifies the decision-making process, and more accurately models the output fractional segment of the monosyllable, thereby improving the accuracy of chord recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of piano sound and chord recognition, and in particular to a two-stage piano sound and chord recognition method based on deep learning. Background Art

[0002] As material life becomes more affluent, people have begun to pursue a higher quality of spiritual life, and learning musical instruments has gradually become a common trend. In the past, learning musical instruments often required on-site guidance from professional teachers. With the rise of artificial intelligence in the Internet era, various online learning courses are becoming increasingly popular, and online learning of musical instruments is no exception. In addition to online video course content, evaluation and feedback during instrument practice is also very important. In order to help instrument practitioners correctly evaluate their current performance level in real time, a performance evaluation system is very important. In order to make a correct evaluation, the first essential step is to identify the notes and chords during the performer's performance.

[0003] Take the piano as an example. Different notes in the piano have corresponding pitches. The traditional algorithm extracts the fundamental frequency of the audio segment based on the autocorrelation function, and completes the note recognition by corresponding the high and low fundamental frequencies with the notes one by one. With the rise of deep learning, deep neural networks are trained as classifiers to recognize notes. Algorithms based on neural networks are superior to traditional algorithms in both recognition rate and noise robustness, and have gradually become the mainstream of the task of note recognition. However, in addition to playing a single note, more often chords composed of multiple notes with an interval relationship are played.

[0004] At present, algorithms based on single notes often cannot achieve good results in identifying chord types. The main reasons are: 1) When multiple notes are played, different notes interfere with each other, making it difficult for the single note recognition model to identify the correct note category; 2) In order to correctly obtain chord recognition results, existing methods often set judgment rules based on the output results of the single note model to determine the chord corresponding to the audio segment, but this method often has a low recognition rate for chords; 3) The chord recognition model that recognizes chords alone cannot simultaneously consider the information at the single note level, nor can it simultaneously output single note results. Therefore, a two-stage piano sound chord recognition method based on deep learning is proposed to solve the above problems. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In view of the shortcomings of the prior art, the present invention provides a two-stage piano sound and chord recognition method based on deep learning, which has the advantages of adding a stage of chord recognition using a neural network on the basis of a monosyllable recognition model. Compared with the method of manually designing and judging chord rules, the proposed neural network chord recognition algorithm can better fit and model the monosyllable output results of the previous stage, thereby improving the classification effect of chords, etc.

[0007] Resolved the following issues:

[0008] 1. When multiple notes are played, different notes interfere with each other, making it difficult for the single note recognition model to identify the correct note category;

[0009] 2. In order to correctly obtain chord recognition results, existing methods often set judgment rules based on the output results of single-note models to determine the chord corresponding to the audio segment, but this method often has a low recognition rate for chords;

[0010] 3. The chord recognition model that recognizes chords alone cannot take into account the information at the single note level at the same time, nor can it output single note results at the same time.

[0011] (II) Technical solution

[0012] In order to achieve the above-mentioned purpose of adding a stage of chord recognition using a neural network on the basis of the monosyllable recognition model, compared with the method of manually designing and determining chord rules, the proposed neural network chord recognition algorithm can better fit and model the monosyllable output results of the previous stage, thereby improving the classification effect of chords. The present invention provides the following technical solutions, the first stage is to train the monosyllable recognition model, and the second stage includes training the syllable recognition model, which is specifically described as follows:

[0013] Phase 1:

[0014] Step 1: Prepare training data: Prepare clean single-note audio data for training. In order to improve the recognition effect under noise, data enhancement is also required. Here, single-note audio data and different types of noise are added with different signal-to-noise ratios to obtain enhanced audio data with noise. These data sets are mixed together and divided into training set, validation set and test set according to 8:1:1;

[0015] Step 2, feature extraction: perform windowing, framing, short-time Fourier transform, and Mel-triangle filtering on the data set to obtain the Mel-domain spectrum feature FBANK feature of the audio segment;

[0016] Step 3: Model training: The extracted audio features are input into the training script of the single-note recognition model according to certain splicing rules for training to obtain a neural network model that can recognize single notes.

[0017] Phase 2:

[0018] Step 1: Chord data preparation: prepare audio data for chord training. Each time period needs to have a corresponding chord type / non-chord label. The training data also needs to be denoised and amplified.

[0019] Step 2: Feature extraction: The clean audio and the noisy audio are windowed, framed, short-time Fourier transformed, and Mel triangle filtered to obtain the FBANK features of the chord audio segment;

[0020] Step 3, forward reasoning: input the FBANK feature data of the chord audio segment into the monosyllable recognition model trained in stage 1, obtain an output score for each input frame, concatenate the output results of adjacent frames, and select them according to certain standards to obtain a dataset of concatenated scores;

[0021] Step 4: Model training: Input the selected concatenated score data set in the previous step into the second-stage chord recognition network for training. Since each concatenated score segment has a corresponding chord label, the chord recognition network used in the second stage can be trained.

[0022] Preferably, in addition to the clean training data with the correct note text, the training data in step 1 of the first stage also needs to add a certain amount of non-piano sound as negative samples, such as conversation sound, TV sound and street noise. Different noises with different signal-to-noise ratios are added in a noise-adding manner to expand the training set under different noise environments. The signal-to-noise ratio is measured by SNR, and the specific calculation formula is as follows:

[0023]

[0024] Among them, P s and P n Represent the effective power of clean audio and noisy audio respectively. When performing data enhancement, the reference signal-to-noise ratio level is between 5-20dB. The noisy audio is randomly selected from the noise set (the noise set contains various types of noise such as conversation, TV noise, street noise, etc.).

[0025] Preferably, in step 2 of the first stage, a 40-dimensional FBANK feature is extracted from the audio signal of each frame. The reference setting is to use a Hamming window when adding a window, set the unit frame length to 25ms, and shift each frame by 10ms. The feature extraction process of this step is used to extract features from all training sample pairs.

[0026] Preferably, in step three of the first stage, since the objects to be identified in the single-note recognition task are audio segment features with obvious temporal relationships, it is considered here to use a recurrent neural network that can model such temporal relationships as the backbone of the single-note recognition network.

[0027] Preferably, in step 1 of the second stage, after the clean chord audio data is prepared, data enhancement and feature extraction are performed on it, and the chord label corresponding to each frame feature is obtained by alignment.

[0028] Preferably, in step three of the second stage, the extracted features are spliced ​​and input into the single-note recognition model to obtain the single-note recognition score of the current frame. The output score is spliced ​​together by 30 frames, that is, the matrix shape of a sample is (61, 89). Since the input features have been spliced ​​when entering the single-note recognition model, the recognition result carries the information of adjacent frames, and the sample matrix is ​​downsampled with a span of 3 frames. The shape of the sample matrix after downsampling is (20, 89).

[0029] Preferably, in step three of the second stage, the chord recognition model can be designed with reference to the monosyllable recognition model of the first stage as a network with time series modeling capabilities and with GRU as the model backbone. After completing data preparation and determining the second stage model structure, the cross entropy loss function is used as the loss to iteratively train the chord recognition model.

[0030] Preferably, after the first stage and the second stage, a monosyllable recognition model and a chord recognition model are obtained. During the first reasoning, the features of the starting audio segment of the first stage are input into the monosyllable recognition model, and the score of the syllable recognition is output. When the input frames accumulate to 61 frames, the chord recognition process of the second stage is performed, and the scores of the 61 frames are downsampled and spliced ​​together and input into the chord recognition model. In order to perform continuous reasoning, the scores that enter first, that is, the scores of the first row of the sample matrix, are eliminated from every 3-frame spliced ​​score sample, and then the scores of the latest frame are added after the last row to perform continuous sample generation and reasoning.

[0031] (III) Beneficial effects

[0032] Compared with the prior art, the present invention provides a two-stage piano sound and chord recognition method based on deep learning, which has the following beneficial effects:

[0033] 1. A neural network model is used to replace human decision-making. The fractional segments output by the monosyllable recognition model are used as input to directly output the results of chord recognition, which simplifies the decision-making process. At the same time, the output fractional segments of monosyllables are modeled more accurately, thereby improving the accuracy of chord recognition.

[0034] 2. When training the second-stage chord recognition model, the training data can be selected and the ratio of difficult and easy samples can be manually divided. This can not only keep the parameters of the first-stage model unchanged, but also improve the recognition accuracy of the second-stage chord recognition model.

[0035] 3. Compared with the method of directly modeling and identifying chord units, the two-stage chord recognition method proposed in the present invention can not only perform more fine-grained modeling of note-level information during chord recognition to improve recognition performance, but also output note recognition results at the same time. (IV) Description of the drawings

[0037] Figure 1 Training flow chart for single note recognition model and chord recognition model;

[0038] Figure 2 This is a schematic diagram of the structure of a monosyllable recognition model;

[0039] Figure 3 Flowchart for chord identification;

[0040] Figure 4 Provides an explanation of the chord identification process;

[0041] Figure 5 It is a schematic diagram of an embodiment of a monosyllable recognition model structure;

[0042] Figure 6 Schematic diagram of an embodiment of a chord recognition model structure. (V) Specific implementation methods

[0044] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The present invention proposes a two-stage chord recognition method, that is, in the first stage, a monosyllable recognition model is used to output a monosyllable recognition score of an audio segment, and in the second stage, the monosyllable recognition scores of a certain length are spliced ​​and sent to another trained chord recognition model, thereby obtaining a chord recognition result of the audio segment. The specific implementation steps of the present invention are as follows:

[0046] Step 1: Prepare piano note training data for training the monosyllable recognition model. When no notes are played, the syllable recognition model will correctly output the label of non-notes at the current moment, so that the monosyllable recognition model of piano sound has good robustness under noise. It is also necessary to perform data enhancement on the clean piano note training data.

[0047] Step 2: Extract features from the audio signal of the training set. After pre-emphasis, framing, windowing, discrete Fourier transform, and Mel triangle filter filtering, a 40-dimensional FBANK feature is extracted from each frame of the audio signal.

[0048] Step 3: Construct note labels. The note recognition model is a typical supervised learning model, so the correct note labels are crucial to the effect of model training. In order to obtain the correct note labels for each frame of the training set, the force-align tool in Kaldi is used to align each frame of the audio according to the text files of all training audios, and finally obtain the classification label of each frame of speech. For illustration, it is assumed that the single note labels used include 88 key labels and one non-note label, totaling 89.

[0049] Step 4: Build a single-note recognition model. Since the objects to be recognized in the single-note recognition task are audio segment features with obvious temporal relationships, we consider using a recurrent neural network that can model this temporal relationship as the backbone of the single-note recognition network. The GRU network is a variant of the recurrent neural network RNN. It has less computational complexity while better retaining the previous moment information. Batch normalization operations are performed between each layer of the GRU network to ensure that the input distribution of each layer is reasonable and to accelerate model convergence. The last GRU unit of the last layer of the GRU network is connected to a fully connected layer. After passing through the fully connected layer, the final classification result is output after a softmax operation. The reason why only the last GRU unit is connected to the fully connected layer is because the last GRU unit not only contains the information of all GRU units in the same layer, but also saves computational effort.

[0050] Step 5, iteratively train the single note model, and concatenate the audio features extracted from the training audio clips into the seven frames before and after as input to the single note recognition model for training. The output result of the FC 1 layer is then subjected to a softmax operation, and the similarity is calculated with the classification label of the input audio feature to obtain the loss function. The loss function used is the most commonly used cross entropy loss function CE for multi-classification problems. The specific calculation formula is as follows:

[0051]

[0052] The network parameters of the voice wake-up model are iteratively updated using the gradient descent method using the loss function until the model reaches the expected convergence effect. The final number of training steps and whether it has converged are determined by the trainer after multiple attempts based on the complexity of the training data.

[0053] Step six, prepare the training data of the second-stage chord recognition model. Different from the general idea of ​​using the frequency domain features of the audio band as input features for chord recognition, the input of the chord recognition model used in the present invention comes from the output score of the first-stage single-note recognition model. This is also the biggest difference between the present invention and the conventional chord recognition idea. This can not only make full use of the output information of the first-stage single-note recognition model, but also manually select difficult chord samples without affecting the results of the single-note model. Here, referring to the common chord recognition system, the chord labels are divided into 22 chord labels and one non-chord label, totaling 23 labels.

[0054] Step seven, train the second stage chord recognition model. The chord recognition model can refer to the monosyllable recognition model of the first stage and be designed as a network with time series modeling capabilities and GRU as the model backbone.

[0055] In summary, compared with the traditional monosyllable recognition model and manual decoding into chords, the two-stage piano sound and chord recognition method based on deep learning adopts a neural network to model the chord type corresponding to the monosyllable output score, which can fit the data more accurately and obtain more accurate chord recognition results; compared with the method of directly modeling chord features, the elimination method of the present invention can not only simultaneously output the results of monosyllable and chord recognition of the current frame, but also the chord recognition model makes full use of the information at the monosyllable level, so it can achieve more accurate results; the chord recognition model proposed in the present invention can control the ratio of difficult and easy samples, so that the chord recognition effect is further improved.

[0056] In addition, the present invention takes the piano as an example when explaining the process, but it is not limited to the piano. The method proposed by the present invention can also be easily extended to instruments such as guitar and recorder.

[0057] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A two-stage piano sound and chord recognition method based on deep learning, characterized in that: The first stage is to train the monosyllable recognition model, and the second stage includes training the syllable recognition model, including the following specific steps: Phase 1: Step 1: Prepare training data: Prepare clean single-note audio data for training. In order to improve the recognition effect under noise, data enhancement is also required. Here, single-note audio data and different types of noise are added with different signal-to-noise ratios to obtain enhanced audio data with noise. These data sets are mixed together and divided into training set, validation set and test set according to 8:1:1; Step 2, feature extraction: perform windowing, framing, short-time Fourier transform, and Mel-triangle filtering on the data set to obtain the Mel-domain spectrum feature FBANK feature of the audio segment; Step 3: Model training: The extracted audio features are input into the training script of the single-note recognition model according to certain splicing rules for training to obtain a neural network model that can recognize single notes. Phase 2: Step 1: Chord data preparation: prepare audio data for chord training. Each time period needs to have a corresponding chord type / non-chord label. The training data also needs to be denoised and amplified. Step 2: Feature extraction: The clean audio and the noisy audio are windowed, framed, short-time Fourier transformed, and Mel triangle filtered to obtain the FBANK features of the chord audio segment; Step 3, forward reasoning: input the FBANK feature data of the chord audio segment into the monosyllable recognition model trained in stage 1, obtain an output score for each input frame, concatenate the output results of adjacent frames, and select them according to certain standards to obtain a dataset of concatenated scores; Step 4: Model training: Input the selected concatenated score data set in the previous step into the second-stage chord recognition network for training. Since each concatenated score segment has a corresponding chord label, the chord recognition network used in the second stage can be trained.

2. A two-stage piano sound and chord recognition method based on deep learning according to claim 1, characterized in that: In the first stage, in addition to the clean training data with the correct note text, the training data in step 1 also needs to add a certain amount of non-piano sound as negative samples, such as conversation, TV and street noise. Different noises with different signal-to-noise ratios are added to expand the training set under different noise environments. The signal-to-noise ratio is measured by SNR. The specific calculation formula is as follows: Among them, P s and P n Represent the effective power of clean audio and noisy audio respectively. When performing data enhancement, the reference signal-to-noise ratio level is between 5-20dB. The noisy audio is randomly selected from the noise set (the noise set contains various types of noise such as conversation, TV noise, street noise, etc.).

3. The two-stage piano sound and chord recognition method based on deep learning according to claim 1 is characterized in that: In step 2 of the first stage, 40-dimensional FBANK features are extracted from the audio signal of each frame. The reference setting is to use the Hamming window when adding windows, set the unit frame length to 25ms, and set the frame shift of each frame to 10ms. The feature extraction process of this step is used to extract features from all training sample pairs.

4. The two-stage piano sound and chord recognition method based on deep learning according to claim 1, characterized in that: In step three of the first stage, since the objects to be identified in the single-note recognition task are audio segment features with obvious temporal relationships, we consider using a recurrent neural network that can model this temporal relationship as the backbone of the single-note recognition network.

5. The two-stage piano sound and chord recognition method based on deep learning according to claim 1, characterized in that: In step 1 of the second stage, after the clean chord audio data is prepared, data enhancement and feature extraction are performed on it, and the chord labels corresponding to each frame feature are aligned.

6. The two-stage piano sound and chord recognition method based on deep learning according to claim 1, characterized in that: In step three of the second stage, the extracted features are spliced ​​and input into the single-note recognition model to obtain the single-note recognition score of the current frame. The output score is spliced ​​together by 30 frames, that is, the matrix shape of a sample is (61, 89). Since the input features have been spliced ​​before entering the single-note recognition model, the recognition results carry the information of adjacent frames. The sample matrix is ​​downsampled with a span of 3 frames, and the shape of the sample matrix after downsampling is (20, 89).

7. The two-stage piano sound and chord recognition method based on deep learning according to claim 1, characterized in that: In step three of the second stage, the chord recognition model can refer to the monosyllable recognition model of the first stage and be designed as a network with time series modeling capabilities and GRU as the model backbone. After completing data preparation and determining the second-stage model structure, the cross entropy loss function is used as the loss to iteratively train the chord recognition model.

8. The two-stage piano sound and chord recognition method based on deep learning according to claim 1, characterized in that: After the first and second stages, a monosyllable recognition model and a chord recognition model are obtained. During the first inference, the features of the starting audio segment of the first stage are input into the monosyllable recognition model, and the syllable recognition score is output. When the input frames accumulate to 61 frames, the second stage of chord recognition is performed. The scores of 61 frames are downsampled and spliced ​​together and input into the chord recognition model. In order to perform continuous inference, the first score that enters every 3-frame spliced ​​score sample will be eliminated, that is, the score of the first row of the sample matrix, and then the score of the latest frame will be added after the last row to perform continuous sample generation and inference.

Images