A method for removing lines from handwritten music scores based on deep learning

Abstract

The present invention discloses a method for deleting handwritten music score lines based on deep learning. The data set used for model training of the present invention is the ICDAR / GREC 2013 handwritten five-line staff line deletion competition data set, and the data set is data enhanced. The present invention uses three different data enhancement methods. The deep convolutional neural network model for deleting handwritten music score lines of the present invention inputs a model containing a music score image, and the model directly outputs a result graph of the music score image without the music score lines. The present method introduces three additional noises, namely rotation, line discontinuity, and line thickening, to simulate the music score in natural scenes, thereby improving the generalization ability of the model. A handwritten music score line deletion model is proposed, which achieves a deletion accuracy rate of 99.3%. In terms of real-time performance, it only takes 14.85 seconds to complete a music score image of size 1024*2048, which is greatly improved in real-time performance compared with the aforementioned model.

Description

Technical Field

[0001] The present invention belongs to the field of music information retrieval, and is a handwritten music score line deletion method based on deep learning, which can be applied to the preprocessing stage of handwritten music score recognition. Background Art

[0002] Optical music score recognition is a technology that converts music score image recognition into a machine-readable format. The general process of optical music score recognition is to delete the clef, then complete the segmentation and classification of notes, and then reconstruct the note information to obtain information such as pitch and duration, and finally convert the score into an editable and playable format, such as MIDI format. Although it is now possible to directly complete the recognition of printed music scores without deleting the clef, for the complex and diverse handwritten music score recognition, the clef and notes and other symbols overlap, making the note recognition difficult, the accuracy is low, and the recognition of the note type is incomplete. Therefore, the deletion of the clef of handwritten music scores has become an important preprocessing link for handwritten music score recognition. However, there are similarities between the clef of handwritten music scores and some handwritten notes (such as legato lines, crescendo and decrescendo symbols), and the quality of the music score image is uneven due to factors such as storage conditions and shooting conditions, which increases the difficulty of deleting the clef of handwritten music scores.

[0003] At present, there are three types of handwriting line deletion technologies: traditional methods based on line morphology feature analysis, machine learning methods, and deep learning methods. However, the steps of these methods are relatively redundant, such as Figure 1 As shown in the figure, the real-time performance is poor. It takes about 200 seconds for the model to complete the line deletion, and it is not possible to directly delete the line of any size score. In addition, the traditional method of deleting the line using the line features and the method based on machine learning can only process binary score images. When the line is irregular or the image quality is reduced, the traditional method of deleting the line is prone to residual line and mistaken deletion of notes.

[0004] As deep learning has made breakthroughs in the field of vision, the application of deep learning methods to the problem of deleting lines in handwritten music scores has mostly achieved good results. However, the current deep learning-based methods have the following problems: the amount of calculation is too large (such as using convolutional neural networks (CNN) to classify music score images pixel by pixel), the model is difficult to train and easy to crash (such as using adversarial generative networks (GAN)), and the model requires manual selection of appropriate thresholds, which can easily lead to image quality degradation (such as using auto-encoders). Summary of the invention

[0005] In order to solve the problems of redundant steps, weak noise adaptability and poor real-time performance in the current handwritten music score line deletion algorithm, an end-to-end handwritten music score line deletion model based on deep learning is designed. Handwritten music score images of any size are input into the model, and the model directly outputs music score images without lines.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is a method for deleting handwritten music lines based on deep learning. The method is divided into two steps:

[0007] (1) Data enhancement: The dataset used for model training in the present invention is the ICDAR / GREC 2013 handwritten stave line deletion competition dataset, and data enhancement is performed on the dataset. The present invention uses three different data enhancement methods.

[0008] (2) End-to-end model: Figure 2 The deep convolutional neural network model for deleting lines from handwritten music scores using the present invention is shown. The model is input with a music score image, and the result graph of the model directly outputting a music score image without lines.

[0009] The core algorithm of the present invention

[0010] (1) Data preprocessing

[0011] The dataset used for model training in the present invention is the ICDAR / GREC 2013 handwritten five-line staff line deletion competition dataset, which is obtained by performing different degradation processes on 1000 handwritten music score images, totaling 4000 training images of size 3400*2200 and 2000 test images. The training images include 1000 3-D distorted images generated by the 3D distortion model, and 1000 noisy images obtained by three different levels of local noise, wherein the higher the local noise level, the more corresponding noise points and the larger the noise area. And 2000 noisy music score images obtained by combining the above two degradation processes. In order to make the model of the present invention have better robustness to low-quality images and irregular music score images, the present invention uses three data enhancement methods, such as Figure 3 As shown, from top to bottom, the image is rotated to simulate different shooting angles, the spectrum lines are thickened to simulate the pollution of the score image, and the spectrum lines are interrupted to simulate the poor preservation quality of the score.

[0012] (2) End-to-end note recognition model

[0013] The process of the handwritten line removal model is as follows: a music score image of any size is input into a convolutional neural network, and after a series of convolution operations, the music score line features are extracted. Then, the extracted line features are subtracted from the input image to finally obtain a music score image without the lines.

[0014] The model structure diagram of the present invention is as follows Figure 4 As shown, it consists of six parts, namely Conv_head, Conv_body_1, Conv_body_2, Conv_body_3, Conv_body_4, and Conv_tail.

[0015] according to Figure 4 Model structure diagram We assume that the model input is Y, Y = X + V, we will train the deep neural network to fit R (Y) ≈ V, and finally output X' = YR (Y). Where X represents the image without staves, V represents the stave lines, R () represents the network model, Y represents the input image with staves, and X' represents the final network output image without staves. Compared with the existing method of using the network to directly fit R (Y) ≈ X, the reason for using the network to fit R (Y) ≈ V is that compared with the diversity of notes, the characteristics of the staves are more simple and obvious, so training the network to obtain the characteristics of the staves is more conducive to the rapid convergence of the model and the improvement of performance.

[0016] In the design of the network structure, the present invention draws on the advantages of the multi-scale neural network model. From Conv_body_2 to Conv_body_4, the size of the feature map is halved and the number of channels is doubled each time a module is passed. Finally, the features of different layers are fused using the concat operation. The advantage of this design is that the model will not learn information at a fixed scale and can combine information at different scales. For music score images with densely distributed notes, this will be beneficial to improving the model's feature extraction and the deletion accuracy.

[0017] In the model structure diagram, Conv_head is a 2-D convolution layer with a convolution kernel size of 3 and a channel number of 64. Conv_body1 consists of 6 residual attention modules, Conv_body2 consists of 8 residual attention modules, Conv_body3 consists of 10 residual attention modules, and Conv_body4 consists of 12 residual attention modules. The residual attention module structure is as follows Figure 5 As shown, we apply the Squeeze-and-Excitation attention module within the layer, as Figure 5 As shown in the figure, firstly, the feature quantity of the feature map is obtained by using the pooling layer, then the weights of different channels are obtained through two fully connected layers and the Sigmoid function, and then the weighted feature map is sent to the next module. The fully connected layer in this module is implemented by a convolution layer with a convolution kernel size of 1*1. The attention module can automatically learn the importance of different channels. The introduction of the attention mechanism will help strengthen important features, improve model accuracy, and reduce training difficulty.

[0018] In the residual attention module, the residual block is as follows Figure 6 As shown. In order to reduce the amount of calculation, the present invention only retains part of the batch normalization layer and the relu layer inside each residual module, wherein the first convolution layer is a convolution layer with a 1*1 convolution kernel, the purpose is to reduce the number of channels to half of the original, and then use the second convolution layer, the second convolution layer is a convolution layer with a convolution kernel size of 3*3 for feature extraction. Prelu indicates that the activation function is Prelu (activation function of learnable parameters), and the residual and represent the residual addition result. Such a design will greatly reduce the amount of calculation and help improve the real-time performance of the model.

[0019] Conv_tail is a 2D convolution layer with 1 channel and a convolution kernel size of 3*3.

[0020] The present invention proposes an end-to-end line deletion model for handwritten music score images, applies a deep convolutional neural network to extract line features, and finally outputs a music score image without line features. The experimental test results show that it only takes 14.85 seconds to recognize a whole music score image, and can obtain a 99.3% deletion accuracy rate. The calculation formula is as follows: Figure 7 , where TP, TN, FP, and FN represent the value of classifying notes as notes, the value of classifying background and spectrum as background and spectrum, the number of classifying notes as background and spectrum, and the number of classifying background and spectrum as notes, respectively.

[0021] The core technologies of this method include:

[0022] (1) Three additional noises, namely rotation, line interruption, and line thickening, are introduced to simulate music scores in natural scenes, thus improving the generalization ability of the model.

[0023] (2) A handwritten music score line deletion model was proposed, which achieved a deletion accuracy of 99.3%. It only took 14.85 seconds to complete a 1024*2048 music score image in real time, which was a significant improvement in real time compared to the previous model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a comparison chart of different methods.

[0025] Figure 2 This is a diagram showing the result of deleting lines of handwritten music score of the model of the present invention.

[0026] Figure 3 It is the data enhancement result graph.

[0027] Figure 4 It is a structural diagram of the present invention.

[0028] Figure 5This is the structure diagram of the attention residual module.

[0029] Figure 6 It is the structural diagram of the residual module.

[0030] Figure 7 This is the accuracy calculation formula diagram. DETAILED DESCRIPTION

[0031] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0032] The technical solution adopted by the present invention is a method for deleting handwritten music score lines based on deep learning, which includes the following steps: (1) data enhancement: for each selected music score image, the entire music score image is randomly cropped into an image of 256*512 size, and this operation is performed 4 times so that the total amount of data is expanded by 4 times. Then, three types of noise, such as rotation, line discontinuity, and line thickening, are introduced to enhance the data, and then the image is input into the neural network model.

[0033] (2) After the data is input into the neural network model, the model is trained using the ADAM optimizer. The selected batch size is 8, the initial learning rate is 0.001, and the learning rate is adjusted using an adaptive method. When the accuracy on the validation set does not decrease for 5 consecutive epochs, the learning rate is adjusted to one-tenth of the original. After about 28 epochs, the model begins to converge. Two 1080Tis are used for training, and the model is trained in about 5 hours.

[0034] (3) After the neural network model is trained, a handwritten music score image of any size is input, and the model can output a music score image without musical lines.

Claims

1. A method for deleting handwritten music lines based on deep learning, characterized in that: This method consists of two steps: (1) Data enhancement: The dataset used for model training is the ICDAR / GREC 2013 handwritten stave line deletion competition dataset, and data enhancement is performed on this dataset; (2) End-to-end model: A deep convolutional neural network model for deleting handwritten music lines is shown. The model inputs a music score image, and the model directly outputs a music score image without the lines. The model structure consists of six parts, namely Conv_head, Conv_body_1, Conv_body_2, Conv_body_3, Conv_body_4, Conv_tail; According to the model structure, the model input is assumed to be Y, Y = X + V, and the deep neural network is trained to fit R (Y) ≈ V, and the final output is X' = YR (Y); where X represents the image without staves, V represents the stave lines, R () represents the network model, Y represents the input image with staves, and X' represents the final network output image without staves; In the design of the network structure, from Conv_body_2 to Conv_body_4, the size of the feature map is halved and the number of channels is doubled after each module. Finally, the features of different layers are fused using the concat operation to combine information at different scales. In the model structure diagram, Conv_head is a 2-D convolution layer with a convolution kernel size of 3 and 64 channels; Conv_body1 consists of 6 residual attention modules, Conv_body2 consists of 8 residual attention modules, Conv_body3 consists of 10 residual attention modules, and Conv_body4 consists of 12 residual attention modules. In the residual attention module structure, the Squeeze-and-Excitation attention module is applied in the layer. First, the feature map is obtained by using the pooling layer, and then the weights of different channels are obtained through two fully connected layers and the Sigmoid function, and then the weighted feature map is sent to the next module; the fully connected layer in the module is realized by a convolution layer with a convolution kernel size of 1*1; In the residual attention module, in order to reduce the amount of calculation, this paper only retains part of the batch normalization layer and the relu layer inside each residual module. The first convolution layer is a convolution layer with a 1*1 convolution kernel, the purpose is to reduce the number of channels to half of the original, and then use the second convolution layer, the second convolution layer is a convolution layer with a convolution kernel size of 3*3 for feature extraction; Prelu means the activation function is Prelu, and the residual sum represents the residual addition result.

2. The method for deleting handwritten music lines based on deep learning according to claim 1, characterized in that: The dataset used for model training is the ICDAR / GREC 2013 handwritten five-line stave line removal competition dataset, which is obtained by performing different degradation treatments on 1000 handwritten music score images, totaling 4000 training images of size 3400*2200 and 2000 test images; the training images include 1000 3D distorted images generated by the 3D distortion model, and 1000 noisy images obtained by three different levels of local noise; three data enhancement methods are used, from top to bottom, respectively, image rotation to simulate different shooting angles, stave line thickening to simulate the pollution of music score images, and stave line interruption to simulate music scores with poor preservation quality.

3. The method for deleting handwritten music lines based on deep learning according to claim 1, characterized in that: The process of the handwritten line deletion model is as follows: a music score image of any size is input into a convolutional neural network, and after a convolution operation, the music score line features are extracted. Then, the extracted line features are subtracted from the input image to finally obtain a music score image without the lines.

4. The method for deleting handwritten music lines based on deep learning according to claim 1, characterized in that: Conv_tail is a 2D convolution layer with 1 channel and a convolution kernel size of 3*3. An end-to-end spectrum line deletion model is proposed for handwritten music score images, and a deep convolutional neural network is used to extract spectrum line features, and finally outputs a music score image without spectrum lines.

Images