Target detection-based Chinese lute music score identification method and system

Abstract

The invention provides a loquat music score recognition method and system based on target detection in the technical field of loquat music score recognition, and the method comprises the steps: S1, obtaining a large number of music score images of a loquat music score, and carrying out the image enhancement preprocessing of each music score image; step S2, music symbols in the preprocessed music score images are labeled through Labe l Img, and corresponding labeled files are obtained; s3, storing the music score images and the annotation files, and constructing a data set based on the music score images and the annotation files; s4, creating a lute music score recognition model based on the YOLOv5 module and the convolution attention module; s5, training a Chinese lute music score recognition model through the data set; and S6, deploying the trained Chinese lute music score recognition model in a server, and calling the deployed Chinese lute music score recognition model online to recognize the Chinese lute music score. The method has the advantage that the Chinese lute music score recognition precision is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of pipa music score recognition, and in particular to a pipa music score recognition method and system based on target detection. Background Art

[0002] With the development of computers, computers are now widely used in the field of music, covering multiple sub-fields such as music entertainment, creation, and teaching, such as multimedia playback, arrangement software, composition software, song recognition, etc., which are inseparable from the digitization of music scores.

[0003] The digitization of sheet music can reduce the corresponding paper materials and thus improve space utilization. It can make full use of the processing power of computers to increase the retrieval speed of sheet music. The conversion of music information into digital information can serve as a reference for the reverse conversion process of the two types of information, which can be used in music production and in forming robot bands, laying the foundation for artificial intelligence to play musical instruments.

[0004] The premise of music score digitization is to identify the music score. However, traditional recognition methods are mainly aimed at five-line notation, while pipa music score is expanded based on the simplified notation form, which has many differences from the five-line notation structure. As a result, when traditional recognition methods are applied to pipa music score recognition, the recognition accuracy is not satisfactory.

[0005] Therefore, how to provide a pipa music score recognition method and system based on target detection to improve the accuracy of pipa music score recognition has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a pipa music score recognition method and system based on target detection, so as to improve the pipa music score recognition accuracy.

[0007] In a first aspect, the present invention provides a method for recognizing pipa music scores based on target detection, comprising the following steps:

[0008] Step S1, obtaining a large number of music score images of pipa music scores, and performing image enhancement preprocessing on each of the music score images;

[0009] Step S2, labeling the music symbols in each of the pre-processed music score images through Label Img to obtain a corresponding labeling file;

[0010] Step S3, storing each of the music score images and the annotation files, and constructing a data set based on each of the music score images and the annotation files;

[0011] Step S4, creating a pipa music score recognition model based on the YOLOv5 module and the convolutional attention module;

[0012] Step S5, training the pipa music score recognition model using the data set;

[0013] Step S6: deploy the trained pipa music score recognition model on the server, and call the deployed pipa music score recognition model online to recognize the pipa music score.

[0014] Furthermore, the step S1 is specifically as follows:

[0015] A large number of music score images of pipa music scores in PNG format are obtained, and noise reduction is performed on each of the music score images by using the maximum inter-class variance method to complete the preprocessing of image enhancement;

[0016] The denoising of each of the music score images by the maximum inter-class variance method is specifically as follows: calculating the grayscale histogram, foreground, background, pixel probability and average grayscale value under each preset threshold of each of the music score images, calculating the inter-class variance of the grayscale histogram, foreground, background, pixel probability and average grayscale value, taking the threshold corresponding to the inter-class variance with the largest value as the binarization threshold, and denoising each of the music score images based on the binarization threshold.

[0017] Furthermore, the step S2 is specifically as follows:

[0018] Labeling the music symbols in each of the preprocessed music score images in turn through Label Img to obtain corresponding label vectors including note category, attribute information and position information;

[0019] The note category is a bar line, a quarter rest, a sharp, a flat, a tie, a plucking, a picking, a double plucking, a double picking, a sweep, a brush, a small sweep, a full round, a round finger, a half round, a three-finger round, a harmonic, a chant, a hit, a band or a stroke; the attribute information includes at least a pitch and a duration; the position information includes at least an x-coordinate of a music symbol, a y-coordinate of a music symbol, a width of a target frame, and a height of a target frame;

[0020] A corresponding annotation file is generated based on each label vector of each of the music score images.

[0021] Furthermore, the step S3 is specifically as follows:

[0022] The same file name is set for each of the score images and the corresponding annotation file samples, each of the score images is stored in a pre-created image folder, each of the annotation files is stored in a pre-created label folder, and a data set is constructed based on each of the score images and the annotation files.

[0023] Further, in step S4, the YOLOv5 module is used to identify music symbols from the music score image, and is provided with four detection heads for identifying music symbols of different scales, and the detection resolutions of the detection heads are 20×20, 40×40, 80×80, and 160×160 respectively; the convolutional attention module is provided with a channel attention unit and a spatial attention unit, which are respectively used to extract channel features and spatial features from the music score image to assist the YOLOv5 module in music symbol recognition;

[0024] The step S5 is specifically as follows:

[0025] The data set is divided into a training set, a validation set and a test set based on a ratio of 7:2:1; after the pipa music score recognition model is trained by the training set, the pipa music score recognition model is verified by the validation set, and based on the verification result, the model parameters of the pipa music score recognition model are fine-tuned to optimize the crossover ratio and confidence, and then the verified pipa music score recognition model is tested by using the test set to complete the training of the pipa music score recognition model.

[0026] In a second aspect, the present invention provides a pipa music score recognition system based on target detection, comprising the following modules:

[0027] A music score image preprocessing module is used to obtain a large number of music score images of pipa music scores and perform image enhancement preprocessing on each of the music score images;

[0028] A music score image annotation module, used to annotate the music symbols in each of the pre-processed music score images through Label Img to obtain a corresponding annotation file;

[0029] A data set construction module, used to store each of the music score images and annotation files, and to construct a data set based on each of the music score images and annotation files;

[0030] A pipa music score recognition model creation module is used to create a pipa music score recognition model based on the YOLOv5 module and the convolutional attention module;

[0031] A pipa music score recognition model training module, used for training a pipa music score recognition model using the data set;

[0032] The pipa music score recognition module is used to deploy the trained pipa music score recognition model on the server, and call the deployed pipa music score recognition model online to recognize the pipa music score.

[0033] Furthermore, the music score image preprocessing module is specifically used for:

[0034] A large number of music score images of pipa music scores in PNG format are obtained, and noise reduction is performed on each of the music score images by using the maximum inter-class variance method to complete the preprocessing of image enhancement;

[0035] The denoising of each of the music score images by the maximum inter-class variance method is specifically as follows: calculating the grayscale histogram, foreground, background, pixel probability and average grayscale value under each preset threshold of each of the music score images, calculating the inter-class variance of the grayscale histogram, foreground, background, pixel probability and average grayscale value, taking the threshold corresponding to the inter-class variance with the largest value as the binarization threshold, and denoising each of the music score images based on the binarization threshold.

[0036] Furthermore, the music score image annotation module is specifically used for:

[0037] Labeling the music symbols in each of the preprocessed music score images in turn through Label Img to obtain corresponding label vectors including note category, attribute information and position information;

[0038] The note category is a bar line, a quarter rest, a sharp, a flat, a tie, a plucking, a picking, a double plucking, a double picking, a sweep, a brush, a small sweep, a full round, a round finger, a half round, a three-finger round, a harmonic, a chant, a hit, a band or a stroke; the attribute information includes at least a pitch and a duration; the position information includes at least an x-coordinate of a music symbol, a y-coordinate of a music symbol, a width of a target frame, and a height of a target frame;

[0039] A corresponding annotation file is generated based on each label vector of each of the music score images.

[0040] Furthermore, the data set construction module is specifically used for:

[0041] The same file name is set for each of the score images and the corresponding annotation file samples, each of the score images is stored in a pre-created image folder, each of the annotation files is stored in a pre-created label folder, and a data set is constructed based on each of the score images and the annotation files.

[0042] Furthermore, in the pipa music score recognition model creation module, the YOLOv5 module is used to recognize music symbols from the music score image, and is provided with four detection heads for recognizing music symbols of different scales, and the detection resolutions of each detection head are 20×20, 40×40, 80×80, and 160×160 respectively; the convolutional attention module is provided with a channel attention unit and a spatial attention unit, which are respectively used to extract channel features and spatial features from the music score image to assist the YOLOv5 module in music symbol recognition;

[0043] The pipa music score recognition model training module is specifically used for:

[0044] The data set is divided into a training set, a validation set and a test set based on a ratio of 7:2:1; after the pipa music score recognition model is trained by the training set, the pipa music score recognition model is verified by the validation set, and based on the verification result, the model parameters of the pipa music score recognition model are fine-tuned to optimize the crossover ratio and confidence, and then the verified pipa music score recognition model is tested by using the test set to complete the training of the pipa music score recognition model.

[0045] The advantages of the present invention are:

[0046] A large number of music score images of pipa music scores are obtained and preprocessed for image enhancement, and the music symbols in each preprocessed music score image are annotated by Label Img to obtain the corresponding annotation file, and each music score image and the annotation file are stored, and a data set is constructed based on each music score image and the annotation file; then a pipa music score recognition model is created based on the YOLOv5 module and the convolutional attention module, and the pipa music score recognition model is trained with the data set, and finally the trained pipa music score recognition model is deployed on the server, and the deployed pipa music score recognition model is called online to recognize the pipa music score; that is, the pipa music score recognition model is trained by obtaining the annotation file and the data set constructed by the music score image by annotating the music symbols, and the pipa music score recognition model is equipped with a YOLOv5 The convolutional attention module extracts channel features and spatial features from the music score image through the channel attention unit and the spatial attention unit respectively, and then assists the YOLOv5 module to recognize music symbols through the channel features and spatial features, so that the pipa music score recognition model can focus on the music symbols. The YOLOv5 module of the pipa music score recognition model is improved, and a detection head with a detection resolution of 160×160 is added to improve the recognition effect of small targets. The music score image is denoised by combining the maximum inter-class variance method to improve the quality of the data set, thereby improving the training effect of the pipa music score recognition model, and ultimately greatly improving the accuracy of drum score recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0048] Figure 1 The present invention is a flow chart of a method for recognizing pipa music scores based on target detection.

[0049] Figure 2 It is a structural schematic diagram of a pipa music score recognition system based on target detection of the present invention. DETAILED DESCRIPTION

[0050] The technical solution in the embodiment of the present application has the following overall idea: the pipa music score recognition model is trained by obtaining a data set of annotation files and music score images obtained by annotating music symbols, and the pipa music score recognition model is provided with a YOLOv5 module and a convolutional attention module. The convolutional attention module extracts channel features and spatial features from the music score image through a channel attention unit and a spatial attention unit respectively, and then assists the YOLOv5 module in music symbol recognition through the channel features and the spatial features, so that the pipa music score recognition model can focus on the music symbols, and the YOLOv5 module of the pipa music score recognition model is improved, and a detection head with a detection resolution of 160×160 is added to improve the recognition effect of small targets, and the music score image is denoised by combining the maximum inter-class variance method to improve the quality of the data set, so as to improve the training effect of the pipa music score recognition model, and then improve the accuracy of drum score recognition.

[0051] Please refer to Figure 1 to Figure 2 As shown, a preferred embodiment of a method for recognizing pipa music scores based on target detection of the present invention comprises the following steps:

[0052] Step S1, obtaining a large number of music score images of pipa music scores, and performing image enhancement preprocessing on each of the music score images;

[0053] Step S2, labeling the music symbols in each of the pre-processed music score images through Label Img to obtain a corresponding labeling file;

[0054] Step S3, storing each of the music score images and the annotation files, and constructing a data set based on each of the music score images and the annotation files;

[0055] Step S4, creating a pipa music score recognition model based on the YOLOv5 module and the convolutional attention module;

[0056] Step S5, training the pipa music score recognition model using the data set;

[0057] Step S6: deploy the trained pipa music score recognition model on the server, and call the deployed pipa music score recognition model online to recognize the pipa music score.

[0058] In the specific implementation, after the music symbols in the pipa music score are recognized and the bounding box is marked in the music score image for visualization, it is also necessary to consider outputting it as a computer symbol in a form close to the music theory logic and saving it in a txt format. The present invention realizes the recognition and extraction of music symbols in the pipa music score by improving the detect.py file of YOLOv5, and outputs it as a txt file in a form consistent with the music theory logic.

[0059] The recognition process is divided into two main parts: first, the rows of pipa music scores are extracted through the detected bar line orientation, and the music symbols are classified into corresponding rows according to the vertical position and spacing of the bar lines; second, within each row, the music symbols are further classified into each bar according to the x-coordinate of the bar line and the position information of the detected music symbols; finally, the text sequence of the pipa music score is output in order from top to bottom and from left to right, taking the bars in each row as the unit to ensure the logic and accuracy of the result.

[0060] The step S1 is specifically as follows:

[0061] A large number of music score images of pipa music scores in PNG format are obtained, and noise reduction is performed on each of the music score images by using the maximum inter-class variance method (OTSU threshold method) to complete the preprocessing of image enhancement;

[0062] Considering that the input music score image of the pipa music score is blurred, noisy or disturbed, it may affect the accuracy and stability of recognition to a certain extent, so image enhancement preprocessing is performed;

[0063] The denoising of each of the music score images by the maximum inter-class variance method is specifically as follows: calculating the grayscale histogram, foreground, background, pixel probability and average grayscale value under each preset threshold of each of the music score images, calculating the inter-class variance of the grayscale histogram, foreground, background, pixel probability and average grayscale value, taking the threshold corresponding to the inter-class variance with the largest value as the binarization threshold, and denoising each of the music score images based on the binarization threshold.

[0064] The step S2 is specifically as follows:

[0065] Labeling the music symbols in each of the preprocessed music score images in turn through Label Img to obtain corresponding label vectors including note category, attribute information and position information;

[0066] The note category is a bar line, a quarter rest, a sharp, a flat, a tie, a plucking, a picking, a double plucking, a double picking, a sweep, a brush, a small sweep, a full round, a round finger, a half round, a three-finger round, a harmonic, a chant, a hit, a band or a stroke; the attribute information includes at least a pitch and a duration; the position information includes at least an x-coordinate of a music symbol, a y-coordinate of a music symbol, a width of a target frame, and a height of a target frame;

[0067] Generate a corresponding annotation file in txt format based on each label vector of each of the music score images.

[0068] Pipa music is mainly based on the form of simplified notation, and adds action symbols such as plucking, picking, and rolling to improve the whole process of pipa performance. Letters or numbers are used to represent pitch, and different symbols and lines are used to represent the duration and rhythm of notes. In terms of pitch, pipa music is sung in the form of tonic sol-fa method, in which the Arabic numerals 1, 2, 3, 4, 5, 6, and 7 correspond to the seven notes do, re, mi, fa, so l, la, and si respectively. The 7 Arabic numerals are equivalent to the natural scale of the major key. Take C major as an example, as shown in the following table:

[0069] scale C D E F G A B name Do Re Mi Fa Sol La Xi symbol 1 2 3 4 5 6 7

[0070] In actual works, in order to indicate more musical symbols of pitch, a high dot "·" is usually added to the Arabic numerals to indicate an octave higher, and a low dot "·" is added under the Arabic numerals to indicate an octave lower, and so on. Adding several high dots or low dots indicates several octaves higher or lower. However, through the collection and research of a large number of pipa scores, it is found that in general pipa scores, it is extremely rare to add more than two "·" above or below the notes. It is more common to add low octave, middle and high octave notes. Take C major as an example, as shown in the following table:

[0071]

[0072] In terms of duration, a single number in the simplified notation represents a quarter note, which is the most basic unit of duration in music. In pipa music, the change of the duration of the note is usually adjusted by adding horizontal lines. When a shorter duration is indicated, it is indicated by adding horizontal lines below the pitch indicated by the Arabic numerals; when a horizontal line is added below the number, the length of the note is halved, that is, an eighth note, which is half the duration of a quarter note; if another horizontal line is added, the duration of the note is further halved, that is, a sixteenth note, which is one-quarter the duration of a quarter note, and so on. When a longer duration needs to be indicated, it is indicated by adding horizontal lines after the pitch indicated by the Arabic numerals. When a horizontal line is added after the number, the length of the note doubles, and the duration becomes twice the duration of a quarter note; when two horizontal lines are added after the number, the length of the note doubles, and the duration becomes three times the duration of a quarter note; and so on, each additional horizontal line at the end increases the duration of the note by the length of a quarter note. For example, if you add a dash after the number 4 (4-), it changes from a quarter note to a half note. If you add three dashes (4---), it becomes a whole note, which is four times the length of a quarter note. Taking a quarter note as one beat, take note "1" as an example, as shown in the following table:

[0073]

[0074] Rests are represented by the number "0" in the simplified musical notation, representing a temporary pause or absence of sound. Rests also have rhythmic properties in music, and their duration can also be adjusted by adding a horizontal line. For example, a single "0" may represent a quarter note rest, while adding a horizontal line below it represents an eighth note rest, and so on. It is worth mentioning that the vast majority of eighth note notes, quarter rests, and larger durations consisting of several quarter rests appear in pipa music scores. In addition, pipa has a variety of playing techniques, including plucking, picking, and rolling. Among them, "plucking" is the most basic technique of pipa, which involves the plucking of fingers to make the strings vibrate and produce sound; "picking" is to use the index finger and thumb to pick up the strings, so that the strings vibrate and produce sound. Based on the above pipa music theory knowledge, the present invention constructs the following rules for the musical note information in the data set of pipa music scores:

[0075] In view of the fact that in the pipa music score, the duration and pitch of each note complement each other, the duration and pitch of the note are usually closely related, and the musical notes of different pitches and durations show diversified forms of expression. Therefore, combining duration and pitch as annotation information helps to more accurately capture the characteristics and changes of the music content, and provides more comprehensive and detailed basic data for subsequent music analysis and processing. Therefore, the present invention uses the common low octave, middle octave, high octave three octave range of the pipa music score note pitch as the pitch range for identification, and the duration and pitch in the dotted quarter note and the note with shorter duration in the pipa music score are classified as information sorting targets, and the musical symbol whose duration is greater than the dotted quarter note, because the short dash is recorded separately from the note, then the duration is judged by the number of short dashes after the note. Therefore, in terms of duration, the present invention records the common duration range, which is divided into sixteenth notes, eighth notes, dotted eighth notes, quarter notes, and dotted quarter notes from small to large, and are recorded as 1, 2, 3, 4, and 5 in the first digit of each symbol code respectively; the pitch is divided into seven types: C, D, E, F, G, A, and B, which correspond to the Arabic numerals 1, 2, 3, 4, 5, 6, and 7 in the pipa music score respectively, and the notes of the lower octave are recorded with 1, the notes of the central octave are recorded with 2, and the notes of the higher octave are recorded with 3. For example, a quarter note with a pitch of 5 is recorded as 4G2; a dotted quarter note with a pitch of 1 in the higher octave is recorded as 5C3.

[0076] In addition, other common musical symbols, such as bar lines, rests, short horizontal lines for extending the duration, sharps and flats, plucking, picking, and rounds, are targeted for classification, so as to classify and mark basic notes more comprehensively. The vast majority of the musical symbols in the pipa score are eighth note symbols, quarter rests, and rests of a larger duration composed of several quarter rests, so the present invention marks quarter rests and eighth rests as musical note primitive units.

[0077] The step S3 is specifically as follows:

[0078] The same file name is set for each of the score images and the corresponding annotation file samples, each of the score images is stored in a pre-created image folder, each of the annotation files is stored in a pre-created label folder, and a data set is constructed based on each of the score images and the annotation files.

[0079] In step S4, the YOLOv5 module is used to identify music symbols from the music score image, and is provided with four detection heads for identifying music symbols of different scales, and the detection resolutions of the detection heads are 20×20, 40×40, 80×80, and 160×160 respectively; the convolutional attention module is provided with a channel attention unit and a spatial attention unit, which are respectively used to extract channel features and spatial features from the music score image to assist the YOLOv5 module in music symbol recognition;

[0080] YOLOv5 is a target detection model based on deep learning. It adopts a single-stage detection architecture and regards the target detection task as a regression problem. It detects directly on the image by dividing the image into grids and predicting bounding boxes and class probabilities for each grid. YOLOv5 introduces a more efficient model structure, including feature fusion of the outputs of CSPNet and PANet for feature extraction, and uses Focal Loss to solve the problem of class imbalance, which can achieve a good balance between real-time and accuracy. In the basic architecture of YOLOv5, there are usually three detection heads, corresponding to feature maps of 80x80, 40x40, and 20x20, responsible for target detection of different scales. Since the music symbols (targets) in the pipa score are usually small, the present invention adds a detection head with a detection resolution of 160x160, which provides a higher resolution, so that small targets occupy more pixels in the feature map, thereby more accurately detecting the music symbols in the pipa score.

[0081] In addition, in order to further improve the accuracy of pipa music score recognition, a convolutional attention module (CBAM) is added. The convolutional attention module is equipped with a channel attention unit and a spatial attention unit. The channel attention unit first aggregates the global features of each channel through global average pooling and maximum pooling, and then generates channel weights through a fully connected layer to enhance the features of important channels and suppress unimportant channels; the spatial attention unit performs maximum pooling and average pooling on the input feature map in the channel dimension, and generates a spatial weight map through a convolutional layer after splicing to emphasize important spatial areas; then the convolutional attention module needs to be inserted into each C3 module in the backbone network (Backbone) of YOLOv5.

[0082] The step S5 is specifically as follows:

[0083] The data set is divided into a training set, a validation set and a test set based on a ratio of 7:2:1; after the pipa music score recognition model is trained by the training set, the pipa music score recognition model is verified by the validation set, and based on the verification result, the model parameters of the pipa music score recognition model are fine-tuned to optimize the crossover ratio and confidence, and then the verified pipa music score recognition model is tested by using the test set to complete the training of the pipa music score recognition model.

[0084] A preferred embodiment of a pipa music score recognition system based on target detection of the present invention comprises the following modules:

[0085] A music score image preprocessing module is used to obtain a large number of music score images of pipa music scores and perform image enhancement preprocessing on each of the music score images;

[0086] A music score image annotation module, used to annotate the music symbols in each of the pre-processed music score images through Label Img to obtain a corresponding annotation file;

[0087] A data set construction module, used to store each of the music score images and annotation files, and to construct a data set based on each of the music score images and annotation files;

[0088] A pipa music score recognition model creation module is used to create a pipa music score recognition model based on the YOLOv5 module and the convolutional attention module;

[0089] A pipa music score recognition model training module, used for training a pipa music score recognition model using the data set;

[0090] The pipa music score recognition module is used to deploy the trained pipa music score recognition model on the server, and call the deployed pipa music score recognition model online to recognize the pipa music score.

[0091] In the specific implementation, after the music symbols in the pipa music score are recognized and the bounding box is marked in the music score image for visualization, it is also necessary to consider outputting it as a computer symbol in a form close to the music theory logic and saving it in a txt format. The present invention realizes the recognition and extraction of music symbols in the pipa music score by improving the detect.py file of YOLOv5, and outputs it as a txt file in a form consistent with the music theory logic.

[0092] The recognition process is divided into two main parts: first, the rows of pipa music scores are extracted through the detected bar line orientation, and the music symbols are classified into corresponding rows according to the vertical position and spacing of the bar lines; second, within each row, the music symbols are further classified into each bar according to the x-coordinate of the bar line and the position information of the detected music symbols; finally, the text sequence of the pipa music score is output in order from top to bottom and from left to right, taking the bars in each row as the unit to ensure the logic and accuracy of the result.

[0093] The music score image preprocessing module is specifically used for:

[0094] A large number of music score images of pipa music scores in PNG format are obtained, and noise reduction is performed on each of the music score images by using the maximum inter-class variance method (OTSU threshold method) to complete the preprocessing of image enhancement;

[0095] Considering that the input music score image of the pipa music score is blurred, noisy or disturbed, it may affect the accuracy and stability of recognition to a certain extent, so image enhancement preprocessing is performed;

[0096] The denoising of each of the music score images by the maximum inter-class variance method is specifically as follows: calculating the grayscale histogram, foreground, background, pixel probability and average grayscale value under each preset threshold of each of the music score images, calculating the inter-class variance of the grayscale histogram, foreground, background, pixel probability and average grayscale value, taking the threshold corresponding to the inter-class variance with the largest value as the binarization threshold, and denoising each of the music score images based on the binarization threshold.

[0097] The music score image annotation module is specifically used for:

[0098] Labeling the music symbols in each of the preprocessed music score images in turn through Label Img to obtain corresponding label vectors including note category, attribute information and position information;

[0099] The note category is a bar line, a quarter rest, a sharp, a flat, a tie, a plucking, a picking, a double plucking, a double picking, a sweep, a brush, a small sweep, a full round, a round finger, a half round, a three-finger round, a harmonic, a chant, a hit, a band or a stroke; the attribute information includes at least a pitch and a duration; the position information includes at least an x-coordinate of a music symbol, a y-coordinate of a music symbol, a width of a target frame, and a height of a target frame;

[0100] Generate a corresponding annotation file in txt format based on each label vector of each of the music score images.

[0101] Pipa music is mainly based on the form of simplified notation, and adds action symbols such as plucking, picking, and rolling to improve the whole process of pipa performance. Letters or numbers are used to represent pitch, and different symbols and lines are used to represent the duration and rhythm of notes. In terms of pitch, pipa music is sung in the form of tonic sol-fa method, in which the Arabic numerals 1, 2, 3, 4, 5, 6, and 7 correspond to the seven notes do, re, mi, fa, so l, la, and si respectively. The 7 Arabic numerals are equivalent to the natural scale of the major key. Take C major as an example, as shown in the following table:

[0102] scale C D E F G A B name Do Re Mi Fa Sol La Xi symbol 1 2 3 4 5 6 7

[0103] In actual works, in order to indicate more musical symbols of pitch, a high dot "·" is usually added to the Arabic numerals to indicate an octave higher, and a low dot "·" is added under the Arabic numerals to indicate an octave lower, and so on. Adding several high dots or low dots indicates several octaves higher or lower. However, through the collection and research of a large number of pipa scores, it is found that in general pipa scores, it is extremely rare to add more than two "·" above or below the notes. It is more common to add low octave, middle and high octave notes. Take C major as an example, as shown in the following table:

[0104]

[0105] In terms of duration, a single number in the simplified notation represents a quarter note, which is the most basic unit of duration in music. In pipa music, the change of the duration of the note is usually adjusted by adding horizontal lines. When a shorter duration is indicated, it is indicated by adding horizontal lines below the pitch indicated by the Arabic numerals; when a horizontal line is added below the number, the length of the note is halved, that is, an eighth note, which is half the duration of a quarter note; if another horizontal line is added, the duration of the note is further halved, that is, a sixteenth note, which is one-quarter the duration of a quarter note, and so on. When a longer duration needs to be indicated, it is indicated by adding horizontal lines after the pitch indicated by the Arabic numerals. When a horizontal line is added after the number, the length of the note doubles, and the duration becomes twice the duration of a quarter note; when two horizontal lines are added after the number, the length of the note doubles, and the duration becomes three times the duration of a quarter note; and so on, each additional horizontal line at the end increases the duration of the note by the length of a quarter note. For example, if you add a dash after the number 4 (4-), it changes from a quarter note to a half note. If you add three dashes (4---), it becomes a whole note, which is four times the length of a quarter note. Taking a quarter note as one beat, take note "1" as an example, as shown in the following table:

[0106]

[0107] Rests are represented by the number "0" in the simplified musical notation, representing a temporary pause or absence of sound. Rests also have rhythmic properties in music, and their duration can also be adjusted by adding a horizontal line. For example, a single "0" may represent a quarter note rest, while adding a horizontal line below it represents an eighth note rest, and so on. It is worth mentioning that the vast majority of eighth note notes, quarter rests, and larger durations consisting of several quarter rests appear in pipa music scores. In addition, pipa has a variety of playing techniques, including plucking, picking, and rolling. Among them, "plucking" is the most basic technique of pipa, which involves the plucking of fingers to make the strings vibrate and produce sound; "picking" is to use the index finger and thumb to pick up the strings, so that the strings vibrate and produce sound. Based on the above pipa music theory knowledge, the present invention constructs the following rules for the musical note information in the data set of pipa music scores:

[0108] In view of the fact that in the pipa music score, the duration and pitch of each note complement each other, the duration and pitch of the note are usually closely related, and the musical notes of different pitches and durations show diversified forms of expression. Therefore, combining duration and pitch as annotation information helps to more accurately capture the characteristics and changes of the music content, and provides more comprehensive and detailed basic data for subsequent music analysis and processing. Therefore, the present invention uses the common low octave, middle octave, high octave three octave range of the pipa music score note pitch as the pitch range for identification, and the duration and pitch in the dotted quarter note and the note with shorter duration in the pipa music score are classified as information sorting targets, and the musical symbol whose duration is greater than the dotted quarter note, because the short dash is recorded separately from the note, then the duration is judged by the number of short dashes after the note. Therefore, in terms of duration, the present invention records the common duration range, which is divided into sixteenth notes, eighth notes, dotted eighth notes, quarter notes, and dotted quarter notes from small to large, and are recorded as 1, 2, 3, 4, and 5 in the first digit of each symbol code respectively; the pitch is divided into seven types: C, D, E, F, G, A, and B, which correspond to the Arabic numerals 1, 2, 3, 4, 5, 6, and 7 in the pipa music score respectively, and the notes of the lower octave are recorded with 1, the notes of the central octave are recorded with 2, and the notes of the higher octave are recorded with 3. For example, a quarter note with a pitch of 5 is recorded as 4G2; a dotted quarter note with a pitch of 1 in the higher octave is recorded as 5C3.

[0109] In addition, other common musical symbols, such as bar lines, rests, short horizontal lines for extending the duration, sharps and flats, plucking, picking, and rounds, are targeted for classification, so as to classify and mark basic notes more comprehensively. The vast majority of the musical symbols in the pipa score are eighth note symbols, quarter rests, and rests of a larger duration composed of several quarter rests, so the present invention marks quarter rests and eighth rests as musical note primitive units.

[0110] The data set construction module is specifically used for:

[0111] The same file name is set for each of the score images and the corresponding annotation file samples, each of the score images is stored in a pre-created image folder, each of the annotation files is stored in a pre-created label folder, and a data set is constructed based on each of the score images and the annotation files.

[0112] In the pipa music score recognition model creation module, the YOLOv5 module is used to recognize music symbols from the music score image, and is provided with four detection heads for recognizing music symbols of different scales, and the detection resolutions of each detection head are 20×20, 40×40, 80×80, and 160×160 respectively; the convolutional attention module is provided with a channel attention unit and a spatial attention unit, which are respectively used to extract channel features and spatial features from the music score image to assist the YOLOv5 module in music symbol recognition;

[0113] YOLOv5 is a target detection model based on deep learning. It adopts a single-stage detection architecture and regards the target detection task as a regression problem. It detects directly on the image by dividing the image into grids and predicting bounding boxes and class probabilities for each grid. YOLOv5 introduces a more efficient model structure, including feature fusion of the outputs of CSPNet and PANet for feature extraction, and uses Focal Loss to solve the problem of class imbalance, which can achieve a good balance between real-time and accuracy. In the basic architecture of YOLOv5, there are usually three detection heads, corresponding to feature maps of 80x80, 40x40, and 20x20, responsible for target detection of different scales. Since the music symbols (targets) in the pipa score are usually small, the present invention adds a detection head with a detection resolution of 160x160, which provides a higher resolution, so that small targets occupy more pixels in the feature map, thereby more accurately detecting the music symbols in the pipa score.

[0114] In addition, in order to further improve the accuracy of pipa music score recognition, a convolutional attention module (CBAM) is added. The convolutional attention module is equipped with a channel attention unit and a spatial attention unit. The channel attention unit first aggregates the global features of each channel through global average pooling and maximum pooling, and then generates channel weights through a fully connected layer to enhance the features of important channels and suppress unimportant channels; the spatial attention unit performs maximum pooling and average pooling on the input feature map in the channel dimension, and generates a spatial weight map through a convolutional layer after splicing to emphasize important spatial areas; then the convolutional attention module needs to be inserted into each C3 module in the backbone network (Backbone) of YOLOv5.

[0115] The pipa music score recognition model training module is specifically used for:

[0116] The data set is divided into a training set, a validation set and a test set based on a ratio of 7:2:1; after the pipa music score recognition model is trained by the training set, the pipa music score recognition model is verified by the validation set, and based on the verification result, the model parameters of the pipa music score recognition model are fine-tuned to optimize the crossover ratio and confidence, and then the verified pipa music score recognition model is tested by using the test set to complete the training of the pipa music score recognition model.

[0117] In summary, the advantages of the present invention are:

[0118] A large number of music score images of pipa music scores are obtained and preprocessed for image enhancement, and the music symbols in each preprocessed music score image are annotated by Label Img to obtain the corresponding annotation file, and each music score image and the annotation file are stored, and a data set is constructed based on each music score image and the annotation file; then a pipa music score recognition model is created based on the YOLOv5 module and the convolutional attention module, and the pipa music score recognition model is trained with the data set, and finally the trained pipa music score recognition model is deployed on the server, and the deployed pipa music score recognition model is called online to recognize the pipa music score; that is, the pipa music score recognition model is trained by obtaining the annotation file and the data set constructed by the music score image by annotating the music symbols, and the pipa music score recognition model is equipped with a YOLOv5 The convolutional attention module extracts channel features and spatial features from the music score image through the channel attention unit and the spatial attention unit respectively, and then assists the YOLOv5 module to recognize music symbols through the channel features and spatial features, so that the pipa music score recognition model can focus on the music symbols. The YOLOv5 module of the pipa music score recognition model is improved, and a detection head with a detection resolution of 160×160 is added to improve the recognition effect of small targets. The music score image is denoised by combining the maximum inter-class variance method to improve the quality of the data set, thereby improving the training effect of the pipa music score recognition model, and ultimately greatly improving the accuracy of drum score recognition.

[0119] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for recognizing pipa music scores based on target detection, characterized in that: The steps include: Step S1, obtaining a large number of music score images of pipa music scores, and performing image enhancement preprocessing on each of the music score images; Step S2, labeling the music symbols in each of the pre-processed music score images through LabelImg to obtain a corresponding labeling file; Step S3, storing each of the music score images and the annotation files, and constructing a data set based on each of the music score images and the annotation files; Step S4, creating a pipa music score recognition model based on the YOLOv5 module and the convolutional attention module; Step S5, training the pipa music score recognition model using the data set; Step S6: deploy the trained pipa music score recognition model on the server, and call the deployed pipa music score recognition model online to recognize the pipa music score.

2. A method for recognizing pipa music scores based on target detection as claimed in claim 1, characterized in that: The step S1 is specifically as follows: A large number of music score images of pipa music scores in PNG format are obtained, and noise reduction is performed on each of the music score images by using the maximum inter-class variance method to complete the preprocessing of image enhancement; The denoising of each of the music score images by the maximum inter-class variance method is specifically as follows: calculating the grayscale histogram, foreground, background, pixel probability and average grayscale value under each preset threshold of each of the music score images, calculating the inter-class variance of the grayscale histogram, foreground, background, pixel probability and average grayscale value, taking the threshold corresponding to the inter-class variance with the largest value as the binarization threshold, and denoising each of the music score images based on the binarization threshold.

3. A method for recognizing pipa music scores based on target detection as claimed in claim 1, characterized in that: The step S2 is specifically as follows: The music symbols in each of the preprocessed music score images are labeled in sequence by LabelImg to obtain corresponding label vectors including note category, attribute information and position information; The note category is a bar line, a quarter rest, a sharp, a flat, a tie, a plucking, a picking, a double plucking, a double picking, a sweep, a brush, a small sweep, a full round, a round finger, a half round, a three-finger round, a harmonic, a chant, a hit, a band or a stroke; the attribute information includes at least a pitch and a duration; the position information includes at least an x-coordinate of a music symbol, a y-coordinate of a music symbol, a width of a target frame, and a height of a target frame; A corresponding annotation file is generated based on each label vector of each of the music score images.

4. A method for recognizing pipa music scores based on target detection as claimed in claim 1, characterized in that: The step S3 is specifically as follows: The same file name is set for each of the score images and the corresponding annotation file samples, each of the score images is stored in a pre-created image folder, each of the annotation files is stored in a pre-created label folder, and a data set is constructed based on each of the score images and the annotation files.

5. A method for recognizing pipa music scores based on target detection as claimed in claim 1, characterized in that: In step S4, the YOLOv5 module is used to identify music symbols from the music score image, and is provided with four detection heads for identifying music symbols of different scales, and the detection resolutions of the detection heads are 20×20, 40×40, 80×80, and 160×160 respectively; the convolutional attention module is provided with a channel attention unit and a spatial attention unit, which are respectively used to extract channel features and spatial features from the music score image to assist the YOLOv5 module in music symbol recognition; The step S5 is specifically as follows: The data set is divided into a training set, a validation set and a test set based on a ratio of 7:2:1; after the pipa music score recognition model is trained by the training set, the pipa music score recognition model is verified by the validation set, and based on the verification result, the model parameters of the pipa music score recognition model are fine-tuned to optimize the crossover ratio and confidence, and then the verified pipa music score recognition model is tested by using the test set to complete the training of the pipa music score recognition model.

6. A pipa music score recognition system based on target detection, characterized in that: Includes the following modules: A music score image preprocessing module is used to obtain a large number of music score images of pipa music scores and perform image enhancement preprocessing on each of the music score images; A music score image labeling module, used to label the music symbols in each of the pre-processed music score images through LabelImg to obtain a corresponding labeling file; A data set construction module, used to store each of the music score images and annotation files, and to construct a data set based on each of the music score images and annotation files; A pipa music score recognition model creation module is used to create a pipa music score recognition model based on the YOLOv5 module and the convolutional attention module; A pipa music score recognition model training module, used for training a pipa music score recognition model using the data set; The pipa music score recognition module is used to deploy the trained pipa music score recognition model on the server, and call the deployed pipa music score recognition model online to recognize the pipa music score.

7. A pipa music score recognition system based on target detection as claimed in claim 6, characterized in that: The music score image preprocessing module is specifically used for: A large number of music score images of pipa music scores in PNG format are obtained, and noise reduction is performed on each of the music score images by using the maximum inter-class variance method to complete the preprocessing of image enhancement; The denoising of each of the music score images by the maximum inter-class variance method is specifically as follows: calculating the grayscale histogram, foreground, background, pixel probability and average grayscale value under each preset threshold of each of the music score images, calculating the inter-class variance of the grayscale histogram, foreground, background, pixel probability and average grayscale value, taking the threshold corresponding to the inter-class variance with the largest value as the binarization threshold, and denoising each of the music score images based on the binarization threshold.

8. A pipa music score recognition system based on target detection as claimed in claim 6, characterized in that: The music score image annotation module is specifically used for: The music symbols in each of the preprocessed music score images are labeled in sequence by LabelImg to obtain corresponding label vectors including note category, attribute information and position information; The note category is a bar line, a quarter rest, a sharp, a flat, a tie, a plucking, a picking, a double plucking, a double picking, a sweep, a brush, a small sweep, a full round, a round finger, a half round, a three-finger round, a harmonic, a chant, a hit, a band or a stroke; the attribute information includes at least a pitch and a duration; the position information includes at least an x-coordinate of a music symbol, a y-coordinate of a music symbol, a width of a target frame, and a height of a target frame; A corresponding annotation file is generated based on each label vector of each of the music score images.

9. A pipa music score recognition system based on target detection as claimed in claim 6, characterized in that: The data set construction module is specifically used for: The same file name is set for each of the score images and the corresponding annotation file samples, each of the score images is stored in a pre-created image folder, each of the annotation files is stored in a pre-created label folder, and a data set is constructed based on each of the score images and the annotation files.

10. The target detection-based pipa score recognition system according to claim 6, characterized in that: In the pipa music score recognition model creation module, the YOLOv5 module is used to recognize music symbols from the music score image, and is provided with four detection heads for recognizing music symbols of different scales, and the detection resolutions of each detection head are 20×20, 40×40, 80×80, and 160×160 respectively; the convolutional attention module is provided with a channel attention unit and a spatial attention unit, which are respectively used to extract channel features and spatial features from the music score image to assist the YOLOv5 module in music symbol recognition; The pipa music score recognition model training module is specifically used for: The data set is divided into a training set, a validation set and a test set based on a ratio of 7:2:1; after the pipa music score recognition model is trained by the training set, the pipa music score recognition model is verified by the validation set, and based on the verification result, the model parameters of the pipa music score recognition model are fine-tuned to optimize the crossover ratio and confidence, and then the verified pipa music score recognition model is tested by using the test set to complete the training of the pipa music score recognition model.

Images