Piano fingering recognition method and computer readable storage medium

Abstract

The present invention discloses a piano fingering recognition method and a computer-readable storage medium, the method comprising: photographing a piano keyboard image and determining the position information of each key area; receiving the MIDI signal corresponding to the pressed piano key in real time, and photographing the playing image corresponding to the MIDI signal; comparing the pitch corresponding to the MIDI signal with the pitch of the corresponding note in the piano score in real time; if the pitch corresponding to the MIDI signal is consistent with the pitch of the corresponding note, performing hand key point detection on the playing image corresponding to the MIDI signal to obtain the hand key point corresponding to the MIDI signal; performing fingering detection according to the hand key point corresponding to the MIDI signal and the position information of each key area. The present invention can realize the automatic recognition of piano playing fingering and reduce the calculation amount of the overall fingering recognition process.

Description

Technical Field

[0001] The present invention relates to the technical field of fingering detection, and in particular to a piano fingering recognition method and a computer-readable storage medium. Background Art

[0002] Traditional piano playing practice or examinations require manual judgment on whether the fingering is correct, which consumes a lot of teacher resources. In addition, due to limited teaching resources, a teacher cannot simultaneously check whether the fingering of multiple students playing the same piano score or different piano scores in the same period of time is correct. Therefore, it is impossible to provide timely and effective guidance to students, resulting in slow learning progress or the formation of bad playing habits.

[0003] Among the existing piano fingering recognition schemes, most of them adopt the method based on feature comparison. First, a standard database of correct hand shapes is established through mathematical models; then, a prediction model is constructed to extract the features of the predicted image, and the features are compared with the standard database to determine whether it is an incorrect playing hand shape. The difficulty of this method is that building a standard database is a complex and inefficient process. Due to the great differences in the size and joint proportions of human hands, it leads to great subjectivity and inaccuracy when using joint angles or joint lengths to construct standard hand shapes. At the same time, due to the changes in the angles between the hand and the camera, even if the hand shapes are very different, a high similarity may be obtained when comparing, leading to wrong conclusions. Therefore, the feature comparison method has poor robustness, high subjectivity, low recognition rate, large amount of calculation, and high requirements for hardware equipment. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a piano fingering recognition method and a computer-readable storage medium, which can realize automatic recognition of piano playing fingerings and reduce the amount of calculation in the overall fingering recognition process.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a piano fingering recognition method, comprising:

[0006] Taking a piano keyboard picture, and determining the position information of each key area according to the piano keyboard picture;

[0007] Receive in real time the MIDI signal corresponding to the pressed piano key, and take a playing picture corresponding to the MIDI signal, wherein the playing picture includes the piano key area and the player's hands;

[0008] Comparing the pitch corresponding to the MIDI signal with the pitch of the corresponding note in the piano score in real time;

[0009] If the pitch corresponding to the MIDI signal is consistent with the pitch of the corresponding note, performing hand key point detection on the playing picture corresponding to the MIDI signal to obtain the hand key points corresponding to the MIDI signal;

[0010] Fingering detection is performed according to the hand key points corresponding to the MIDI signal and the position information of each key area.

[0011] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and the program implements the above method when executed by a processor.

[0012] The beneficial effects of the present invention are as follows: by first comparing the MIDI signal output by the piano playing with the music score, fingering detection is performed only when the pitch is consistent, that is, fingering detection is performed only when the sound played by the performer is correct, which can reduce the calculation amount of the overall recognition and reduce the requirements for hardware devices, so that it can run in real time on embedded devices; by performing hand key point detection on the playing picture, the finger position of the user when playing is determined, and it is compared with the position of each predetermined key area to determine whether a specific finger falls into the corresponding key area, thereby realizing automatic recognition of fingering when playing the piano, and having good robustness and high recognition rate, it has a great auxiliary effect on piano beginners practicing fingering and piano teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flow chart of a piano fingering recognition method of the present invention;

[0014] Figure 2 This is a flow chart of a method according to Embodiment 1 of the present invention;

[0015] Figure 3 A schematic diagram of playing a picture in Embodiment 1 of the present invention;

[0016] Figure 4 is a method flow chart of step S1 in embodiment 2 of the present invention;

[0017] Figure 5 is a schematic diagram of a piano keyboard area in a second embodiment of the present invention;

[0018] Figure 6 is a schematic diagram of a complete sound group in Embodiment 2 of the present invention;

[0019] Figure 7 A schematic diagram of a piano keyboard comprising 88 keys. DETAILED DESCRIPTION

[0020] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the implementation methods and the accompanying drawings.

[0021] See also Figure 1 , a piano fingering recognition method, comprising:

[0022] Taking a piano keyboard picture, and determining the position information of each key area according to the piano keyboard picture;

[0023] Receive in real time the MIDI signal corresponding to the pressed piano key, and take a playing picture corresponding to the MIDI signal, wherein the playing picture includes the piano key area and the player's hands;

[0024] Comparing the pitch corresponding to the MIDI signal with the pitch of the corresponding note in the piano score in real time;

[0025] If the pitch corresponding to the MIDI signal is consistent with the pitch of the corresponding note, performing hand key point detection on the playing picture corresponding to the MIDI signal to obtain the hand key points corresponding to the MIDI signal;

[0026] Fingering detection is performed according to the hand key points corresponding to the MIDI signal and the position information of each key area.

[0027] From the above description, it can be seen that the beneficial effects of the present invention are: it can realize automatic recognition of piano playing fingerings and can reduce the amount of calculation in the overall fingering recognition process.

[0028] Furthermore, the real-time comparison of the pitch corresponding to the MIDI signal with the pitch of the corresponding note in the piano score is specifically as follows:

[0029] Obtaining a piano score to be played, and determining the order of playing each note in the piano score, and obtaining a sequence number of each note;

[0030] Determining the sequence number of the MIDI signal according to the order in which the MIDI signal is received;

[0031] According to the sequence number of the MIDI signal, obtaining the note with the same sequence number in the piano score as the note corresponding to the MIDI signal;

[0032] The pitch corresponding to the MIDI signal is compared with the pitch of the corresponding note in real time.

[0033] As can be seen from the above description, the notes corresponding to the MIDI signals are determined according to the playing order of the MIDI signals and the order of the notes in the piano score, and the pitch comparison is performed, so as to determine whether the notes played by the performer are correct.

[0034] Furthermore, the photographing obtains a piano keyboard picture, and according to the piano keyboard picture, the position information of each key area is specifically:

[0035] Taking a piano keyboard image, wherein the piano keyboard image includes a piano keyboard area;

[0036] Determine a piano keyboard area according to the piano keyboard picture;

[0037] The piano keyboard area is divided to obtain each key area, and the position information of each key area is determined.

[0038] It can be seen from the above description that by first determining the piano keyboard area in the piano keyboard image and then dividing the piano keyboard area, more accurate key areas can be obtained, thereby improving the accuracy of the key area position.

[0039] Furthermore, the piano keyboard area is determined based on the piano keyboard image as follows:

[0040] Perform edge detection on the piano keyboard image by using an edge detection algorithm, and detect the upper and lower boundaries and left and right boundaries of the piano keyboard area by using Hough transform, wherein the upper and lower boundaries are boundaries in the length direction of the piano keyboard area, and the left and right boundaries are boundaries in the width direction of the piano keyboard area;

[0041] The piano keyboard area is determined according to the upper and lower boundaries and the left and right boundaries of the piano keyboard area.

[0042] It can be seen from the above description that the four boundaries of the piano keyboard area are detected through edge detection and Hough transform, and then the piano keyboard area is determined according to the four boundaries.

[0043] Furthermore, before the piano keyboard area is divided to obtain each key area and the position information of each key area is determined, the method further includes:

[0044] Calculating the angle between the length direction and the horizontal direction of the piano keyboard area;

[0045] If the angle is greater than a preset first threshold, the piano keyboard image is rotated according to the angle, and the length direction of the piano keyboard area in the rotated piano keyboard image is parallel to the horizontal direction, and the angle is less than 90 degrees.

[0046] From the above description, it can be seen that by rotating the piano keyboard image, a piano keyboard area parallel to the horizontal direction is obtained, thereby solving the problem of difficult or inaccurate segmentation caused by the non-horizontal piano keyboard area captured by the camera.

[0047] Further, the left and right sides of the piano keyboard area correspond to the left and right sides of the piano keyboard respectively, the left side of the piano keyboard is the side located on the left hand side of the performer when the performer plays the piano, and the right side of the piano keyboard is the side located on the right hand side of the performer;

[0048] The piano keyboard area is divided to obtain each key area, and the position information of each key area is determined as follows:

[0049] Binarization is performed on the piano keyboard area, and each black key area is determined by a connected domain detection algorithm;

[0050] According to the note corresponding to the first left key of the preset piano keyboard, the number of black keys of the first black key group on the left is determined;

[0051] According to the number of black keys in the first black key group and the number of black keys in the complete tone group, the black key areas are grouped in sequence from the left to the right of the piano keyboard area to obtain at least two black key groups, and the black key group whose number of black key areas is consistent with the number of black keys in the complete tone group is used as the black key group corresponding to the complete tone group, wherein the complete tone group includes 5 black keys and 7 white keys;

[0052] The distance between the same position points of the two black key areas on the left of the black key group corresponding to each complete tone group is calculated respectively, or the distance between the same position points of the two adjacent black key areas on the right of the three black key areas in the black key group corresponding to each complete tone group is calculated respectively, so as to obtain the black key distance corresponding to each complete tone group one by one, and the white key width corresponding to each complete tone group is calculated respectively according to each black key distance and a preset proportional coefficient, wherein the preset proportional coefficient is 23 / 41;

[0053] According to the positions of the black key areas in the black key groups corresponding to the complete tone groups and the widths of the white keys corresponding to the complete tone groups, the piano keyboard area is divided to obtain seven white key areas corresponding to the complete tone groups;

[0054] Determine the keyboard area corresponding to the incomplete tone group according to the piano keyboard area and the white key areas corresponding to each complete tone group, and divide the keyboard area corresponding to the incomplete tone group according to the white key width corresponding to the complete tone group adjacent to the incomplete tone group to obtain the white key area corresponding to the incomplete tone group;

[0055] The coordinate values ​​of the same vertices of each black key region and each white key region in the piano keyboard image and the length and width of each key region are recorded respectively to obtain the position information of each key region.

[0056] From the above description, it can be seen that by calculating the width of white keys in groups and dividing the white key areas in groups, the problem of different widths of each white key in the piano keyboard area caused by the shooting angle of the camera can be solved, and a more accurate division result can be obtained.

[0057] Furthermore, the piano keyboard area is divided according to the position of the black key area in the black key group corresponding to each complete tone group and the width of the white key corresponding to each complete tone group, and the seven white key areas corresponding to each complete tone group are obtained as follows:

[0058] Determine the center line of the fourth black key area from left to right in the black key group corresponding to each complete tone group along its length direction, and divide the piano keyboard area into five white key areas on the left side of the center line and two white key areas on the right side of the center line according to the width of the white keys corresponding to each complete tone group, as the seven white key areas corresponding to each complete tone group;

[0059] If the boundary lines of two adjacent white key areas corresponding to two adjacent complete tone groups do not overlap, the center line of the boundary lines of the two white key areas is first used as a new boundary line.

[0060] It can be seen from the above description that by redefining the boundary line of the white key area with overlap, it is ensured that each pixel point in the key area corresponds to only one key area.

[0061] Furthermore, the shooting position and shooting angle of the piano keyboard picture and the piano playing picture are consistent;

[0062] Before detecting the key points of the hand on the playing picture corresponding to the MIDI signal to obtain the key points of the hand corresponding to the MIDI signal, the method further comprises:

[0063] If the angle is greater than a preset first threshold, the playing picture is rotated according to the angle.

[0064] It can be seen from the above description that when the piano keyboard image is rotated, the playing image is rotated synchronously to ensure that the accurate key positions can be determined in the playing image based on the position information of each key area.

[0065] Furthermore, the hand key points are detected for the playing picture corresponding to the MIDI signal, and the hand key points corresponding to the MIDI signal are obtained as follows:

[0066] Perform hand detection on the playing picture to obtain positions of each hand in the playing picture according to a preset first deep learning model, wherein the first deep learning model is a target detection model;

[0067] According to a preset second deep learning model, hand key point detection is performed on each hand position to obtain the hand key points in each hand position, and the second deep learning model is a classification model.

[0068] From the above description, it can be seen that by detecting the key points of the hands in the playing picture, it is convenient to subsequently determine the fingers of the performer playing the piano.

[0069] Furthermore, after performing hand key point detection on the playing picture corresponding to the MIDI signal and obtaining the hand key points corresponding to the MIDI signal, the method further includes:

[0070] Get the coordinates of the wrist key points and the coordinates of the key points of each fingertip;

[0071] According to the wrist key point coordinates and the coordinates of each fingertip key point, the average value of the angles between adjacent fingertips and the wrist is calculated, or the angles between the thumb and the little finger and the wrist are calculated to obtain the fingertip coordinate confidence;

[0072] If the fingertip coordinate confidence is greater than or equal to a preset second threshold, the step of performing fingering detection according to the hand key points corresponding to the MIDI signal and the position information of each key area is executed.

[0073] From the above description, it can be seen that the confidence of the fingertip coordinates is calculated to determine whether the fingers overlap. If they overlap, they are discarded and no subsequent finger detection is performed. If they do not overlap, subsequent finger detection is performed to ensure the accuracy of finger detection.

[0074] Furthermore, the fingering detection is performed according to the hand key points corresponding to the MIDI signal and the position information of each key area as follows:

[0075] Determine the area range of each key area in the playing picture according to the position information of each key area respectively, and determine the key area corresponding to the MIDI signal according to the pitch corresponding to the MIDI signal and the preset pitch corresponding to each key area;

[0076] Obtain fingertip key points among the hand key points corresponding to the MIDI signal, and determine whether any fingertip key points fall within the piano key area corresponding to the MIDI signal;

[0077] If yes, determine the finger corresponding to the fingertip key point falling within the key area corresponding to the MIDI signal, and compare the corresponding finger with the standard playing finger of the note corresponding to the MIDI signal;

[0078] If the corresponding finger is consistent with the standard playing finger, the fingering is determined to be correct, otherwise the fingering is determined to be incorrect.

[0079] From the above description, it can be seen that the fingering of the piano player is automatically recognized by comparing the position of the finger pressing the piano key with the position of the key area corresponding to the piano key.

[0080] Further, the hand key point detection is performed on the playing picture corresponding to the MIDI signal to obtain the hand key point corresponding to the MIDI signal; and the fingering detection is performed according to the hand key point corresponding to the MIDI signal and the position information of each key area as follows:

[0081] If the time interval between continuous MIDI signals is less than a preset third threshold, hand key point detection is performed on the playing picture corresponding to the last MIDI signal in the continuous MIDI signals to obtain the hand key points corresponding to the continuous MIDI signals, and fingering detection is performed based on the hand key points corresponding to the continuous MIDI signals and the position information of each key area.

[0082] It can be seen from the above description that by merging MIDI signals whose time intervals are less than a certain threshold, and only performing subsequent detection on the playing picture corresponding to the last MIDI signal, the amount of calculation for overall recognition can be further reduced.

[0083] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and the program implements the above method when executed by a processor.

[0084] Embodiment 1

[0085] Please refer to Figure 2-3 , Embodiment 1 of the present invention is: a piano fingering recognition method, which can be applied to embedded devices and is suitable for piano teaching or training and other scenarios.

[0086] The piano in this embodiment is an electric piano or an electronic keyboard, and each time a key is pressed, a MIDI signal will be sent out accordingly.

[0087] Different pianos have different numbers of keys. For example, currently common pianos usually have 88 keys (including 36 black keys and 52 white keys). There are also a small number of pianos that only have 83, 85, 92 or 97 keys. There are also some special pianos (such as children's pianos) with fewer keys. This embodiment is described by taking 88 keys as an example.

[0088] like Figure 2 As shown, the following steps are included:

[0089] S1: photographing a piano keyboard image, and determining position information of each key area according to the piano keyboard image.

[0090] In this embodiment, the camera is located above the piano and faces the piano keyboard area, ensuring that the entire piano keyboard area can be clearly photographed, and the performer's hands can also be clearly photographed subsequently.

[0091] S2: When the performer plays the piano, the MIDI signals corresponding to the pressed piano keys are received in real time.

[0092] S3: Real-time shooting to obtain a playing picture corresponding to the MIDI signal.

[0093] Among them, the playing picture includes the complete piano key area and the player's hands, which is convenient for subsequent fingering detection, such as Figure 3 In addition, the shooting position and shooting angle of the piano keyboard image and the playing image are consistent, that is, the state of the camera is kept unchanged to ensure that the position of each key in the piano keyboard image is consistent with the position of each key in the playing image.

[0094] S4: Compare the pitch corresponding to the MIDI signal with the pitch of the corresponding note in the piano score in real time to determine whether the pitch corresponding to the MIDI signal is consistent with the pitch of the corresponding note. If so, execute step S5.

[0095] In this embodiment, the note corresponding to the current MIDI signal is determined according to the playing order of the notes in the piano score.

[0096] Specifically, this step includes the following steps:

[0097] S401: Obtain a piano score to be played, and determine the order of playing each note in the piano score to obtain the sequence number of each note. The sequence numbers of the notes played at the same time point need to be consistent; for example, for a double-line staff, the sequence numbers of the treble notes and the bass notes played at the same time point need to be consistent.

[0098] S402: Determine the sequence number of the MIDI signal according to the order in which the MIDI signal is received; that is, the sequence number of the MIDI signal is determined according to the order in which the MIDI signal is played. Similarly, the sequence numbers of the MIDI signals corresponding to the piano keys pressed at the same time point are consistent.

[0099] S403: According to the serial number of the MIDI signal, obtain the note with the same serial number in the piano score as the note corresponding to the MIDI signal; that is, the MIDI signal with the same serial number corresponds to the note.

[0100] S404: Determine whether the pitch corresponding to the MIDI signal is consistent with the pitch of the corresponding note, if so, execute step S5. That is, fingering detection is performed only when the player plays the right note, thereby reducing the overall number of calculations.

[0101] S5: Perform hand key point detection on the playing picture corresponding to the MIDI signal to obtain the hand key points corresponding to the MIDI signal.

[0102] Specifically, the first deep learning model is used to perform hand detection on the playing picture to detect all hand positions in the playing picture; then the second deep learning model is used to perform hand key point detection on all hand positions to detect all hand key points (skeleton points).

[0103] Among them, the first deep learning model can use the existing deep model of target detection, such as yolov3~v5, ssd, etc., and the second deep learning model can use the existing classification model, such as ResNet, ShuffleNet, Mobilenet, Rexnet, etc.

[0104] S6: Calculate the fingertip coordinate confidence according to the hand key points corresponding to the MIDI signal.

[0105] Specifically, according to the detected key points of the hand, the coordinates of the wrist key points and the coordinates of the fingertip key points are obtained; then the average value of the angles of adjacent fingertips relative to the wrist is calculated, that is, the angles of the thumb and index finger relative to the wrist, the angles of the index finger and middle finger relative to the wrist, the angles of the middle finger and ring finger relative to the wrist, and the angles of the ring finger and little finger relative to the wrist are calculated respectively, and then the average value of these angles is calculated as the fingertip confidence.

[0106] In other embodiments, the angle between the thumb and the little finger relative to the wrist may also be directly calculated as the fingertip confidence.

[0107] Among them, in the detection of human hand key points, each hand key point detected has a certain serial number. For example, the hand key point with serial number 0 is the wrist key point, and the hand key points with serial numbers 4, 8, 12, 16, and 20 are the thumb tip key point, index finger tip key point, middle finger tip key point, ring finger tip key point, and little finger tip key point, respectively. Therefore, the corresponding finger can be determined by the serial number of the hand key point.

[0108] S7: Determine whether the fingertip coordinate confidence is greater than or equal to a preset second threshold. If so, execute step S8.

[0109] Since fingers may overlap when playing the piano, the accuracy of fingering detection will be reduced. Therefore, the confidence of the fingertip coordinates is calculated to determine whether the fingers overlap. If they overlap, they are discarded and no subsequent fingering detection is performed. If they do not overlap, subsequent fingering detection is performed to ensure the accuracy of fingering detection.

[0110] S8: Perform fingering detection according to the hand key points corresponding to the MIDI signal and the position information of each key area.

[0111] Specifically, this step includes the following steps:

[0112] S801: Determine the area range of each key area in the playing picture according to the position information of each key area, and determine the key area corresponding to the MIDI signal according to the pitch corresponding to the MIDI signal and the preset pitch corresponding to each key area. That is, determine the key area corresponding to the key pressed by the performer in the playing picture.

[0113] S802: Obtain the fingertip key points in the hand key points corresponding to the MIDI signal, and determine whether there are fingertip key points falling within the key area corresponding to the MIDI signal, and if so, execute step S803.

[0114] S803: Determine the finger corresponding to the fingertip key point falling within the key area corresponding to the MIDI signal, and compare the corresponding finger with the standard playing finger of the note corresponding to the MIDI signal.

[0115] In the above step S4, the notes in the piano score corresponding to the MIDI signal have been determined, and each note in the piano score has a corresponding standard playing finger (determined by the piano score with fingering, or pre-associated). This step is to first determine the finger of the performer pressing the piano key, and then compare it with the standard playing finger.

[0116] S804: Determine whether the corresponding finger is consistent with the standard playing finger, if so, execute step S805, if not, execute step S806.

[0117] S805: Determine whether the fingering is correct.

[0118] S806: Determine that the fingering is wrong and give a prompt, for example, by a prompt tone, voice or human-computer interface.

[0119] Furthermore, since multiple fingers are often pressed simultaneously during piano playing, the time intervals of the corresponding multiple MIDI signals are relatively short. If fingering detection is performed on each MIDI signal and its corresponding playing picture, the overall number of calculations will increase. Therefore, in this embodiment, if the time interval of continuous MIDI signals is less than a preset third threshold, only the playing picture corresponding to the last MIDI signal among these continuous MIDI signals is detected for hand key points, and the hand key points corresponding to these continuous MIDI signals are obtained. Fingering detection is performed based on the hand key points corresponding to these continuous MIDI signals and the position information of each key area.

[0120] This embodiment compares the MIDI signal output by the piano playing with the music score first, and only performs fingering detection when the pitch is consistent, that is, fingering detection is performed only when the performer plays the right note, which can reduce the calculation amount of overall recognition and reduce the requirements for hardware equipment; by merging MIDI signals with a time interval less than a certain threshold, only performing subsequent detection on the playing picture corresponding to the last MIDI signal, the calculation amount of overall recognition can be further reduced; by performing hand key point detection on the playing picture, the finger position of the performer when playing is determined, and comparing it with the position of each predetermined key area to determine whether a specific finger falls into the corresponding key area, thereby realizing automatic recognition of fingering when playing the piano; by calculating the confidence of the fingertip coordinates to determine whether the fingers overlap, if there is overlap, they are discarded and no subsequent fingering detection is performed, and if there is no overlap, subsequent fingering detection is performed to ensure the accuracy of fingering detection.

[0121] The method of this embodiment is simple, efficient, robust, and has a high recognition rate. It can be run in real time on an embedded device and has a great auxiliary effect on piano beginners practicing fingering and piano teaching.

[0122] Embodiment 2

[0123] Please refer to Figure 4-7 This embodiment is a further extension of step S1 in the first embodiment. Specifically, Figure 4 As shown, step S1 includes the following steps:

[0124] S101: photographing and obtaining a piano keyboard image, where the piano keyboard image includes a complete piano keyboard area.

[0125] S102: Determine a piano keyboard area according to the piano keyboard picture.

[0126] Specifically, edge detection is performed on the piano keyboard image through an edge detection algorithm, and the upper and lower boundaries and left and right boundaries of the piano keyboard area are detected through Hough transform, wherein the upper and lower boundaries are boundaries in the length direction of the piano keyboard area, i.e., boundaries with longer lengths; the left and right boundaries are boundaries in the width direction of the piano keyboard area, i.e., boundaries with shorter lengths. Then, the piano keyboard area is determined based on the upper and lower boundaries and left and right boundaries of the piano keyboard area; that is, based on the four boundaries of the piano keyboard area, the piano keyboard area is segmented from the piano keyboard image, and the segmented piano keyboard area is as follows: Figure 5 shown.

[0127] In this embodiment, the Canny operator is used for edge detection.

[0128] S103: Binarization is performed on the piano keyboard area, and each black key area is determined by a connected domain detection algorithm.

[0129] That is, an adaptive threshold is first used for binarization processing to separate the black keys from the white keys, and then the connected domain detection algorithm is used to first detect the circumscribed rectangle of each black key to determine the area of ​​each black key.

[0130] S104: Determine the number of black keys of the first black key group on the left according to the note corresponding to the first key on the left of the preset piano keyboard.

[0131] In this embodiment, the shooting angle of the camera is adjusted in advance so that the left side of the piano keyboard area in the captured piano keyboard image corresponds to the left side of the real piano keyboard, and the right side of the piano keyboard area corresponds to the right side of the real piano keyboard, wherein the left side of the piano keyboard is the side located on the left hand side of the performer when the performer plays the piano, and the right side of the piano keyboard is the side located on the right hand side of the performer when the performer plays the piano, that is, the left and right sides when the performer faces the piano keyboard.

[0132] For a piano with any number of keys, the keys can be divided into a left key group, at least one middle key group and a right key group, where each middle key group corresponds to a complete tone group, including 5 black keys and 7 white keys (the 7 white keys correspond to C, D, E, F, G, A, and B respectively), such as Figure 6 As shown. The left key group and the right key group may be a complete sound group or an incomplete sound group. When it is an incomplete sound group, the left key group includes several keys on the right of the complete sound group, and the right key group includes several keys on the left of the complete sound group. In other words, the incomplete sound group is a part of the complete sound group.

[0133] For example, for a piano with 88 keys, the piano keyboard is as follows Figure 7As shown, it contains 9 tone groups, among which the seven middle tone groups (large character group one, large character group, small character group, small character group one, small character group two, small character group three and small character group four) are complete tone groups, and the two tone groups on the left and right sides (large character group two and small character group five) are incomplete tone groups. Large character group two only contains two white keys (corresponding to A and B respectively) and one black key, and small character group five only contains one white key (corresponding to C).

[0134] By observing the complete and incomplete sound groups, we can know that for an incomplete sound group, as long as we know the note corresponding to the first key on its left (i.e., the first white key), we can determine the number of black keys in the incomplete sound group. For example, if the first white key corresponds to the D note, the incomplete sound group contains 4 black keys; if the first white key corresponds to the E note or the F note, the incomplete sound group contains 3 black keys; if the first white key corresponds to the G note, the incomplete sound group contains 2 black keys; if the first white key corresponds to the A note, the incomplete sound group contains 1 black key; if the first white key corresponds to the B note, the incomplete sound group contains 0 black keys.

[0135] Therefore, by obtaining the note corresponding to the first key on the left side of the piano keyboard, the number of black keys in the first black key group can be determined, wherein the first black key group corresponds to the black keys in the first key group on the left side.

[0136] S105: According to the number of black keys in the first black key group and the number of black keys in the complete tone group (i.e., 5), group the black key areas in order from the left to the right of the piano keyboard area to obtain at least two black key groups, and take the black key group whose number of black key areas is consistent with the number of black keys in the complete tone group as the black key group corresponding to the complete tone group.

[0137] For example, for a piano with 88 keys, the note corresponding to the first key on the left side of the piano keyboard is A, so the number of black keys in the first black key group on the left side is 1. When grouping, the first black key area on the left side of the piano keyboard area is used as the first black key group, the second to sixth black key areas on the left side of the piano keyboard area are used as the second black key group, and the seventh to eleventh black key areas on the left side of the piano keyboard area are used as the third black key group, and so on. The 36 black key areas can be divided into 8 black key groups, among which the first black key group includes 1 black key area, and the other seven black key groups each include 5 black key areas, that is, the other seven black key groups are all black key groups corresponding to the complete tone group.

[0138] S106: Calculate the distance between the same position points of the two black key areas on the left of the black key group corresponding to each complete tone group, or calculate the distance between the same position points of two adjacent black key areas on the right of the three black key areas in the black key group corresponding to each complete tone group, to obtain the black key distance corresponding to each complete tone group.

[0139] The five black key areas in the black key group corresponding to each complete tone group can be divided into two black key areas on the left and three black key areas on the right. In this embodiment, the distance between the upper left corner points of the two black key areas on the left can be calculated, or the distance between the upper left corner points of two adjacent black key areas in the three black key areas on the right can be calculated to obtain the black key distance.

[0140] S107: Calculate the width of the white keys corresponding to each complete tone group according to the distance between each black key and the preset ratio coefficient between the width of the white keys and the distance between the black keys, wherein the ratio coefficient is 23 / 41.

[0141] According to the actual size of the piano, the distance between black keys is about 41 mm and the width of white keys is about 23 mm. Therefore, by calculating the distance between black keys, the width of white keys in the piano keyboard area can be calculated.

[0142] S108: Divide the piano keyboard area according to the positions of the black key areas in the black key groups corresponding to each complete tone group and the widths of the white keys corresponding to each complete tone group, and obtain seven white key areas corresponding to each complete tone group.

[0143] By observing the actual piano keyboard, we can know that for each complete tone group, the center line of the fourth black key on the left is the dividing line between the two white keys on the left and right sides of the black key.

[0144] Therefore, in this embodiment, the center lines of the fourth black key area from left to right in the black key group corresponding to each complete tone group along its length direction are first determined respectively, and then according to the width of the white keys corresponding to each complete tone group, five white key areas are obtained on the left side of the center line of the fourth black key area in each complete tone group, and two white key areas are obtained on the right side of the center line of the fourth black key area in each complete tone group. At this time, seven white key areas corresponding to each complete tone group can be obtained.

[0145] Since the shooting angle of the camera may cause the width of each white key in the photographed piano keyboard area to be different, a more accurate division result can be obtained by calculating the white key widths by grouping and dividing the white key areas by grouping.

[0146] Furthermore, if the boundary lines of two adjacent white key areas corresponding to two adjacent complete tone groups do not overlap, the center lines of the boundary lines of the two white key areas are first used as the new boundary line.

[0147] By redefining the boundary line of the white key area with overlap, it is ensured that each pixel point in the key area corresponds to only one key area.

[0148] S109: Determine the keyboard area corresponding to the incomplete tone group according to the piano keyboard area and the white key areas corresponding to each complete tone group, and divide the keyboard area corresponding to the incomplete tone group according to the white key widths corresponding to the complete tone group adjacent to the incomplete tone group to obtain the white key areas corresponding to the incomplete tone group.

[0149] Since the complete tone groups are in the middle of the piano keyboard area and the incomplete tone groups are on both sides of the piano keyboard area, the keyboard area corresponding to the incomplete tone group on the left can be determined based on the left border of the piano keyboard area and the left border of the first white key area on the left in the first complete tone group from left to right; the keyboard area corresponding to the incomplete tone group on the right can be determined based on the right border of the piano keyboard area and the right border of the last white key area in the last complete tone group from left to right.

[0150] Furthermore, when the last tone group from left to right is an incomplete tone group that does not include black keys (i.e., includes only one white key), it is impossible to determine whether there is an incomplete tone group on the right side based on the black key group. At this time, the distance between the right boundary of the piano keyboard area and the right boundary of the last white key area in the last complete tone group from left to right can be used for determination. When the distance is less than a certain threshold (e.g., 0.8 times the width of the white key), it is considered that there is no incomplete tone group on the right side of the piano keyboard area. In other words, the last tone group from left to right in the piano keyboard area is the last complete tone group from left to right. At this time, the piano keyboard area only has an incomplete tone group on the left side.

[0151] After determining the keyboard area corresponding to the incomplete tone group, the keyboard area corresponding to the incomplete tone group is divided according to the white key width corresponding to the complete tone group adjacent to the incomplete tone group to obtain the white key area corresponding to the incomplete tone group.

[0152] The number of white key areas of the incomplete tone group on the left can be determined according to the note corresponding to the first key on the left of the preset piano keyboard. The number of white key areas of the incomplete tone group on the right can be determined according to the note corresponding to the first key on the right of the preset piano keyboard, or by calculating the quotient of the length of the incomplete tone group on the right in the left-right direction and the width of the white key corresponding to the complete tone group on the left, and then determining the number according to the integer closest to the quotient.

[0153] For example, for a piano with 88 keys, the keyboard area corresponding to the second group of large characters is divided according to the width of the white keys corresponding to the first group of large characters, thus obtaining two white key areas for the second group of large characters; the keyboard area corresponding to the fifth group of small characters is divided according to the width of the white keys corresponding to the fourth group of small characters, thus obtaining one white key area for the fifth group of small characters.

[0154] S110: Record the coordinate values ​​of the same vertices of each black key region and each white key region in the piano keyboard image and the length and width of each key region respectively, and obtain the position information of each key region.

[0155] Through the above steps, all the key areas can be obtained. For a piano with 88 keys, 36 black key areas and 52 white key areas can be obtained. The coordinate values ​​of the same vertex (such as the upper left corner point) of all the key areas in the piano keyboard image and the length and width of each key area are recorded respectively to obtain the position information of each key area.

[0156] This embodiment calculates the width of white keys by grouping and divides the white key areas by grouping, which can solve the problem of different widths of each white key in the piano keyboard area due to the shooting angle of the camera and obtain a more accurate division result.

[0157] Embodiment 3

[0158] This embodiment is a further development of the second embodiment, and the similarities are not repeated here. The difference is that, in this embodiment, before step S103, the following steps are also included:

[0159] Calculate the angle between the length direction of the piano keyboard area and the horizontal direction; that is, calculate the angle between the upper boundary or the lower boundary of the piano keyboard area and the horizontal line, and the angle is less than 90 degrees.

[0160] If the angle is greater than a preset first threshold, the piano keyboard image is rotated according to the angle, and the length direction of the piano keyboard area in the rotated piano keyboard image is parallel to the horizontal direction.

[0161] Furthermore, in step S110, the coordinate values ​​of the same vertices of each black key region and each white key region in the rotated piano keyboard image and the length and width of each key region may be recorded respectively to obtain the position information of each key region.

[0162] At this time, since the position information records the coordinate value of a vertex of the key area in the rotated piano keyboard image, in order to ensure that the accurate key position can be determined in the playing image according to the position information of each key area, therefore, before step S5, if the above-calculated angle is greater than the preset first threshold, the playing image corresponding to the MIDI signal is rotated according to the angle, so that the upper and lower boundaries of the piano keyboard area in the rotated playing image are parallel to the horizontal direction. Then, the hand detection is performed on the rotated playing image through the first deep learning model. And, in step S801, the area range of each key area is determined in the rotated playing image according to the position information of each key area.

[0163] This embodiment rotates the piano keyboard image to obtain a piano keyboard area parallel to the horizontal direction, thereby solving the problem of difficult or inaccurate segmentation caused by the non-horizontal piano keyboard area captured by the camera.

[0164] Embodiment 4

[0165] The present embodiment is a computer-readable storage medium corresponding to the above-mentioned embodiment, on which a computer program is stored. When the computer program is executed by the processor, the various processes in the above-mentioned piano fingering recognition method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0166] In summary, the piano fingering recognition method and computer-readable storage medium provided by the present invention can obtain a piano keyboard area parallel to the horizontal direction by rotating the piano keyboard image, thereby solving the problem of difficult or inaccurate segmentation caused by the non-horizontal piano keyboard area photographed by the camera; by grouping and calculating the white key widths, and dividing the white key areas by groups, the problem of different widths of each white key in the piano keyboard area caused by the camera's shooting angle can be solved, and a more accurate segmentation result can be obtained; by first comparing the MIDI signal output by the piano playing with the music score, and only performing fingering detection when the pitch is consistent, that is, only when the performer plays the right note, the fingering detection is performed, which can reduce the overall recognition calculation time. The amount of calculation can be reduced, and the requirements for hardware equipment can be reduced; by merging MIDI signals with a time interval less than a certain threshold, only the playing picture corresponding to the last MIDI signal is subsequently detected, which can further reduce the amount of calculation for overall recognition; by detecting the key points of the hand in the playing picture, the finger position of the performer when playing is determined, and it is compared with the position of each predetermined key area to determine whether a specific finger falls into the corresponding key area, thereby realizing automatic recognition of fingering when playing the piano; by calculating the confidence of the fingertip coordinates to determine whether the fingers overlap, if there is an overlap, it is discarded and no subsequent fingering detection is performed. When there is no overlap, subsequent fingering detection is performed to ensure the accuracy of fingering detection.

[0167] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A piano fingering recognition method, characterized in that: include: Taking a piano keyboard picture, and determining the position information of each key area according to the piano keyboard picture; Receive in real time the MIDI signal corresponding to the pressed piano key, and take a playing picture corresponding to the MIDI signal, wherein the playing picture includes the piano key area and the player's hands; Comparing the pitch corresponding to the MIDI signal with the pitch of the corresponding note in the piano score in real time; If the pitch corresponding to the MIDI signal is consistent with the pitch of the corresponding note, performing hand key point detection on the playing picture corresponding to the MIDI signal to obtain the hand key points corresponding to the MIDI signal; Perform fingering detection according to the hand key points corresponding to the MIDI signal and the position information of each key area; The piano keyboard image is obtained by shooting, and the position information of each key area is determined according to the piano keyboard image: Taking a piano keyboard image, wherein the piano keyboard image includes a piano keyboard area; Determine a piano keyboard area according to the piano keyboard picture; Dividing the piano keyboard area to obtain various key areas, and determining position information of each key area; The left side and the right side of the piano keyboard area correspond to the left and right sides of the piano keyboard respectively, the left side of the piano keyboard is the side located on the left hand side of the performer when the performer plays the piano, and the right side of the piano keyboard is the side located on the right hand side of the performer; The piano keyboard area is divided to obtain each key area, and the position information of each key area is determined as follows: Binarization is performed on the piano keyboard area, and each black key area is determined by a connected domain detection algorithm; According to the note corresponding to the first left key of the preset piano keyboard, the number of black keys of the first black key group on the left is determined; According to the number of black keys in the first black key group and the number of black keys in the complete tone group, the black key areas are grouped in sequence from the left to the right of the piano keyboard area to obtain at least two black key groups, and the black key group whose number of black key areas is consistent with the number of black keys in the complete tone group is used as the black key group corresponding to the complete tone group, wherein the complete tone group includes 5 black keys and 7 white keys; The distance between the same position points of the two black key areas on the left of the black key group corresponding to each complete tone group is calculated respectively, or the distance between the same position points of the two adjacent black key areas on the right of the three black key areas in the black key group corresponding to each complete tone group is calculated respectively, so as to obtain the black key distance corresponding to each complete tone group one by one, and the white key width corresponding to each complete tone group is calculated respectively according to each black key distance and a preset proportional coefficient, wherein the preset proportional coefficient is 23 / 41; According to the positions of the black key areas in the black key groups corresponding to the complete tone groups and the widths of the white keys corresponding to the complete tone groups, the piano keyboard area is divided to obtain seven white key areas corresponding to the complete tone groups; Determine the keyboard area corresponding to the incomplete tone group according to the piano keyboard area and the white key areas corresponding to each complete tone group, and divide the keyboard area corresponding to the incomplete tone group according to the white key width corresponding to the complete tone group adjacent to the incomplete tone group to obtain the white key area corresponding to the incomplete tone group; The coordinate values ​​of the same vertices of each black key region and each white key region in the piano keyboard image and the length and width of each key region are recorded respectively to obtain the position information of each key region.

2. The piano fingering recognition method according to claim 1, characterized in that: The real-time comparison of the pitch corresponding to the MIDI signal with the pitch of the corresponding note in the piano score is specifically as follows: Obtaining a piano score to be played, and determining the order of playing each note in the piano score, and obtaining a sequence number of each note; Determining the sequence number of the MIDI signal according to the order in which the MIDI signal is received; According to the sequence number of the MIDI signal, obtaining the note with the same sequence number in the piano score as the note corresponding to the MIDI signal; The pitch corresponding to the MIDI signal is compared with the pitch of the corresponding note in real time.

3. The piano fingering recognition method according to claim 1, characterized in that: The piano keyboard area is determined according to the piano keyboard picture as follows: Perform edge detection on the piano keyboard image by using an edge detection algorithm, and detect the upper and lower boundaries and left and right boundaries of the piano keyboard area by using Hough transform, wherein the upper and lower boundaries are boundaries in the length direction of the piano keyboard area, and the left and right boundaries are boundaries in the width direction of the piano keyboard area; The piano keyboard area is determined according to the upper and lower boundaries and the left and right boundaries of the piano keyboard area.

4. The piano fingering recognition method according to claim 1, characterized in that: Before the piano keyboard area is divided to obtain each key area and the position information of each key area is determined, the method further includes: Calculating the angle between the length direction and the horizontal direction of the piano keyboard area; If the angle is greater than a preset first threshold, the piano keyboard image is rotated according to the angle, and the length direction of the piano keyboard area in the rotated piano keyboard image is parallel to the horizontal direction, and the angle is less than 90 degrees.

5. The piano fingering recognition method according to claim 1, characterized in that: According to the positions of the black key areas in the black key groups corresponding to the complete tone groups and the widths of the white keys corresponding to the complete tone groups, the piano keyboard area is divided to obtain seven white key areas corresponding to the complete tone groups: Determine the center line of the fourth black key area from left to right in the black key group corresponding to each complete tone group along its length direction, and divide the piano keyboard area into five white key areas on the left side of the center line and two white key areas on the right side of the center line according to the width of the white keys corresponding to each complete tone group, as the seven white key areas corresponding to each complete tone group; If the boundary lines of two adjacent white key areas corresponding to two adjacent complete tone groups do not overlap, the center line of the boundary lines of the two white key areas is first used as a new boundary line.

6. The piano fingering recognition method according to claim 4, characterized in that: The shooting positions and angles of the piano keyboard picture and the piano playing picture are consistent; Before detecting the key points of the hand on the playing picture corresponding to the MIDI signal to obtain the key points of the hand corresponding to the MIDI signal, the method further comprises: If the angle is greater than a preset first threshold, the playing picture is rotated according to the angle.

7. The piano fingering recognition method according to claim 1 or 6, characterized in that: The hand key points corresponding to the playing picture corresponding to the MIDI signal are detected, and the hand key points corresponding to the MIDI signal are obtained as follows: Perform hand detection on the playing picture to obtain positions of each hand in the playing picture according to a preset first deep learning model, wherein the first deep learning model is a target detection model; According to a preset second deep learning model, hand key point detection is performed on each hand position to obtain the hand key points in each hand position, and the second deep learning model is a classification model.

8. The piano fingering recognition method according to claim 1 or 6, characterized in that: After detecting the key points of the hand on the playing picture corresponding to the MIDI signal and obtaining the key points of the hand corresponding to the MIDI signal, the method further includes: Get the coordinates of the wrist key points and the coordinates of the key points of each fingertip; According to the wrist key point coordinates and the coordinates of each fingertip key point, the average value of the angles between adjacent fingertips and the wrist is calculated, or the angles between the thumb and the little finger and the wrist are calculated to obtain the fingertip coordinate confidence; If the fingertip coordinate confidence is greater than or equal to a preset second threshold, the step of performing fingering detection according to the hand key points corresponding to the MIDI signal and the position information of each key area is executed.

9. The piano fingering recognition method according to claim 1 or 6, characterized in that: The fingering detection is specifically performed according to the hand key points corresponding to the MIDI signal and the position information of each key area: Determine the area range of each key area in the playing picture according to the position information of each key area respectively, and determine the key area corresponding to the MIDI signal according to the pitch corresponding to the MIDI signal and the preset pitch corresponding to each key area; Obtain fingertip key points among the hand key points corresponding to the MIDI signal, and determine whether any fingertip key points fall within the piano key area corresponding to the MIDI signal; If yes, determine the finger corresponding to the fingertip key point falling within the key area corresponding to the MIDI signal, and compare the corresponding finger with the standard playing finger of the note corresponding to the MIDI signal; If the corresponding finger is consistent with the standard playing finger, the fingering is determined to be correct, otherwise the fingering is determined to be incorrect.

10. The piano fingering recognition method according to claim 1 or 6, characterized in that: The step of performing hand key point detection on the playing picture corresponding to the MIDI signal to obtain the hand key points corresponding to the MIDI signal; and performing fingering detection according to the hand key points corresponding to the MIDI signal and the position information of each key area is specifically as follows: If the time interval between continuous MIDI signals is less than a preset third threshold, hand key point detection is performed on the playing picture corresponding to the last MIDI signal in the continuous MIDI signals to obtain the hand key points corresponding to the continuous MIDI signals, and fingering detection is performed based on the hand key points corresponding to the continuous MIDI signals and the position information of each key area.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images