Method and system for identifying piano playing error
By using video analysis to identify piano playing errors, common problems encountered by piano beginners during the playing process have been solved, improving playing accuracy and teaching efficiency, and reducing the burden on teachers.
Patent Information
- Application Number
- CN202411171828.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Common mistakes made by piano beginners include incorrect hand position, misreading notes, inaccurate rhythm, and improper speed control, which affect the playing effect and teaching efficiency.
By acquiring videos of performers playing piano, breaking down the image frames one by one, performing preprocessing and motion recognition model analysis, extracting hand and key movement features, and comparing them with the sheet music, playing errors can be identified.
Improve playing accuracy, provide personalized teaching feedback, reduce teacher workload, and enhance practice efficiency.
Smart Images

Figure CN119007295A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of piano intelligent teaching, and in particular to a method and system for identifying piano playing errors. Background Art
[0002] Common mistakes made by piano beginners during playing mainly include the following aspects: incorrect hand shape, fingers too bent or straight, palm collapse, wrist too high or too low, etc., which will affect the flexibility and strength control of fingers, and thus affect the playing effect; wrong note reading, wrong recognition of notes on the score, resulting in inaccurate playing pitch, too much reliance on memorization, lack of the habit of looking up at the score, affecting the cultivation of sight-reading ability; inaccurate rhythm control, unable to play accurately according to the rhythm markings on the score, resulting in fast and slow situations; improper speed control, playing too fast, easily leading to wrong notes, and also affecting the rhythm and integrity of the playing. Therefore, a method and system for identifying piano playing errors are urgently needed to solve them. Summary of the invention
[0003] Based on this, it is necessary to provide a method and system for identifying piano playing errors in response to the problems existing in the current machine tool positioning fixtures, so as to eliminate or improve one or more defects existing in the prior art.
[0004] The above purpose is achieved through the following technical solutions: A method for identifying piano playing errors comprises the following steps: Obtain a video of a performer playing the piano, and decompose the video frame by frame into multiple image frames; Preprocessing the image frame to obtain a processed image frame; The processed image frames are analyzed using a motion recognition model to obtain hand motion features and piano key motion features; The hand movement features and the key movement features are respectively compared with the correct hand movement features and key movement features corresponding to the music score to obtain a comparison result to identify whether the performer's playing is wrong.
[0005] Preferably, the image frames are subjected to image correction, denoising and contrast enhancement processing; the denoising processing comprises the following steps: obtaining a denoising template and determining the size and shape of the denoising template; roaming the denoising template on the image frame and overlapping with each pixel point; calculating the grayscale average value of the pixel points in the image frame area covered by the denoising template; and assigning the calculated average value to the pixel point corresponding to the center of the denoising template.
[0006] Preferably, the method for constructing an action recognition model comprises the following steps: Based on the human hand model, the hand motion in the processed image frame is estimated to obtain the hand motion features; Obtaining piano key action features based on image recognition method; Based on the hand motion features and the piano key motion features, a motion recognition model is constructed using a feature fusion network and a classifier.
[0007] Preferably, the method for obtaining hand motion features includes: Through camera calibration and background modeling, the parameter information and posture of the human hand model are initialized to obtain the initialized human hand model; Extract the underlying features of the processed image frames, perform feature matching on the underlying features between adjacent image frames, and obtain initial action features; The initial motion features are matched to the initialized human hand model, the hand motion feature posture is estimated, and the hand motion features of the current image frame are obtained.
[0008] Preferably, an image of the piano keyboard is captured to extract the outline of the keyboard and the position information of each key, and based on the outline of the keyboard and the position information of each key, the depression state information and the depression duration information of each key are obtained, and based on the depression state information and the depression duration information of the key, the key action characteristics are obtained.
[0009] Preferably, the method for constructing the feature fusion network comprises the following steps: An improved recurrent neural network is used to align the time series of the hand movement features and the key movement features, and the hand movement features and key movement features after the time series alignment are integrated through an attention mechanism to complete the construction of a feature fusion network.
[0010] Preferably, the hand motion features include the positions of finger joints and the posture of the palm.
[0011] Preferably, the key action characteristics include key number, key pressing depth, and start and end time of pressing.
[0012] A system for identifying piano playing errors, comprising: An image acquisition module is used to acquire a video of a performer playing the piano, and decompose the video frame by frame into multiple image frames; A preprocessing module, used for preprocessing the image frame to obtain a processed image frame; The motion recognition module is used to analyze the processed image frames using the motion recognition model to identify the hand motion features and the piano key motion features; The action comparison module is used to compare the recognized hand action features and key action features with the correct hand action features and key action features corresponding to the music score, and obtain the comparison result to identify whether the performer's playing is wrong.
[0013] Preferably, the preprocessing module comprises: A denoising template acquisition unit, used to obtain a denoising template and determine the size and shape of the denoising template; A roaming unit, used for roaming the denoising template on the image frame and making it overlap with each pixel point; A calculation unit, used for calculating the grayscale average value of the pixels in the image frame area covered by the denoising template; The assignment unit is used to assign the calculated average value to the pixel point corresponding to the center of the denoising template.
[0014] The beneficial effects of the present invention are: 1. Improve playing accuracy: By accurately identifying the performer's hand movements and key movements and comparing them with the correct movements in the score, it can promptly discover and point out errors in the performance, help the performer correct the errors, and improve playing accuracy.
[0015] 2. Personalized teaching feedback: This method can provide personalized feedback to each performer. Since the recognition is based on video analysis, it can capture each performer's unique playing habits and problems, thereby giving more targeted improvement suggestions.
[0016] 3. Reduce the burden on teachers: In traditional piano teaching, teachers need to spend a lot of time observing and correcting students' mistakes. This method can automatically complete part of the error correction work, reducing the burden on teachers and allowing them to focus more on the formulation of teaching plans and the teaching of advanced techniques.
[0017] 4. Improve practice efficiency: By quickly identifying playing errors, performers can practice more specifically and avoid repeating work on the parts they have already mastered, thereby improving practice efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 A flowchart of a method for identifying piano playing errors according to an embodiment of the present invention; Figure 2 4 is a system structure diagram of a system for identifying piano playing errors according to an embodiment of the present invention. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below through embodiments and in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0020] The serial numbers assigned to the components herein, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings). In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. are based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present invention.
[0021] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.
[0022] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.
[0023] The embodiment of the present invention provides a method for identifying piano playing errors, referring to Figure 1 , including the following steps: S1, obtaining a video of a performer playing the piano, and decomposing the video frame by frame into multiple image frames; S2, preprocessing the image frame to obtain a processed image frame; S3, using the action recognition model to analyze the processed image frames to obtain hand action features and piano key action features; S4. Compare the hand movement features and the key movement features with the correct hand movement features and the key movement features corresponding to the music score, and obtain a comparison result to identify whether the performer's playing is wrong.
[0024] The purpose of the present invention is to improve the accuracy and stability of piano playing, so as to make piano playing more perfect and pleasant. By using the method, the mistakes made by the performer during the playing process can be quickly and accurately identified and corrected in time to avoid unnecessary mistakes, and at the same time, it can also help learners better understand and master the correct method of piano playing.
[0025] Specifically, in step 1, the playing process of the performer can be captured by a camera or a mobile phone. In step 2, the video is decomposed into multiple image frames for subsequent processing. In step 3, various preprocessing techniques, such as denoising, contrast enhancement, edge detection, etc., can be used to process the image frames to reduce noise interference and improve image quality. In step 4, various action recognition models, such as deep neural networks, recurrent neural networks, support vector machines, etc., can be used to analyze the processed image frames to extract the hand action features and key action features contained therein. These action recognition models can be selected and designed according to different application scenarios and requirements. In step 5, the extracted hand action features and key action features can be compared with the correct hand action features and key action features in the music score to obtain a comparison result. Finally, in step 6, according to the comparison results, it can be determined whether there are any errors in the performer's playing, and corresponding adjustments and error corrections can be made.
[0026] The above embodiments have the following effects: Improve playing accuracy: By accurately identifying the performer's hand and key movements and comparing them with the correct movements in the score, it can promptly discover and point out errors in the performance, help the performer correct the errors, and improve playing accuracy.
[0027] Personalized teaching feedback: This method can provide personalized feedback to each performer. Since the recognition is based on video analysis, it can capture each performer's unique playing habits and problems, thus giving more targeted improvement suggestions.
[0028] Reduce the burden on teachers: In traditional piano teaching, teachers need to spend a lot of time observing and correcting students' mistakes. This method can automatically complete part of the error correction work, reducing the burden on teachers and allowing them to focus more on the formulation of teaching plans and the teaching of advanced techniques.
[0029] Improve practice efficiency: By quickly identifying playing errors, performers can practice more targeted and avoid repeating work on parts that have already been mastered, thereby improving practice efficiency.
[0030] The scheme is further optimized to perform image correction, denoising, and contrast enhancement on each image frame; the denoising process includes the following steps: obtaining a denoising template and determining the size and shape of the denoising template; roaming the denoising template on the image frame and overlapping it with each pixel; calculating the grayscale average value of the pixels in the image frame area covered by the denoising template; and assigning the calculated average value to the pixel corresponding to the center of the denoising template.
[0031] Further optimizing the scheme, the method for constructing the action recognition model includes the following steps: performing posture estimation on the hand actions in the processed image frames based on the human hand model to obtain hand action features; obtaining piano key action features based on the image recognition method; and constructing the action recognition model based on the hand action features and the piano key action features using a feature fusion network and a classifier.
[0032] Further optimizing the scheme, the method for obtaining hand motion features includes: initializing the parameter information and posture of the human hand model through camera calibration and background modeling to obtain the initialized human hand model; extracting the underlying features of the processed image frame, performing feature matching on the underlying features between adjacent image frames to obtain the initial motion features; matching the initial motion features to the initialized human hand model, performing hand motion feature posture estimation, and obtaining the hand motion features of the current image frame.
[0033] As an implementation, camera calibration is used to determine the camera's internal and external parameters, as well as the depth information in the scene. Background modeling is then used to build a depth image that includes the background model. In each image frame, a pre-trained human hand model can be used to identify and track the hand. Parameter information includes point clouds or vectors at the skeletal joint level. Feature matching is performed on adjacent image frames to find a continuous sequence of actions. Use nearest neighbor search or other similarity measurement methods to find matching features. Match the obtained action features with the initialized human hand model. This is achieved by calculating the probability distribution predicted by the model or by directly comparing the model with the feature space. Use maximum likelihood estimation or other probabilistic models to predict the state transition probability of the model. Pose estimation is performed on the matched action features to determine the exact position of the hand in each frame. This is achieved by rotating and translating the model to a matching pose, or using a deep learning model to predict the hand motion trajectory.
[0034] Specifically, the method of hand motion feature posture estimation includes: segmenting the hand area from the image through image processing techniques such as edge detection and contour extraction. Extracting the key feature points of the hand using feature extraction algorithms such as SIFT and SURF. Estimating the hand posture by matching feature points or using machine learning / deep learning models.
[0035] For example: Hand motion feature posture estimation is based on the GCN network structure, which specifically includes: Graph construction: First, it is necessary to build a graph based on the hand structure. The nodes in the graph represent the hand joints, and the edges represent the connection relationship between the joints. Graph convolution operation: GCN updates the feature representation of the node through the graph convolution operation. During the graph convolution process, the features of each node are updated according to the features of its neighboring nodes, thereby capturing the interaction between the nodes. Posture estimation: After several layers of graph convolution operations, the final feature representation of each node can be obtained. These feature representations can be used to estimate the posture of the hand, such as outputting the coordinates of key points through a regression model.
[0036] To further optimize the solution, image acquisition is performed on the piano keyboard to extract the outline of the keyboard and the position information of each key, based on the outline of the keyboard and the position information of each key, the pressing state information and pressing duration information of each key are obtained, and based on the key pressing state information and pressing duration information, the key action characteristics are obtained.
[0037] Specifically, a camera is placed in front of the piano to capture real-time images of the piano keyboard. The outline of the keyboard and the position information of each key are extracted from these images through computer vision and image processing technology. Then, this information is used to calculate the real-time position and movement trajectory of each key.
[0038] In order to obtain the information of the pressed state and the pressed duration of each key, the movement of each key is detected in the captured image sequence. This can be achieved by analyzing the motion trajectory of the key. When a key is pressed, its motion trajectory changes, and the information of the pressed state and duration of each key is determined based on these changes.
[0039] Once we have this information, we can calculate the motion characteristics of each key. These characteristics can be expressed as a series of numbers, where each number represents the movement pattern and speed of the key. These motion characteristics can reflect the player's playing skills and style, and can provide useful data for machine learning and artificial intelligence applications. In general, obtaining a real-time image of the piano keyboard and extracting the pressing state information and motion trajectory from it can more accurately identify the key action and playing state, thereby improving the efficiency and quality of music learning.
[0040] To further optimize the scheme, the method for constructing the feature fusion network includes the following steps: using an improved recurrent neural network (RNN) to align the time series of hand movement features and key movement features, integrating the hand movement features and key movement features after the time series alignment through the attention mechanism, and completing the construction of the feature fusion network.
[0041] Specifically, the improved recurrent neural network is a bidirectional RNN (Bi-RNN), which combines the forward and reverse RNNs, allowing the model to consider both past and future information. This architecture is effective in processing hand motion feature and key motion feature sequence data with forward and backward dependencies, and they can more comprehensively capture the contextual information in the sequence.
[0042] To further optimize the scheme, the hand motion features include the positions of the finger joints and the posture of the palm.
[0043] To further optimize the scheme, the key action characteristics include the key number, the key pressing depth, and the start and end time of the pressing.
[0044] Reference Figure 2 The embodiment of the present invention also provides a system for identifying piano playing errors, including: an image acquisition module, used to acquire a video of a performer playing the piano, and decompose the video frame by frame into multiple image frames; a preprocessing module, used to preprocess the image frames to obtain processed image frames; a motion recognition module, used to analyze the processed image frames using a motion recognition model to identify hand motion features and key motion features; a motion comparison module, used to compare the identified hand motion features and key motion features with the correct hand motion features and key motion features corresponding to the piano score, respectively, to obtain a comparison result to identify whether the performer's playing is incorrect.
[0045] To further optimize the scheme, the preprocessing module includes: a denoising template acquisition unit, used to obtain the denoising template and determine the size and shape of the denoising template; a roaming unit, used to roam the denoising template on the image frame and overlap it with each pixel; a calculation unit, used to calculate the grayscale average of the pixels in the image frame area covered by the denoising template; and an assignment unit, used to assign the calculated average value to the pixel corresponding to the center of the denoising template.
[0046] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0047] The above-described embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for identifying piano playing errors, characterized in that: The following steps are involved: Obtain a video of a performer playing the piano, and decompose the video frame by frame into multiple image frames; Preprocessing the image frame to obtain a processed image frame; The processed image frames are analyzed using a motion recognition model to obtain hand motion features and piano key motion features; The hand movement features and the key movement features are respectively compared with the correct hand movement features and key movement features corresponding to the music score to obtain a comparison result to identify whether the performer's playing is wrong.
2. The method for identifying piano playing errors according to claim 1, characterized in that: The image frames are subjected to image correction, denoising and contrast enhancement processing; the denoising processing comprises the following steps: obtaining a denoising template and determining the size and shape of the denoising template; roaming the denoising template on the image frame and overlapping each pixel point; calculating the grayscale average value of the pixel points in the image frame area covered by the denoising template; and assigning the calculated average value to the pixel point corresponding to the center of the denoising template.
3. The method for identifying piano playing errors according to claim 1, characterized in that: The method of building an action recognition model includes the following steps: Based on the human hand model, the hand motion in the processed image frame is estimated to obtain the hand motion features; Obtaining piano key action features based on image recognition method; Based on the hand motion features and the piano key motion features, a motion recognition model is constructed using a feature fusion network and a classifier.
4. The method for identifying piano playing errors according to claim 3, characterized in that: Methods for obtaining hand motion features include: Initialize the human hand model parameter information and posture through camera calibration and background modeling to obtain the initialized human hand model; Extract the underlying features of the processed image frames, perform feature matching on the underlying features between adjacent image frames, and obtain initial action features; The initial motion features are matched to the initialized human hand model, the hand motion feature posture is estimated, and the hand motion features of the current image frame are obtained.
5. The method for identifying piano playing errors according to claim 4, characterized in that: The piano keyboard is imaged to extract the outline of the keyboard and the position information of each key, and the pressing state information and pressing duration information of each key are obtained based on the outline of the keyboard and the position information of each key, and the key action characteristics are obtained based on the pressing state information and the pressing duration information of the key.
6. The method for identifying piano playing errors according to claim 4, characterized in that: The method for constructing the feature fusion network comprises the following steps: An improved recurrent neural network is used to align the time series of the hand movement features and the key movement features, and the hand movement features and key movement features after the time series alignment are integrated through an attention mechanism to complete the construction of a feature fusion network.
7. The method for identifying piano playing errors according to claim 1, characterized in that: The hand motion features include the positions of finger joints and the posture of the palm.
8. The method for identifying piano playing errors according to claim 1, characterized in that: The key action characteristics include key number, key pressing depth, and the start and end time of pressing.
9. A system for identifying piano playing errors, characterized in that: include: An image acquisition module is used to acquire a video of a performer playing the piano, and decompose the video frame by frame into multiple image frames; A preprocessing module, used for preprocessing the image frame to obtain a processed image frame; The motion recognition module is used to analyze the processed image frames using the motion recognition model to identify the hand motion features and the piano key motion features; The action comparison module is used to compare the recognized hand action features and key action features with the correct hand action features and key action features corresponding to the music score, and obtain the comparison result to identify whether the performer's playing is wrong.
10. The system for identifying piano playing errors according to claim 9, characterized in that: The preprocessing module comprises: A denoising template acquisition unit, used to obtain a denoising template and determine the size and shape of the denoising template; A roaming unit, used for roaming the denoising template on the image frame and making it overlap with each pixel point; A calculation unit, used for calculating the grayscale average value of the pixels in the image frame area covered by the denoising template; The assignment unit is used to assign the calculated average value to the pixel point corresponding to the center of the denoising template.
Citation Information
Patent Citations
Error correction method and system for piano playing
CN108074440A
Intelligent auxiliary method and device for piano practice and medium
CN113657185A
Fitness action recognition model, method of training model, and method of recognizing fitness action
US11854306B1
Intelligent identification method and system for giving assistance with piano teaching, and intelligent piano training method and system
WO2022052941A1
Cited By
Motion capture method, system, storage medium, and program product
WO2026108965A1
Played key display method and intelligent musical instrument
WO2026153473A1
