Piano playing posture real-time correction system based on hand shape recognition

Abstract

This invention discloses a real-time piano playing posture correction system based on hand shape recognition, specifically in the field of music-assisted teaching technology. It addresses the problems of joint positioning loss and inaccurate dynamic force assessment under occlusion during playing. First, a primary contour is generated by fusing visual flow entities and reflection edges to construct a key mapping mesh. Then, inverse kinematics deduction is performed using skeletal proportion constraints to reconstruct the three-dimensional skeleton sequence under occlusion. Next, focusing on the key sinking time window, the vertical displacement gradient of the metacarpophalangeal joints and the rate of change of interphalangeal joint angles are compared to quantify joint motion values ​​and construct a force deformation feature vector. The extracted feature vectors are input into a standardized database for comparison, and posture error labels are parsed out. Finally, graphic and synchronous speech correction signals are generated at rests or long notes in the electronic score, constructing a closed-loop correction system that does not interrupt the playing rhythm, providing scientific support for piano posture-assisted teaching.

Description

Technical Field

[0001] This invention relates to the field of music-assisted teaching technology, specifically to a real-time piano playing posture correction system based on hand shape recognition. Background Technology

[0002] With the popularization of music education, the number of people learning piano continues to expand. In the long-term training of piano performance, maintaining a scientific hand position and force application posture is the physiological foundation for improving playing skills, ensuring tone quality, and preventing chronic hand strain. However, most daily practice sessions are conducted independently in a home environment without the supervision of a professional teacher. Beginners are highly susceptible to developing poor hand position due to improper force application habits. Traditional periodic offline instruction struggles to provide immediate supervision and correction during daily practice. The education market urgently needs an auxiliary teaching tool that can provide non-contact, automated professional posture guidance in everyday practice settings.

[0003] Currently, methods for assisting in the correction of piano playing posture mainly rely on wearable sensors or two-dimensional image comparison technology. However, both methods have significant limitations in practical teaching applications. Wearable data gloves or motion sensors alter the actual physical feedback when the player touches the keys, increasing hand load and interfering with normal muscle memory for playing, thus violating the objective physical laws of instrument teaching. Furthermore, conventional visual recognition technology, when faced with complex hand crossings and chord playing, suffers from severe optical occlusion between the fingers and the back of the hand, leading to frequent loss of joint positioning signals and tracking interruptions. Simultaneously, static planar contour feature extraction cannot capture the transient micromechanical changes of fingertip touch, thus failing to accurately determine underlying postural errors from a physiological force perspective. This results in delayed assessments and a very high misjudgment rate in assisted teaching evaluations. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a real-time piano playing posture correction system based on hand shape recognition, which solves the problems mentioned above.

[0005] To achieve the above objectives, this invention provides the following technical solution: a real-time piano playing posture correction system based on hand shape recognition, comprising the following modules: an environment mapping module, used to acquire multi-source visual flows including the keyboard and the player's hands, extract the physical contour pixels of the hand and the reflection edge pixels under bright conditions and fuse them to generate a primary hand contour topology vector, and identify the coordinates of the intersection points of the black and white keys to construct a key space mapping mesh; a skeleton deduction module, used to extract the center of the key surface that produces vertical depth changes within the key space mapping mesh and mark it as the fingertip touch space coordinates, read the wrist joint space coordinates within the primary hand contour topology vector, set the fingertip touch space coordinates and wrist joint space coordinates as fixed anchor points, introduce hand bone node length ratio constraints, reconstruct the three-dimensional space coordinate sequence of the metacarpophalangeal joints and the first interphalangeal joint through inverse kinematic deduction, and combine them to generate a real-time three-dimensional skeleton. The topology sequence and posture analysis module are used to extract the spatial displacement trajectories of the metacarpophalangeal joints and the first interphalangeal joint relative to the wrist joint in the real-time 3D skeleton topology sequence within the time window of the vertical depth change of the piano keys. It compares the vertical displacement gradient of the metacarpophalangeal joints with the rate of change of the flexion and extension angle of the first interphalangeal joint. Under the condition that the rate of change of the flexion and extension angle of the first interphalangeal joint is negatively reversed and the 3D height of the metacarpophalangeal joint is lower than the 3D height of the wrist joint, it quantifies the relative acceleration of the corresponding joints and the height difference values ​​to construct the force deformation feature vector. The posture correction module is used to input the force deformation feature vector into the touch posture specification database for comparison and parsing to obtain posture error labels. It extracts the rest time nodes or long note sustain time nodes in the associated electronic music score. At the rest time nodes or long note sustain time nodes, it generates a graphical user interface rendering signal containing the correction text corresponding to the posture error label and a synchronous voice broadcast signal.

[0006] Furthermore, the specific process of acquiring multi-source visual flows including the keyboard and the practitioner's hands, extracting the hand entity contour pixels and reflection edge pixels under bright conditions, and fusing them to generate a primary hand contour topology vector is as follows: The image acquisition device is called to simultaneously capture the top-down visual flow facing the keyboard plane and the reflected visual flow including the piano dust cover plane. The skin color pixel clusters in the top-down visual flow are separated to form the hand entity contour pixels, and the mirror boundary features in the reflected visual flow are tracked to form the reflection edge pixels. The hand entity contour pixels and reflection edge pixels are aligned along the physical bottom intersection line of the piano dust cover. The three-dimensional spatial edge extreme points of the aligned pixel clusters are extracted to construct a topological envelope map including the fingertips to the wrist, and the primary hand contour topology vector containing the spatial positions of the three-dimensional spatial edge extreme points is output.

[0007] Furthermore, the specific process of identifying the coordinates of the intersection points of the black and white keys and constructing the key space mapping mesh is as follows: Filter out dynamic background pixels in the multi-source visual flow, locate the corner point clusters with right-angled geometric features at the intersection of the black and white keys, fit a system of straight line equations parallel to and perpendicular to the keyboard edge along the direction of the line connecting adjacent corner point clusters, solve the mutual intersection points of the straight line equations to form the coordinates of the intersection points of the black and white keys; introduce the physical size ratio parameters of the standard piano keyboard, project the coordinates of the intersection points of the black and white keys onto the three-dimensional spatial coordinate system to divide the independent key touch areas, and connect the boundary nodes of each independent key touch area to construct the key space mapping mesh.

[0008] Furthermore, the specific process of extracting the center of the piano key surface that produces vertical depth changes within the piano key space mapping mesh and calibrating it as the fingertip touch space coordinates, and reading the wrist joint space coordinates within the primary hand contour topology vector is as follows: Monitor the surface pixel grayscale gradient abrupt change features of each independent piano key touch area within the piano key space mapping mesh, and lock the independent piano key touch areas that exhibit continuous sinking depth features; map the geometric center of the independent piano key touch areas that exhibit continuous sinking depth features to the three-dimensional space coordinate system and calibrate it as the fingertip touch space coordinates; scan towards the proximal end along the principal axis direction of the primary hand contour topology vector, extract the midpoint position of the contour boundary where the curvature change tends to be gentle, and read the three-dimensional space coordinates corresponding to the midpoint position of the contour boundary and mark them as the wrist joint space coordinates.

[0009] Furthermore, the fingertip touch space coordinates and wrist joint space coordinates are set as fixed anchor points. Hand bone node length ratio constraints are introduced, and the three-dimensional spatial coordinate sequences of the metacarpophalangeal joints and the first interphalangeal joint are reconstructed through inverse kinematics deduction. The specific process of generating a real-time three-dimensional skeleton topology sequence is as follows: In the three-dimensional spatial coordinate system, the relative positions of the topological ends of the fingertip touch space coordinates and wrist joint space coordinates are established to form fixed spatial anchor points. Pre-set human physiological hand bone linkage parameters are retrieved, and the length ratios of each phalanx segment and the physiological rotation limit angles of each joint are extracted from the linkage parameters as constraints on the length ratios of hand bone node lengths. These are substituted into the inverse kinematics equations to solve for the optimal spatial solution of the intermediate joint points that satisfies the fixed spatial anchor point positions and constraints. The optimal spatial solution is analyzed to reconstruct the three-dimensional spatial coordinate sequences of the metacarpophalangeal joints and the first interphalangeal joint. The joint nodes within the three-dimensional spatial coordinate sequence are connected according to the hand bone topology hierarchy rules to generate a real-time three-dimensional skeleton topology sequence.

[0010] Furthermore, within the time window of the piano key longitudinal depth change, the spatial displacement trajectories of the metacarpophalangeal joints and the first interphalangeal joint relative to the wrist joint in the real-time 3D skeleton topology sequence are extracted, and the specific process of comparing the vertical displacement gradient of the metacarpophalangeal joints with the rate of change of the flexion-extension angle of the first interphalangeal joint is as follows: Synchronize the global clock of the system with the piano key longitudinal depth change time window, extract the real-time 3D skeleton topology sequence within the longitudinal depth change time window, and establish a local reference coordinate system with the wrist joint spatial coordinates as the origin; map the 3D spatial coordinate sequence of the metacarpophalangeal joints and the first interphalangeal joint to the local reference coordinate system to output the spatial displacement trajectory, extract the displacement differential parameter of the metacarpophalangeal joint along the direction of gravity in the spatial displacement trajectory and output it as the vertical displacement gradient; extract the temporal derivative of the angle between the proximal and middle phalanges of the first interphalangeal joint and output it as the rate of change of the flexion-extension angle, and perform concurrent feature comparison by aligning the timestamps of the vertical displacement gradient and the rate of change of the flexion-extension angle.

[0011] Furthermore, under the condition that the rate of change of the flexion-extension angle of the first interphalangeal joint exhibits a negative reversal and the three-dimensional height of the metacarpophalangeal joint is lower than that of the wrist joint, the specific process of quantifying the relative acceleration of the corresponding joint and the height difference value to construct the force deformation feature vector is as follows: Real-time detection of the vector direction of the rate of change of the flexion-extension angle and the vertical extreme value of the three-dimensional height of the metacarpophalangeal joint; Under the triggering condition that the rate of change of the flexion-extension angle exhibits a negative reversal characteristic that breaks through the physiological positive flexion limit and the spatial projection of the three-dimensional height of the metacarpophalangeal joint is lower than that of the three-dimensional height of the wrist joint, the second-order time derivative parameter of the spatial displacement trajectory of the corresponding joint is extracted and quantified as relative acceleration; The spatial distance parameter between the three-dimensional height of the metacarpophalangeal joint and the three-dimensional height of the wrist joint in the direction of gravity is measured and quantified as the height difference value, and the relative acceleration and height difference values ​​are fused into the multi-dimensional feature matrix to construct the force deformation feature vector.

[0012] Furthermore, the specific process of inputting the force deformation feature vector into the touch posture specification database for comparison and parsing to obtain posture error labels, and extracting rest time nodes or long note sustain time nodes from the associated electronic music score is as follows: retrieve the pre-stored typical teaching method error force feature clusters in the touch posture specification database, measure the spatial similarity distance between the force deformation feature vector and the typical teaching method error force feature clusters in the multi-dimensional feature space; parse the identifier code bound to the feature cluster with the smallest spatial similarity distance and output it as the posture error label; parse the machine-readable note timing file of the associated electronic music score, scan the pronunciation gap interval along the time axis of the machine-readable note timing file to extract the rest time node, and scan the pronunciation interval whose time value span covers multiple standard note cycles to extract the long note sustain time node.

[0013] Furthermore, at rest time nodes or long note sustain time nodes, the specific process of generating a graphical user interface rendering signal containing the corrective text corresponding to the posture error label and a synchronous voice broadcast signal is as follows: Bind the system process timestamp and the electronic music score playback progress timestamp; within the response period when the system process timestamp matches the rest time node or long note sustain time node, retrieve the text sequence associated with the posture error label and read it as the corrective text; encapsulate the corrective text and preset visual layout coordinate parameters and convert them into a graphical user interface rendering signal; import the corrective text into the speech synthesis engine to transcribe it into an audio waveform data stream and output it as a synchronous voice broadcast signal.

[0014] The present invention has the following beneficial effects: (1) A real-time piano playing posture correction system based on hand shape recognition generates a topological vector by fusing entity contours and reflection edges from multi-source visual streams. It then uses a key space mapping mesh to calibrate the touch center and wrist joint coordinates and performs inverse kinematics deduction using constraints on the length ratio of hand bone nodes. This effectively overcomes the problem of severe visual occlusion in complex playing scenarios. This design can accurately reconstruct the three-dimensional spatial coordinate sequence of metacarpophalangeal joints and interphalangeal joints within the blind spot of vision without any contact or interference with the player's real physical tactile feedback. This provides highly coherent dynamic skeleton basic data for subsequent scientific evaluation of playing posture.

[0015] (2) A real-time piano playing posture correction system based on hand shape recognition compares the vertical displacement gradient of the metacarpophalangeal joints with the rate of change of flexion and extension angles of the interphalangeal joints concurrently within the transient time window of key sinking. It quantifies the relative acceleration of the corresponding joints and the height difference to construct a force deformation feature vector, and introduces a standardized database to compare and analyze error labels. Finally, it performs synchronous correction of voice and interface at rest or sustained note nodes. This design abandons the conventional method of static image appearance comparison. From the perspective of combining human biomechanics and music teaching methods, it accurately detects stubborn incorrect force habits and provides correction guidance at appropriate teaching times without interrupting the rhythm of the student's continuous musical phrase playing, which greatly improves the scientific nature and user experience of auxiliary teaching.

[0016] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0017] Figure 1 This is a flowchart of the real-time piano playing posture correction system based on hand shape recognition according to the present invention.

[0018] Figure 2 This is a schematic diagram of the temporal changes in joint displacement and angle during key press transients.

[0019] Figure 3A schematic diagram of multidimensional spatial clustering matching for force deformation features.

[0020] Figure 4 This is a schematic diagram of electronic music score timing and multimodal signal gating triggering. Detailed Implementation

[0021] This application's embodiments address the problems of existing auxiliary teaching technologies, such as the easy loss of joint positioning in complex playing occlusion environments and the inability of static visual methods to accurately assess dynamic force application posture, through a real-time piano playing posture correction system based on hand shape recognition.

[0022] The overall approach of the solution in this application is as follows: First, the optical reflection characteristics of the piano's glossy surface are used to extract the physical hand and its reflection pixels. A spatial grid is constructed by combining the intersection of the black and white keys to establish the absolute spatial reference points for the fingertips and wrists. Then, human physiological skeletal proportion constraints are introduced, and the three-dimensional topological sequence of the occluded finger joints is deduced based on the set fixed anchor points using the principle of inverse kinematics. Next, the focus is on the microscopic transient time window of key sinking, and the three-dimensional spatial displacement trajectory and angular derivative change characteristics of each joint are deeply tracked and compared. The deformation feature vector of abnormal force is extracted from the kinematic dimension. Finally, the feature vector is compared and analyzed with the standard music teaching database, and the corresponding corrective text and voice signal are output at the natural pause time of the musical phrase to achieve an interference-free automated posture guidance closed loop.

[0023] Please see Figure 1This invention provides a technical solution: a real-time piano playing posture correction system based on hand shape recognition, comprising the following modules: an environment mapping module, used to acquire multi-source visual streams including the keyboard and the player's hands, extract the physical contour pixels of the hand and the reflection edge pixels under bright conditions and fuse them to generate a primary hand contour topology vector, and identify the coordinates of the intersection points of the black and white keys to construct a key space mapping mesh; a skeleton deduction module, used to extract the center of the key surface that produces vertical depth changes within the key space mapping mesh and mark it as the fingertip touch space coordinates, read the wrist joint space coordinates within the primary hand contour topology vector, set the fingertip touch space coordinates and wrist joint space coordinates as fixed anchor points, introduce hand bone node length ratio constraints, reconstruct the three-dimensional space coordinate sequence of the metacarpophalangeal joints and the first interphalangeal joint through inverse kinematic deduction, and combine them to generate a real-time three-dimensional skeleton topology sequence. The posture analysis module is used to extract the spatial displacement trajectory of the metacarpophalangeal joints and the first interphalangeal joint relative to the wrist joint in the real-time 3D skeleton topology sequence within the time window of the vertical depth change of the piano keys. It compares the vertical displacement gradient of the metacarpophalangeal joints with the rate of change of the flexion and extension angle of the first interphalangeal joint. Under the condition that the rate of change of the flexion and extension angle of the first interphalangeal joint is negatively reversed and the 3D height of the metacarpophalangeal joint is lower than the 3D height of the wrist joint, it quantifies the relative acceleration of the corresponding joints and the height difference value to construct the force deformation feature vector. The posture correction module is used to input the force deformation feature vector into the touch posture specification database for comparison and parsing to obtain posture error labels. It extracts the rest time nodes or long note sustain time nodes in the associated electronic music score. At the rest time nodes or long note sustain time nodes, it generates a graphical user interface rendering signal containing the correction text corresponding to the posture error label and a synchronous voice broadcast signal.

[0024] In this implementation scheme, the environment mapping module is primarily responsible for capturing practice footage from multiple angles and establishing a digital reference system for the physical space. This module acquires the visual flow facing the keyboard and including the dust cover, extracts the actual edges of the hand and their reflected edges on the dust cover, and merges them into a primary hand contour topological vector. Simultaneously, it extracts the intersection points of the black and white keys to construct a key space mapping mesh. Here, the primary hand contour topological vector refers to the mathematical expression of the complete external contour of the hand formed by combining the actual hand image with its mirror reflection. The key space mapping mesh is a three-dimensional digital chessboard built based on the corner points of the black and white keys of a fixed size. Its technical advantage lies in effectively compensating for the visual blind spot where the inner shape of the palm cannot be observed from a top-down view by introducing the reflective surface of the piano's surface. Simultaneously, the key mesh provides a high-precision physical teaching environment benchmark for accurately calculating the absolute spatial position of the practitioner's hands on the keyboard.

[0025] The skeleton deduction module is primarily responsible for reconstructing the inherent skeletal movement of the hand under complex playing conditions. This module defines the center of the dropped key as the fingertip coordinate and the wrist joint in the contour vector as the wrist coordinate. Using these two points as fixed anchor points, and combined with the constraints of human skeletal length proportions, it calculates the spatial coordinate sequence of the metacarpophalangeal joints and the first interphalangeal joint through inverse kinematics deduction. Here, the fixed anchor points refer to three-dimensional coordinate points whose positions are already clear in a single spatial calculation and serve as the deduction benchmark. Inverse kinematics deduction is a mathematical algorithm that, based on the known end position (fingertip touch point) and the starting position (wrist point), and combined with the physiological limits of natural finger joint bending, reversely calculates the positions of intermediate obscured joints. Its technical advantage lies in completely solving the problem of visual loss caused by severe finger obstruction when the hands are crossed or playing chords in piano teaching. It accurately reconstructs a three-dimensional digital skeleton for posture assessment without requiring the learner to wear any invasive sensors, ensuring the continuity of teaching evaluation data.

[0026] The posture analysis module is primarily responsible for diagnosing the practitioner's force application habits and muscle posture health at the moment of key touch. During the extremely short time the keys sink, this module tracks the displacement trajectory of each joint in the skeletal sequence relative to the wrist, comparing the vertical drop of the metacarpophalangeal joints with the angular changes of the interphalangeal joints. When it detects negative bending of the interphalangeal joints and the metacarpophalangeal joints being lower than the wrist, it extracts acceleration and height difference to construct a force deformation feature vector. Here, negative bending represents a typical finger bending phenomenon in piano teaching, i.e., the finger joints failing to maintain their supporting curvature and instead concave towards the back of the hand—a physiological distortion. The force deformation feature vector transforms this unscientific degree of muscle exertion into a multi-dimensional numerical matrix that can be quantified by a computer. Its technical advantage lies in enabling the auxiliary system to move beyond superficial visual comparison based solely on external shape, delving into the dynamics of key touch, and accurately detecting fundamental force errors such as wrist collapse and finger bending that lead to inaccurate sound production or hand strain, much like a professional piano teacher.

[0027] The posture correction module is primarily responsible for transforming the force application defects diagnosed by the underlying algorithm into non-intrusive teaching guidance movements. This module substitutes the force application deformation feature vectors obtained from previous steps into the touch posture specification database to identify specific posture error labels. Then, it searches for rests or sustained note durations in the associated electronic score, simultaneously triggering text and voice prompts at these points. The touch posture specification database is a pre-recorded digital collection of correct and incorrect force application models from standard piano pedagogy. Rests or sustained note durations refer to the musical intervals in the current piece where there are natural pauses or periods without dense finger movement. Its technical advantage lies in completing a closed-loop teaching process from identifying the causes of wrong notes to implementing physical correction, while fully adhering to the principles of music education. It avoids abruptly interrupting the learner's rhythm with sudden vocalizations during continuous musical phrases, achieving highly targeted and intelligent guidance that aligns with the logic of human teacher-led practice.

[0028] Specifically, the process of acquiring multi-source visual flows including the keyboard and the practitioner's hands, extracting the physical outline pixels of the hand and the reflection edge pixels under bright conditions, and fusing them to generate a primary hand outline topological vector is as follows: The image acquisition device is called to simultaneously capture the top-down visual flow facing the keyboard plane and the reflected visual flow including the piano dust cover plane. The skin color pixel clusters in the top-down visual flow are separated to form the physical outline pixels of the hand, and the mirror boundary features in the reflected visual flow are tracked to form the reflection edge pixels. The physical outline pixels of the hand and the reflection edge pixels are aligned along the physical bottom intersection line of the piano dust cover. The three-dimensional spatial edge extreme points of the aligned pixel clusters are extracted to construct a topological envelope map including the fingertips to the wrist, and the primary hand outline topological vector containing the spatial positions of the three-dimensional spatial edge extreme points is output.

[0029] In this implementation scheme, to comprehensively capture the hand shape of the learner during complex playing states, the system overcomes the limitation that a single top-down camera cannot observe the palm of the hand. The system first captures images from two perspectives simultaneously using an image acquisition device. On one hand, it extracts the skin color pixels captured directly from the top-down view as the real hand contour. On the other hand, it utilizes the optical reflective properties of the glossy lacquer material of the piano dust cover to trace the mirrored edge of the hand reflected in the light. Subsequently, using the boundary line between the bottom of the dust cover and the keyboard as the axis of symmetry and physical alignment line, the system stitches and flips the real pixels and mirrored pixels in three-dimensional space. To extract an envelope map from the stitched data that accurately describes the undulations from the fingers to the wrist, the system performs spatial extremum point fusion calculations. The specific formula for calculating the coordinates of the primary hand contour topological extremum points is expressed as follows: ;in, : The coordinates of the m-th extreme point of the three-dimensional space edge in the primary hand contour topological vector; The m-th extreme point corresponds to the sharpness weight coefficient of the top-view visual flow. This weight coefficient is obtained by calculating the Laplacian variance matrix of the corresponding pixel neighborhood of the top-view image. The larger the variance value, the clearer and unobstructed the current view is, and the closer the weight is to a constant. Extreme pixel points of entity outlines in the top-down visual flow Perspective mapping function for converting to three-dimensional space; Extreme pixel points at the edge of the reflection in the visual flow of reflection. Based on the physical bottom line of the dustproof board The mirror inverse transformation function; The three-dimensional Euclidean distance parameter between the m-th extreme point and the physical ground intersection line; The reflection deformation attenuation threshold is obtained by fitting the limit distribution of curvature deformation error of the dustproof panel reflection as distance increases, measured in a standard piano environment. Through this deep spatial fusion algorithm, the system reconstructs the three-dimensional structure of the hand with height undulation information using the instrument's inherent optical environment, providing complete physiological morphological data support for subsequent judgment of whether the interphalangeal joints have collapsed or broken.

[0030] Specifically, the process of identifying the coordinates of the intersection points of the black and white keys and constructing the key space mapping mesh is as follows: Filter out dynamic background pixels in the multi-source visual flow, locate the corner point clusters with right-angled geometric features at the intersection of the black and white keys, fit a system of straight line equations parallel to and perpendicular to the keyboard edge along the direction of the line connecting adjacent corner point clusters, solve the mutual intersection points of the straight line equations to form the coordinates of the intersection points of the black and white keys; introduce the physical size ratio parameters of the standard piano keyboard, project the coordinates of the intersection points of the black and white keys onto the three-dimensional spatial coordinate system to divide the independent key touch areas, and connect the boundary nodes of each independent key touch area to construct the key space mapping mesh.

[0031] In this implementation scheme, after clarifying the hand shape, the system also needs to construct a data-driven base that maps one-to-one with the real keyboard to accurately determine which note the finger touched. The system first uses a background dynamic filtering algorithm to remove moving finger and sleeve pixels, exposing the naturally occurring right-angled gaps between the keys and identifying the corner points at the intersections of these black and white keys. Next, the system fits a grid of horizontally parallel and vertically perpendicular virtual straight lines along these regularly arrayed corner points, calculating the exact two-dimensional intersection points of the lines. To give the pixel coordinates on the plane a realistic physical meaning, the system incorporates the manufacturing dimensions of a standard piano for dimensionality reduction projection. The specific formula for the derivation of the three-dimensional spatial coordinates of the key boundary nodes is expressed as follows: ;in, : The three-dimensional coordinates of the boundary node of the independent piano key touch area in the a-th row and b-th column of the constructed piano key space mapping mesh; The coordinate matrix of the two-dimensional black and white bond edge intersection points obtained by solving the system of fitted linear equations; The intrinsic perspective operator for transforming a two-dimensional image pixel plane into a three-dimensional physical plane; The system's preset standard piano keyboard physical width dimension reference parameter; The difference in horizontal pixel spacing between the coordinates of the intersection points of two adjacent two-dimensional black and white key edges; Longitudinal depth of field distortion compensation coefficient: This coefficient is determined by pre-placing a checkerboard calibration plate of a specific ratio on the surface of a standard piano keyboard, measuring the pixel stretching ratio of different depth areas from the center of the field of view to the edge, and establishing a nonlinear regression curve to solve the problem. The system defines the absolute origin vector of the three-dimensional physical space of the piano key space mapping grid. Through this calculation, the system transforms the planar video image into a three-dimensional keyboard digital sandbox with real physical space scales. This not only delineates extremely precise note trigger boundaries but also provides an indispensable environmental reference system for subsequent reverse engineering to calculate the absolute height of each finger joint when hovering and exerting force above the keys.

[0032] Specifically, the process of extracting the center of the piano key surface that produces vertical depth changes within the piano key space mapping mesh and calibrating it as the fingertip touch space coordinates, and reading the wrist joint space coordinates within the primary hand contour topology vector, is as follows: Monitor the surface pixel grayscale gradient abrupt change features of each independent piano key touch area within the piano key space mapping mesh, and lock the independent piano key touch areas that exhibit continuous sinking depth features; map the geometric center of the independent piano key touch areas exhibiting continuous sinking depth features to the three-dimensional space coordinate system and calibrate it as the fingertip touch space coordinates; scan along the principal axis direction of the primary hand contour topology vector towards the proximal end, extract the midpoint position of the contour boundary where the curvature change tends to be gentle, and read the three-dimensional space coordinates corresponding to the midpoint position of the contour boundary and mark them as the wrist joint space coordinates.

[0033] In this implementation scheme, to accurately determine when and which specific key the practitioner pressed and the precise point of wrist force application, the system performs in-depth environmental feature extraction and coordinate calibration. Piano keys are typical lever mechanisms; when a finger presses a key, the reflective angle of the key surface changes instantaneously, resulting in a sharp drop in pixel grayscale in a specific touch area within the multi-source visual stream. The system precisely locates the key that experienced the downward movement by monitoring this abrupt change in grayscale gradient. Subsequently, the system projects the geometric center of that key into three-dimensional space as the endpoint of the hand movement, and simultaneously scans backward along the hand contour towards the arm, intercepting the midpoint where the curvature of the contour transitions from a complex palm shape to a gentle straight line, using this as the starting point of wrist force application. The physical mapping calculation formula for the core fingertip key touch spatial coordinate extraction is expressed as: ;in, The 3D spatial coordinates of the kth fingertip touch key were determined. : The three-dimensional position parameter of the static geometric center of the k-th independent piano key touch area when no longitudinal deformation occurs; : The unit normal vector perpendicular to the plane of the piano keyboard pointing downwards; and The start and end times of the longitudinal depth change monitoring time window; The mapping conversion coefficient of the pixel grayscale gradient of the piano key surface to the physical depth is established by using a robotic arm to press black and white keys with known depth under the factory standard illumination environment and establishing a linear regression equation between the grayscale attenuation value and the sinking depth value. The grayscale rate of change of a pixel with coordinates c and r at the center of the piano key surface over time is the derivative of this rate of change. Through this calculation, the system can obtain highly accurate three-dimensional coordinates of the key touch without installing any pressure sensors inside the keys, relying solely on non-contact visual-physical mapping. Simultaneously, by reading wrist joint coordinates, it successfully constructs two physical anchor points—far and proximal—essential for assessing hand force application posture, laying a spatial foundation for subsequently reconstructing details of occluded fingers.

[0034] Specifically, the process of setting the fingertip touch space coordinates and wrist joint space coordinates as fixed anchor points, introducing hand bone node length ratio constraints, and reconstructing the three-dimensional spatial coordinate sequence of the metacarpophalangeal joint and the first interphalangeal joint through inverse kinematics deduction to generate a real-time three-dimensional skeleton topology sequence is as follows: In the three-dimensional spatial coordinate system, the relative positions of the topological ends of the fingertip touch space coordinates and wrist joint space coordinates are established to form fixed spatial anchor points. Pre-set human physiological hand bone link parameters are retrieved, and the length ratio of each phalanx segment and the physiological rotation limit angle of each joint are extracted from the link parameters as constraints on the length ratio of hand bone node lengths. These are then substituted into the inverse kinematics equations to solve for the optimal spatial solution of the intermediate joint points that satisfies the fixed spatial anchor point positions and constraints. The optimal spatial solution is analyzed to reconstruct the three-dimensional spatial coordinate sequence of the metacarpophalangeal joint and the first interphalangeal joint. The joint nodes within the three-dimensional spatial coordinate sequence are connected according to the hand bone topology hierarchy rules to generate a real-time three-dimensional skeleton topology sequence.

[0035] In this implementation scheme, when faced with visual blind spots caused by finger joints being severely obscured by the back of the hand during hand-crossed playing or complex chords spanning octaves, the system utilizes inverse kinematics principles to reconstruct the topology of the hidden skeleton. Since the preceding steps have firmly locked the fingertip touch position and wrist joint position as two spatially fixed anchor points, and the length ratios of the phalanges of the human finger remain constant over a short period, while finger flexion is constrained by the extreme angles of physiological ligaments, the system transforms these physiological characteristics into mathematical constraints. It continuously deduces the optimal positions of the metacarpophalangeal joints and the first interphalangeal joint in virtual space until the deduced fingertip perfectly coincides with the actually calibrated fingertip coordinates. Its core inverse kinematics optimal spatial reconstruction formula is expressed as: ;in, : The reconstructed set of joint angle states containing the optimal spatial solutions of intermediate joint points; The multidimensional matrix to be solved for the relative rotation angles of the finger bone segments; : Spatial coordinates of the wrist joint read by the front-facing scanner; A forward spatial topology derivation function based on the constraint of the ratio of wrist joint spatial coordinates to the length of human hand bone nodes is used to calculate the theoretical fingertip free coordinates under the current prediction angle matrix. The established absolute coordinates of the fingertip touch in three-dimensional space; : Physiological limit violation penalty weight coefficient, which is determined empirically by analyzing the mechanical damping characteristics of non-natural bending states in a large number of non-destructive piano playing hand medical images; h: the h-th independent rotational degree of freedom in the hand topology; N: the total number of degrees of freedom of the fingers from the metacarpophalangeal joint to the first interphalangeal joint. For the h-th rotation angle With respect to the preset physiological rotation limit angle The system employs an over-limit constraint penalty function. By performing this complex inverse skeleton derivation calculation, the system, much like having X-ray vision, can accurately reconstruct the three-dimensional coordinate sequence of joints hidden deep within blind spots without relying on any wearable sensors. This approach completely solves the technical bottleneck of traditional visual recognition frequently losing tracking targets in complex piano teaching, ensuring the consistency and accuracy of subsequent dynamic force posture evaluation.

[0036] Please see Figure 2Specifically, within the time window of the vertical depth change of the piano keys, the spatial displacement trajectories of the metacarpophalangeal joints and the first interphalangeal joint relative to the wrist joint in the real-time three-dimensional skeleton topology sequence are extracted, and the process of comparing the vertical displacement gradient of the metacarpophalangeal joints with the rate of change of the flexion-extension angle of the first interphalangeal joint is as follows: Synchronize the global clock of the system with the time window of the vertical depth change of the piano keys, extract the real-time three-dimensional skeleton topology sequence within the time window of the vertical depth change, and establish a local reference coordinate system with the spatial coordinates of the wrist joint as the origin; map the three-dimensional spatial coordinate sequence of the metacarpophalangeal joints and the first interphalangeal joint to the local reference coordinate system to output the spatial displacement trajectory, extract the displacement differential parameter of the metacarpophalangeal joint along the direction of gravity in the spatial displacement trajectory and output it as the vertical displacement gradient; extract the temporal derivative of the angle between the proximal phalanx and the middle phalanx of the first interphalangeal joint and output it as the rate of change of the flexion-extension angle, and perform concurrent feature comparison by aligning the timestamps of the vertical displacement gradient and the rate of change of the flexion-extension angle.

[0037] In this implementation scheme, to accurately assess the muscle support state of the learner during the extremely short time it takes for their fingers to touch and press the keys, the system strictly aligns the global clock with the key-pressing time window, specifically capturing the skeletal movement at the instant of the key press. Since the hand continuously translates and crosses the keyboard during performance, using only absolute spatial coordinates cannot accurately reflect the bending shape of each finger. Therefore, the system establishes a local reference coordinate system with the wrist joint as the absolute stationary origin, mapping the three-dimensional coordinates of the metacarpophalangeal joints and the first interphalangeal joint into it, effectively eliminating displacement interference caused by the overall arm movement. Subsequently, the system extracts the vertical downward velocity change of the metacarpophalangeal joints and the rate of change of the angle between the phalanges at both ends of the first interphalangeal joint, comparing these two dynamic features at the same absolute time stamp concurrently. The core calculation formula for the concurrent feature comparison state quantity is expressed as: ;in, : Concurrent feature comparison state quantity of the u-th finger at time t; : The vertical spatial position parameter of the u-th metacarpophalangeal joint along the direction of gravity after mapping to the local reference coordinate system; : The directional feature vector of the proximal phalanx of the u-th finger in three-dimensional space; : The directional feature vector of the middle phalanx of the u-th finger in three-dimensional space; Vertical displacement sensitivity amplification factor, which is determined by calculating the reciprocal of the average vertical pressing physical stroke of a standard piano key; : Angular velocity normalization coefficient, which is determined by collecting the maximum instantaneous angular velocity of the first interphalangeal joint of a professional pianist when performing staccato and extracting its median envelope; : Timestamp alignment and feature cascading operators. Through deep spatial isolation and feature alignment parsing in this step, the system can, like a professional piano teacher, accurately perceive the microscopic dynamic changes of the fingers when they are subjected to the physical reaction force of the piano keys, providing crucial underlying data support for accurately determining whether the force-generating structure is healthy.

[0038] Specifically, under the condition that the rate of change of the flexion-extension angle of the first interphalangeal joint exhibits a negative reversal and the three-dimensional height of the metacarpophalangeal joint is lower than that of the wrist joint, the specific process of quantifying the relative acceleration of the corresponding joint and the height difference value to construct the force deformation feature vector is as follows: Real-time detection of the vector direction of the rate of change of the flexion-extension angle and the vertical extreme value of the three-dimensional height of the metacarpophalangeal joint; Under the triggering condition that the rate of change of the flexion-extension angle exhibits a negative reversal characteristic that breaks through the physiological positive flexion limit and the spatial projection of the three-dimensional height of the metacarpophalangeal joint is lower than that of the three-dimensional height of the wrist joint, the second-order time derivative parameter of the spatial displacement trajectory of the corresponding joint is extracted and quantified as relative acceleration; The spatial distance parameter between the three-dimensional height of the metacarpophalangeal joint and the three-dimensional height of the wrist joint in the direction of gravity is measured and quantified as the height difference value, and the relative acceleration and height difference values ​​are fused into a multi-dimensional feature matrix to construct the force deformation feature vector.

[0039] In this implementation plan, after capturing the dynamic characteristics at the moment of force application, the system needs to conduct in-depth qualitative and quantitative analysis of the incorrect key-touching posture. In standard piano pedagogy, a negative backward bend at the first interphalangeal joint represents a typical pathological posture of finger collapse, where the fingertip joint fails to maintain the supporting arc of a semi-grip ball and collapses unnaturally towards the back of the hand. Conversely, a metacarpophalangeal joint height lower than the wrist height represents a typical pathological posture of wrist collapse. The simultaneous occurrence of these two physiological phenomena indicates a complete breakdown of the force transmission chain from the arm to the fingertips. When this concurrent error condition is triggered, the system calculates the relative acceleration of the falling metacarpophalangeal joint to assess the degree of muscle loss of support, and simultaneously measures the height difference to assess the physical depth of wrist collapse. These parameters representing the degree of error are then mathematically fused, and the formula for constructing the force deformation feature vector is expressed as follows: ;in, Regarding the first The force deformation feature vector constructed from a target finger that triggers an incorrect posture; : No. The spatial absolute displacement trajectory vector of a target finger in the local reference coordinate system; : Acceleration scaling factor, which is determined by measuring the inertial damping response of a physical piano key action at a standard key strike. Multidimensional matrix tensor product concatenation operator; : Height drop penalty coefficient, which is determined by mapping the biomechanical tensile strain limit threshold of human hand tendons when the wrist is below the physical plane of the palm. The measured absolute spatial distance between the three-dimensional heights of the metacarpophalangeal joints and the three-dimensional heights of the wrist joints along the gravity axis; The rate of change of flexion-extension angle exhibits a negative reflex characteristic that breaks through the physiological positive flexion limit, triggering the absolute moment of initiation; The absolute moment when the three-dimensional height of the metacarpophalangeal joints returns to above the three-dimensional height of the wrist joint; The muscle relaxation time constant is determined by extracting the average muscle tension recovery cycle time of novice piano learners after an incorrect exertion of force. By constructing this highly condensed feature vector that encapsulates the severity of force dissipation, the system successfully transforms the invisible physiological muscle error state into a multidimensional matrix that can be precisely read by a computer. This allows subsequent comparisons with standardized databases to directly pinpoint the root causes of educational pathologies, thereby significantly improving the scientific accuracy of the teaching assistance and posture correction system.

[0040] Please see Figure 3 Specifically, the process of inputting the force deformation feature vector into the touch posture specification database for comparison and parsing to obtain posture error labels, and extracting rest time nodes or long note sustain time nodes from the associated electronic music score is as follows: Retrieve the pre-stored typical teaching method error force feature clusters from the touch posture specification database, measure the spatial similarity distance between the force deformation feature vector and the typical teaching method error force feature clusters in the multi-dimensional feature space; parse the identifier code bound to the feature cluster with the smallest spatial similarity distance and output it as the posture error label; parse the machine-readable note timing file of the associated electronic music score, scan the pronunciation gap interval along the time axis of the machine-readable note timing file to extract the rest time node, and scan the pronunciation interval whose time value span covers multiple standard note cycles to extract the long note sustain time node.

[0041] In this implementation plan, to transform the obscure underlying physical motion data into music teaching guidance that learners can understand, the system establishes a bridge connecting biomechanics and piano pedagogy. The system first retrieves a built-in standard touch posture specification database, which pre-stores a large number of standardized feature multidimensional matrices of typical force application errors, such as finger bending leading to joint depression or wrist collapse causing a downward shift in the center of gravity. The system diagnoses the specific error type by measuring the spatial similarity distance between the currently extracted force deformation feature vector and each known error feature cluster in the database. To make this matching more consistent with the tolerance characteristics of human physical motion and to eliminate dimensional interference, the spatial similarity distance metric and the analytical calibration formula for posture error labels are expressed as follows: ;in, : Best matching posture error label identifier code in the parsed output; g: Index number of the typical teaching method error force application feature cluster in the touch posture specification database; The total number of typical error feature clusters stored in the database; : The force deformation feature vector that is extracted and combined in real time; : The standard cluster center vector of the g-th typical teaching method error feature cluster in the multidimensional feature space; The inverse covariance matrix of the feature dimension is determined by multivariate statistical distribution analysis of a large number of normal and incorrect key touch feature samples covering piano beginners of different ages collected before the system leaves the factory. It is used to eliminate the deviation interference caused by the different physical units of acceleration and height difference. : Prior penalty coefficient for high-frequency and easily occurring errors. This coefficient is determined by mapping the statistical probability distribution of various touch errors in authoritative piano education statistical literature. The system represents the historical prior probability distribution function of type g posture errors in the daily practice group of beginners. By comprehensively comparing and calculating the Mahalanobis distance and prior probability in this multidimensional space, the system can pinpoint the specific pathological errors leading to the collapse of force exertion with remarkable precision. Subsequently, instead of immediately issuing an error alarm, the system, in accordance with the objective educational principle of not arbitrarily interrupting the practitioner's complete musical expression, deeply analyzes the note timing file of the electronic score, specifically seeking rest intervals without vocalization tasks, or long note durations where the notes are long enough that the fingers do not need to move quickly, extracting these natural pauses in the musical beat as the optimal time points for implementing auxiliary teaching feedback.

[0042] Please see Figure 4 Specifically, at rest time nodes or long note sustain time nodes, the process of generating a graphical user interface rendering signal containing the corrective text corresponding to the posture error label and a synchronous voice broadcast signal is as follows: Bind the system process timestamp and the electronic music score playback progress timestamp; within the response period when the system process timestamp matches the rest time node or long note sustain time node, retrieve the text sequence associated with the posture error label and read it as the corrective text; encapsulate the corrective text and preset visual layout coordinate parameters and convert them into a graphical user interface rendering signal; import the corrective text into the speech synthesis engine to transcribe it into an audio waveform data stream and output it as a synchronous voice broadcast signal.

[0043] In this implementation scheme, after accurately diagnosing postural pathology and finding the optimal feedback timing, the system begins to provide non-invasive closed-loop guidance to the learner. The system deeply binds its own system process clock to the progress clock of the electronic music score playback. When the time scale advances to the previously locked rest or sustained note node, it immediately retrieves the instructional text corresponding to the error label, such as "pay attention to maintaining a semi-clenched ball position." The system converts this text into a rendering signal that can be displayed on the screen and simultaneously imports it into the speech engine for transcription into an audio waveform stream. To ensure that the audiovisual dual-modal guidance signal is accurately triggered within the optimal response period and does not interfere with the normal playing of subsequent notes, the generation formula for the synchronous multimodal teaching correction signal is expressed as: ;in, The system's absolute timestamp in the current process. The output synchronous multimodal teaching and correction integrated signal; Step-type time-gated trigger operator, used to open the signal output channel within the safe response period when the current time is at a rest or long tone delay node; : The starting absolute timestamp of the rest time node or long note duration node extracted from the preceding sequence; The maximum effective duration threshold for visual and voice feedback signals is dynamically calculated by dynamically reading the tempo of the current electronic score, identifying the beat count, and extracting the absolute physical seconds occupied by the current rest, ensuring that the voice broadcast ends completely before the next note is played. : Graphical user interface rendering signal conversion engine functions; : Retrieved and mapped from the database, along with the associated corrective text sequence; The preset visual layout coordinate parameters are used to control the absolute floating position of the text on the music score display screen to avoid the line of sight of the staff score track that the current learner is reading; System-level synchronous parallel fusion operator for audiovisual multimodal signals; : A speech synthesis engine function that converts a corrected text sequence into an audio waveform data stream; The fundamental frequency pitch parameter of the voice broadcast; : The smooth fade-out attenuation coefficient of the corrected signal, which is determined by fitting the acoustic attenuation characteristic curve that simulates the natural weakening of the sound at the end of a guided musical phrase by a human music teacher. The current guidance signal has been continuously output for a cumulative period of time since it was triggered. By executing this highly synchronized signal generation and gating limit calculation, the system perfectly simulates the teaching process of a professional piano teacher providing precise guidance during the pauses between musical phrases. This effectively conveys targeted posture correction information while greatly protecting the musical continuity of the learner during immersive performance, fully demonstrating the guiding value of the intelligent system in the field of music-assisted education.

[0044] In summary, this application has at least the following effects: This real-time piano playing posture correction system based on hand shape recognition integrates entities and reflection edges from multi-source visual streams to construct a key space mapping mesh. Combined with human skeletal length constraints, it performs inverse kinematics deduction, accurately reconstructing the three-dimensional hand skeleton sequence under complex playing occlusion conditions in a completely non-contact state. This effectively avoids interference from wearable hardware devices on the player's actual physical feedback from key touch. Furthermore, the system focuses on the transient window of key sinking, deeply comparing the vertical displacement gradient of the metacarpophalangeal joints and the rate of change of flexion and extension angles of the interphalangeal joints to quantify the force deformation feature vector. This overcomes the technical limitations of traditional static two-dimensional vision, which cannot perceive microscopic mechanical errors in force application habits. Finally, the force deformation feature vector is input into a standardized database for comparison and analysis of posture error labels. Simultaneously, graphic and speech correction signals are triggered at rests or natural pauses in the associated electronic score, constructing an automated correction closed loop that highly conforms to the principles of professional music education. This achieves highly accurate and intelligent music-assisted teaching guidance that absolutely does not interrupt the player's immersive playing rhythm.

[0045] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0046] This invention is described with reference to flowchart illustrations and / or block diagrams of systems, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0047] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0048] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0049] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0050] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A real-time piano playing posture correction system based on hand shape recognition, characterized in that, Includes the following modules: The environment mapping module is used to acquire multi-source visual streams including the keyboard and the practitioner's hands, extract the physical outline pixels of the hands and the reflection edge pixels in a bright environment and fuse them to generate a primary hand outline topology vector, and identify the coordinates of the intersection points of the black and white keys to construct a keyboard space mapping mesh. The skeleton derivation module is used to extract the center of the piano key surface that produces vertical depth changes within the piano key space mapping mesh and calibrate it as the fingertip touch space coordinates. It reads the wrist joint space coordinates within the primary hand contour topology vector, sets the fingertip touch space coordinates and wrist joint space coordinates as fixed anchor points, introduces hand bone node length ratio constraints, and reconstructs the three-dimensional space coordinate sequence of the metacarpophalangeal joint and the first interphalangeal joint through inverse kinematic derivation. The sequence is then combined to generate a real-time three-dimensional skeleton topology sequence. The posture analysis module is used to extract the spatial displacement trajectory of the metacarpophalangeal joint and the first interphalangeal joint relative to the wrist joint in the real-time 3D skeleton topology sequence within the time window of the longitudinal depth change of the piano keys. It compares the vertical displacement gradient of the metacarpophalangeal joint with the rate of change of the flexion and extension angle of the first interphalangeal joint. Under the condition that the rate of change of the flexion and extension angle of the first interphalangeal joint is negatively reversed and the 3D height of the metacarpophalangeal joint is lower than the 3D height of the wrist joint, it quantifies the relative acceleration of the corresponding joint and the height difference value to construct the force deformation feature vector. The posture correction module is used to input the force deformation feature vector into the touch posture specification database for comparison and parsing to obtain posture error labels, extract rest time nodes or long note duration time nodes in the associated electronic music score, and generate a graphical user interface rendering signal and a synchronous voice broadcast signal containing the posture error label and the corresponding correction text at the rest time node or long note duration time node.

2. The real-time piano playing posture correction system based on hand shape recognition according to claim 1, characterized in that: The specific process of acquiring multi-source visual streams containing the keyboard and the practitioner's hands, extracting the entity contour pixels of the hand and the reflection edge pixels in a bright environment, and fusing them to generate a primary hand contour topology vector is as follows: The image acquisition device is invoked to simultaneously capture the top-down visual flow facing the keyboard plane and the reflected visual flow including the piano dust cover plane. The skin color pixel clusters in the top-down visual flow are separated to form the outline pixels of the hand entity, and the mirror boundary features in the reflected visual flow are traced to form the reflection edge pixels. Align the physical outline pixels of the hand with the reflection edge pixels along the physical bottom intersection line of the piano dust cover. Extract the three-dimensional spatial edge extreme points of the aligned pixel cluster to construct a topological envelope map containing the fingertips to the wrist. Output a primary hand outline topological vector containing the spatial positions of the three-dimensional spatial edge extreme points.

3. The real-time piano playing posture correction system based on hand shape recognition according to claim 2, characterized in that: The specific process of identifying the coordinates of the intersection points of the black and white keys and constructing the key space mapping mesh is as follows: Filter out dynamic background pixels in the multi-source visual stream, locate the corner point clusters with right-angled geometric features at the junction of black and white keys, fit a set of straight line equations parallel to and perpendicular to the keyboard edge along the direction of the line connecting adjacent corner point clusters, and solve the mutual intersection points of the straight line equations to form the coordinates of the intersection points of the black and white key edges. By introducing the physical size ratio parameters of a standard piano keyboard, the coordinates of the intersection points of the black and white keys are projected onto a three-dimensional spatial coordinate system to divide the independent key touch areas. The boundary nodes of each independent key touch area are connected to construct a key space mapping mesh.

4. The real-time piano playing posture correction system based on hand shape recognition according to claim 1, characterized in that: The process of extracting the center coordinates of the piano key surfaces that produce vertical depth variations within the piano key space mapping mesh and calibrating them as the fingertip touch space coordinates, and then reading the wrist joint space coordinates within the primary hand contour topology vector, is as follows: Monitor the surface pixel grayscale gradient abrupt change characteristics of each independent piano key touch area within the piano key space mapping grid, and lock the independent piano key touch areas that exhibit continuous sinking depth characteristics; The geometric center of the independent piano key touch area, which exhibits a continuous sinking depth characteristic, is mapped to a three-dimensional spatial coordinate system and calibrated as the fingertip touch space coordinate. The system scans proximally along the principal axis of the primary hand contour topology vector, extracts the midpoint of the contour boundary where the curvature change tends to be gentle, and reads the three-dimensional spatial coordinate corresponding to the midpoint of the contour boundary and marks it as the wrist joint space coordinate.

5. The real-time piano playing posture correction system based on hand shape recognition according to claim 4, characterized in that: The specific process of setting the fingertip touch space coordinates and wrist joint space coordinates as fixed anchor points, introducing hand bone node length ratio constraints, reconstructing the three-dimensional space coordinate sequence of the metacarpophalangeal joints and the first interphalangeal joint through inverse kinematic deduction, and combining them to generate a real-time three-dimensional skeleton topology sequence is as follows: In a three-dimensional spatial coordinate system, the relative positions of the topological ends of the fingertip touch space coordinate and the wrist joint space coordinate are established to form a fixed spatial anchor point. Pre-set human physiological hand bone linkage parameters are retrieved, and the length ratio of each finger bone segment and the physiological rotation limit angle of each joint in the linkage parameters are extracted as constraints on the length ratio of hand bone nodes. Substitute the inverse kinematics equations to find the optimal spatial solution for the intermediate joints that satisfy the fixed anchor point position and constraints. Analyze the optimal spatial solution to reconstruct the three-dimensional spatial coordinate sequence of the metacarpophalangeal joint and the first interphalangeal joint. Connect the joint nodes in the three-dimensional spatial coordinate sequence according to the topological hierarchy rules of the hand skeleton to generate a real-time three-dimensional skeleton topology sequence.

6. The real-time piano playing posture correction system based on hand shape recognition according to claim 1, characterized in that: Within the time window of the longitudinal depth change of the piano keys, the spatial displacement trajectories of the metacarpophalangeal joints and the first interphalangeal joint relative to the wrist joint in the real-time 3D skeleton topology sequence are extracted, and the specific process of comparing the vertical displacement gradient of the metacarpophalangeal joints with the rate of change of the flexion-extension angle of the first interphalangeal joint is as follows: Synchronize the global clock of the system with the longitudinal depth change time window of the piano keys, extract the real-time three-dimensional skeleton topology sequence within the longitudinal depth change time window, and establish a local reference coordinate system with the wrist joint spatial coordinate as the origin; The three-dimensional spatial coordinate sequence of the metacarpophalangeal joint and the first interphalangeal joint is mapped to the local reference coordinate system to output the spatial displacement trajectory. The displacement differential parameter of the metacarpophalangeal joint along the direction of gravity in the spatial displacement trajectory is extracted and output as the vertical displacement gradient. The temporal derivative of the angle between the proximal and middle phalanges of the first interphalangeal joint is extracted and output as the rate of change of flexion-extension angle. Concurrent feature comparison is performed by aligning the timestamps of the vertical displacement gradient and the rate of change of flexion-extension angle.

7. The real-time piano playing posture correction system based on hand shape recognition according to claim 6, characterized in that: Under the condition that the rate of change of the flexion-extension angle of the first interphalangeal joint is negatively reversed and the three-dimensional height of the metacarpophalangeal joint is lower than that of the wrist joint, the specific process of quantifying the combination of the relative acceleration of the corresponding joint and the height difference to construct the force deformation feature vector is as follows: Real-time detection of the vector direction of the rate of change of flexion and extension angles and the vertical extreme value of the three-dimensional height of the metacarpophalangeal joint; Under the triggering conditions that the rate of change of flexion and extension angles exhibits negative reflexion characteristics that exceed the physiological positive flexion limit and the spatial projection of the three-dimensional height of the metacarpophalangeal joint is lower than the spatial projection of the three-dimensional height of the wrist joint, the second-order time derivative parameter of the corresponding joint spatial displacement trajectory is extracted and quantified into relative acceleration. The spatial distance parameters of the three-dimensional height of the metacarpophalangeal joint and the three-dimensional height of the wrist joint along the gravity axis are measured and quantified into height difference values. The relative acceleration and height difference values ​​are then integrated into a multi-dimensional feature matrix to construct a force deformation feature vector.

8. The real-time piano playing posture correction system based on hand shape recognition according to claim 1, characterized in that: The specific process of inputting the force deformation feature vector into the touch posture specification database for comparison and parsing to obtain posture error labels, and extracting rest time nodes or sustained note time nodes from the associated electronic music score is as follows: Retrieve the typical teaching method error force application feature clusters pre-stored in the touch posture specification database, and measure the spatial similarity distance between the force application deformation feature vector and the typical teaching method error force application feature clusters in the multi-dimensional feature space. The identifier code bound to the feature cluster with the smallest spatial similarity distance is output as a pose error label. The machine-readable note timing file associated with the electronic musical score is parsed, and the pronunciation gap interval is extracted as a rest time node by scanning along the time axis of the machine-readable note timing file. The pronunciation interval with a time value span covering multiple standard note cycles is extracted as a long note duration time node.

9. The real-time piano playing posture correction system based on hand shape recognition according to claim 8, characterized in that: The specific process of generating the graphical user interface rendering signal and the synchronous voice broadcast signal containing the posture error label and the corresponding corrective text at the rest time node or long note persistence time node is as follows: Bind the system process timestamp and the electronic music score playback progress timestamp. Within the response period when the system process timestamp matches the rest time node or the long note duration time node, retrieve the text sequence associated with the posture error label and read it as the correction text. The encapsulated corrected text and preset visual layout coordinate parameters are converted into graphical user interface rendering signals, and the corrected text is imported into the speech synthesis engine to be transcribed into an audio waveform data stream and output as a synchronous speech broadcast signal.

Images