A wrist plucking touch key action measurement platform, measurement method and data analysis method based on leap motion
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2023-05-09
- Publication Date
- 2026-06-23
Smart Images
Figure CN116481575B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of motion learning and measurement, and in particular to a wrist-based piano-playing touch motion measurement platform and method based on Leap Motion. Background Technology
[0002] Piano playing is a relatively specialized hand skill, requiring beginners to practice repeatedly under the guidance of a professional teacher to master it. Maintaining correct hand posture is crucial for powerful key strikes and preventing hand injuries from prolonged playing. Therefore, analyzing the hand movement patterns during piano playing based on standard human key-striking motions has become a fundamental part of piano skill training and teaching, and even robotic piano instruction. In piano performance, the wrist maintains coordination and balance, playing a pivotal role in power transmission. [1~2] Only by correctly using the coordinated movements of the wrist, fingers, and arm can one play steadily, achieve a good playing state, and thus produce a smooth, solid, and powerful sound.
[0003] Currently, there are various methods for capturing human hand movements, with data glove acquisition and surface electromyography (EMG) signal measurement being the most common. [3] However, wearing data gloves can interfere with normal hand movements while playing the piano, causing the measured results to fail to reflect standard piano touch. Furthermore, surface electromyography (EMG) signals require attaching acquisition points to nerves in the arm, and prolonged movement may cause the electrodes to slip, degrading the measurement results. The Leap Motion sensor, a professional hand information capture device, can directly identify hand joints and fingertip feature points without requiring any attachments or wearing on the hand, thus not affecting piano playing movements. [4] Moreover, Leap Motion offers fast frame rates and high accuracy, making it well-suited for measuring hand movements while playing the piano.
[0004] The optimal measurement position for Leap Motion is 20-30cm directly below the hand. As an optical sensor, it requires no obstruction between the sensor and the hand during measurement, thus necessitating a transparent platform to complete the hand motion measurement. [5~6] Even with this approach, optical sensors still have their limitations. Using transparent glass or acrylic plates can cause refraction, reducing the reliability of hand information measured by Leap Motion. Therefore, using multiple sensors to compensate for the shortcomings of a single Leap Motion sensor and building a new measurement platform to measure human hand movements while playing the piano, processing the measurement data and performing data fusion, is crucial for obtaining accurate hand movement information. Only with accurate hand movement information can the correct patterns of hand movements while playing the piano be derived through analysis.
[0005] [1] Huang Yan. Analysis of wrist technique in piano playing [J]. Yellow River Voice. 2019(16):39
[0006] [2] Xiao Ziming. Exploring wrist techniques in piano performance [J]. Northern Music, 2018, 38(22):63-64.
[0007] [3] Zhu Ying, Yang Dikai, Bing Linyuan, et al. Research on robotic hand based on data glove somatosensory control [J]. Electronic Design Engineering, 2021, 29(12): 61-66.
[0008] [4] Peng Miao, Chen Guoli, Jiang Yu. Research on hand position information acquisition method based on Leap Motion somatosensory sensor [J]. Southern Agricultural Machinery, 2021, 52(07): 28-33.
[0009] [5] Weichert F, Bachmann D, Rudak B, et al. Analysis of theAccuracy and Robustness of the Leap Motion Controller[J]. Sensors, 2013, 13(5):6380-6393.
[0010] [6] Nizamis K, Rijken N, Mendes A, et al. A Novel Setup andProtocol to Measure the Range of Motion of the Wrist and the Hand[J]. Sensors, 2018, 18(10). Summary of the Invention
[0011] The purpose of this invention is to overcome the shortcomings of existing technologies and address the limitations of single-leap motion in measuring wrist-based piano playing movements. This invention adds a new Leap Motion sensor to compensate for the deficiencies of single-sensor measurement. Combining this with the posture habits of the human hand when playing the piano, a dual-leap motion-based platform for measuring wrist-based piano playing movements and a data fusion analysis method are proposed. This platform can adjust the sensor's pose to find the optimal measurement position and complete the measurement of piano playing movement data. Furthermore, it synchronously fuses the measurement data from the two sensors, reducing the influence of other factors on the measurement process and improving data accuracy.
[0012] The objective of this invention is achieved through the following technical solution:
[0013] A Leap Motion-based wrist piano playing touch motion measurement platform includes a sensor position adjustment device, a measurement platform frame, an upper platform glass surface, a fixed glass surface, a lower platform, a touch key limiting structure, and a Leap Motion sensor; the measurement platform frame is a cuboid three-dimensional frame structure made of aluminum profiles, and a support frame is provided in the middle of the measurement platform frame at a distance of 20~30cm from the top and bottom of the measurement platform frame.
[0014] The sensor position adjustment device consists of a corner bracket, a connecting rod, and a movable hinge. The connecting rod is fixed to the top of the measurement platform frame by the corner bracket, and a movable hinge is provided on the connecting rod. A LeapMotion sensor is installed on the movable hinge.
[0015] The support frame has a fixed glass surface and an upper platform glass surface, and the fixed glass surface is provided with piano key position marks; the touch key limiting structure is provided between the fixed glass surface and the upper platform glass surface;
[0016] The bottom of the measurement platform frame is provided with a lower platform made of acrylic plate; a Leap Motion sensor is installed on the lower platform.
[0017] Furthermore, the touch limit structure consists of a right-angle connector, a screw with holes, a set screw, and a steel wire rope. The steel wire rope is fixed to the right-angle connector by the screw with holes, and the right-angle connector is fixed to the measuring platform frame by the set screw. The right-angle connector is provided with a built-in corner groove.
[0018] Furthermore, the distance between the upper platform glass surface and the steel wire rope is 9.5mm, which meets the standard key depth requirements for piano keys.
[0019] Furthermore, the thickness of the glass surface of the upper platform is 2mm.
[0020] Furthermore, the measuring platform frame and support frame are both made of 15×15mm aluminum profiles.
[0021] This invention also provides a method for measuring wrist touch motion while playing the piano based on Leap Motion, comprising:
[0022] (1) Place the Leap Motion sensor located on the lower platform directly below the palm of the hand, and set the Leap Motion sensor located above to a tilt angle of 10-30°. On the horizontal plane, the distance from the middle finger metacarpophalangeal joint along the direction of the fingers is 5-10cm to ensure that each joint of the fingers can be measured.
[0023] (2) The hands should maintain the standard movements required when playing the piano, and place them on the upper platform glass surface and the fixed glass surface;
[0024] (3) Simulate staccato action; place the middle finger on the steel wire rope inside the key-limiting structure, and place the other fingers on the glass planes on both sides to perform the staccato action of the middle finger touching the key;
[0025] (4) Description of the simulated key crossing action: Place the four fingers other than the thumb on the fixed glass surface, corresponding to the piano key marks on the fixed glass surface respectively; the thumb performs the key crossing action, crossing one, two, and three piano key marks respectively, which corresponds to crossing one, two, and three white keys in the actual piano playing action; the other fingers keep their positions unchanged during the thumb's key crossing process, simulating the key crossing action in the actual piano playing process; during the key crossing process, the thumb first lifts the finger to the corresponding position of the piano key mark, the wrist moves in the direction of movement to assist the key crossing, pauses and then drops the finger onto the steel wire, stays for the corresponding time according to normal piano playing habits, and repeats the key crossing action in the above manner;
[0026] (5) Collect the motions of several sets of staccato and cross-key movements, taking 20 staccato notes as a set of staccato movements and 20 cross-key movements as a set of staccato movements.
[0027] This invention also provides a method for analyzing wrist-based piano-playing touch motion data based on Leap Motion, including: synchronizing sensor measurement data, which is divided into spatial synchronization and temporal synchronization; spatial synchronization unifies the data of two upper and lower Leap Motion sensors into the same coordinate system; temporal synchronization locates the measurement points of the two upper and lower Leap Motion sensors at the same time; and by finding the patterns in the data movement process, the starting point and ending point of each cycle are determined.
[0028] A weighted fusion method is adopted, using the length of the finger joint as the fusion standard. The finger joint lengths measured by the upper and lower LeapMotion sensors are calculated. When the difference between the length measured by one sensor and the actual finger length exceeds 1mm, the measurement value at that moment is considered unreliable, and the measurement result of the other sensor is selected as the final result. When both sensors exceed the actual value by 1mm, they are considered unreliable, and the data at the corresponding time point is skipped. When the difference between the measured length and the actual length is within 1mm, a weighted method is used, with the ratio of the measured finger joint length to the actual length as the weight, and the fused data is obtained after weighting.
[0029] Spatial synchronization is as follows: With the poses of the upper and lower Leap Motion sensors unchanged, the hand is placed in four different positions, and the hand position information at each position is measured. Based on the MCP position data of the middle finger metacarpophalangeal joint, the coordinate transformation matrix R from the upper Leap Motion sensor coordinate system to the lower Leap Motion sensor coordinate system is calculated. The coordinate transformation matrix includes 3D angle transformation and displacement components, therefore R is a 4th order matrix, as shown in the following equation:
[0030] (1)
[0031] in This is the MCP position data of the middle finger metacarpophalangeal joint measured by the sensor above. The data is the position data of the metacarpophalangeal joint (MCP) of the middle finger measured by the sensor below;
[0032] The data on the position of the metacarpophalangeal joint (MCP) of the middle finger were processed. The average value of every ten data sets was calculated, and data exceeding the average value by 1 mm were filtered out. The average value was filtered, and the coordinate transformation equations of four different hand positions were solved to obtain the coordinate transformation matrix, thus completing spatial synchronization.
[0033] The time synchronization is as follows:
[0034] Based on the movement patterns of playing the piano, the human hand's piano playing touch action can be summarized as a process of touching the key, pausing, and returning to the starting position; the middle finger's staccato touch action is manifested as a staccato touch, pressing the key and pausing, and the hand jumping up; the thumb's straddling action is manifested as a thumb straddling the key, pressing the key and pausing, and the thumb lifting and returning to the starting position; by grasping the regularity points, the start and end points of the same cycle are synchronized.
[0035] In one motion cycle, the measured angle changes in a sawtooth pattern. The start and end points of each cycle, i.e., the peaks and troughs, are synchronized to obtain the data a and b of the same cycle from the two sensors, where a > b. Assuming the motion cycle time is 1, the data corresponding to b is used as the reference, and the data with a time difference less than 1 / 3b are fused. The data with the smallest difference is selected for fusion. After completing the subsequent fusion process, the redundant data in the data group a is compared with the actual finger joint length, and the data with a difference less than 0.5mm is inserted into the corresponding time point.
[0036] Compared with the prior art, the beneficial effects of the technical solution of the present invention are as follows:
[0037] 1. This invention relates to a wrist-based piano-playing touch motion measurement platform based on dual Leap Motion sensors. This platform can collect data on human wrist movements during normal piano playing, process and analyze the collected data to obtain the laws governing piano-playing touch motion. Using two Leap Motion sensors reduces the influence of environmental factors such as position and lighting on a single sensor, effectively improving the measurement range. The most significant feature of the experimental platform is its adjustability, including the adjustability of the upper sensor's pose and the position of the touch-limiting structure. This makes the platform suitable for measurements in different situations and environments, greatly improving the effectiveness and accuracy of the collected motion data.
[0038] 2. The present invention provides a method for analyzing wrist motion data of human piano playing based on dual Leap Motion. This method can identify the start and end points of each key-touching motion based on the hand's touch data, and synchronize the two sensors (upper and lower) simultaneously. To address the issue of unstable frame rates in Leap Motion, measurement data from similar moments are synchronized. The finger joint length of the finger to be analyzed is used as the fusion standard, and the weights of the two sensors are calculated based on their correlation data. This approach reduces measurement errors and improves data accuracy. Attached Figure Description
[0039] Figure 1 This is a schematic diagram of the measurement platform structure of the present invention;
[0040] Figure 2 This is a schematic diagram of the sensor position adjustment device of the present invention;
[0041] Figure 3 This is a schematic diagram of the measurement platform frame structure of the present invention;
[0042] Figure 4 This is a schematic diagram of the upper platform glass surface and the fixed glass surface of the present invention;
[0043] Figure 5 This is a schematic diagram of the key limiting structure of the present invention;
[0044] Figure 6 , Figure 7 This is a diagram of the piano key touch measurement method of the present invention;
[0045] Figure 8 It is a graph showing the approximate changes in finger rotation angle over multiple measurement cycles.
[0046] Reference numerals: 1-Sensor position adjustment device; 2-Measuring platform frame; 3-Upper platform glass surface; 4-Fixed glass surface; 5-Touch limit structure; 6-Leap Motion sensor; 7-Lower platform; 101-Angle bracket; 102-Connecting rod; 103-Modible hinge; 201-Aluminum profile; 202-Bolt; 203-Angle bracket Detailed Implementation
[0047] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.
[0048] This embodiment provides a Leap Motion-based wrist piano-playing touch motion measurement platform, including a sensor position adjustment device 1, a measurement platform frame 2, an upper platform glass surface 3, a fixed glass surface 4, a lower platform 7, a touch key limiting structure 5, and a Leap Motion sensor 6; the measurement platform frame 2 is a cuboid three-dimensional frame structure made of aluminum profiles. Figure 1 As shown, the wrist piano playing touch motion measurement platform based on Leap Motion mainly consists of 6 parts: 1. sensor position adjustment device, 2. measurement platform frame, 3. upper platform glass surface, 4. fixed glass surface, 5. touch key limiting structure, 6. Leap Motion sensor, and 7. lower platform.
[0049] like Figure 2 As shown, the sensor position adjustment device 1 consists of a corner bracket 101, a connecting rod 102, and a movable hinge 103. The connecting rod 102 is made of a 15×15cm aluminum profile. The connecting rod 102 is fixed to the top of the measurement platform frame 2 via a 90° corner bracket 101. A movable hinge 103 is installed on the connecting rod 102. A Leap Motion sensor 6 is mounted on the movable hinge 103. The position of the sensor above is adjusted using the movable hinge 103. The movable hinge 103 itself can adjust the angle of the sensor. By adjusting the position of the movable hinge and the aluminum profile above, the horizontal position of the sensor can be adjusted. In this way, the pose of the upper Leap Motion sensor 6 can be adjusted to meet the sensor position requirements of the measurement, and the optimal measurement position of the Leap Motion sensor can be found in this way.
[0050] like Figure 3 As shown, the measuring platform frame 2 uses 15×15mm aluminum profile 201, and is secured with bolts 202 and... Connect the corner code 203, and set the distance between the upper and lower Leap Motion sensors and the human hand between 20-30cm, so that the human hand is within the optimal measurement range of Leap Motion during measurement.
[0051] like Figure 4 As shown, both the upper platform glass surface 3 and the fixed glass surface 4 are made of high-transparency glass with a thickness of 2mm, which reduces refraction by the Leap Motion sensor and improves its recognition accuracy. The fixed glass surface has piano key position markings to realistically simulate the position of piano keys and standardize the human hand's touch motion. The upper platform glass surface is detachable and serves only a supporting function, suitable for measuring staccato movements.
[0052] The lower platform 7 is a single piece of acrylic sheet, providing a flat surface for the Leap Motion sensor 6 and reducing the impact of uneven desktop on the Leap Motion sensor during measurement.
[0053] like Figure 5 As shown, the key limiting structure 5 includes a built-in corner groove right-angle connector, a set screw, a screw with holes, and a steel wire rope. The steel wire rope is fixed to the built-in corner groove right-angle connector by the screw with holes. The set screw is used to fix the built-in corner groove right-angle connector to the upper aluminum profile of the measuring platform frame, so that the steel wire rope is in the corner groove position of the upper aluminum profile, making the distance between the upper platform glass surface 3 and the steel wire rope 9.5mm, which meets the standard key depth requirements of piano keys. The front and rear positions of the steel wire rope can be adjusted arbitrarily by the set screw to meet the needs of different people's piano playing habits.
[0054] One Leap Motion sensor is placed on the top and one on the bottom of the platform to detect hand joint information and record motion data of relevant joint feature points of the hand and wrist during the wrist's action of playing the piano.
[0055] Specifically, the wrist touch measurement method based on the above measurement platform is as follows:
[0056] Methods for measuring wrist playing skills, such as Figure 6 , Figure 7 As shown, hand movement measurement includes the following steps:
[0057] (1) The Leap Motion sensor 6 is placed on the upper part of the lower platform and the measuring platform frame respectively. The Leap Motion sensor is about 20-30cm away from the glass surface of the upper platform. The lower sensor is placed directly below the palm of the hand, and the upper sensor is at a tilt angle of 10-30°. The measurement effect is best when the distance from the middle finger metacarpophalangeal joint point to the middle finger tip point on the horizontal plane is 5-10cm.
[0058] (2) The hand should maintain the standard playing motion and be placed on the platform. Depending on the action being measured, there are two measurement methods: measurement of the thumb and measurement of the striking motion of the other fingers. The hand position is as follows: Figure 6 , Figure 7 As shown.
[0059] (3) The human hand should be as close as possible to the lower Leap Motion sensor in the horizontal direction to ensure that the lower Leap Motion sensor can directly identify the MCP joint of the middle finger, and to minimize the impact of light emitted by the Leap Motion sensor on motion measurement caused by refraction when passing through the glass of the measurement platform.
[0060] (4) Description of staccato movement: The human hand is like Figure 6 As shown in the diagram, the middle finger rests on the steel cable, while the other fingers rest on the fixed glass surface and the upper platform glass surface, mimicking the staccato motion of the middle finger striking the keys. The middle finger and wrist exert force, causing the hand to jump up and then quickly fall back down, with the middle finger landing on the steel cable, simulating the staccato motion of the middle finger striking the keys in real piano playing.
[0061] (5) Description of cross-key movement: The human hand is like Figure 7 As shown in the diagram, the four fingers excluding the thumb are placed on a fixed glass surface, corresponding to the key markings on the glass. The thumb performs a straddle motion, crossing one, two, and three key markings respectively, corresponding to crossing one, two, and three white keys in real piano playing. The other fingers remain in position as much as possible, simulating the straddle motion in real piano playing. During the straddle motion, the thumb first lifts to the corresponding key marking, the wrist moves in the direction of movement to assist the straddle, pauses briefly, and then lowers the finger onto the wire, pausing for the appropriate time according to normal piano playing habits, and repeats the straddle motion in the above manner.
[0062] (6) Collect the motions of multiple sets of staccato and cross-key movements, taking 20 staccato notes as a set of staccato movements and 20 cross-key movements as a set of staccato movements.
[0063] This embodiment also provides a method for fusing piano playing touch motion data based on Leap Motion, as follows:
[0064] As an optical sensor, Leap Motion's measurement reliability is significantly reduced when the finger is obstructed, the light intensity is too strong or too weak, or reflections and refractions occur. In general, the problem stems from the Leap Motion's measurement angle causing partial obstruction of the finger joints and data distortion due to reflections and refractions. To address this issue, adding another Leap Motion sensor to measure from a different location expands the measurement range and improves data reliability.
[0065] 1. Synchronization of positioning points for upper and lower Leap Motion measurement data
[0066] Before data fusion, the sensor measurement data needs to be synchronized, which consists of two parts: spatial synchronization and temporal synchronization. Spatial synchronization unifies the data from the upper and lower sensors into the same coordinate system for easier subsequent processing. With the poses of the upper and lower sensors unchanged, the hand is placed in four different positions, and the hand position information at each position is measured. Based on the position data of the metacarpophalangeal joint (MCP) of the middle finger, the coordinate transformation matrix R from the upper sensor coordinate system to the lower sensor coordinate system is calculated. The coordinate transformation matrix includes 3D angle transformation and displacement components, therefore R is a 4th order matrix, as shown in the following equation:
[0067] (1)
[0068] in It is the position data of the metacarpophalangeal joint (MCP) of the middle finger measured by the sensor above. This is the position data of the metacarpophalangeal joint (MCP) of the middle finger measured by the sensor below.
[0069] The data on the metacarpophalangeal joint (MCP) position of the middle finger are processed to filter out data with excessive deviations. The average value of the data is filtered, and the coordinate transformation equations for four different hand positions are solved to obtain the coordinate transformation matrix, thus achieving spatial synchronization.
[0070] Time synchronization involves locating the measurement points of two Leap Motion sensors at the same moment. By identifying patterns in the data movement, the start and end points of each cycle are determined. Based on the movement patterns of playing piano keys, the human hand's piano playing motion can be summarized as a process of striking the key—pausing—returning to its original position. The middle finger's staccato touch is characterized by a staccato touch—pressing the key and pausing—the hand jumping up, while the thumb's straddle motion is characterized by the thumb straddling the key—pressing the key and pausing—the thumb lifting and returning to its original position. By capturing these patterns, the start and end points of the same cycle are synchronized.
[0071] Furthermore, Leap Motion suffers from frame rate instability, resulting in different data measurements from the upper and lower sensors within the same cycle. Therefore, synchronization of similar time points is necessary. Within a single motion cycle, the measured angle exhibits a roughly sawtooth-like pattern, such as... Figure 8 As shown, the start and end points of each cycle, i.e., the peak and trough, are synchronized. The data volume of the same cycle from the two sensors above and below is compared, and data with similar time intervals are fused.
[0072] The fusion method employs a weighted fusion approach, using the length of the finger joints as the fusion standard. While the length of a human finger joint is a fixed value, differences in measurement between the upper and lower sensors result in discrepancies between the measured and actual joint lengths. A weighted approach is used to derive weights by comparing the measured and actual joint lengths, thus obtaining the fused data.
[0073] In conclusion, correct movements and hand positions are indispensable aspects of piano playing. Beginners often encounter difficulties in these two areas, resulting in stiff movements and unsatisfactory tone. While the fingers, as the primary sound-producing organ in piano playing, play a crucial role, the wrist's role is equally important. In summary, the wrist supports the fingers, transmits power, and assists in movement during piano playing, making it relatively difficult for beginners to master. Therefore, analyzing wrist-related piano playing movements and techniques to derive standard movement patterns is of great significance for piano education.
[0074] Leap Motion, as an optical sensor for measuring hand joints, offers advantages such as high cost-effectiveness and high measurement accuracy. However, when measuring piano playing movements on a platform with a single sensor, issues such as obstruction and refraction introduce certain errors into the measurement data. Therefore, multiple sensors are used to measure hand movements from different positions to avoid the shortcomings of a single sensor. Thus, combining piano playing habits and the need for dual-sensor measurement, a wrist piano playing technique measurement platform with dual Leap Motion sensors was built. This measurement platform can measure staccato and thumb crossing movements, and by setting up a sensor position adjustment device, the position of the upper sensor can be adjusted to find the optimal measurement position based on actual conditions. Placing the hand in a suitable position ensures that Leap Motion can directly measure each joint of the fingers being measured, as well as the metacarpophalangeal joint (MCP) of the middle finger, reducing data errors caused by light refraction in the Leap Motion sensor.
[0075] Furthermore, synchronizing and fusing data from the two sensors yields more accurate data. Based on data from the metacarpophalangeal joints (MCPs) of the upper and lower sensors, a transformation matrix from the upper sensor coordinate system to the lower sensor coordinate system is calculated. According to the characteristics of each piano-playing touch, the start and end points of each set of movements are identified, and each set of movements is synchronized, fusing data from similar time points. Using finger joint length as the fusion standard, the measurement data from the upper and lower sensors are fused, effectively improving the reliability of the data.
[0076] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate and explain the technical solutions of this invention. The specific embodiments described above are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.
Claims
1. A method for analyzing wrist-based piano-playing motion data based on Leap Motion, characterized in that, include: Synchronizing sensor measurement data is divided into two parts: spatial synchronization and temporal synchronization. Spatial synchronization unifies the data from two upper and lower Leap Motion sensors into the same coordinate system. Temporal synchronization locates the measurement points of the two upper and lower Leap Motion sensors at the same moment. By finding the patterns in the data movement process, the start and end points of each cycle are determined. A weighted fusion method is employed, using the length of the finger joint as the fusion standard. The finger joint lengths measured by the upper and lower Leap Motion sensors are calculated. If the difference between the length measured by one sensor and the actual finger length exceeds 1mm, the measurement value at that moment is considered unreliable, and the result from the other sensor is selected as the final result. If both sensors exceed the actual value by 1mm, both are considered unreliable, and the data at the corresponding time point is skipped. When the difference between the measured length and the actual length is within 1mm, a weighted method is used, with the ratio of the measured finger joint length to the actual length as the weight, and the fused data is obtained after weighting. Based on the movement patterns of playing the piano, the human hand's piano playing touch action can be summarized as a process of touching the key, pausing, and returning to the starting position; the middle finger's staccato touch action is manifested as a staccato touch, pressing the key and pausing, and the hand jumping up; the thumb's straddling action is manifested as a thumb straddling the key, pressing the key and pausing, and the thumb lifting and returning to the starting position; by grasping the regularity points, the start and end points of the same cycle are synchronized. In one motion cycle, the measured angle changes in a sawtooth pattern. The start and end points of each cycle, i.e., the peaks and troughs, are synchronized to obtain the data a and b of the same cycle from the two sensors, where a > b. Assuming the motion cycle time is 1, the data corresponding to b is used as the reference, and the data with a time difference less than 1 / 3b are fused. The data with the smallest difference is selected for fusion. After completing the subsequent fusion process, the redundant data in the data group a is compared with the actual finger joint length, and the data with a difference less than 0.5mm is inserted into the corresponding time point.
2. The method for analyzing wrist-based piano playing motion data based on Leap Motion according to claim 1, characterized in that, Spatial synchronization is as follows: With the poses of the upper and lower Leap Motion sensors unchanged, the hand is placed in four different positions, and the hand position information at each position is measured. Based on the MCP position data of the middle finger metacarpophalangeal joint, the coordinate transformation matrix R from the upper Leap Motion sensor coordinate system to the lower Leap Motion sensor coordinate system is calculated. The coordinate transformation matrix includes 3D angle transformation and displacement components, therefore R is a 4th order matrix, as shown in the following equation: (1)； in This is the MCP position data of the middle finger metacarpophalangeal joint measured by the sensor above. The data is the position data of the metacarpophalangeal joint (MCP) of the middle finger measured by the sensor below; The data on the position of the metacarpophalangeal joint (MCP) of the middle finger were processed. The average value of every ten data sets was calculated, and data exceeding the average value by 1 mm were filtered out. The average value was filtered, and the coordinate transformation equations of four different hand positions were solved to obtain the coordinate transformation matrix, thus completing spatial synchronization.
3. A motion measurement platform for implementing the wrist-playing piano touch motion data analysis method according to any one of claims 1-2, characterized in that, It includes a sensor position adjustment device, a measurement platform frame, an upper platform glass surface, a fixed glass surface, a lower platform, a touch key limiting structure, and a Leap Motion sensor; the measurement platform frame is a cuboid three-dimensional frame structure made of aluminum profiles, and a support frame is provided in the middle of the measurement platform frame at a distance of 20~30cm from the top and bottom of the measurement platform frame. The sensor position adjustment device consists of a corner bracket, a connecting rod, and a movable hinge. The connecting rod is fixed to the top of the measurement platform frame by the corner bracket, and a movable hinge is provided on the connecting rod. A LeapMotion sensor is installed on the movable hinge. The support frame has a fixed glass surface and an upper platform glass surface, and the fixed glass surface is provided with piano key position marks; the touch key limiting structure is provided between the fixed glass surface and the upper platform glass surface; The bottom of the measurement platform frame is provided with a lower platform made of acrylic plate; a LeapMotion sensor is installed on the lower platform.
4. The motion measurement platform according to claim 3, characterized in that, The touch limit structure consists of a right-angle connector, a screw with holes, a set screw, and a steel wire rope. The steel wire rope is fixed to the right-angle connector by the screw with holes, and the right-angle connector is fixed to the measuring platform frame by the set screw. The right-angle connector is provided with a built-in corner groove.
5. The motion measurement platform according to claim 4, characterized in that, The distance between the glass surface of the upper platform and the steel wire rope is 9.5mm, which meets the standard key depth requirements for piano keys.
6. The motion measurement platform according to claim 3, characterized in that, The thickness of the glass surface of the upper platform is 2mm.
7. The motion measurement platform according to claim 3, characterized in that, The measuring platform frame and support frame are both made of 15×15mm aluminum profiles.
8. A method for measuring wrist-based piano-playing touch motion based on Leap Motion, using the motion measurement platform described in any one of claims 3-7, characterized in that... include: (1) Place the Leap Motion sensor located on the lower platform directly below the palm of the hand, and set the Leap Motion sensor located above to a tilt angle of 10-30°. On the horizontal plane, the distance from the middle finger metacarpophalangeal joint along the direction of the fingers is 5-10cm to ensure that each joint of the fingers can be measured. (2) The hands should maintain the standard movements required when playing the piano, and place them on the upper platform glass surface and the fixed glass surface; (3) Simulate staccato action; place the middle finger on the steel wire rope inside the key-limiting structure, and place the other fingers on the glass planes on both sides to perform the staccato action of the middle finger touching the key; (4) Description of the simulated key crossing action: Place the four fingers other than the thumb on the fixed glass surface, corresponding to the piano key marks on the fixed glass surface respectively; the thumb performs the key crossing action, crossing one, two, and three piano key marks respectively, which corresponds to crossing one, two, and three white keys in the actual piano playing action; the other fingers keep their positions unchanged during the thumb's key crossing process, simulating the key crossing action in the actual piano playing process; during the key crossing process, the thumb first lifts the finger to the corresponding position of the piano key mark, the wrist moves in the direction of movement to assist the key crossing, pauses and then drops the finger onto the steel wire, stays for the corresponding time according to normal piano playing habits, and repeats the key crossing action in the above manner; (5) Collect the motions of several sets of staccato and cross-key movements, taking 20 staccato notes as a set of staccato movements and 20 cross-key movements as a set of staccato movements.