A multimodal interactive piano teaching and training system

Abstract

The present application provides a multimodal interactive piano teaching and training system, which relates to the technical field of piano teaching and training. The system performs collaborative fitting on the reciprocating motion trajectory to obtain the collaborative contraction rules of the player's fingers on the piano keyboard. The fingering control labels during fingering rhythm synchronization are determined according to the collaborative contraction rules, and then the fingering control labels are used to guide the fingering decision attributes during playing by tactile paths; the rhythm synchronization data and dynamic playing information are cross-compared to obtain the rhythm deviation identifier during music style conversion, and then the beat timing spacing during playing is auditory perceptually regulated by the rhythm compensation constraint; the fingering decision attributes after tactile path guidance and the beat timing spacing after auditory perceptual regulation are interactively fused to generate a playing manifold of an unseen piece of music and project it onto the key surface in real time. The present application can form real-time guidance for piano teaching in multimodal interactive fusion teaching to improve the player's training adaptability.

Description

Technical Field

[0001] The present application relates to the technical field of piano teaching and training, and more specifically, to a multimodal interactive piano teaching and training system. Background Art

[0002] Piano teaching and training refers to the process of using a systematic method to guide and provide feedback to learners' playing process through multiple sensory channels in order to improve piano playing skills and musical expression ability. With the development of information technology, multimodal interactive piano teaching and training has emerged. This method integrates multiple sensory modes such as vision, hearing and touch. Multimodal interaction not only improves the intuitiveness and effectiveness of training, but also helps learners accurately master fingering and rhythm through tactile feedback and visual cues, promoting the coordinated development of perception and movement.

[0003] However, existing multimodal interactive piano teaching and training mostly focuses on the static analysis of single-modal data, lacks the fine capture and collaborative modeling of the dynamic trajectory of the player's fingers and rhythmic behavior, making it impossible to effectively identify the linkage relationship and rhythmic synchronization characteristics between complex fingerings. As a result, when faced with music with frequent rhythm changes or complex style switching, the training feedback lacks pertinence and real-time adaptability, thereby limiting the learner's ability to understand and adapt to unseen repertoires. Therefore, how to form real-time guidance for piano teaching in the integrated teaching of multimodal interaction to improve the player's training adaptability is a difficult problem faced by the industry. Summary of the Invention

[0004] The present application provides a multimodal interactive piano teaching and training system, which can form real-time guidance of piano teaching in the fusion teaching of multimodal interaction to improve the training adaptability of the player.

[0005] In a first aspect, the present application provides a multimodal interactive piano teaching and training system, the teaching and training system comprising:

[0006] The information acquisition module is used to capture the reciprocating motion trajectory of the piano player's fingers when playing in real time, and collect the dynamic playing information of the player during the playing;

[0007] a path guidance module for collaboratively fitting the reciprocating motion trajectory to obtain a collaborative contraction rule of the player's fingers on the piano keyboard, determining a fingering control label for fingering rhythm synchronization based on the collaborative contraction rule, and then using the fingering control label to perform tactile path guidance on the fingering decision attributes during playing;

[0008] a perception and control module, configured to obtain rhythm synchronization data during playing, cross-compare the rhythm synchronization data with the dynamic playing information, obtain a rhythm deviation identifier during music style conversion, determine a rhythm compensation constraint during playing based on the rhythm deviation identifier, and then perform auditory perception control on the beat timing interval during playing based on the rhythm compensation constraint;

[0009] The interactive teaching module is used to interactively integrate the fingering decision attributes guided by the tactile path and the beat timing spacing regulated by auditory perception, generate the playing manifold of unseen music and project it onto the keyboard surface in real time.

[0010] In this embodiment, the reciprocating motion trajectory represents the three-dimensional path of the fingers in the process of pressing and lifting keys during piano playing.

[0011] In this embodiment, the dynamic playing information refers to multi-dimensional data related to keystrokes in piano playing.

[0012] In this embodiment, the coordinated contraction rules of the player's fingers on the piano keyboard are obtained by performing collaborative fitting on the reciprocating motion trajectory, specifically including:

[0013] Determining the finger position timing information of the player according to the reciprocating motion trajectory;

[0014] Performing collaborative analysis on the finger position timing information to obtain dynamic collaborative features during finger movement;

[0015] The coordinated contraction rules of the player's fingers on the piano keyboard are determined according to the dynamic coordinated features.

[0016] In this embodiment, the fingering control tag performs tactile path guidance on the fingering decision attributes during playing, specifically including:

[0017] determining a tactile response strategy for tactile feedback according to the fingering control label;

[0018] determining a tactile intensity distribution of a finger movement path according to the tactile response strategy;

[0019] Determine the fingering decision attributes when playing;

[0020] The tactile feedback unit is driven by the tactile intensity distribution to output a path guidance signal synchronized with the rhythm to the tactile perception layer for information collection on the piano keyboard in real time, and to provide prompts for the fingering decision attributes.

[0021] In this embodiment, the rhythm synchronization data represents a set of information describing the temporal correspondence between actual keystroke events and target rhythm references during piano playing.

[0022] In this embodiment, cross-comparing the rhythm synchronization data with the dynamic playing information to obtain a rhythm deviation identifier during music style conversion specifically includes:

[0023] extracting a beat interval feature value from the rhythm synchronization data;

[0024] determining, according to the dynamic playing information, a linked rhythm feature of playing when converting the music style;

[0025] Determine the dynamic deviation index of playing when changing music styles;

[0026] The linked rhythm feature is mapped to the dynamic deviation index to obtain a rhythm deviation identifier during music style conversion.

[0027] In this embodiment, the rhythm deviation indicator refers to the deviation of the rhythm execution during style conversion.

[0028] In this embodiment, determining the rhythm compensation constraint during playing by using the rhythm deviation identifier specifically includes:

[0029] determining a rhythm feedback feature when the rhythm changes during playing according to the rhythm deviation identifier;

[0030] Determining the interactive compensation rules between fingering and rhythm during playing by using the rhythm feedback features;

[0031] The rhythm compensation constraint during playing is determined by the interactive compensation rule.

[0032] In this embodiment, the auditory perception control of the beat timing intervals during playing by the rhythm compensation constraint specifically includes:

[0033] An auditory perception control strategy for generating a beat timing sequence based on the rhythm compensation constraint;

[0034] Determine the beat correction rules during dynamic beat stretching while playing;

[0035] The auditory perception control strategy is compensated according to the beat correction rule, and a synchronously compensated beat pulse signal is output and the auditory perception consistency is verified.

[0036] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects:

[0037] The invention captures the reciprocating motion trajectory of the pianist's fingers when playing in real time, and collects the dynamic playing information of the pianist when playing; performs collaborative fitting on the reciprocating motion trajectory to obtain the collaborative contraction rule of the player's fingers on the piano keyboard, determines the fingering control label during fingering rhythm synchronization according to the collaborative contraction rule, and then uses the fingering control label to perform tactile path guidance on the fingering decision-making attributes during playing; obtains rhythm synchronization data during playing, cross-compares the rhythm synchronization data with the dynamic playing information, obtains the rhythm deviation identifier during music style conversion, determines the rhythm compensation constraint during playing through the rhythm deviation identifier, and then uses the rhythm compensation constraint to perform auditory perception regulation on the beat timing spacing during playing; interactively fuses the fingering decision-making attributes after tactile path guidance and the beat timing spacing after auditory perception regulation to generate a playing manifold of an unseen piece of music and projects it onto the key surface in real time.

[0038] It can be seen that in this application, the teaching and training of the player can be linked and adapted based on the fusion of multi-source dynamic data; among them, by capturing the finger movement trajectory and collecting dynamic playing information, the real movement pattern and force change of the player in the actual performance process can be obtained with high precision and low latency, and a comprehensive dynamic perception of the individual performance behavior can be achieved; by fitting the coordinated contraction rules of the fingers on the keyboard, the multi-finger linkage mechanism can be effectively explored, and then a fingering control label with rhythm synchronization capability can be formed, so that the teaching system can achieve fingering guidance that is more in line with the actual performance rhythm; by imposing rhythm compensation constraints, the timing deviation in the rhythm execution can be accurately identified during the conversion of music style, and a rhythm compensation mechanism adapted to different music styles can be automatically formed, thereby ensuring the rhythm continuity and style consistency during the performance process; by integrating tactile and auditory feedback, the playing manifold of unseen repertoire is generated and projected onto the surface of the keys in real time, which not only improves the player's learning intuitiveness and rhythm control ability, but also significantly enhances the rapid understanding and performance adaptability of new music.

[0039] In summary, the technical solution adopted in this application can form real-time guidance for piano teaching in the fusion teaching of multimodal interaction, so as to improve the player's training adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1This is a module structure diagram of the multimodal interactive piano teaching and training system provided by this application;

[0042] Figure 2 is a schematic diagram of a process for determining a fingering control tag according to the present application;

[0043] Figure 3 This is a flow chart of determining a rhythm deviation indicator according to the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0045] The embodiment of the present application provides a multimodal interactive piano teaching and training system, the core of which is to capture the reciprocating motion trajectory of a pianist's fingers when playing in real time, and collect the dynamic playing information of the player when playing; perform collaborative fitting on the reciprocating motion trajectory to obtain the collaborative contraction rule of the player's fingers on the piano keyboard, determine the fingering control label during fingering rhythm synchronization according to the collaborative contraction rule, and then use the fingering control label to perform tactile path guidance on the fingering decision attributes during playing; obtain rhythm synchronization data during playing, cross-compare the rhythm synchronization data with the dynamic playing information to obtain a rhythm deviation identifier during music style conversion, determine the rhythm compensation constraint during playing through the rhythm deviation identifier, and then use the rhythm compensation constraint to perform auditory perception regulation on the beat timing spacing during playing; interactively fuse the fingering decision attributes after tactile path guidance and the beat timing spacing after auditory perception regulation to generate a playing manifold of an unseen piece of music and project it onto the key surface in real time. The above scheme can form real-time guidance of piano teaching in the fusion teaching of multimodal interaction, so as to improve the player's training adaptability.

[0046] In order to better understand the above technical solution, the following will be described in detail with reference to the accompanying drawings and specific implementation methods. Figure 1 As shown in FIG, this figure is a module structure diagram of a multimodal interactive piano teaching and training system according to this embodiment of the present application. The teaching and training system includes: an information acquisition module 100, a path guidance module 200, a perception and control module 300, and an interactive teaching module 400, which are described as follows:

[0047] The information acquisition module 100 is used to capture the reciprocating motion trajectory of the piano player's fingers when playing in real time, and to collect the dynamic playing information of the piano player when playing.

[0048] In this embodiment, real-time capture of the reciprocating motion trajectory of a pianist's fingers while playing can be achieved in the following manner: the player wears data gloves with integrated high-precision inertial measurement units on both hands, and each finger is equipped with a three-degree-of-freedom micro accelerometer and gyroscope to obtain the acceleration and angular velocity changes of the fingertips in space. At the same time, a set of inertial reference modules is set on the back of the hand or wrist to provide a coordinate reference and correct the overall motion offset. All sensor data is synchronously transmitted to the central processing system via a low-latency Bluetooth or Wi-Fi module, and the original trajectory data is smoothed and reconstructed by posture fusion, so as to restore the reciprocating motion trajectory formed by the fingers between pressing the keys and lifting them in real time.

[0049] It should be noted that, in this application, the reciprocating motion trajectory represents the three-dimensional path of the fingers in the process of pressing keys and lifting fingers during piano playing.

[0050] In addition, in specific implementation, the following method can be used to collect the dynamic playing information of the player during playing: a high-sensitivity piezoelectric pressure sensor and photoelectric switch module can be placed under each key to record key parameters such as the initial trigger time, continuous pressing time, keystroke pressure, and release speed when the key is pressed. At the same time, a displacement encoder is introduced into the key travel path to accurately capture the velocity curve and keystroke depth during the keystroke process. Then, a pickup microphone array is installed inside the piano body. Combined with the amplitude envelope and attack transient characteristics of the audio signal, multi-modal auxiliary verification of the keystroke intensity and expressed emotion is performed. Finally, all data is transmitted in real time and in parallel to the central processing unit through a multi-channel data acquisition card. The central processing unit obtains the player's dynamic playing information during playing by reading the data.

[0051] It should be noted that, in this application, dynamic playing information refers to multidimensional data related to keystroke actions in piano playing.

[0052] The path guidance module 200 is used to perform collaborative fitting on the reciprocating motion trajectory to obtain the collaborative contraction rules of the player's fingers on the piano keyboard, determine the fingering control labels during fingering rhythm synchronization based on the collaborative contraction rules, and then use the fingering control labels to perform tactile path guidance on the fingering decision attributes during playing.

[0053] In this embodiment, the reciprocating motion trajectory is collaboratively fitted to obtain the collaborative contraction rules of the player's fingers on the piano keyboard in the following manner, namely:

[0054] Determining the finger position timing information of the player according to the reciprocating motion trajectory;

[0055] Performing collaborative analysis on the finger position timing information to obtain dynamic collaborative features during finger movement;

[0056] The coordinated contraction rules of the player's fingers on the piano keyboard are determined according to the dynamic coordinated features.

[0057] In the specific implementation, first, the three-dimensional coordinate data of each finger at different time points captured by the data glove or inertial measurement system are used to construct a finger position sequence with the timestamp as the main axis. Specifically, the acquisition frequency is set to 100 Hz to ensure the continuity and accuracy of the motion trajectory. An index is established for each finger in the central processing unit to form a matrix with the structure of [finger number, timestamp, x-coordinate, y-coordinate, z-coordinate], which is used as the finger position timing information; then, a multi-channel dynamic time warping method is used for the finger position timing information to align the finger trajectories in time, identify finger pairs with similar or complementary motion trends in the same time period, and use the mutual information function calculation method to measure the motion correlation of any two fingers in multiple time periods, identify finger groups with coupled relationships, and use the movements between finger groups with coupled relationships as dynamic collaborative features during finger movement. In addition, to avoid short-term noise interference, the minimum collaborative window is set to 250 milliseconds. Only collaborative relationships that appear repeatedly in multiple consecutive windows are considered valid patterns. Finally, the identified dynamic collaborative features and their corresponding action time windows are used as input, and clustering algorithms such as the DBSCAN density clustering method are used to summarize different types of collaborative action patterns. Subsequently, the Apriori frequent item set mining algorithm is used to extract frequently occurring collaborative combinations and their triggering conditions, such as rhythm type, speed level, etc., and finally a "cooperative contraction rule" is formed. Each cooperative contraction rule contains the fields of "participating finger group", "cooperative start time", "action direction relationship", "cooperative duration" and "structural category".

[0058] It should be noted that, in the present application, the finger position timing information represents a sequence of three-dimensional spatial coordinates of fingers arranged in chronological order; the dynamic collaborative feature represents the joint motion relationship of multiple fingers during the performance process with temporal synchronization, spatial coupling and motion complementarity; the collaborative contraction rule represents the coordinated motion characteristics between multiple fingers in a specific performance segment.

[0059] Preferably, in this embodiment, the fingering control label when the fingering rhythm is synchronized is determined according to the co-contraction rule, referring to Figure 2 As shown in FIG, this figure is a schematic diagram of a process for determining a fingering control label in some embodiments of the present application. In this embodiment, determining a fingering control label can be implemented by the following steps:

[0060] In step S21, the synchronization adaptation amount during fingering rhythm synchronization is extracted from the co-contraction rule;

[0061] In step S22, rhythm matching information of the target music is collected;

[0062] In step S23, determining a fingering adjustment attribute during fingering rhythm synchronization according to the synchronization adaptation amount and the rhythm matching information;

[0063] In step S24, the fingering control tag for fingering rhythm synchronization is determined according to the fingering adjustment attribute.

[0064] In the specific implementation, first, a multi-finger coordination pattern that matches the current target phrase structure is selected from the coordinated contraction rules, and the rhythm window sliding algorithm is used to compare the time point of the coordinated action with the standard beat point, and the multi-index synchronization error is calculated, such as the average key-on delay, the staggered key-touch ratio, and the standard deviation of the inter-finger offset. The calculated multi-index synchronization error is used as the synchronization adaptation amount during fingering rhythm synchronization; then, the input digital music score (such as MusicXML or MIDI format) is subjected to time signature extraction, rhythm section division and speed calibration through the spectrum analysis algorithm, and the beat tracking algorithm, such as the beat detection technology based on wavelet transform, is combined to extract the beat signature of each note. The theoretical beat and rhythmic distribution structure of the notes is determined by constructing a data table structured as [beat position, rhythm type, note density]. This information is used as the rhythmic matching information for the target piece. Next, the synchronization adaptation amount is cross-compared with the rhythmic matching information. If significant timing deviations are found in the original coordinated movements (such as premature touch or fingering conflicts), local fingering rearrangements are performed using a motion control model. A hybrid optimization model based on heuristic rules and genetic algorithms is employed. Subject to constraints such as minimum finger span and fingering sequence continuity, fingering adjustment properties that satisfy the rhythmic structure are output, including fingering substitution, time delay interpolation, and touch sequence reconstruction. Finally, the adjusted fingerings are mapped into a structured label, including fields such as fingering sequence number, movement type (keystroke, release, hover), trigger delay, and target key number. This structured label serves as a fingering control label, which is embedded as a node on the performance model timeline. A graph structure is then used to construct a fingering control chain for the entire piece of music.

[0065] It should be noted that, in this application, the synchronization adaptation amount refers to the parameter that describes the degree of matching between the multi-finger coordinated action and the target rhythm; the rhythm matching information refers to the correlation regularity information between the rhythm changes and the playing actions of the target music during the playing process; the fingering adjustment attribute represents the behavioral characteristics of the fingers when adjusting in response to rhythm changes; the fingering control label refers to the structural label of the specific fingering action combination used on the beat point.

[0066] In this embodiment, the fingering control tag can be used to guide the fingering decision attributes during playing by performing tactile path guidance, specifically by the following steps:

[0067] determining a tactile response strategy for tactile feedback according to the fingering control label;

[0068] determining a tactile intensity distribution of a finger movement path according to the tactile response strategy;

[0069] Determine the fingering decision attributes when playing;

[0070] The tactile feedback unit is driven by the tactile intensity distribution to output a path guidance signal synchronized with the rhythm to the tactile perception layer for information collection on the piano keyboard in real time, and to provide prompts for the fingering decision attributes.

[0071] In specific implementation, the system first searches the policy library for matching tactile response rules based on parameters such as the finger number, target key position, and touch time window in the fingering control tag. For example, if a note is labeled "right hand second finger, keystroke after 0.3 seconds," the system generates a tactile response policy: "Apply a short-period vibration with an amplitude of 1.5mN and a frequency of 200Hz at the target key position, delaying 0.2 seconds from the current time, for 100ms." This tactile response policy forms a control instruction queue to drive subsequent tactile output. It then compares the execution priority and rhythmic urgency of different finger positions in the fingering path, and generates a tactile intensity distribution map based on the corresponding tactile response policy. The tactile response policy uses a Gaussian distribution or exponential decay model to ensure that key positions on the fingering path have higher stimulation intensity, while auxiliary paths have weaker stimulation, forming a gradient tactile guidance, which is used as the tactile intensity distribution. For example, a central stimulus of 2.0 mN is set at the primary keystroke point, decreasing to 0.3 mN within a 0.5 cm radius, simulating a clear "landing point." Next, after receiving feedback signals and generating tactile cues, wearable sensors (such as capacitive fingertips or inertial measurement modules) record the user's actual finger motion data and compare it with the ideal path. Based on trajectory deviation, movement time difference, and target misalignment, the player's fingering decision-making behavior at that node is extracted, such as "0.12 second delay," "path deviation 2 mm to the left," or "finger sequence swap 3 → 2." This fingering decision-making behavior is used as a fingering decision attribute. Finally, the tactile intensity distribution is converted into a PWM (pulse width modulation) signal, which is transmitted to the tactile sensing layer via a vibration motor array controller integrated into the key surface or wearable feedback band. This tactile sensing layer, composed of thin-film piezoelectric vibrators or micro-electric heating elements, outputs rhythmically synchronized local vibration or slight heating signals when the user touches or approaches a key, indicating path node events such as "prepare for keystroke," "prepare for finger rotation," or "finger approaching the path."

[0072] It should be noted that, in the present application, the tactile response strategy represents the correlation between the finger movement path and the physical tactile feedback; the tactile intensity distribution represents a configuration diagram that provides differentiated tactile guidance stimulation for different fingering decision points within the playing area to assist users in making correct finger movement movements; the finger movement decision attributes refer to the dynamic movement parameters of the fingers in path planning, key touch sequence, speed control, etc. during playing; the path guidance signal refers to the tactile output control signal used to prompt the finger movement direction and rhythm beat; the tactile perception layer refers to a perceptible micro-tactile output layer arranged on the surface of the piano keyboard, whose function is to receive system control signals and transmit path information to the user's fingers through local vibration, temperature difference or micro-pressure deformation.

[0073] The perception and control module 300 is used to obtain the rhythm synchronization data during playing, cross-compare the rhythm synchronization data with the dynamic playing information, obtain the rhythm deviation identifier during music style conversion, determine the rhythm compensation constraint during playing through the rhythm deviation identifier, and then use the rhythm compensation constraint to perform auditory perception control on the beat timing spacing during playing.

[0074] In this embodiment, acquiring rhythmic synchronization data during playing can be achieved in the following manner: first, a photoelectric sensor array or a built-in pressure sensor on the keys accurately records the start and release time of each keystroke, generating a sequence of actual keystroke times. Second, the rhythmic model of the target piece (including beat position, note length, syncopated rhythm, etc.) is converted into a reference rhythm track and time-synchronized. Subsequently, the actual keystroke sequence is mapped to the rhythm track, and the "expected time" and "actual trigger time" of each corresponding note are compared to extract their time difference, rhythmic drift direction, and continuity changes. This constructs rhythmic synchronization data that includes beat error, timing offset, and synchronization accuracy.

[0075] It should be noted that, in the present application, rhythm synchronization data refers to a set of information describing the timing correspondence between actual keystroke events and target rhythm references during piano playing.

[0076] Preferably, in this embodiment, the rhythm synchronization data and the dynamic playing information are cross-compared to obtain the rhythm deviation mark when the music style is converted, and the reference Figure 3 As shown in FIG. 1 , this figure is a schematic diagram of a process for determining a rhythm deviation flag in some embodiments of the present application. In this embodiment, determining the rhythm deviation flag can be implemented using the following steps:

[0077] In step S31, a beat interval feature value is extracted from the rhythm synchronization data;

[0078] In step S32, determining the linked rhythm features of the music played during the music style conversion according to the dynamic playing information;

[0079] In step S33, determining a dynamic deviation index of the music played during the music style conversion;

[0080] In step S34, the linked rhythm feature is mapped to the dynamic deviation index to obtain a rhythm deviation identifier during music style conversion.

[0081] In its implementation, the system first extracts the timestamp sequence of consecutive keystrokes from the rhythm synchronization data, calculates the time intervals between adjacent keystrokes, and applies statistical analysis methods (such as mean and standard deviation calculation within a sliding window) to extract the characteristic values ​​of the beat intervals. Then, based on dynamic playing information, a multi-sensor fusion algorithm (integrating pressure sensor, motion capture, and accelerometer data) is employed. Machine learning methods (such as support vector machines or convolutional neural networks) are used to identify and extract the dynamic rhythm changes, finger movement frequency, and rhythm swing patterns corresponding to different musical styles, thereby forming style-specific rhythm linkage features. Next, by comparing the ideal rhythm behavior of the target style with the rhythm synchronization data and dynamic playing characteristics of the actual playing, the timing deviation, dynamic error, and motion trajectory differences are calculated. A multi-dimensional deviation index is established, including the root mean square of the beat error, the dynamic error percentage, and the trajectory offset distance. This multi-dimensional deviation index serves as an indicator of the dynamic deviation of playing during musical style transitions. Finally, a mapping model (such as multivariate linear regression or deep learning model) is used to take the linkage rhythm features as input and the dynamic deviation index as output to train a generative model for the rhythm deviation identification during style conversion. The rhythm deviation identification represents the type, degree and time distribution of deviations in rhythm execution in the form of structured data, which is used to guide subsequent rhythmic adjustment.

[0082] It should be noted that in this application, the beat interval feature value refers to the statistic of the time interval between adjacent keystrokes when playing the piano; the linkage rhythm feature refers to the rhythm behavior indicator that reflects the characteristics of the musical style; the dynamic deviation index refers to the degree of difference in rhythm and dynamic performance during the style conversion process; the rhythm deviation indicator refers to the offset in rhythm execution during style conversion.

[0083] In this embodiment, the rhythm compensation constraint during playing can be determined by the rhythm deviation identifier in the following manner, namely:

[0084] determining a rhythm feedback feature when the rhythm changes during playing according to the rhythm deviation identifier;

[0085] Determining the interactive compensation rules between fingering and rhythm during playing by using the rhythm feedback features;

[0086] The rhythm compensation constraint during playing is determined by the interactive compensation rule.

[0087] In its implementation, the system first uses rhythm deviation identifiers as input and utilizes a pre-defined rhythmic feedback feature library to identify the type of rhythmic variation (e.g., acceleration, deceleration, pause, etc.) and its intensity level through fuzzy matching or rule-based reasoning. A machine learning model (e.g., recurrent neural network) is then used to analyze the time series characteristics of rhythmic deviations and extract corresponding feedback patterns, including rhythmic fluctuation amplitude, frequency, and duration, to form quantifiable rhythmic feedback features. Then, based on these rhythmic feedback features and combined with a piano fingering kinematic model and a rhythmic dynamics model, a mapping relationship between fingering movement adjustments and rhythmic feedback is established. Specifically, a parameterized motion compensation model is constructed to convert rhythmic fluctuation information into adjustment instructions for fingering velocity, velocity, and timing. These adjustment instructions serve as the interactive compensation rules between fingering and rhythm during playing. Finally, the interactive compensation rules are formulated into a set of rhythmic compensation constraint parameters, including the fingering adjustment range, time window limit, and velocity change threshold. Combined with real-time performance data, a constrained optimization algorithm (e.g., gradient descent-based real-time optimization) is used to dynamically adjust the compensation strategy and output the rhythmic compensation constraints during playing, ensuring that the compensation behavior meets the requirements of musical expression without affecting the naturalness of the performance.

[0088] It should be noted that in this application, rhythmic feedback characteristics refer to the feedback properties and force patterns that describe the rhythmic fluctuations during playing; interactive compensation rules represent the rule body that represents the interactive relationship between fingering movements and rhythmic changes; and rhythmic compensation constraints refer to the rhythm correction behaviors used to standardize and regulate the playing process.

[0089] In this embodiment, the auditory perception control of the beat timing interval during playing by the rhythm compensation constraint can be specifically implemented in the following manner, namely:

[0090] An auditory perception control strategy for generating a beat timing sequence based on the rhythm compensation constraint;

[0091] Determine the beat correction rules during dynamic beat stretching while playing;

[0092] The auditory perception control strategy is compensated according to the beat correction rule, and a synchronously compensated beat pulse signal is output and the auditory perception consistency is verified.

[0093] In its implementation, a dynamic beat spacing adjustment model is first designed based on rhythm compensation constraints, combined with psychoacoustic theory and musical cognition. This dynamic adjustment model utilizes time series signal processing techniques, such as weighted moving average and dynamic time warping algorithms, to smoothly adjust the beat spacing curve, preventing auditory discomfort caused by sudden changes. By combining the auditory masking effect and rhythm perception threshold, the adjustment amplitude and frequency are optimized, forming an auditory perception control strategy for beat timing. Then, based on the real-time monitoring of the dynamic rhythmic state of the playing, time series analysis and pattern recognition techniques are used to determine the specific requirements for beat stretching. A beat correction rule is then established, including a maximum limit on beat extension or compression, a principle for smooth transitions between consecutive beats, and a response priority strategy for rapid rhythm changes. A rule engine or decision tree-based algorithm can be used to adjust and modify the beat correction rule in real time to ensure rhythmic coherence and expressiveness during beat stretching. Finally, the beat correction rule is integrated into the auditory perception control strategy, using real-time signal processing to adjust the beat pulse timing. The compensated beat pulse signal is output and synchronously fed back through a built-in audio system or haptic feedback device. The adjusted beat is verified in real time using a subjective evaluation model and auditory consistency detection algorithm to ensure that the rhythm adjustment is consistent with the natural perception of the human ear.

[0094] It should be noted that in this application, the auditory perception regulation strategy refers to a method of dynamically adjusting the beat timing spacing so that the adjusted rhythm is more in line with the human ear's rhythm; the beat correction rule represents the adjustment principle defined in the rhythm stretching and compression process; the beat pulse signal refers to the adjusted beat time node signal.

[0095] The interactive teaching module 400 is used to interactively integrate the fingering decision attributes guided by the tactile path and the beat timing spacing regulated by the auditory perception, generate the playing manifold of the unseen piece and project it onto the keyboard surface in real time.

[0096] In this embodiment, the fingering decision attributes guided by the tactile path and the beat timing intervals regulated by the auditory perception are interactively integrated to generate a playing manifold of an unseen piece of music and project it onto the surface of the piano keys in real time. Specifically, the following methods can be used, namely:

[0097] Determining the amount of playing perception feedback according to the fingering decision attributes after the tactile path guidance;

[0098] Determining interactive control properties of the key surface based on the temporal spacing of beats modulated by auditory perception;

[0099] The playing manifold of the unseen music piece is determined by the playing perception feedback amount and the interactive control attribute, and the projection feedback unit is driven to perform real-time projection onto the surface of the piano keys.

[0100] In its implementation, the fingering decision attributes collected under tactile guidance are first quantified, and key movement features are extracted using feature extraction algorithms (such as principal component analysis or convolutional neural networks). Based on movement amplitude, velocity variation, and force distribution, the perceived playing feedback is calculated. This feedback is then mapped to a perceptual feedback value domain using a machine learning model, reflecting the contribution and accuracy of the current fingering movement to the musical performance. The temporal spacing of the beats, regulated by auditory perception, is converted into key surface feedback parameters, including projection brightness, color variation, and tactile feedback intensity. Using temporal signal conversion technology, the time interval information is encoded into dynamic visual and tactile signals, generating interactive control properties that regulate the response pattern of the key surface device. The control properties are then adjusted in conjunction with real-time sensor data to achieve high synchronization between rhythm and interactive feedback. Finally, the perceived playing feedback and interactive control properties are integrated to construct a multidimensional playing manifold model that encompasses spatial motion trajectories and temporal rhythms. Manifold learning algorithms (such as t-SNE or locally linear embedding) are used to perform data dimensionality reduction and structure mapping, generating a visual playing trajectory pattern. This multidimensional playing manifold model is input into the projection feedback unit, which uses high-precision laser or LED projection technology to project the manifold pattern onto the piano key surface in real time. Simultaneously, the haptic feedback module outputs tactile signals synchronously, achieving multimodal interaction between vision and touch.

[0101] It should be noted that in this application, the playing perception feedback amount refers to the perceptual contribution of the current playing action to the musical performance; the interactive control attribute is a parameter indicator used to adjust the visual and tactile feedback on the key surface; and the playing manifold refers to a multimodal model that reflects the playing trajectory and rhythm pattern of unseen music.

[0102] It can be seen that in this application, the teaching and training of the player can be linked and adapted based on the fusion of multi-source dynamic data; among them, by capturing the finger movement trajectory and collecting dynamic playing information, the real movement pattern and force change of the player in the actual performance process can be obtained with high precision and low latency, and a comprehensive dynamic perception of the individual performance behavior can be achieved; by fitting the coordinated contraction rules of the fingers on the keyboard, the multi-finger linkage mechanism can be effectively explored, and then a fingering control label with rhythm synchronization capability can be formed, so that the teaching system can achieve fingering guidance that is more in line with the actual performance rhythm; by imposing rhythm compensation constraints, the timing deviation in the rhythm execution can be accurately identified during the conversion of music style, and a rhythm compensation mechanism adapted to different music styles can be automatically formed, thereby ensuring the rhythm continuity and style consistency during the performance process; by integrating tactile and auditory feedback, the playing manifold of unseen repertoire is generated and projected onto the surface of the keys in real time, which not only improves the player's learning intuitiveness and rhythm control ability, but also significantly enhances the rapid understanding and performance adaptability of new music.

[0103] In summary, the technical solution adopted in this application can form real-time guidance for piano teaching in the fusion teaching of multimodal interaction, so as to improve the player's training adaptability.

[0104] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, magnetic disk storage, or magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0106] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

Claims

1. A multimodal interactive piano teaching and training system, characterized in that: The teaching and training system comprises: The information acquisition module is used to capture the reciprocating motion trajectory of the piano player's fingers when playing in real time, and collect the dynamic playing information of the player during the playing; a path guidance module for collaboratively fitting the reciprocating motion trajectory to obtain a collaborative contraction rule of the player's fingers on the piano keyboard, determining a fingering control label for fingering rhythm synchronization based on the collaborative contraction rule, and then using the fingering control label to perform tactile path guidance on the fingering decision attributes during playing; The coordinated contraction rules of the player's fingers on the piano keyboard are obtained by performing collaborative fitting on the reciprocating motion trajectory, specifically including: Determining the finger position timing information of the player according to the reciprocating motion trajectory; Performing collaborative analysis on the finger position timing information to obtain dynamic collaborative features during finger movement; determining the coordinated contraction rules of the player's fingers on the piano keyboard according to the dynamic coordinated features; The co-contraction rule represents the coordinated action characteristics between multiple fingers in a specific performance segment; a perception and control module, configured to obtain rhythm synchronization data during playing, cross-compare the rhythm synchronization data with the dynamic playing information, obtain a rhythm deviation identifier during music style conversion, determine a rhythm compensation constraint during playing based on the rhythm deviation identifier, and then perform auditory perception control on the beat timing interval during playing based on the rhythm compensation constraint; The interactive teaching module is used to interactively integrate the fingering decision attributes guided by the tactile path and the beat timing spacing regulated by auditory perception, generate the playing manifold of unseen music and project it onto the keyboard surface in real time.

2. A multimodal interactive piano teaching and training system as claimed in claim 1, characterized in that: The reciprocating motion trajectory represents the three-dimensional path of the fingers in the process of pressing and lifting keys during piano playing.

3. A multimodal interactive piano teaching and training system as claimed in claim 1, characterized in that: The dynamic playing information refers to multi-dimensional data related to the keystroke action in piano playing.

4. A multimodal interactive piano teaching and training system as claimed in claim 1, characterized in that: The fingering control tag guides the fingering decision attributes during playing by the tactile path, specifically including: determining a tactile response strategy for tactile feedback according to the fingering control label; determining a tactile intensity distribution of a finger movement path according to the tactile response strategy; Determine the fingering decision attributes when playing; The tactile feedback unit is driven by the tactile intensity distribution to output a path guidance signal synchronized with the rhythm to the tactile perception layer for information collection on the piano keyboard in real time, and to provide prompts for the fingering decision attributes.

5. A multimodal interactive piano teaching and training system as claimed in claim 1, characterized in that: The rhythm synchronization data represents a set of information describing the timing correspondence between actual keystroke events and target rhythm references during piano playing.

6. A multimodal interactive piano teaching and training system as claimed in claim 1, characterized in that: Cross-comparing the rhythm synchronization data with the dynamic playing information to obtain a rhythm deviation identifier during music style conversion specifically includes: extracting a beat interval feature value from the rhythm synchronization data; determining, according to the dynamic playing information, a linked rhythm feature of playing when converting the music style; Determine the dynamic deviation index of playing when changing music styles; The linked rhythm feature is mapped to the dynamic deviation index to obtain a rhythm deviation identifier during music style conversion.

7. A multimodal interactive piano teaching and training system as claimed in claim 1, characterized in that: The rhythm deviation indicator refers to the deviation of the rhythm execution during style conversion.

8. A multimodal interactive piano teaching and training system as claimed in claim 1, characterized in that: Determining the rhythm compensation constraint during playing by using the rhythm deviation identifier specifically includes: determining a rhythm feedback feature when the rhythm changes during playing according to the rhythm deviation identifier; Determining the interactive compensation rules between fingering and rhythm during playing by using the rhythm feedback features; The rhythm compensation constraint during playing is determined by the interactive compensation rule.

9. A multimodal interactive piano teaching and training system as claimed in claim 1, characterized in that: The auditory perception control of the beat timing intervals during playing by the rhythm compensation constraint specifically includes: An auditory perception control strategy for generating a beat timing sequence based on the rhythm compensation constraint; Determine the beat correction rules during dynamic beat stretching while playing; The auditory perception control strategy is compensated according to the beat correction rule, and a synchronously compensated beat pulse signal is output and the auditory perception consistency is verified.

Images