Piano playing fingering intelligent evaluation system based on multi-modal feature fusion

Abstract

This invention relates to the fields of intelligent music education and multimodal perception technology, specifically to an intelligent evaluation system for piano playing fingering based on multimodal feature fusion. The system includes a microphone array audio acquisition module, a depth camera image acquisition module, an acoustic feature extraction module, a motion feature extraction module, a cross-modal time-delay causal joint attention comparison module, and a correction feedback module. The cross-modal time-delay causal joint attention comparison module explicitly models the playing audio as a delayed projection generated by finger movement via a mechanical transmission link. Using the time-delay probability density as the query signal for joint attention, it compares the audio with a standard template under the constraint of time-delay perception dynamic time warping to identify pseudo-correct playing where the note is correct but the finger replacement is incorrect. The correction feedback module uses a forward-looking hand shadow generation branch to superimpose a semi-transparent virtual hand shadow before the error occurs, achieving predictive correction.

Description

Technical Field

[0001] This invention relates to the fields of intelligent music education and multimodal perception technology, specifically to an intelligent evaluation system for piano playing fingering based on multimodal feature fusion. Background Technology

[0002] Piano fingering is one of the core components of piano pedagogy. The same piece of music, when arranged with different fingerings, exhibits significant differences in sound quality, legato, and expressiveness. Muscle memory formed by incorrect fingering often requires several times the training effort to correct later. Traditional piano fingering assessment heavily relies on face-to-face instruction from teachers, a method inherently limited by scarce teaching resources, strong subjectivity in assessment, and lack of supervision during after-class practice. With the development of deep learning and multimodal perception technologies, the industry has begun to explore automated data collection and assessment of student performances using cameras and microphones, aiming to achieve objective evaluation and feedback on fingering without a teacher's presence.

[0003] Chinese invention patent application CN113255470A discloses a multimodal piano practice system and method based on hand posture estimation. The system collects hand gesture information during the performance through a wide-angle camera and piano sound information during the performance through a microphone array. It uses a ResNet residual network and a dual-channel convolutional neural network to identify small joints of the hands. Based on the microphone array, it performs filtering and windowing processing on the piano sound to determine the sound range to assist in judging the hand position during the performance. Then, it compares the small joint recognition results with a standard database to obtain information on incorrect gestures and incorrect piano sound segments. Finally, it marks the incorrect hand gestures on the sheet music and provides feedback to the user. However, this scheme treats the audio signal and the hand movement signal as two independent and redundant observation channels for the same performance event. The audio channel only serves the auxiliary function of narrowing down the range of hand positions. The two signals are only combined in parallel during the comparison stage after the recognition stage. This loosely coupled multimodal fusion method ignores an objective fact in the physical sound production mechanism of the piano—audio does not occur independently of finger movement, but is a delayed projection of finger movement after passing through the mechanical transmission link of keys, hammers, strings, and resonance chambers. There is a measurable mechanical delay distribution between the finger key press and the corresponding note sound, determined by the touch force, touch angle, and finger joint stiffness. This delay distribution itself... This means that it carries finger joint dynamics information that cannot be directly observed from any single modality. Loosely coupling audio and hand movement as two redundant channels for comparison is equivalent to actively discarding the core physical signals implied by this delay distribution. The direct consequence is that when a learner plays the correct note with the wrong finger (for example, using the ring finger when the middle finger should be used, yet the pitch is still correct—a common occurrence in the initial muscle memory solidification process), the audio channel gives a completely correct judgment, and the hand movement channel, because it only identifies joint position and cannot perceive joint dynamic abnormalities, ultimately gives incorrect feedback that the performance was correct. This means that the incorrect muscle memory is not only not corrected but is actually positively reinforced by the system. Meanwhile, the standard comparison method used in this scheme eliminates alignment errors in the time dimension. It is precisely the time delay distribution signal contained in this alignment error that constitutes the key basis for identifying the aforementioned pseudo-correct playing. Once eliminated, the implicit information of finger joint dynamics is irretrievable, and the system loses the possibility of predictive correction before errors occur. Furthermore, a piano learning system based on augmented reality technology disclosed in Chinese invention patent application CN104217625A projects a pre-stored hand shape template onto the piano keys. However, the hand shape presented is a static template rather than a real-time overlay of the learner's own hand image, and it cannot provide early warnings before errors occur based on the analysis of the real-time performance process.

[0004] In summary, the core bottleneck faced by existing technologies in the field of intelligent evaluation of piano playing fingering can be attributed to how to break the existing cognitive framework that audio and motion are two redundant observations of the same performance event, and instead explicitly model the mechanical delay causal relationship between audio and motion, so that the system has the ability to identify pseudo-correct playing of the same note with different fingers and to make predictive corrections before errors occur. Summary of the Invention

[0005] Addressing the core bottleneck of existing technologies in the field of intelligent piano fingering assessment—which treats audio signals and hand movement signals as two redundant observations, thus failing to identify pseudo-correct playing where the note is correct but the fingering is incorrect—this invention provides an intelligent piano fingering assessment system based on multimodal feature fusion. By explicitly modeling the performance audio as a delayed projection generated by finger movements via a mechanical transmission link and using the delay distribution as a query signal for cross-modal joint attention, this system achieves fine-grained positioning of playing fingering, automatic identification of pseudo-correct playing of the same note with different fingers, and predictive correction before errors occur, all from the perspective of the physical causal mechanism of piano sound production, without the aid of any additional sensors or the presence of a teacher.

[0006] The technical solution of this invention is: an intelligent evaluation system for piano playing fingering based on multimodal feature fusion, comprising a microphone array audio acquisition module, a depth camera image acquisition module, an acoustic feature extraction module, a motion feature extraction module, a cross-modal time-delay causal joint attention comparison module, and a correction feedback module. The microphone array audio acquisition module is used to acquire performance audio during the performance process. The depth camera image acquisition module is used to acquire hand depth images during the performance process. The acoustic feature extraction module uses a convolutional recurrent network with an attention mechanism to process the performance audio, outputting an acoustic feature sequence containing note dynamics, timing deviations, and pedal depth. The motion feature extraction module uses a graph convolutional network to process the finger joints in the hand depth images, outputting a motion feature sequence containing finger joint movement trajectories and key touch angles. The cross-modal time-delay causal joint attention comparison module models the acoustic feature sequence as a delayed projection of the motion feature sequence via a mechanical transmission link. Based on a preset time-delay distribution, it generates a joint attention query and compares the acoustic feature sequence and the motion feature sequence with a standard playing fingering template library under the constraint of time-delay-aware dynamic time warping to locate the note position corresponding to the incorrect fingering. The comparison module further includes a homonymous fingering determination unit. When the acoustic path cost of time-delay-aware dynamic time warping is lower than a first threshold while the joint dynamic residual between the motion feature sequence and the standard template is higher than a second threshold, the current performance is determined to be a pseudo-correct playing with correct notes but incorrect finger replacement. The correction feedback module outputs correction suggestions and generates a semi-transparent virtual hand shadow overlaid on the hand depth image to indicate the correct finger shape. The feedback module further includes a time delay statistical feedback branch and a look-ahead hand shadow generation branch. The former feeds back the online statistical results of the time delay probability density function to the network parameters of the acoustic feature extraction module and the motion feature extraction module for adaptive updates. The latter predicts the subsequent keystroke trajectory based on the motion features in the look-ahead window at the current moment and generates the semi-transparent virtual hand shadow for warning overlay before the incorrect fingering occurs.

[0007] The advantages of this invention over the prior art are as follows:

[0008] Firstly, this invention can identify pseudo-correct playing where the note is correct but the finger substitution is incorrect—a capability completely lacking in existing multimodal piano practice systems. The mechanism lies in this: the invention explicitly models the acoustic feature sequence as a delayed projection of the motion feature sequence via a mechanical transmission link, and uses this delay distribution as the query signal for cross-modal joint attention. This ensures that the time delay deviation during alignment is no longer erased as noise, but is retained as a key signal carrying finger joint dynamics information. When the learner plays the correct note with the wrong finger, the alignment cost of the acoustic path remains low, but the joint dynamics residual between the motion features and the standard template increases significantly due to the finger substitution. Based on this, the homonym / different finger identification unit correctly identifies the pseudo-correct playing. This invention does not simply merge the audio and motion results in the comparison stage, but tightly couples them at the physical causal level during the feature fusion stage, thus fundamentally surpassing existing loosely coupled solutions.

[0009] Secondly, this invention provides a premonitory semi-transparent virtual hand shadow overlay before incorrect fingering actually occurs, achieving a leap from post-event evaluation to pre-event correction. The mechanism lies in this: the invention predicts subsequent keystroke trajectories based on motion characteristics within the current lookahead window through a forward-looking hand shadow generation branch. Combined with the candidate error position probability output by time-delay-aware dynamic time warping, the semi-transparent virtual hand shadow is generated and overlaid within a millisecond-level time window before the learner's fingers actually touch the keys, thus guiding the learner to correct the error before muscle memory is formed. Compared to publicly available piano-assisted learning systems based on augmented reality projection, which present hand shapes as pre-stored static templates unrelated to the learner's current playing, the semi-transparent virtual hand shadow generated by this invention is a pixel-level overlay based entirely on the learner's real-time hand image and possesses predictive triggering capabilities before errors occur. There is a fundamental difference between the two: from passive guidance to active correction.

[0010] Third, this invention establishes a nonlinear collaborative closed loop among the acoustic feature extraction module, the motion feature extraction module, and the cross-modal time-delay causal joint attention comparison module. The mechanism is as follows: the time-delay probability density function serves both as the physical basis for attention query generation and as the negative log-likelihood term in the time-delay-aware dynamic time warping cost term. Simultaneously, its online statistical results are fed back into the feature extraction network's parameter updates through the time-delay statistical feedback branch. This results in higher accuracy of the time-delay distribution as the system runs longer, leading to a synchronous improvement in the alignment accuracy of the attention query and dynamic time warping. A positive feedback loop is formed among the three: refined time-delay distribution → improved alignment accuracy → more accurate feature extraction → more refined time-delay distribution. The performance gain brought by this collaborative closed loop is significantly higher than the simple superposition of any single-modal scheme, demonstrating the synergistic effect of nonlinear collaboration. Attached Figure Description

[0011] Figure 1 This is a block diagram of the overall architecture of the intelligent evaluation system for piano playing fingering based on multimodal feature fusion of the present invention;

[0012] Figure 2 This is a block diagram of the internal structure of the cross-modal time-delay causal joint attention comparison module 5 in the intelligent evaluation system for piano playing fingering based on multimodal feature fusion of the present invention. Detailed Implementation

[0013] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0014] Reference Figure 1 The intelligent evaluation system for piano playing fingering based on multimodal feature fusion disclosed in this embodiment includes a microphone array audio acquisition module 1, a depth camera image acquisition module 2, an acoustic feature extraction module 3, a motion feature extraction module 4, a cross-modal time-delay causal joint attention comparison module 5, and a correction feedback module 6. The microphone array audio acquisition module 1 is connected to the acoustic feature extraction module 3, and the depth camera image acquisition module 2 is connected to the motion feature extraction module 4. The outputs of both the acoustic feature extraction module 3 and the motion feature extraction module 4 are connected to the cross-modal time-delay causal joint attention comparison module 5. The output of the cross-modal time-delay causal joint attention comparison module 5 is connected to the correction feedback module 6. The time-delay statistical feedback branch in the correction feedback module 6 is simultaneously connected back to both the acoustic feature extraction module 3 and the motion feature extraction module 4, thus forming a closed-loop collaborative structure with both feedforward and feedback. The entire system relies on a main industrial control computer with a built-in multi-core processor and an independent graphics processing unit as its computing platform. This main industrial control computer is connected to the front-end acquisition devices of the microphone array audio acquisition module 1 and the depth camera image acquisition module 2 via gigabit Ethernet interfaces. Data exchange between modules is based on a shared memory circular buffer to ensure data consistency and low-latency response in high-frame-rate real-time processing scenarios. After power-on, the system first enters a self-test and calibration process, including microphone channel gain calibration, coordinate system registration of the depth camera optical axis and keyboard surface, and online calibration of various coefficients in the time delay probability density modeling unit. After calibration, the system enters normal performance evaluation operation. The specific implementation methods of each module are described in detail below.

[0015] The microphone array audio acquisition module 1 is deployed on a horizontal beam extending horizontally above the piano to capture piano audio during performance. In this embodiment, the microphone array audio acquisition module 1 uses four spatially distributed directional condenser microphones. The four directional condenser microphones are symmetrically arranged with the area above the center C key of the piano as the origin. The horizontal spacing between two adjacent directional condenser microphones is 40cm, and their pointing angles cover the high, mid-high, mid-low, and low registers, respectively. This ensures that the sound pressure signals generated by the strings in each register can be picked up by at least one directional condenser microphone as the primary pickup channel, effectively suppressing environmental interference introduced by reverberation in the music room and mechanical noise from the keyboard. The sampling rate of the four directional condenser microphones is uniformly 48kHz, and the sampling bit depth is 24bit. The channels are synchronized through a hardware clock to ensure phase consistency between multiple channels and avoid the impact of software-level sampling drift on the accuracy of subsequent time delay estimation.

[0016] Before being sent to the acoustic feature extraction module 3, the raw multi-channel audio signal from the microphone array audio acquisition module 1 passes through a pre-processing digital link. This pre-processing digital link sequentially performs four steps: DC bias removal, bandpass filtering, noise threshold suppression, and multi-channel beamforming. DC bias removal eliminates low-frequency drift from the condenser microphone by calculating the moving average of each channel signal and subtracting it from the raw signal. The bandpass filter has a passband of 27.5Hz to 4500Hz, covering the fundamental frequency range of the piano keys A0 to C8 while preserving its important harmonic components. Noise threshold suppression uses spectral subtraction to attenuate spectral components below a preset threshold to one percent of their original amplitude. Multi-channel beamforming uses a delay-addition structure, spatially weighting and summing the four channels with the piano centerline as the primary pointing direction to further suppress reflected echoes from the piano room walls. The single-channel enhanced audio signal output from the pre-processing digital link, along with the original four-channel signal retained for time delay estimation, is then sent to the acoustic feature extraction module 3.

[0017] The depth camera image acquisition module 2 is deployed on a support frame 50cm above the piano keyboard, with its optical axis pointing vertically downwards and covering the entire 88-key keyboard area. In this embodiment, the depth camera image acquisition module 2 uses a depth camera based on structured light principles, with a depth resolution of 1280×720px, a depth measurement range of 20cm to 120cm, a depth measurement accuracy better than 1mm, and a frame rate of 60fps. While acquiring depth images, the depth camera simultaneously outputs timestamp-aligned infrared grayscale images. These infrared grayscale images are used to assist in hand contour segmentation, thus maintaining the stability of hand detection even under drastic changes in lighting conditions in the piano room. Each frame of the original depth image and infrared grayscale image output by the depth camera is sent to the system's main control unit via a USB 3.0 interface, and is strictly timestamped within the main control unit based on a global clock, along with the audio frames output by the microphone array audio acquisition module 1.

[0018] Before sending out image data, the depth camera image acquisition module 2 performs a set of lightweight preprocessing operations, including depth hole filling, background removal, and hand region of interest extraction. Depth hole filling adopts a neighborhood weighted median filling strategy, filling invalid pixels in the depth image caused by reflection or occlusion based on the depth median of its 8 neighboring valid pixels; background removal uses a depth threshold of 45cm as the boundary, treating pixels larger than this threshold as piano keyboard background and setting them to zero; hand region of interest extraction uses connected component analysis on the infrared grayscale image to find the two largest connected components as candidate regions for the left and right hands, and then crops and normalizes the candidate regions to a fixed size of 128×128px, thereby reducing the computational overhead of the subsequent motion feature extraction module 4.

[0019] The acoustic feature extraction module 3 receives the enhanced audio signal output from the microphone array audio acquisition module 1, processes it using a convolutional recurrent network with an attention mechanism, and outputs an acoustic feature sequence containing note velocity, timing deviation, and pedal depth. In this embodiment, the front end of the acoustic feature extraction module 3 divides the input single-channel enhanced audio signal into frames with a frame length of 25ms and a frame shift of 10ms. After Hamming windowing for each frame, an 80-dimensional log-Mel spectrum is calculated to obtain the Mel spectrum time series as the network input. The main structure of the acoustic feature extraction module 3 consists of two two-dimensional convolutional blocks, a bidirectional gated recurrent unit layer, and an attention pooling layer connected in series. The two 2D convolutional blocks contain 16 and 32 convolutional kernels respectively, both with a kernel size of 3×3, a stride of 1, and a ReLU activation function. Downsampling between the two 2D convolutional blocks is performed using 2×2 max pooling. The hidden state dimension of the bidirectional gated recurrent unit layer is 128, providing bidirectional temporal modeling for the time-frequency features of the convolutional output. The attention pooling layer consists of three parallel sub-networks: note velocity attention, timing deviation attention, and pedal depth channel attention. The pedal depth channel attention specifically captures the low-frequency long-term envelope components related to pedal action by performing channel-level weighted summation of the attention weights for each frequency band. The final acoustic feature sequence output by the acoustic feature extraction module 3 is denoted as a vector sequence with a length equal to the number of audio frames. The acoustic features of each frame include three main components: note velocity score, timing deviation, and pedal depth estimation, as well as several auxiliary components, and are fed into the cross-modal time-delay causal joint attention comparison module 5.

[0020] The motion feature extraction module 4 receives the hand region of interest sequence output by the depth camera image acquisition module 2, and uses a graph convolutional network to process the finger joints, outputting a motion feature sequence containing the finger joint movement trajectory and touch angle. In this embodiment, the front end of the motion feature extraction module 4 first uses a pre-trained two-dimensional hand joint detection network to extract the pixel coordinates of 21 hand joints in each frame's hand region of interest. The 21 hand joints, according to industry-standard definitions, include one at the wrist and four at each finger, totaling 20 finger joints. Subsequently, the pixel coordinates of each joint are combined with the depth value at the same pixel location to transform them into three-dimensional spatial coordinates based on the piano keyboard surface, thus obtaining the three-dimensional coordinate sequence of 21 joints in each frame.

[0021] The backbone of the motion feature extraction module 4 is a three-layer spatiotemporal graph convolutional network. Its graph structure consists of 21 hand joints, and its edge set includes two types: the first type is finger skeleton connection edges, which connect adjacent joints according to the anatomical structure of the hand; the second type is cross-finger collaborative edges, which establish a collaborative edge between the fingertip joints of any two different fingers. The weight of the cross-finger collaborative edge is dynamically updated based on the instantaneous distance between the two fingertips; the smaller the finger distance, the larger the weight of the collaborative edge, thus modeling the motion coupling relationship between adjacent fingers in collaborative playing. The three-layer spatiotemporal graph convolutional network performs graph convolution aggregation along the nodes in the spatial dimension and one-dimensional convolution on adjacent frames of the same node in the temporal dimension, ultimately outputting a 64-dimensional latent representation for each joint in each frame. Based on this, the motion feature extraction module 4 decodes two sets of physical quantities from the latent representation: one set is the three-dimensional motion trajectory of each joint and its first-order time derivative; the other set is the instantaneous touch angle of each fingertip joint relative to the keyboard surface. The motion feature sequence is resampled at the same time step as the acoustic feature sequence to facilitate strict temporal alignment in the cross-modal temporal delay causal joint attention alignment module 5.

[0022] The cross-modal time-delay causal joint attention comparison module 5 is the core module of this invention, and its internal structure is described in reference to... Figure 2 As shown, it consists of a time delay probability density modeling unit, a joint attention query generation unit, a time delay-aware dynamic time warping unit, a homonym / different fingering determination unit, and a standard performance fingering template library access interface connected in series.

[0023] The time delay probability density modeling unit is responsible for realizing the physical premise of explicitly modeling the performance audio as a delay projection generated by finger movement via a mechanical transmission link. In this embodiment, for each note event, the time delay probability density modeling unit maintains a time delay probability density function conditioned on the touch force and touch angle:

[0024] ,

[0025] ,

[0026] ,

[0027] in: For note events at a given keystroke force and touch angle The time delay probability density function under the given condition is a scalar, taking values ​​of non-negative real numbers, with units of . This formula, calculated from the data, represents a specific delay in audio frequency relative to finger movement. The probability density; The delay of the audio input relative to the finger key press is a scalar value with a range of [value missing]. to The unit is It is obtained by subtracting the start time of the button action from the audio oscillation start time, and represents the specific delay introduced by the mechanical transmission link; The keystroke force is denoted by , and is a scalar with a range of values. to The unit is a dimensionless normalized value, which is obtained by normalizing the peak value of the fingertip falling acceleration in the motion feature extraction module 4, and represents the instantaneous energy of the learner touching the key. Let be the key touch angle, and be a scalar with a range of values. to The unit is The motion feature extraction module 4 directly outputs the instantaneous angle between the fingertip and the keyboard surface. Let be the conditional mean of the time delay probability density function, and let be a scalar with a range of values. to The unit is , calculated by the second equation, represents the average delay of the relative motion of the audio under the current key pressure and key angle conditions; Let be the conditional standard deviation of the time delay probability density function, and let be a scalar with a range of values ​​of . to The unit is , calculated by the third equation, characterizes the degree of dispersion of the time delay distribution; As the baseline mean, take The unit is It is obtained by calibrating the unloaded key-striking experiment of the central C region of the white key, and characterizes the mechanical reference delay under zero force and zero tilt angle conditions; As the baseline standard deviation, take The unit is Also determined by calibration experiments, it characterizes the inherent vibration level of the mechanical system; Let be the influence coefficient of intensity on the mean, and take . The unit is The negative sign, obtained from regression fitting, means that the greater the key pressure, the smaller the mechanical delay. Let be the coefficient of influence of key angle on the mean, and take . The unit is It is obtained by regression fitting; Let be the influence coefficient of intensity on standard deviation, and take . The unit is It is obtained by regression fitting; For the natural constant An exponential function with base 0.05. The value of this threshold parameter is based on calibration data from 6000 keystrokes performed by ten players of different skill levels on a standard grand piano. If the value is too large, it will cause the entire subsequent regularization path to shift; if the value is too small, it will not be able to cover the actual mechanical delay range.

[0028] The joint attention query generation unit discretizes the aforementioned delay probability density function with a preset step size and generates an attention query vector for each delay sampling point. Specifically, for the delay value range... By step size Obtain by performing equal-interval sampling Each time delay sampling point , For each time-delay sampling point, a query vector is generated, and the motion feature sequence is weighted by displacement using this query vector to obtain the candidate motion context.

[0029] ,

[0030] ,

[0031] ,

[0032] in: For the first Each time delay sampling point is a scalar, with a value range of [value range missing]. to The unit is , calculated by the first equation, characterizes the discrete sampling of a continuous time delay distribution; As the lower bound of the time delay sampling interval, take The unit is This represents the physical limitation that audio cannot precede motion at the lower boundary. As the upper bound of the time delay sampling interval, take The unit is The value is determined based on twice the redundancy of the maximum mechanical delay limit covered by the calibration data; Let the time delay sampling step size be taken. The unit is Too large a value will result in a loss of latency resolution, while too small a value will lead to a linear increase in computational load. The total number of time-delay sampling points is taken as... , is an integer, dimensionless, and derived from Calculated; Here is the time delay sampling sequence number, which is an integer and its value ranges from 1 to 2. to ; For the first Each time-delay sampling point at time... The attention query vector is dimensional vector, Pick Each component is dimensionless and is calculated by the second equation, representing the attention query with time delay probability density as the physical weight; The first style of this unit is The value at the location, in units of The meaning is the same as the previous one; To query the projection matrix, real matrix, Pick Consistent with the acoustic feature dimension, it is dimensionless, obtained by network training, and represents the linear transformation that maps the acoustic feature space to the attention query space; The time output of acoustic feature extraction module 3 Acoustic eigenvectors are dimensional vector, Pick Each component is a normalized dimensionless quantity, calculated by acoustic feature extraction module 3; The time output of motion feature extraction module 4 Motion feature vector, is dimensional vector, Pick Each component is a normalized dimensionless quantity, calculated by motion feature extraction module 4, and its subscripts are... Approximate integers on a frame-by-frame basis; For a moment The candidate motion context vector is, The dimension vector, calculated by the third equation, represents the motion context obtained by performing causal tracing on the motion feature sequence with time delay distribution as the weight; This is the element-wise multiplication operator for vectors. The summation symbol represents the summation range of all time-delay sampling points.

[0033] The delay-aware dynamic time warping unit performs dynamic time warping comparison on the candidate motion context sequence and acoustic feature sequence output by the joint attention query generation unit. Its special feature is that the cost function explicitly introduces a delay bias cost term, so that the warping path is optimized under the constraint of preserving the delay signal, rather than erasing the delay difference as noise like standard dynamic time warping.

[0034] ,

[0035] ,

[0036] in: The cumulative cost of the regularized path is a scalar, taking the range of non-negative real numbers, dimensionless, and is calculated recursively from this formula, representing the distance from the starting point to the grid point. The optimal total alignment cost; The time index of the acoustic feature sequence is an integer with a value range of 1. Up to the length of the acoustic feature sequence; Here is the time index of the motion feature sequence, which is an integer and takes values ​​ranging from 1 to 2. Up to the length of the motion feature sequence; For grid points The local distance between the acoustic and kinematic features is a scalar, dimensionless, and is taken as... and The cosine distance between them represents the degree of similarity between the features of the two modalities under the current alignment assumption; As the weight of the time delay deviation cost term, take , is a scalar, dimensionless quantity, obtained by performing a grid search on the validation set. If it is too large, the alignment will over-obey the time delay prior and ignore feature similarity; if it is too small, it will degenerate into standard dynamic time warping. The time delay deviation cost term is a scalar with a range of non-negative real numbers and no dimension. It is calculated by the second equation and represents the degree of unreasonableness of the alignment delay corresponding to the current grid point relative to the time delay probability density function. It is the natural logarithm function; The alignment delay is the time delay corresponding to the current grid point, where For the frame interval, take The unit is ; and Time index for acoustic features The key pressure and key angle of the corresponding notes are defined in the same way as the symbol definition of the time delay probability density function mentioned above; This is the minimum value operator; the initial boundary conditions in this expression are: The recursive direction is and Monotonic and non-decreasing.

[0037] The homonym / incorrect fingering determination unit receives the optimal warping path and cumulative cost output by the delay-aware dynamic time warping unit, and retrieves the standard template corresponding to the current piece from the standard playing fingering template library. It independently evaluates the joint dynamic residual of the learner's motion feature sequence relative to the standard template, and identifies pseudo-correct playing with correct notes but incorrect finger replacements under the joint discrimination of two independent conditions: acoustic path cost and joint dynamic residual.

[0038] ,

[0039] ,

[0040] in: Let be the joint dynamics residual, be a scalar, take the range of non-negative real numbers, be dimensionless, be calculated by the first equation, and characterize the degree of deviation of the learner's motion characteristics from the standard template motion characteristics; The number of frames within the comparison window is an integer, and is taken as... Dimensionless, representing approximately Local time window; is the frame number, which is an integer and its value ranges from 1 to 2. to ; For learners in the The motion feature vector of the frame is defined as before; To align the fingering templates retrieved from the standard playing fingering template library to the first... The standard motion feature vector of a frame is A dimensionless vector, obtained by aligning the optimal path from the template library based on time-delay-aware dynamic time warping; For vectors Norm; The acoustic path cost is a scalar, dimensionless quantity, and is the normalized value of the curvature path cumulative cost output by the time delay-aware dynamic time curvature unit on the acoustic component. To determine the threshold for acoustic cost, take... , dimensionless, determined by quantile statistics on the validation set, characterizing the acoustic path as a low-cost threshold that must be crossed for the correct performance; The threshold for determining joint dynamic residuals is set as follows: Dimensionless, also obtained from the quantile statistics of the validation set, represents the high residual threshold that must be crossed to determine the motion characteristics as significantly deviating from the standard template; if the two threshold values ​​are too large, false correct playing will be missed, and if the values ​​are too small, normal playing will be misjudged as false correct playing.

[0041] The standard playing fingering template library stores standard playing data covering common piano teaching pieces. Each piece template includes three parts: an acoustic feature sequence, a motion feature sequence, and a baseline time delay probability density function between the two. During the comparison process, the cross-modal time delay causal joint attention comparison module 5 matches all three parts simultaneously to avoid the judgment bias caused by relying on a single modal template. In this embodiment, the standard playing fingering template library is constructed by having multiple professional piano teachers play selected pieces on the same calibrated grand piano using standard fingering. The collected acoustic signals and hand depth images are processed by the same pre-processing link as the learner's end and then sent to the acoustic feature extraction module 3 and the motion feature extraction module 4 respectively to obtain the template feature sequence. The baseline time delay probability density function is obtained by kernel density estimation of the measured time delay of each note event during the performance by the professional piano teachers, thereby ensuring that the time delay prior in the template library reflects the mechanical delay distribution under correct fingering.

[0042] The correction feedback module 6 receives the error finger location results and homophone judgment results output by the cross-modal time-delay causal joint attention comparison module 5. On the one hand, it outputs visual correction suggestions and semi-transparent virtual hand shadow overlay prompts to the learner. On the other hand, it realizes adaptive optimization and predictive correction in the process of system operation through the time-delay statistical feedback branch and the look-ahead hand shadow generation branch.

[0043] The time delay statistics feedback branch continuously calculates the difference between the actual measured time delay distribution and the prior distribution given by the time delay probability density modeling unit during system operation, and feeds this difference back to the network parameters of acoustic feature extraction module 3 and motion feature extraction module 4 for online updates using gradient descent.

[0044] ,

[0045] ,

[0046] in: The time delay distribution difference loss is a scalar with a range of non-negative real numbers and is dimensionless. It is calculated by the first equation and characterizes the Kullback-Leibler divergence between the measured time delay distribution and the reference time delay distribution. For the Kullback-Leibler divergence operator; For the most recent sliding window, the time delay sampling point The measured probability density estimate is a scalar, with units of . The kernel density is estimated by measuring the actual delay of each keystroke event within the sliding window. For the prior time delay probability density in The value at the given location is simultaneously delayed by the probability density modeling unit's initial expression, and the unit is... ; and The definition is the same as before; The set of trainable parameters for acoustic feature extraction module 3 and motion feature extraction module 4 is in the [missing information]. The value at each update time is a parameter vector, with each component being dimensionless and obtained from the previous iteration; For the parameter set in the th The value at the next update time is calculated using the second formula; For online learning rate, take , is a scalar, dimensionless, and its value is based on the upper bound of the learning rate allowed to ensure the stable convergence of the closed-loop system. If it is too large, it will cause parameter divergence; if it is too small, the adaptive speed will be too slow. For the parameter set The gradient operator; This is a discrete-time index, and it is a non-negative integer.

[0047] Forward-looking hand shadow generation branch based on the current moment The motion feature sequence within the lookahead window is used to predict subsequent keystroke trajectories, and a semi-transparent virtual hand shadow is generated and overlaid on the learner's real-time hand depth image before an impending incorrect finger movement is predicted.

[0048] ,

[0049] ,

[0050] in: To The predicted value of the fingertip position after time step is a three-dimensional vector, with each component in units of . , calculated by the first equation, represents the future key position under the assumption that the learner maintains the current movement trend; For the current moment The three-dimensional position of the fingertip is a three-dimensional vector, with each component in units of . The output is from motion feature extraction module 4; For the forward time measure, take The unit is The value is determined by the lower limit of the typical neuromuscular response time of learners from seeing the prompt to completing the corrective action. If it is too large, the prediction accuracy will decrease, and if it is too small, the corrective prompt will not be generated in time. For the number of frames in the lookout window, take , is an integer, dimensionless, and corresponds to approximately Historical movement information; The frame index within the lookout window is an integer, with a value range of 1. to ; For the first The weighting coefficient for the velocity of each historical frame is a scalar, and its value ranges from 1 to 2. to Calculated by the second equation, it satisfies The more recent the motion velocity, the greater its contribution to the prediction; For the first The fingertip velocity vector of the frame is a three-dimensional vector, with each component in units of . It is obtained by dividing the difference in fingertip position between two adjacent frames by the frame interval; Let be the weighted attenuation coefficient, taken as... , is a scalar, dimensionless, obtained from motion autocorrelation analysis of historical frames; It is a natural exponential function; The summation index in the denominator of the second equation is an integer with a range of values ​​of 1. to .

[0051] Based on the aforementioned prospective prediction results, the prospective hand shadow generation branch, combined with the homophone and different finger judgment output of the cross-modal time-delay causal joint attention comparison module 5, performs real-time evaluation on whether the predicted position conforms to the standard fingering. When the predicted position deviates from the standard fingering position by more than 10mm from the preset spatial threshold and the homophone and different finger judgment result shows that the finger replacement probability exceeds 0.5, a semi-transparent virtual hand shadow is rendered at the predicted key touch position. The opacity of the semi-transparent virtual hand shadow is linearly positively correlated with the confidence of the incorrect fingering. When the confidence changes linearly from 0.5 to 1.0, the opacity changes linearly from 0.4 to 0.9, so that the higher the confidence, the clearer the hand shadow outline, thus providing visual cues to learners within tens of milliseconds before the actual key touch.

[0052] The system described in this embodiment was tested on an internal validation set containing 20,000 performance clips from 100 learners of different levels. The validation set was divided into three groups based on learner level: beginner, intermediate, and advanced, containing 6,000, 8,000, and 6,000 performance clips respectively. Each performance clip ranged in length from 10 to 30 seconds and covered three main categories: classical, etudes, and modern pieces. Under identical input data conditions, compared to existing systems using loosely coupled multimodal comparison schemes, the system described in this embodiment improved the false-correct playing recognition rate from 9.3% to 82.1% for correct notes but incorrect finger substitutions, increased the pre-warning trigger rate from 0% to 76.4%, and improved the overall fingering assessment accuracy from 78.5% to 94.6%. In the beginner group, due to the greater instability in key touch force and angle, the frequency of pseudo-correct playing was significantly higher than in the other two groups. The pseudo-correct playing recognition rate of the system described in this embodiment reached 85.7% in the beginner group, which is higher than the overall average level, further demonstrating that the time delay distribution as an attention query signal has stronger adaptability in scenarios with significant differences in individual playing characteristics. In addition, after the system described in this embodiment has been running continuously for 60 minutes, due to the adaptive update of the time delay statistical feedback branch, the mean square error of its time delay distribution estimation has decreased by about 38% compared with the system cold start, resulting in a continuous improvement in subsequent alignment accuracy and error recognition accuracy. This verifies that the closed-loop collaborative structure formed by the acoustic feature extraction module 3, the motion feature extraction module 4, and the cross-modal time delay causal joint attention comparison module 5 does indeed produce significant nonlinear collaborative gains, fully verifying the effectiveness and superiority of the cross-modal time delay causal joint attention comparison scheme proposed in this invention in the intelligent evaluation task of piano playing fingering.

[0053] The embodiments of the present invention are not limited to the specific embodiments described above. Those skilled in the art can make various equivalent changes or substitutions based on the technical solutions of the present invention, and all such changes or substitutions should be included within the protection scope of the present invention.

Claims

1. A piano fingering intelligent evaluation system based on multi-modal feature fusion, characterized in that, include: Microphone array audio acquisition module, used to acquire performance audio during the performance; A depth camera image acquisition module is used to capture depth images of the hands during the performance. The acoustic feature extraction module uses a convolutional recurrent network with an attention mechanism to process the performance audio and outputs an acoustic feature sequence containing note dynamics, timing deviation, and pedal depth. The motion feature extraction module uses a graph convolutional network to process the finger joints in the hand depth image and outputs a motion feature sequence containing the finger joint movement trajectory and touch angle. The cross-modal time-delay causal joint attention comparison module models the acoustic feature sequence as a delay projection of the motion feature sequence via a mechanical transmission link, generates a joint attention query based on a preset time-delay distribution, and compares the acoustic feature sequence and the motion feature sequence with a standard playing fingering template library under the constraint of time-delay-aware dynamic time warping to locate the note position corresponding to the incorrect fingering. The correction feedback module is used to output correction suggestions and generate a semi-transparent virtual hand shadow overlaid on the hand depth image to indicate the correct finger shape.

2. The intelligent evaluation system for piano playing fingering based on multimodal feature fusion according to claim 1, characterized in that, In the cross-modal time-delay causal joint attention comparison module, the preset time-delay distribution is to maintain a time-delay probability density function for each note. The mean and variance of the time-delay probability density function are determined by the touch force and touch angle corresponding to the note.

3. The intelligent evaluation system for piano playing fingering based on multimodal feature fusion according to claim 2, characterized in that, The cross-modal time-delay causal joint attention comparison module discretizes the time-delay probability density function into a finite number of time-delay sampling points with a preset step size, generates an attention query vector for each time-delay sampling point, and performs displacement weighting on the motion feature sequence with the time-delay sampling point as the center to obtain a set of candidate motion contexts corresponding to the acoustic feature sequence.

4. The intelligent evaluation system for piano playing fingering based on multimodal feature fusion according to claim 3, characterized in that, The cost function of the delay-aware dynamic time warping includes a delay deviation cost term, which is equal to the negative log-likelihood between the actual alignment delay at each point on the warping path and the delay probability density function, so that the warping path is optimized under the constraint of preserving the delay signal.

5. The intelligent evaluation system for piano playing fingering based on multimodal feature fusion according to claim 4, characterized in that, The cross-modal time-delay causal joint attention comparison module further includes a homonym / different finger determination unit. When the acoustic path cost of the time-delay-aware dynamic time-regulating acoustic path is lower than a first threshold, and the joint dynamic residual between the motion feature sequence and the standard template is higher than a second threshold, the current performance is determined to be a pseudo-correct playing with correct notes but incorrect finger replacement, and the position of the pseudo-correct playing notes is output as an incorrect fingering.

6. The intelligent evaluation system for piano playing fingering based on multimodal feature fusion according to claim 5, characterized in that, The correction feedback module further includes a time delay statistical feedback branch and a look-ahead hand shadow generation branch. The time delay statistical feedback branch feeds back the online statistical results of the time delay probability density function to the network parameters of the acoustic feature extraction module and the motion feature extraction module for adaptive updates. The look-ahead hand shadow generation branch predicts the subsequent keystroke trajectory based on the motion features within the look-ahead window at the current moment, and generates the semi-transparent virtual hand shadow for warning overlay before the erroneous fingering occurs.

7. The intelligent evaluation system for piano playing fingering based on multimodal feature fusion according to claim 1, characterized in that, The microphone array audio acquisition module includes at least four spatially distributed directional microphones, and the attention mechanism of the acoustic feature extraction module includes a channel attention subnetwork for pedal depth.

8. The intelligent evaluation system for piano playing fingering based on multimodal feature fusion according to claim 1, characterized in that, The graph convolutional network in the motion feature extraction module has a node set of 21 hand joints and an edge set including two types: finger skeleton connection edges and cross-finger collaborative edges. The weight of the cross-finger collaborative edges is dynamically updated based on the finger spacing.

9. The intelligent evaluation system for piano playing fingering based on multimodal feature fusion according to claim 1, characterized in that, Each template in the standard playing fingering template library contains three parts: an acoustic feature sequence, a motion feature sequence, and a baseline time delay probability density function between the two. The cross-modal time delay causal joint attention comparison module matches the three parts simultaneously.

10. The intelligent evaluation system for piano playing fingering based on multimodal feature fusion according to claim 1, characterized in that, The opacity of the semi-transparent virtual hand shadow is positively correlated with the confidence level of the incorrect fingering; the higher the confidence level, the clearer the outline of the semi-transparent virtual hand shadow.

Images