Quantization method for evaluating influence of small drum visual playing on coordination of two hands

Abstract

The invention discloses a quantitative method for evaluating the influence of small drum visual playing on the coordination of both hands, and the quantitative evaluation result of the influence of small drum visual playing on the coordination of both hands is obtained through machine learning by fusing the visual playing behavior characteristics of a user and the physiological characteristics of a nervous system. Guiding a user to perform small drum visual playing in an animation form, and collecting audio and video signals in a visual playing process to extract visual playing behavior characteristics; pPG signals of both hands are measured through a multi-point PPG signal acquisition system, and physiological features of a nervous system are extracted from the PPG signals after multi-channel fusion; a two-hand coordination test is performed through guidance of the computer, and task performance is obtained; and fusing the visual playing behavior characteristics, the physiological characteristics of the nervous system and the task performance, carrying out regression analysis, and carrying out tracking training to obtain a quantitative evaluation result of influence of small drum visual playing on coordination of both hands. According to the invention, an objective quantification method can be provided for evaluating the influence of the small drum visual playing on the coordination of the two hands.

Description

Technical Field

[0001] The present invention relates to the fields of signal processing and machine learning, and in particular to a quantitative method for evaluating the influence of snare drum sight-playing on the coordination of both hands. Background Art

[0002] Many tasks in people's daily lives require the coordination of both hands in time and space. Therefore, bimanual coordination is very important to people, and quantitative evaluation of bimanual coordination is of great significance.

[0003] Traditional methods for evaluating bimanual coordination mainly include the scale method and the task performance method. The scale method refers to a method in which professional evaluators observe the user's bimanual coordination performance in daily life and then score and rate the user's bimanual coordination ability. The task performance method refers to a method of evaluating bimanual coordination by recording the user's completion of a task within a specified time or the time required to complete a specified task, as well as performance indicators such as accuracy. Neither of these two evaluation methods includes the evaluation of physiological indicators. In addition, the above methods generally need to be carried out in hospitals or medical institutions, and professional guidance and observation are required during the evaluation process. The results are easily affected by subjective and environmental factors, making them inconvenient for application in daily monitoring.

[0004] Using a two-handed instrument, such as the snare drum, for bimanual coordination training offers advantages such as small size, portability, low cost, and wide reach. Therefore, utilizing modern signal processing techniques combined with machine learning models to explore an objective and quantitative method for evaluating the impact of snare drum sight-reading on bimanual coordination based on behavioral and physiological characteristics of sight-reading is of great significance for assisting in the daily monitoring of cardio-cerebral health. Summary of the Invention

[0005] The purpose of the present invention is to solve the above-mentioned defects in the prior art and provide an objective quantitative method for evaluating the influence of snare drum sight-reading on the coordination of both hands.

[0006] The purpose of the present invention can be achieved by taking the following technical solutions:

[0007] A quantitative method for evaluating the effect of snare drum sight-playing on hand coordination, the method comprising the following steps:

[0008] S1, guiding the user to sight-read the snare drum in an animation form, and collecting audio and video signals of the sight-reading process;

[0009] S2, extracting the time sequence of the user's drumming beats from the audio signal of the sight-reading process;

[0010] S3, extracting the time sequence of the user's left and right hands beating the drum from the video signal of the sight-reading process;

[0011] S4, comparing the time sequence of the drum beats and the time sequence of the left and right hand drum beats with a preset standard drum beat sequence to extract the user's sight-reading behavior characteristics;

[0012] S5. Measure the PPG signals of the user's hands using a multi-point PPG signal acquisition system, and extract the user's nervous system physiological characteristics after performing multi-channel fusion on the PPG signals;

[0013] S6. Conducting a two-hand coordination test on the user under computer guidance to obtain task performance of the two-hand coordination test on the user;

[0014] S7. Perform a regression analysis on the user's two-hand coordination based on the sight-reading behavior characteristics, the nervous system physiological characteristics, and the task performance, and obtain a quantitative evaluation result of the effect of snare drum sight-reading on two-hand coordination through follow-up training.

[0015] Furthermore, the step S1 includes the following steps:

[0016] S101, extracting a drum beat sequence from the small drum musical notation, and distinguishing the drum beats, accents, rests, and beat lengths of the left and right hands according to the musical notation rules; this step structures the drum beat information according to the musical notation rules to provide standardized input for subsequent quantitative analysis.

[0017] S102. Displaying a drum beat sequence in the form of an animated drum beat picture on a controllable display screen; wherein the drum beat picture above the controllable display screen displays the drum beat sequence for the right hand, and the drum beat picture below the controllable display screen displays the drum beat sequence for the left hand; marking the accents of the drum beats on the drum beat pictures in the form of specific symbols; a horizontal straight line is connected after the drum beat picture, and the total width of the drum beat picture and the horizontal straight line indicates the beat length of the drum beat to be played; if there is no horizontal straight line, it represents a rest, and this is an empty beat that does not need to be played; this step intuitively guides the user to sight-read the snare drum, lowers the threshold for sight-reading the snare drum, and is interesting.

[0018] S103: The user sight-reads the snare drum by striking the snare drum surface with drumsticks using both hands according to the animation instructions. A controllable microphone and camera are used to collect audio and video signals during the snare drum sight-reading process until the sight-reading is completed. This step uses the microphone and camera to synchronously record the audio and video signals, providing a data basis for subsequent feature extraction.

[0019] Furthermore, step S2 includes the following steps:

[0020] S201, preprocessing the audio signal of the sight-reading process; the preprocessing includes removing environmental noise, framing and windowing the audio, and performing short-time Fourier transform to obtain a time-frequency diagram; this step effectively suppresses noise and retains the time-frequency characteristics of the audio signal.

[0021] S202, performing drum beat detection on the pre-processed audio signal; performing a differential operation on each window of the time-frequency graph to obtain a spectral flux representing the volume value, and determining whether each window contains a drum beat using a threshold method; this step can quickly locate the moment of the drum beat, and has high robustness and reliability.

[0022] S203: For all drum beats obtained in the audio signal in step S202, the time corresponding to the drum beat is calculated based on the position of the drum beat in the window and the position of the window in the audio signal, thereby obtaining a time sequence of the drum beats during the user's sight-reading process. This step achieves temporal alignment between the user's sight-reading behavior and the standard drum beat sequence, facilitating subsequent calculation of sight-reading behavior features.

[0023] Furthermore, step S3 includes the following steps:

[0024] S301. Input the video signal of the sight-reading process into an open-source gesture recognition machine learning framework to extract the time series of changes in the bending angles of the user's left and right arms. This step can accurately identify the user's gestures and reduce costs.

[0025] S302: Detect the user's behavior of drumming with both hands; detect the bending angles of the user's left and right arms in each frame of the video, and use a threshold method to determine whether the user is drumming at this time; this step is reasonable and efficient, does not require complex algorithms, and is suitable for the scenario of snare drum sight-reading.

[0026] S303: Obtain the corresponding moments of the user's left and right hands beating the drums in the video signal through step S302, and after matching them with the video time nodes, output the time series of the left and right hand beating the drums, respectively. This step can accurately distinguish the moments of the left and right hand beating the drums, ensuring the reliability of the subsequent sight-playing behavior feature calculation.

[0027] Furthermore, the sight-reading behavior characteristics of the user in step S4 include time accuracy and force accuracy; the time accuracy needs to be calculated in combination with the moments of the left and right hands hitting the drum, and the force accuracy needs to normalize the volume to eliminate individual differences; the process of extracting the sight-reading behavior characteristics of the user is as follows:

[0028] S401. Align the time sequence of the user's drum beats obtained in step S2 with a preset standard drum beat sequence, and judge the accuracy in combination with the time sequence of the user's left and right hand drum beats; calculate the proportion of the number of correct user drum beats as the time accuracy of the user's snare drum sight-reading; this step balances the fault tolerance and accuracy requirements in the actual sight-reading process.

[0029] S402: The volume of the correct drum beats struck by the user in step S401 is normalized, and 1.5 times the mean of the normalized volume is taken as the accent threshold. A drum beat exceeding this threshold is considered a correct accent, while a drum beat below this threshold is considered an incorrect accent. The proportion of correct accents is calculated as the velocity accuracy of the user's snare drum sight-reading. This step eliminates volume variations caused by individual differences and ensures the reliability of the velocity accuracy calculation.

[0030] Furthermore, step S5 includes the following steps:

[0031] S501. When the user is in a resting state, they place their left and right hands on the sensors of the multi-point PPG signal acquisition system. The multi-point PPG signal acquisition system will simultaneously collect multi-channel PPG signals from both hands and upload them to the host computer for processing. This step reduces the interference of motion artifacts, and multi-point acquisition improves the reliability of extracting physiological characteristics of the nervous system.

[0032] S502. Use wavelet threshold denoising to remove baseline drift and high-frequency noise in the PPG signal to obtain a pure PPG signal; select the single-channel signal with the largest relative power in the pulse wave frequency band as the reference signal; perform autocorrelation alignment on the other single-channel signals and the reference signal, and then perform amplitude transformation based on the least squares method; perform arithmetic averaging on all channel signals to obtain a fused unilateral PPG signal; select the side with higher signal quality from the left and right PPG signals for feature extraction; this step can fully utilize the effective components in the multi-channel signal and improve the accuracy of feature calculation.

[0033] S503: Extract time-domain features, frequency-domain features, and nonlinear features from the PPG signal obtained in step S502 as the physiological features of the nervous system. This step extracts traditional PPG time-frequency domain features and advanced entropy features to more comprehensively characterize the state of the nervous system.

[0034] Furthermore, step S6 includes the following steps:

[0035] S601. The user sits in front of a controllable display screen and uses the left and right hands to manipulate two virtual joysticks respectively to control the cursor on the controllable display screen to track the target point along the target trajectory line, and makes the cursor and the target point overlap as much as possible to form a tracking trajectory line; the task parameters of this step can be dynamically adjusted to avoid the resistance caused by repeated testing.

[0036] S602: Calculate the integral of the sum of the offsets between the tracking trajectory and the target trajectory to obtain an absolute offset error as the performance of the two-hand coordination test task. This step uses a computer to implement automated testing.

[0037] S603: Scoring the performance of multiple users on the bimanual coordination test task according to the law of large numbers serves as a marker for the bimanual coordination regression analysis. This step standardizes the scores based on the group data distribution and eliminates the impact of individual differences on the bimanual coordination assessment.

[0038] Furthermore, step S7 includes the following steps:

[0039] S701. The sight-reading behavior characteristics are combined with the physiological characteristics of the nervous system to obtain a feature data set. After dimensionality reduction processing is performed on the feature data set, the optimal feature subset is selected as input. The annotations of the bimanual coordination regression analysis corresponding to step S603 are used as output, and a machine learning model is applied to perform regression analysis. This step is based on feature layer fusion and combines dimensionality reduction technology to improve the generalization ability of the model.

[0040] S702: Tracking training is performed based on how the output of the machine learning model changes over time to obtain quantitative evaluation results of the effect of snare drum sight-reading on hand coordination. This step dynamically tracks changes in hand coordination by inputting multiple sight-reading data into the model, supporting verification of the effectiveness of snare drum sight-reading training.

[0041] The present invention has the following advantages and effects compared to the prior art:

[0042] (1) The present invention extracts musical score fragments from a snare drum score according to the measure, and displays the score title in an easily understandable animated form on a computer-controlled screen. The animation is generated based on the snare drum beat sequence, clearly and easily presenting the key information required for sight-reading, such as the drum beat, dynamics, and rhythm, thus solving the problem of difficulty in recognizing music in traditional music score performance. The wide variety of music score titles extracted based on computer technology solves the problem of lack of scalability and lack of interest caused by repeated titles in the evaluation process.

[0043] (2) The present invention incorporates physiological parameters into the process of evaluating the coordination of both hands, which is an objective, efficient, non-invasive and non-sensory method. It improves the problems of traditional evaluation methods such as the scale method and the task performance method, which lack objectivity and are limited by environmental factors. The multi-point PPG signal acquisition system adopts the "end-network-cloud" architecture. The computer network quickly responds to the data collected by the sensor and performs certain preprocessing. Combined with cloud computing technology, it can customize the snare drum sight-reading training program, solving the problem of lack of personalized solutions in the existing technology.

[0044] (3) Based on the multi-point PPG signal acquisition system, the present invention proposes a method for extracting physiological features from it by multi-channel fusion. The channel signal with the largest relative power in the pulse wave frequency band is selected as the reference signal. After the multi-point PPG signals are aligned, linear fusion is performed to obtain the pulse wave signal, which solves the problem that the single-point signal is susceptible to interference and leads to insufficient accuracy.

[0045] (4) The present invention proposes a method for regression analysis of bimanual coordination through feature layer fusion, which integrates sight-reading features with physiological features, solving the problem that existing evaluation methods fail to consider the influence of training and physiological factors; through tracking training, the influence of snare drum sight-reading on bimanual coordination is quantified in an objective way. The results of quantitative evaluation can provide a reference basis for the clinical application of bimanual coordination evaluation methods based on instrument training, and provide scientific and technological means for preventing neurological diseases and popularizing aesthetic education among the public, which is highly practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0047] Figure 1 1 is a flow chart of a quantitative method for evaluating the effect of snare drum sight-playing on hand coordination disclosed in an embodiment of the present invention;

[0048] Figure 2 It is an easy-to-understand animated diagram displayed on the screen in an embodiment of the present invention;

[0049] Figure 3 is a diagram of a process for sight-reading a snare drum in an embodiment of the present invention;

[0050] Figure 4 is a process diagram of drum beat detection in an embodiment of the present invention;

[0051] Figure 5 is a schematic diagram of a multi-point PPG signal acquisition system according to an embodiment of the present invention;

[0052] Figure 6 is a schematic diagram of a two-hand coordination test task in an embodiment of the present invention;

[0053] Figure 7 is a schematic diagram of a regression analysis method based on feature fusion in an embodiment of the present invention;

[0054] Figure 8 Schematic diagram of multi-channel signals after autocorrelation alignment in an embodiment of the present invention;

[0055] Figure 9 Schematic diagram of a PPG signal used for feature extraction after multi-channel fusion in an embodiment of the present invention;

[0056] Figure 10 3 is a reference diagram for objectively and quantitatively evaluating the effect of snare drum sight-playing on the coordination of both hands in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0058] Example 1

[0059] This embodiment discloses a quantitative method for evaluating the effect of snare drum sight-playing on the coordination of both hands. Figure 1 The specific steps are as follows:

[0060] S1, guiding the user to sight-read the snare drum in an animated form, and collecting audio and video signals of the sight-reading process;

[0061] S101, extracting a drum beat sequence from a small drum musical notation; distinguishing left and right hand drum beats, accents, rests, beat length, and other information according to the musical notation rules;

[0062] S102, displaying a drum beat sequence in the form of a picture on a controllable display screen; the drum beat picture is a rectangular picture, wherein the right-hand drum beat picture sequence is displayed at the top of the screen, and the left-hand drum beat picture sequence is displayed at the bottom of the screen, and a specific symbol is marked on the drum beat picture to indicate that the drum beat is an accent; a horizontal straight line is connected after the drum beat picture, and the total width of the drum beat picture and the horizontal straight line indicates the beat length of the drum beat to be played. If there is no horizontal straight line, it represents a rest, which is an empty beat that does not need to be played. The start time and beat length of each drum beat to be played are indicated by the movement of a vertical line; Figure 2 Shown is a schematic diagram of the aforementioned easy-to-understand animated form;

[0063] S103, using a controllable microphone and camera to collect audio and video signals during the snare drum sight-reading process; the user uses the snare drum sticks with both hands to hit the snare drum surface to sight-read the snare drum according to the guidance of the animation; the computer-controlled microphone and camera synchronously collect audio and video until the sight-reading ends. Figure 3 Shown is a diagram of the process of sight-reading the snare drum.

[0064] S2, extracting the time sequence of the user's drumming beats from the audio signal of the sight-reading process;

[0065] S201. Preprocess the audio signal of the sight-reading process. The preprocessing includes removing environmental noise, framing and windowing the audio, and then performing a short-time Fourier transform (STFT) to obtain a time-frequency diagram. The short-time Fourier transform X(m, k) of the audio signal x[n] is calculated as follows:

[0066]

[0067] Wherein, w[n] is a window function with a length of N. In this embodiment, a Hanning window is selected, m is a time step index, R is a window moving step, and k is a frequency index.

[0068] S202, performing drum beat detection on the pre-processed audio signal; performing a differential operation on each window of the time-frequency graph to obtain a spectral flux representing the volume value, and determining whether each window contains a drum beat using a threshold method; Figure 4 Shown is a process diagram of drum beat detection.

[0069] S203, extracting the time sequence of the drum beats of the user; obtaining all the drum beats in the audio signal through step S202, calculating the time corresponding to the drum beats according to the position of the drum beats in the window and the position of the window in the audio signal, and then obtaining the time sequence of the drum beats during the user's sight-reading process.

[0070] S3, extracting the time sequence of the user's left and right hands drumming from the video signal of the sight-reading process;

[0071] S301, inputting the video signal of the sight-reading process into an open source gesture recognition machine learning framework to extract the time series of changes in the bending angles of the user's left and right arms;

[0072] S302, detecting the user's behavior of drumming with both hands; detecting the bending angles of the user's left and right arms in each frame of the video, and using a threshold method to determine whether the user is drumming at this time;

[0073] S303, extracting the time sequence of the user's left and right hands beating the drum; obtaining the corresponding moments when the user's left and right hands beat the drum in the video signal through step S302, and after corresponding to the video time nodes, outputting the time sequence of the left and right hands beating the drum respectively.

[0074] S4. Combining the user's drum beat time sequence and the left and right hand drum beat time sequence, and comparing them with the standard drum beat sequence for sight-reading, extract the user's sight-reading behavior characteristics; the user's sight-reading behavior characteristics include time accuracy and force accuracy; time accuracy needs to be calculated based on the moments of the left and right hand drum beats, and force accuracy requires normalization of volume to eliminate individual differences; the user's sight-reading behavior characteristics extraction process is as follows:

[0075] S401, extracting the time accuracy of the user's snare drum sight-reading; aligning the user's drum beat time sequence obtained in step S2 with the standard drum beat sequence, and judging the accuracy in combination with the user's left and right hand drum beat time sequences; if the user's drum beat exists within the standard drum beat tolerance window, the tolerance window length is 100 milliseconds, and the hand used by the user is the same as that in the standard drum beat sequence, then the user's drum beat is correct; otherwise, it is incorrect; calculating the proportion of the number of correct user drum beats as the time accuracy of the user's snare drum sight-reading;

[0076] S402, extract the user's snare drum sight-reading force accuracy; normalize the user's drum beat sequence v[n] judged as correct in step S401, and normalize the sequence v norm [n] is calculated as follows:

[0077]

[0078] Where v max Indicates the maximum value of v[n], v min represents the minimum value of v[n], and takes 1.5 times the mean of the normalized volume as the accent threshold. A sound exceeding this threshold is considered correct, otherwise it is considered incorrect. The proportion of the number of correct accented drum beats is calculated as the velocity accuracy of the user's snare drum sight-reading.

[0079] S5. Measure the PPG signals of both hands through a multi-point PPG signal acquisition system, perform multi-channel fusion on the PPG signals, and extract physiological characteristics of the nervous system from them;

[0080] S501. Measure the PPG signals of the user's hands using a multi-point PPG signal acquisition system. When the user is in a resting state, the user places their left and right hands on the sensors of the multi-point PPG signal acquisition system respectively. The multi-point PPG signal acquisition system simultaneously acquires multi-channel PPG signals from both hands and uploads them to a host computer for processing. Figure 5 It is a schematic diagram of the multi-point PPG signal acquisition system.

[0081] S502, perform multi-channel fusion on the PPG signal; after performing a 7-layer wavelet transform on the PPG signal, set the approximate coefficient corresponding to the baseline drift to zero, and set the detail coefficient corresponding to the power frequency interference to zero, so as to remove the baseline drift and high-frequency noise and obtain a pure PPG signal.

[0082] For a sampling frequency of f s The PPG signal with a length of N is calculated by the Welch method to obtain the power spectral density P x (f), relative power P of pulse wave frequency band rel The calculation method is as follows:

[0083]

[0084] Where f1 and f2 in the numerator are the frequency ranges of the pulse wave band, f1 is 0.6Hz, and f2 is 4Hz; the denominator is the average power of the signal; Δf = f s / N is the frequency resolution. The channel signal with the largest relative power in the pulse wave frequency band is selected as the reference signal r[n].

[0085] Perform autocorrelation alignment on other single-channel signals and the reference signal. Suppose there are M signals with a channel length of N, where the signal of the mth channel is denoted as x m [n] (n = 1, 2, ..., N), and the cross-correlation function between the reference signal r[n] and any other channel signal is assumed to be R[τ], which is calculated as follows:

[0086]

[0087] Where τ represents the time shift, and the cross-correlation function R[τ] quantifies the time shift when x m The degree of similarity between the reference signal and the other channel signals when time-shifted by τ is applied to r[n]. A larger value of R[τ] indicates a higher similarity between the reference signal and the other channel signals when time-shifted by τ. The goal of signal alignment is to maximize the similarity between the compared signals by applying a time shift to the compared signals.

[0088] Then, the amplitude transformation is performed based on the least squares method; the goal of the least squares method is to minimize the sum of square errors between the other channel signals and the reference signal. The amplitude transformation of the other channel signals is performed, and the amplitude transformation coefficient of the mth channel is defined as α m , let the transformed signal α m x m [n] is as close as possible to the reference signal r[n], and the error function E(α m ) as shown in the following formula:

[0089]

[0090] The goal of the least squares method is to find the optimal coefficients corresponding to each channel Minimize the error function, so for each channel, let the error function be about α m The derivative is zero, find the optimal coefficient Such as the formula:

[0091]

[0092] Solve the equation to get the optimal coefficients The expression is as follows:

[0093]

[0094] Calculate the optimal coefficient After that, the amplitude of each channel is transformed, and then the arithmetic average of all channel signals is performed to obtain the unilateral PPG signal after multi-channel fusion The calculation method is as follows:

[0095]

[0096] From the left and right PPG signals, the side with relatively higher pulse wave frequency power is selected for feature extraction; Figure 9 Shown is the PPG signal after multi-channel fusion.

[0097] S503 : Extracting time domain features, frequency domain features, and nonlinear features from the PPG signal obtained in step S502 as physiological features of the nervous system.

[0098] S6. Conduct a two-hand coordination test under computer guidance to obtain the user's task performance of the two-hand coordination test;

[0099] S601. Use a computer to guide the user through a bimanual coordination test. This test (Bimanual Tracking Task) is based on authoritative neuropsychological research. The user sits in front of a controllable display and uses two virtual joysticks with their left and right hands, respectively, to control an on-screen cursor to track a target point along a target trajectory, trying to ensure that the cursor and the target point overlap as much as possible to form a tracking trajectory. Figure 6 Shown is a schematic diagram of the bimanual coordination test task;

[0100] S602: After the test, the user's bimanual coordination test task performance is obtained; the integral of the sum of the offsets between the tracking trajectory line and the target trajectory line is calculated to obtain the absolute offset error, which is used as the bimanual coordination test task performance;

[0101] S603: Scoring the performance of the two-hand coordination test task of multiple users according to the law of large numbers, and using the scores as labels for the two-hand coordination regression analysis.

[0102] S7. Integrate the user's snare drum sight-playing behavioral characteristics, nervous system physiological characteristics, and task performance to conduct a regression analysis of bimanual coordination. Through follow-up training, obtain quantitative evaluation results of the impact of snare drum sight-playing on bimanual coordination.

[0103] S701, combining sight-reading behavior features with nervous system physiological features; performing dimensionality reduction on the feature dataset, selecting the optimal feature subset as input, taking the corresponding hand coordination annotations in step S6 as output, and applying a machine learning model for regression analysis; Figure 7 The figure shows the principle diagram of the regression analysis method based on feature fusion; Figure 7 The physiological characteristics in the above are the physiological characteristics of the nervous system. Figure 7 The sight-reading features refer to the sight-reading behavior features.

[0104] S702. Quantitative evaluation of the effect of snare drum sight-reading on hand coordination.

[0105] Example 2

[0106] This embodiment further discloses the implementation process of a quantitative method for evaluating the effect of snare drum sight-playing on bimanual coordination. The specific steps are as follows:

[0107] S1, guiding the user to sight-read the snare drum in an animated form, and collecting audio and video signals of the sight-reading process;

[0108] S101, refer to the corresponding steps in Example 1, which will not be repeated here;

[0109] S102, refer to the corresponding steps in Example 1, which will not be repeated here;

[0110] S103, refer to the corresponding steps in Example 1, which will not be repeated here;

[0111] S2, extracting the time sequence of the user's drumming beats from the audio signal of the sight-reading process;

[0112] S201, refer to the corresponding steps in Example 1, which will not be repeated here;

[0113] S202: Refer to the corresponding steps in Example 1 and do not repeat them here;

[0114] S203, refer to the corresponding steps in Example 1, which will not be repeated here;

[0115] S3, extracting the time sequence of the user's left and right hands drumming from the video signal of the sight-reading process;

[0116] S301, refer to the corresponding steps in Example 1, which will not be repeated here;

[0117] S302: Refer to the corresponding steps in Example 1 and do not repeat them here;

[0118] S303, refer to the corresponding steps in Example 1, which will not be repeated here;

[0119] S4, combining the user's drum beat time sequence and the left and right hand drum beat time sequence, and comparing them with the standard drum beat sequence of sight-reading to extract the user's sight-reading behavior characteristics;

[0120] S401, refer to the corresponding steps in Example 1, which will not be repeated here;

[0121] S402: Refer to the corresponding steps in Example 1 and do not repeat them here;

[0122] S5. Measure the PPG signals of both hands through a multi-point PPG signal acquisition system, perform multi-channel fusion on the PPG signals, and extract physiological characteristics of the nervous system from them;

[0123] S501, refer to the corresponding steps in Example 1, which will not be repeated here;

[0124] S502: Refer to the corresponding steps in Example 1 and do not repeat them here; Figure 8 Shown are the multi-channel signals after autocorrelation alignment;

[0125] S503: Refer to the corresponding steps in Example 1 and will not be repeated here; Figure 9 Shown is the PPG signal used for feature extraction after multi-channel fusion.

[0126] S6. Conduct a two-hand coordination test under computer guidance to obtain the user's task performance of the two-hand coordination test;

[0127] S601, refer to the corresponding steps in Example 1, which will not be repeated here;

[0128] S602: Refer to the corresponding steps in Example 1 and do not repeat them here;

[0129] S603: Refer to the corresponding steps in Example 1 and do not repeat them here;

[0130] S7. Integrate the user's snare drum sight-playing behavioral characteristics, nervous system physiological characteristics, and task performance to conduct a regression analysis of bimanual coordination. Through follow-up training, obtain quantitative evaluation results of the impact of snare drum sight-playing on bimanual coordination.

[0131] S701, refer to the corresponding steps in Example 1, which will not be repeated here;

[0132] S702. Quantitatively evaluate the impact of snare drum sight-reading on bimanual coordination. The user sight-reads the snare drum for a certain number of days, each time inputting audio and video signals, as well as PPG signals collected by a multi-point PPG signal acquisition system, into the model. The changes in the model output over time can be used to quantitatively evaluate the impact of snare drum sight-reading on the user's bimanual coordination. Figure 10 Shown is a reference diagram for quantitatively evaluating the impact of snare drum sight-reading on hand coordination.

Claims

1. A quantitative method for evaluating the effect of snare drum sight-playing on hand coordination, characterized in that: The method comprises the following steps: S1, guiding the user to sight-read the snare drum in an animation form, and collecting audio and video signals of the sight-reading process; S2, extracting the time sequence of the user's drumming beats from the audio signal of the sight-reading process; S3, extracting the time sequence of the user's left and right hands beating the drum from the video signal of the sight-reading process; S4, comparing the time sequence of the drum beats and the time sequence of the left and right hand drum beats with a preset standard drum beat sequence to extract the user's sight-reading behavior characteristics; S5. Measure the PPG signals of the user's hands using a multi-point PPG signal acquisition system, and extract the user's nervous system physiological characteristics after performing multi-channel fusion on the PPG signals; S6. Conducting a two-hand coordination test on the user under computer guidance to obtain task performance of the two-hand coordination test on the user; S7. Perform a regression analysis on the user's two-hand coordination based on the sight-reading behavior characteristics, the nervous system physiological characteristics, and the task performance, and obtain a quantitative evaluation result of the effect of snare drum sight-reading on two-hand coordination through follow-up training.

2. The quantitative method for evaluating the effect of snare drum sight-reading on hand coordination according to claim 1, characterized in that: The step S1 comprises the following steps: S101, extracting a drum beat sequence from the small drum musical notation, and distinguishing the drum beats, accents, rests, and beat lengths of the left and right hands according to the musical notation rules; S102, displaying a drum beat sequence in the form of an animated drum beat picture on a controllable display screen; wherein the drum beat picture above the controllable display screen displays the drum beat sequence for the right hand, and the drum beat picture below the controllable display screen displays the drum beat sequence for the left hand; marking the accents of the drum beats on the drum beat pictures in the form of specific symbols; a horizontal straight line is connected after the drum beat pictures, and the total width of the drum beat picture and the horizontal straight line indicates the beat length of the drum beat to be played; if there is no horizontal straight line, it represents a rest, which is an empty beat that does not need to be played; S103: The user uses drumsticks with both hands to strike the snare drum surface according to the guidance of the animation to sight-read the snare drum; and uses a controllable microphone and camera to collect audio signals and video signals during the snare drum sight-reading process until the sight-reading is completed.

3. The quantitative method for evaluating the effect of snare drum sight-reading on hand coordination according to claim 1, characterized in that: The step S2 comprises the following steps: S201, pre-processing the audio signal of the sight-reading process; the pre-processing includes removing environmental noise, framing and windowing the audio, and performing short-time Fourier transform to obtain a time-frequency graph; S202, performing drum beat detection on the pre-processed audio signal; performing a differential operation on each window of the time-frequency graph to obtain a spectrum flux representing a volume value, and determining whether each window contains a drum beat using a threshold method; S203: For all drum beats in the audio signal obtained in step S202, the time corresponding to the drum beat is calculated according to the position of the drum beat in the window and the position of the window in the audio signal, thereby obtaining a time sequence of the drum beats during the user's sight-reading process.

4. The quantitative method for evaluating the effect of snare drum sight-reading on hand coordination according to claim 1, characterized in that: The step S3 comprises the following steps: S301, inputting the video signal of the sight-reading process into an open source gesture recognition machine learning framework to extract the time series of changes in the bending angles of the user's left and right arms; S302, detecting the user's behavior of drumming with both hands; detecting the bending angles of the user's left and right arms in each frame of the video, and using a threshold method to determine whether the user is drumming at this time; S303 , obtaining the corresponding moments when the user's left and right hands beat the drum in the video signal through step S302 , and outputting the time series of the left and right hands beating the drum respectively after corresponding them to the time nodes of the video.

5. The quantitative method for evaluating the effect of snare drum sight-reading on hand coordination according to claim 1, characterized in that: The sight-reading behavior characteristics of the user in step S4 include time accuracy and force accuracy; the time accuracy needs to be calculated in combination with the moments of the left and right hands hitting the drum, and the force accuracy needs to normalize the volume to eliminate individual differences; the process of extracting the sight-reading behavior characteristics of the user is as follows: S401, aligning the time sequence of the user's drum beats obtained in step S2 with a preset standard drum beat sequence, and evaluating the accuracy based on the time sequence of the user's left and right hand drum beats; Calculating the proportion of the number of correct drum beats struck by the user as the time accuracy of the user's snare drum sight-reading; S402. Normalize the volume of the correct drum beats struck by the user in step S401, and take 1.5 times the average of the normalized volume as the accent threshold. A drum beat exceeding the threshold is judged as a correct accent drum beat, otherwise it is an incorrect accent drum beat; calculate the proportion of the number of correct accent drum beats as the velocity accuracy of the user's snare drum sight-reading.

6. The quantitative method for evaluating the effect of snare drum sight-reading on hand coordination according to claim 1, characterized in that: The step S5 comprises the following steps: S501: The user, in a resting state, places his or her left and right hands on the sensors of the multi-point PPG signal acquisition system. The multi-point PPG signal acquisition system will simultaneously collect multi-channel PPG signals from both hands and upload them to the host computer for processing. S502: Use wavelet threshold denoising to remove baseline drift and high-frequency noise from the PPG signal to obtain a pure PPG signal; select the single-channel signal with the highest relative power in the pulse wave frequency band as the reference signal; perform autocorrelation alignment on the other single-channel signals and the reference signal, and then perform amplitude transformation based on the least squares method; perform arithmetic averaging on all channel signals to obtain a fused unilateral PPG signal; and select the PPG signal with higher signal quality from the left and right PPG signals for feature extraction; S503 , extracting time domain features, frequency domain features, and nonlinear features from the PPG signal obtained in step S502 as the physiological features of the nervous system.

7. The quantitative method for evaluating the effect of snare drum sight-reading on hand coordination according to claim 1, characterized in that: The step S6 comprises the following steps: S601: A user sits in front of a controllable display screen and uses their left and right hands to manipulate two virtual joysticks respectively to control a cursor on the controllable display screen to track a target point along a target trajectory line, and tries to make the cursor and the target point coincide with each other as much as possible to form a tracking trajectory line; S602, calculating the integral of the sum of the offsets between the tracking trajectory and the target trajectory to obtain an absolute offset error as the performance of the two-hand coordination test task; S603: Scoring the performance of the two-hand coordination test task of multiple users according to the law of large numbers as a label for the two-hand coordination regression analysis.

8. The quantitative method for evaluating the effect of snare drum sight-reading on hand coordination according to claim 1, characterized in that: The step S7 comprises the following steps: S701, combining the sight-reading behavior features with the physiological features of the nervous system to obtain a feature data set, performing dimensionality reduction processing on the feature data set, selecting an optimal feature subset as input, using the annotations of the bimanual coordination regression analysis corresponding to step S603 as output, and applying a machine learning model to perform regression analysis; S702. Perform tracking training based on how the output of the machine learning model changes over time to obtain a quantitative evaluation result of the effect of snare drum sight-reading on the coordination of both hands.

Images