A wood piano playing detection method based on photoelectric information

Abstract

This application relates to the field of piano teaching technology, and more particularly to a method for detecting the playing of an acoustic piano based on photoelectric information. It includes: installing sensors; sampling a reference playing process using the sensors to obtain a set of photoelectric reference signals; performing intensity and rhythm analysis on each photoelectric reference signal to obtain the photoelectric reference signal analysis results and establishing a playing detection model; sampling the actual playing process using the sensors to obtain a set of actual photoelectric signals; inputting the actual photoelectric signal set into the playing detection model, which uses the photoelectric reference signal set as the detection basis to perform intensity and rhythm detection on each actual photoelectric signal in the actual photoelectric signal set, and outputting the timing-based detection results of the piano keys during the playing process. The playing detection model of this application detects whether the piano keys were pressed accurately, and the intensity and rhythm of the corresponding actual photoelectric signals after the keys were pressed, thereby achieving comprehensive detection.

Description

Technical Field

[0001] This application relates to the field of piano teaching technology, and in particular to a method for detecting the playing of an acoustic piano based on photoelectric information. Background Technology

[0002] In the field of piano education, performers often find it difficult to quantitatively grasp elements such as the correctness, dynamics, and rhythm of their playing, whether practicing alone or in class. Therefore, some testing technologies have emerged on the market that use the piano keys to detect whether a performance is correct.

[0003] However, most products in the same field on the market focus on the devices used in digital pianos and electronic keyboards; while acoustic pianos, due to the special nature of their manufacturing process, make it difficult to add relevant testing devices. Even if such devices were to be added to acoustic pianos, the following defects would occur:

[0004] 1. The testing device is too closely coupled with the original mechanical structure of the acoustic piano. There is a risk of damaging the original mechanical structure when modifying the acoustic piano.

[0005] 2. Limited information elements collected. Current piano or electronic keyboard key detection technology can only provide feedback on the accuracy and rhythm of the sound, but cannot detect information elements such as dynamics and emotional tone. Summary of the Invention

[0006] This application provides a method for detecting the playing of a wooden piano based on photoelectric information, which can solve the problems of excessive coupling with the wooden piano and insufficient information elements collected by existing playing detection devices applied to wooden pianos.

[0007] The technical solution of this application is a method for detecting the playing of an acoustic piano based on photoelectric information. The method includes:

[0008] S1: Install sensors based on the keys in a wooden piano;

[0009] S2: Determine several target pieces, and based on a preset sampling period, use sensors to sample the comparison playing process when the target pieces are played correctly on the acoustic piano, and obtain a set of photoelectric comparison signals based on time sequence about the pressed keys during the comparison playing process of each target piece;

[0010] S3: Perform strength and rhythm analysis on each photoelectric reference signal in the photoelectric reference signal set to obtain the photoelectric reference signal analysis results, and establish a playing detection model based on several target pieces based on the photoelectric reference signal analysis results;

[0011] S4: Based on a preset sampling period, the actual playing process of the acoustic piano when the target piece is actually played is sampled by the sensor, and the actual photoelectric signal set of the pressed keys based on the time sequence is obtained during the actual playing process.

[0012] S5: Input the actual photoelectric signal set to the playing detection model. The playing detection model uses the photoelectric reference signal set as the detection basis to perform strength and rhythm detection on each actual photoelectric signal in the actual photoelectric signal set, and outputs the detection results of the piano keys based on the time sequence during the playing process.

[0013] Optionally, step S1 includes:

[0014] S11: Install sensors according to the keys in the acoustic piano, with two sensors corresponding to each key;

[0015] And, step S2 includes:

[0016] S21: Determine the target piece, and based on the preset sampling period, sample the comparison playing process when the target piece is played correctly on the acoustic piano using a sensor, and obtain an initial photoelectric comparison signal set based on the time sequence of the pressed keys during the comparison playing process, including several initial photoelectric comparison signals.

[0017] Each initial photoelectric reference signal includes: the initial photoelectric reference sub-signals obtained by the two sensors corresponding to the keys pressed during the playing process in a preset sampling period;

[0018] S22: Denoising, standardization and integration are performed sequentially on several initial photoelectric control sub-signals in the initial photoelectric control signal set to obtain several photoelectric control signals and a corresponding photoelectric control signal set;

[0019] And, step S4 includes:

[0020] S41: Based on a preset sampling period, the actual playing process of the acoustic piano when the target piece is actually played is sampled by the sensor, and an initial photoelectric actual signal set including several initial photoelectric actual signals is obtained based on the time sequence of the pressed keys during the actual playing process.

[0021] Each initial photoelectric actual signal includes: the initial photoelectric actual sub-signals obtained by the two sensors corresponding to the keys pressed during the playing process in the preset sampling period;

[0022] S42: Perform denoising, standardization and integration processing on several initial photoelectric actual sub-signals in the initial photoelectric actual signal set in sequence to obtain several photoelectric actual signals and the corresponding photoelectric actual signal set.

[0023] Optionally, step S22 includes:

[0024] S221: A denoising formula based on the wavelet transform principle, which denoises several initial photoelectric control sub-signals to obtain several denoised photoelectric control sub-signals and the corresponding denoised photoelectric control signal set;

[0025] The denoising formula based on the wavelet transform principle is shown below:

[0026] S(t)=f(t)+ke(t), t=0, 1, 2,...n;

[0027] In the formula, S(t) represents the initial photoelectric control sub-signal containing noise or the initial photoelectric actual sub-signal containing noise, f(t) represents the denoised photoelectric control sub-signal or the denoised photoelectric actual sub-signal, e(t) is the noise signal, k is the noise intensity, and t is the number of wavelet decomposition layers.

[0028] During the calculation, the wavelet transform principle is used to eliminate e(t) in the formula and reconstruct f(t) so that S(t) = f(t);

[0029] S222: Standardize the denoised photoelectric control sub-signals according to the standardization formula of photoelectric signals to obtain several standard photoelectric control sub-signals and the corresponding standard photoelectric control signal set;

[0030] The standardized formula for photoelectric signals is as follows:

[0031] f(t)'=(f(t)-max) / (max-min);

[0032] Where max represents the maximum value of the denoised photoelectric control sub-signal or the maximum value of the denoised photoelectric actual sub-signal, min represents the minimum value of the denoised photoelectric control sub-signal or the minimum value of the denoised photoelectric actual sub-signal; f(t)' represents the standard photoelectric control sub-signal or the standard photoelectric actual sub-signal;

[0033] S223: Integrate the two standard photoelectric reference sub-signals corresponding to each piano key to obtain several photoelectric reference signals and the corresponding photoelectric reference signal set.

[0034] Optionally, step S223 includes:

[0035] S2231: Determine the two standard photoelectric control sub-signals corresponding to each piano key, and calculate the correlation between the two standard photoelectric control sub-signals corresponding to each piano key to obtain the correlation matrix;

[0036] The formula for calculating the correlation matrix is ​​shown below:

[0037]

[0038] In the formula, d12 represents the correlation matrix between two standard photoelectric control sub-signals corresponding to each piano key or the correlation matrix between two standard photoelectric actual sub-signals corresponding to each piano key;

[0039] x 1k and x 2k These represent either the two standard photoelectric reference sub-signals corresponding to each piano key, or the two standard photoelectric actual sub-signals corresponding to each piano key.

[0040] k represents the kth key, and n represents the total number of keys;

[0041] S2232: Based on the correlation matrix, establish the corresponding covariance matrix, and normalize the covariance matrix to obtain the absolute value of the covariance.

[0042] S2233: Determine the absolute value threshold of the covariance as 0.8, and compare the absolute value of the covariance with 0.8. If it is greater than 0.8, merge the two standard photoelectric reference signals corresponding to each piano key to obtain the photoelectric reference signal.

[0043] If it is less than 0.8, calculate the fusion value of the two standard photoelectric reference signals corresponding to each piano key, and use the fusion value as the photoelectric reference signal;

[0044] The formula for calculating the fusion value is shown below:

[0045] U = min(U1, U2);

[0046] U1 = A1 + (a2 - a1)(a2 - A1) T U2 = A2 + (a1 - a2)(a1 - A2) T ;

[0047] In the formula, U represents the fusion value of the two standard photoelectric reference sub-signals corresponding to each piano key, or the fusion value of the two standard photoelectric actual sub-signals corresponding to each piano key;

[0048] (a1, A1) and (a2, A2) represent the mean and variance of the two standard photoelectric control sub-signals corresponding to each piano key, or the mean and variance of the two standard photoelectric actual sub-signals corresponding to each piano key, respectively.

[0049] U1 and U2 represent the estimated values ​​of the two standard photoelectric control sub-signals corresponding to each piano key, or the estimated values ​​of the two standard photoelectric actual sub-signals corresponding to each piano key.

[0050] Optionally, step S5 includes:

[0051] S51: Input the actual photoelectric signal set into the playing detection model. The playing detection model uses a preset sampling period as the mapping basis between the photoelectric reference signal set and the actual photoelectric signal set, determines the similarity between each photoelectric reference signal and the corresponding actual photoelectric signal, and obtains several similarity values ​​respectively.

[0052] S52: Determine the similarity threshold, and compare the similarity value with the similarity threshold respectively. If the similarity value is greater than the similarity threshold, confirm that the photoelectric actual signal corresponding to the similarity value is the correct playing signal, and correspondingly confirm that the key pressed during the playing process is correct.

[0053] S53: Repeat step S52 until all similarity values ​​have been compared with the similarity threshold, and output the detection results of correctly pressed keys and incorrectly pressed keys based on the time sequence during the playing process.

[0054] Optionally, step S51 includes:

[0055] S551: Input the actual photoelectric signal set to the playing detection model. The playing detection model uses a preset sampling period as the mapping basis between the photoelectric reference signal set and the actual photoelectric signal set, and determines the similarity of the strength features and rhythm features between each photoelectric reference signal and the corresponding actual photoelectric signal.

[0056] S552: Based on the similarity of strength features and rhythm features between each photoelectric reference signal and the corresponding actual photoelectric signal, determine the comprehensive similarity index between each photoelectric reference signal and the corresponding actual photoelectric signal, and obtain several similarity values ​​respectively;

[0057] The formula for calculating the comprehensive similarity index is as follows:

[0058] Y = Y Amp ω1+Y Beat ω2;

[0059] In the formula, Y Amp Y represents the similarity between strong and weak features. Best Let ω1 and ω2 represent the similarity of rhythmic features, respectively, and let ω1 ≥ ω2.

[0060] If Y≥0.75, it is confirmed that the actual photoelectric signal corresponding to the comprehensive similarity index is the correct playing signal, and correspondingly, it is confirmed that the key pressed during the playing process is correct;

[0061] If Y < 0.75, it is confirmed that the actual photoelectric signal corresponding to the comprehensive similarity index is an incorrect playing signal, and correspondingly, it is confirmed that the key pressed during the playing process is incorrect.

[0062] Optionally, the formula for calculating the similarity between strong and weak features is as follows:

[0063]

[0064]

[0065] In the formula, x pp Indicates the peak-to-peak value of the photoelectric contrast signal; x represents the mean of all photoelectric reference signals in the photoelectric reference signal set; Amp Characteristic values ​​representing the strength of actual photoelectric signals; z represents the mean of the strength characteristic values ​​of all actual photoelectric signals in the photoelectric signal set; Amp It represents the standard deviation of the strength characteristic values ​​of all photoelectric reference signals in the photoelectric reference signal set.

[0066] Optionally, the formula for calculating the similarity of the rhythmic features is:

[0067]

[0068] X Beat =ω;

[0069] In the formula, ω represents the peak-to-peak frequency of the photoelectric control signal per unit time; x Beat Represents the rhythmic characteristic value of the actual photoelectric signal; The mean value of the rhythmic characteristic values ​​of all photoelectric control signals in the photoelectric control signal set; z Beat It represents the standard deviation of the rhythmic characteristic values ​​of all photoelectric control signals in the photoelectric control signal set.

[0070] Optionally, the playing detection model includes: flashing lights of different colors, used to display different colors based on whether the piano key is pressed correctly or incorrectly;

[0071] The method includes:

[0072] S6: Play back the target track and display the detection results based on the time sequence using flashing lights of different colors during the playback process.

[0073] Beneficial effects:

[0074] This application uses photoelectric sensors installed at the keys in a wooden piano, thus eliminating the need for excessive coupling with the piano's mechanical structure. The photoelectric sensors only need to be installed at locations where relevant key information can be accurately obtained.

[0075] Furthermore, this application accurately detects the actual playing process by setting up a set of photoelectric reference signals and establishing a playing detection model. In addition to detecting whether the piano keys are pressed accurately, it can also detect the strength and rhythm of the corresponding photoelectric signals after the keys are pressed, thus achieving comprehensive detection and ensuring the detection effect.

[0076] In summary, this application can solve the problems of excessive coupling with the acoustic piano and insufficient information elements collected in existing playing detection devices applied to acoustic pianos. Attached Figure Description

[0077] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 This is a flowchart illustrating a method for detecting the playing of a wooden piano based on photoelectric information, as described in an embodiment of this application. Detailed Implementation

[0079] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0080] This application provides a method for detecting the playing of an acoustic piano based on photoelectric information, such as... Figure 1 As shown, Figure 1 This is a flowchart illustrating a method for detecting acoustic piano playing based on photoelectric information, as described in an embodiment of this application, including:

[0081] S1: Install sensors based on the keys in a wooden piano.

[0082] Step S1 includes:

[0083] S11: Install sensors according to the keys in the acoustic piano, with two sensors corresponding to each key.

[0084] Specifically, multiple photoelectric sensors are installed on the thick plush support of the acoustic piano to collect photoelectric signals generated by pressing the keys during piano playing. Each photoelectric sensor is configured to correspond to a key in the acoustic piano, meaning each sensor collects the photoelectric signal generated by one key. In this embodiment, to improve the accuracy of the collected signals, each photoelectric sensor contains two photoelectric sensor units, and these two units simultaneously collect the photoelectric signal corresponding to one key; that is, in each sampling period, two photoelectric signals are collected for the same key.

[0085] S2: Determine several target pieces and, based on a preset sampling period, sample the comparison playing process of the acoustic piano when the target pieces are played correctly using a sensor, to obtain a set of photoelectric comparison signals based on time sequence about the pressed keys during the comparison playing process of each target piece.

[0086] Specifically, the photoelectric comparison signal is transmitted to the FPGA platform through the communication module, and the photoelectric comparison signal is preprocessed and integrated to obtain the photoelectric comparison signal set.

[0087] The communication module uses a BT04-E Bluetooth module, which is installed on the photoelectric sensor circuit board and the FPGA platform respectively, and transmits data through the Bluetooth serial port protocol.

[0088] The FPGA platform uses the XILINX ZYNQ-7020 chip to carry out photoelectric information preprocessing algorithms.

[0089] The preset sampling period is set to 0.25 seconds, which is intended to ensure the accuracy and response speed of the photoelectric sensor.

[0090] Step S2 includes:

[0091] S21: Determine the target piece, and based on the preset sampling period, sample the comparison playing process when the target piece is played correctly on the acoustic piano using a sensor, and obtain an initial photoelectric comparison signal set based on the time sequence of the pressed keys during the comparison playing process, including several initial photoelectric comparison signals.

[0092] Each initial photoelectric reference signal includes: the initial photoelectric reference sub-signals obtained by the two sensors corresponding to the keys pressed during the playing process within a preset sampling period.

[0093] S22: Denoising, standardization and integration are performed sequentially on several initial photoelectric reference sub-signals in the initial photoelectric reference signal set to obtain several photoelectric reference signals and a corresponding photoelectric reference signal set.

[0094] Step S22 includes:

[0095] S221: A denoising formula based on the wavelet transform principle is used to denoise several initial photoelectric control sub-signals to obtain several denoised photoelectric control sub-signals and the corresponding denoised photoelectric control signal set.

[0096] The denoising formula based on the wavelet transform principle is shown below:

[0097] S(t)=f(t)+ke(t), t=0, 1, 2,...n.

[0098] In the formula, S(t) represents the initial photoelectric control sub-signal containing noise or the initial photoelectric actual sub-signal containing noise, f(t) represents the denoised photoelectric control sub-signal or the denoised photoelectric actual sub-signal, e(t) is the noise signal, k is the noise intensity, and t is the number of wavelet decomposition layers.

[0099] During the calculation, the wavelet transform principle is used to eliminate e(t) in the formula and reconstruct f(t) so that S(t) = f(t).

[0100] Specifically, the denoising formula denoises the initial photoelectric control signal containing noise to obtain the true denoised photoelectric control signal.

[0101] In this embodiment, the initial photoelectric control sub-signal acquired is regarded as S(t) using the wavelet transform method, which includes the real denoised photoelectric control sub-signal f(t) and the noise signal e(t).

[0102] The denoising formula based on the wavelet transform principle is shown below:

[0103] S(t)=f(t)+ke(t), t=0,1,2,...n.

[0104] Wavelet transform can be used to eliminate e(t) in the formula to reconstruct the real photoelectric signal f(t) so that S(t) = f(t).

[0105] The basic principle is that wavelet transform has different properties in the normal band and white noise. By using different decomposition coefficients, the denoised photoelectric control signal can be distinguished from the noise signal.

[0106] The specific steps are as follows:

[0107] A1: Select appropriate orthogonal wavelets and wavelet layers, perform orthogonal wavelet transform on the original signal (real photoelectric signal) to obtain the wavelet decomposition coefficients w for each dimension;

[0108] A2: Perform nonlinear thresholding on the wavelet transform coefficients of the measured signal; process the high-frequency coefficients of each layer from the 1st to the Nth layer using a threshold function, and leave the low-frequency coefficients of each layer unprocessed.

[0109] A3: Reconstruct the processed wavelet coefficients. Reconstruct the signal based on the low-frequency coefficients of the Nth layer of wavelet decomposition and the processed high-frequency coefficients from the 1st to the Nth layer to obtain an estimate of the original signal (the real photoelectric signal).

[0110] S222: Standardize the denoised photoelectric reference sub-signals according to the standardization formula of photoelectric signals to obtain several standard photoelectric reference sub-signals and the corresponding standard photoelectric reference signal set.

[0111] The standardized formula for photoelectric signals is as follows:

[0112] f(t)'=(f(t)-max) / (max-min).

[0113] Where max represents the maximum value of the denoised photoelectric control sub-signal or the maximum value of the denoised photoelectric actual sub-signal, and min represents the minimum value of the denoised photoelectric control sub-signal or the minimum value of the denoised photoelectric actual sub-signal. f(t)' represents the standard photoelectric control sub-signal or the standard photoelectric actual sub-signal.

[0114] S223: Integrate the two standard photoelectric reference sub-signals corresponding to each piano key to obtain several photoelectric reference signals and the corresponding photoelectric reference signal set.

[0115] Step S223 includes:

[0116] S2231: Determine the two standard photoelectric reference signals corresponding to each piano key, and calculate the correlation between the two standard photoelectric reference signals corresponding to each piano key to obtain the correlation matrix.

[0117] The formula for calculating the correlation matrix is ​​shown below:

[0118]

[0119] In the formula, d 12 This represents the correlation matrix between two standard photoelectric control sub-signals corresponding to each piano key, or the correlation matrix between two standard photoelectric actual sub-signals corresponding to each piano key.

[0120] x 1k and x 2k These represent either the two standard photoelectric reference sub-signals corresponding to each piano key, or the two standard photoelectric actual sub-signals corresponding to each piano key.

[0121] k represents the k-th key, and n represents the total number of keys.

[0122] S2232: Based on the correlation matrix, establish the corresponding covariance matrix, and normalize the covariance matrix to obtain the absolute value of the covariance.

[0123] S2233: Determine the absolute value threshold of the covariance as 0.8, and compare the absolute value of the covariance with 0.8. If it is greater than 0.8, merge the two standard photoelectric reference signals corresponding to each piano key to obtain the photoelectric reference signal.

[0124] If it is less than 0.8, calculate the fusion value of the two standard photoelectric reference signals corresponding to each piano key, and use the fusion value as the photoelectric reference signal.

[0125] The formula for calculating the fusion value is shown below:

[0126] U = min(U1, U2).

[0127] U1 = A1 + (a2 - a1)(a2 - A1) T U2 = A2 + (a1 - a2)(a1 - A2) T .

[0128] In the formula, U represents the fusion value of the two standard photoelectric reference sub-signals for each key, or the fusion value of the two standard photoelectric actual sub-signals for each key.

[0129] (a1, A1) and (a2, A2) represent the mean and variance of the two standard photoelectric control sub-signals corresponding to each piano key, or the mean and variance of the two standard photoelectric actual sub-signals corresponding to each piano key.

[0130] U1 and U2 represent the estimated values ​​of the two standard photoelectric control sub-signals corresponding to each piano key, or the estimated values ​​of the two standard photoelectric actual sub-signals corresponding to each piano key.

[0131] Specifically, the standard photoelectric reference sub-signals are integrated through the above integration process, and two standard photoelectric reference sub-signals corresponding to the same piano key in the same sampling period are integrated into one photoelectric reference signal. The photoelectric reference signal in the integrated photoelectric reference signal set has a one-to-one correspondence with the piano key.

[0132] S3: Perform strength and rhythm analysis on each photoelectric reference signal in the photoelectric reference signal set to obtain the photoelectric reference signal analysis results, and establish a playing detection model based on several target pieces based on the photoelectric reference signal analysis results.

[0133] The playing detection model includes flashing lights of different colors, used to display different colors based on whether the keys are pressed correctly or incorrectly.

[0134] Furthermore, the playing detection model is loaded into the FPGA platform before being applied.

[0135] S4: Based on a preset sampling period, the actual playing process of the acoustic piano when the target piece is actually played is sampled by the sensor, and the actual photoelectric signal set of the pressed keys based on the time sequence is obtained during the actual playing process.

[0136] Step S4 includes:

[0137] S41: Based on a preset sampling period, the actual playing process of the acoustic piano when the target piece is actually played is sampled by the sensor, and an initial photoelectric actual signal set including several initial photoelectric actual signals is obtained based on the time sequence of the pressed keys during the actual playing process.

[0138] Each initial photoelectric actual signal includes: the initial photoelectric actual sub-signals obtained by the two sensors corresponding to the keys pressed during the playing process within a preset sampling period.

[0139] S42: Perform denoising, standardization and integration processing on several initial photoelectric actual sub-signals in the initial photoelectric actual signal set in sequence to obtain several photoelectric actual signals and the corresponding photoelectric actual signal set.

[0140] Specifically, the denoising, standardization, and integration processes performed on the initial photoelectric actual sub-signal can all refer to the processing of the initial photoelectric reference sub-signal to the photoelectric control signal, and the corresponding formulas can also be applied, which will not be elaborated here.

[0141] The final result of the processing is a set of actual photoelectric signals. This set includes several actual photoelectric signals, each corresponding one-to-one with a piano key.

[0142] S5: Input the actual photoelectric signal set to the playing detection model. The playing detection model uses the photoelectric reference signal set as the detection basis to detect the actual photoelectric signal set and outputs the detection results of correctly pressed keys and incorrectly pressed keys based on the timing during the playing process.

[0143] Specifically, the playing detection model uses the photoelectric reference signal set as the detection basis to detect the actual photoelectric signal set, determines the actual photoelectric signal corresponding to the piano key, and then transmits the actual photoelectric signal corresponding to the piano key to the host computer through the communication module.

[0144] Step S5 includes:

[0145] S51: Input the actual photoelectric signal set into the playing detection model. The playing detection model uses a preset sampling period as the mapping basis between the photoelectric reference signal set and the actual photoelectric signal set, determines the similarity between each photoelectric reference signal and the corresponding actual photoelectric signal, and obtains several similarity values ​​respectively.

[0146] Step S51 includes:

[0147] S551: Input the actual photoelectric signal set into the playing detection model. The playing detection model uses a preset sampling period as the mapping basis between the photoelectric reference signal set and the actual photoelectric signal set, and determines the similarity of the strength features and rhythm features between each photoelectric reference signal and the corresponding actual photoelectric signal.

[0148] Among them, 1) the formula for calculating the similarity between strong and weak features is as follows:

[0149]

[0150]

[0151] In the formula, x pp This indicates the peak-to-peak value of the photoelectric contrast signal. This represents the mean of all photoelectric reference signals in the photoelectric reference signal set. Amp It represents the strength characteristic value of the actual photoelectric signal. This represents the mean of the strength characteristic values ​​of all actual photoelectric signals in the photoelectric signal set. Amp It represents the standard deviation of the strength characteristic values ​​of all photoelectric reference signals in the photoelectric reference signal set.

[0152] 2) The formula for calculating the similarity of rhythmic features is:

[0153]

[0154] X Beat =ω.

[0155] In the formula, ω represents the peak-to-peak frequency of the photoelectric reference signal per unit time. Beat This represents the rhythmic characteristic value of the actual photoelectric signal. This represents the mean of the rhythmic characteristic values ​​of all photoelectric reference signals in the photoelectric reference signal set. Beat It represents the standard deviation of the rhythmic characteristic values ​​of all photoelectric control signals in the photoelectric control signal set.

[0156] S552: Based on the similarity of strength features and rhythm features between each photoelectric reference signal and the corresponding actual photoelectric signal, determine the comprehensive similarity index between each photoelectric reference signal and the corresponding actual photoelectric signal, and obtain several similarity values ​​respectively.

[0157] The formula for calculating the comprehensive similarity index is as follows:

[0158] Y = Y Amp ω1+Y Beat ω2.

[0159] In the formula, Y Amp Y represents the similarity between strong and weak features. Beat Let ω1 and ω2 represent the similarity of rhythmic features, respectively, and let ω1 ≥ ω2.

[0160] If Y≥0.75, it confirms that the actual photoelectric signal corresponding to the comprehensive similarity index is the correct playing signal, and correspondingly confirms that the key pressed during the playing process is correct.

[0161] If Y < 0.75, it is confirmed that the actual photoelectric signal corresponding to the comprehensive similarity index is an incorrect playing signal, and correspondingly, it is confirmed that the key pressed during the playing process is incorrect.

[0162] S52: Determine the similarity threshold and compare the similarity value with the similarity threshold respectively. If the similarity value is greater than the similarity threshold, confirm that the actual photoelectric signal corresponding to the similarity value is the correct playing signal, and correspondingly confirm that the key pressed during the playing process is correct.

[0163] S53: Repeat step S52 until all similarity values ​​have been compared with the similarity threshold. The output includes the detection results of correctly pressed keys and incorrectly pressed keys based on the time sequence during the playing process.

[0164] S6: Play back the target track and display the detection results based on the time sequence using flashing lights of different colors during the playback process.

[0165] Specifically, during playback, different colored LEDs flash to indicate whether the playing was correct or incorrect in each sampling cycle. The specific process involves building SPI and DMA peripherals on the FPGA, controlling the high and low level ratios to be 40% and 60% respectively.

[0166] The host computer plays back the piece of music played, and the FPGA platform transmits the judgment results in the form of a data packet to the light strip through the communication module. The different flashing colors of the light strip reflect the correctness of the piano notes. At the same time, the host computer displays the detection results on the terminal scoring interface.

[0167] During the display of the light strip, if the performer plays incorrectly, the light strip will display red during the corresponding playback; if the performance is correct, the light strip will display green during the corresponding playback, according to the above process.

[0168] The embodiments of this application have been described in detail above, but the content is only a preferred embodiment of this application and should not be considered as limiting the scope of this application. All equivalent changes and improvements made within the scope of this application should still fall within the patent coverage of this application.

Claims

1. A method for detecting the playing of a wooden piano based on photoelectric information, characterized in that, The method includes: S1: Install sensors based on the keys in a wooden piano; S2: Determine several target pieces, and based on a preset sampling period, use sensors to sample the comparison playing process when the target pieces are played correctly on the acoustic piano, and obtain a set of photoelectric comparison signals based on time sequence about the pressed keys during the comparison playing process of each target piece; S3: Perform strength and rhythm analysis on each photoelectric reference signal in the photoelectric reference signal set to obtain the photoelectric reference signal analysis results, and establish a playing detection model based on several target pieces based on the photoelectric reference signal analysis results; S4: Based on a preset sampling period, the actual playing process of the acoustic piano when the target piece is actually played is sampled by the sensor, and the actual photoelectric signal set of the pressed keys based on the time sequence is obtained during the actual playing process. S5: Input the actual photoelectric signal set to the playing detection model. The playing detection model uses the photoelectric reference signal set as the detection basis to perform strength and rhythm detection on each actual photoelectric signal in the actual photoelectric signal set, and outputs the detection results of the piano keys based on the time sequence during the playing process. Step S5 includes: S51: Input the actual photoelectric signal set into the playing detection model. The playing detection model uses a preset sampling period as the mapping basis between the photoelectric reference signal set and the actual photoelectric signal set, determines the similarity between each photoelectric reference signal and the corresponding actual photoelectric signal, and obtains several similarity values ​​respectively. S52: Determine the similarity threshold, and compare the similarity value with the similarity threshold respectively. If the similarity value is greater than the similarity threshold, confirm that the photoelectric actual signal corresponding to the similarity value is the correct playing signal, and correspondingly confirm that the key pressed during the playing process is correct. S53: Repeat step S52 until all similarity values ​​have been compared with the similarity threshold, and output the detection results of correctly pressed keys and incorrectly pressed keys based on the time sequence during the playing process. Step S51 includes: S551: Input the actual photoelectric signal set to the playing detection model. The playing detection model uses a preset sampling period as the mapping basis between the photoelectric reference signal set and the actual photoelectric signal set, and determines the similarity of the strength features and rhythm features between each photoelectric reference signal and the corresponding actual photoelectric signal. S552: Based on the similarity of strength features and rhythm features between each photoelectric reference signal and the corresponding actual photoelectric signal, determine the comprehensive similarity index between each photoelectric reference signal and the corresponding actual photoelectric signal, and obtain several similarity values ​​respectively; The formula for calculating the comprehensive similarity index is as follows: ； In the formula, Indicates the similarity between strong and weak features. Indicates the similarity of rhythmic features. and These represent the weight coefficients for similarity in strong and weak features and the weight coefficients for similarity in rhythmic features, respectively. ; if Y ≥0.75 confirms that the actual photoelectric signal corresponding to the comprehensive similarity index is the correct playing signal, and correspondingly confirms that the key pressed during the playing process is correct; if Y <0.75, confirming that the actual photoelectric signal corresponding to the comprehensive similarity index is an incorrect playing signal, and correspondingly confirming that the key pressed during the playing process is incorrect.

2. The method for detecting acoustic piano playing based on photoelectric information according to claim 1, characterized in that, Step S1 includes: S11: Install sensors according to the keys in the acoustic piano, with two sensors corresponding to each key; And, step S2 includes: S21: Determine several target pieces, and based on a preset sampling period, use sensors to sample the comparison playing process when the target pieces are played correctly on the acoustic piano, and obtain an initial photoelectric comparison signal set based on time sequence about the pressed keys during the comparison playing process of each target piece, including several initial photoelectric comparison signals. Each initial photoelectric reference signal includes: the initial photoelectric reference sub-signals obtained by the two sensors corresponding to the keys pressed during the playing process in a preset sampling period; S22: Denoising, standardization and integration are performed sequentially on several initial photoelectric control sub-signals in the initial photoelectric control signal set to obtain several photoelectric control signals and a corresponding photoelectric control signal set; And, step S4 includes: S41: Based on a preset sampling period, the actual playing process of the acoustic piano when the target piece is actually played is sampled by the sensor, and an initial photoelectric actual signal set including several initial photoelectric actual signals is obtained based on the time sequence of the pressed keys during the actual playing process. Each initial photoelectric actual signal includes: the initial photoelectric actual sub-signals obtained by the two sensors corresponding to the keys pressed during the playing process in the preset sampling period; S42: Perform denoising, standardization and integration processing on several initial photoelectric actual sub-signals in the initial photoelectric actual signal set in sequence to obtain several photoelectric actual signals and the corresponding photoelectric actual signal set.

3. The method for detecting acoustic piano playing based on photoelectric information according to claim 2, characterized in that, Step S22 includes: S221: A denoising formula based on the wavelet transform principle, which denoises several initial photoelectric control sub-signals to obtain several denoised photoelectric control sub-signals and the corresponding denoised photoelectric control signal set; The denoising formula based on the wavelet transform principle is shown below: S(t)=f(t)+ke(t), t=0, 1, 2,...n; In the formula, S(t) represents the initial photoelectric control sub-signal containing noise or the initial photoelectric actual sub-signal containing noise, f(t) represents the denoised photoelectric control sub-signal or the denoised photoelectric actual sub-signal, e(t) is the noise signal, k is the noise intensity, and t is the number of wavelet decomposition layers. During the calculation, the wavelet transform principle is used to eliminate e(t) in the formula and reconstruct f(t) so that S(t) = f(t); S222: Standardize the denoised photoelectric control sub-signals according to the standardization formula of photoelectric signals to obtain several standard photoelectric control sub-signals and the corresponding standard photoelectric control signal set; The standardized formula for photoelectric signals is as follows: f(t)'=(f(t)-max) / (max-min); Where max represents the maximum value of the denoised photoelectric control sub-signal or the maximum value of the denoised photoelectric actual sub-signal, min represents the minimum value of the denoised photoelectric control sub-signal or the minimum value of the denoised photoelectric actual sub-signal; f(t)' represents the standard photoelectric control sub-signal or the standard photoelectric actual sub-signal; S223: Integrate the two standard photoelectric reference sub-signals corresponding to each piano key to obtain several photoelectric reference signals and the corresponding photoelectric reference signal set.

4. The method for detecting acoustic piano playing based on photoelectric information according to claim 3, characterized in that, Step S223 includes: S2231: Determine the two standard photoelectric control sub-signals corresponding to each piano key, and calculate the correlation between the two standard photoelectric control sub-signals corresponding to each piano key to obtain the correlation matrix; The formula for calculating the correlation matrix is ​​shown below: ； In the formula, This represents the correlation matrix between two standard photoelectric control sub-signals corresponding to each piano key, or the correlation matrix between two standard photoelectric actual sub-signals corresponding to each piano key. and These represent either the two standard photoelectric reference sub-signals corresponding to each piano key, or the two standard photoelectric actual sub-signals corresponding to each piano key. Indicates the first One piano key, Indicates the total number of piano keys; S2232: Based on the correlation matrix, establish the corresponding covariance matrix, and normalize the covariance matrix to obtain the absolute value of the covariance. S2233: Determine the absolute value threshold of the covariance as 0.8, and compare the absolute value of the covariance with 0.

8. If it is greater than 0.8, merge the two standard photoelectric reference signals corresponding to each piano key to obtain the photoelectric reference signal. If it is less than 0.8, calculate the fusion value of the two standard photoelectric reference signals corresponding to each piano key, and use the fusion value as the photoelectric reference signal; The formula for calculating the fusion value is shown below: ； ， ； In the formula, U represents the fusion value of the two standard photoelectric reference sub-signals corresponding to each piano key, or the fusion value of the two standard photoelectric actual sub-signals corresponding to each piano key; , These represent the mean and variance of the two standard photoelectric control sub-signals corresponding to each piano key, or the mean and variance of the two standard photoelectric actual sub-signals corresponding to each piano key. and These represent the estimated values ​​of the two standard photoelectric control sub-signals corresponding to each piano key, or the estimated values ​​of the two standard photoelectric actual sub-signals corresponding to each piano key.

5. The method for detecting acoustic piano playing based on photoelectric information according to claim 1, characterized in that, The formula for calculating the similarity between strong and weak features is as follows: ； ； In the formula, Indicates the peak-to-peak value of the photoelectric contrast signal; This represents the mean of all photoelectric control signals in the photoelectric control signal set; Characteristic values ​​representing the strength of actual photoelectric signals; It represents the mean of the strength characteristic values ​​of all actual photoelectric signals in the actual photoelectric signal set; It represents the standard deviation of the strength characteristic values ​​of all photoelectric reference signals in the photoelectric reference signal set.

6. The method for detecting acoustic piano playing based on photoelectric information according to claim 5, characterized in that, The formula for calculating the similarity of the rhythmic features is: ； ； In the formula, This indicates the peak-to-peak frequency of the photoelectric reference signal per unit time. Represents the rhythmic characteristic value of the actual photoelectric signal; This represents the mean of the rhythmic characteristic values ​​of all photoelectric control signals in the photoelectric control signal set; It represents the standard deviation of the rhythmic characteristic values ​​of all photoelectric reference signals in the photoelectric reference signal set.

7. The method for detecting acoustic piano playing based on photoelectric information according to claim 1, characterized in that, The playing detection model includes: flashing lights of different colors, used to display different colors based on whether the keys are pressed correctly or incorrectly; The method includes: S6: Play back the target track and display the detection results based on the time sequence using flashing lights of different colors during the playback process.

Images