Tone identification-based practice timing method, terminal and system

Abstract

According to the practice timing method, terminal and system based on timbre identification, real-time audio acquisition is carried out at a preset frequency, and the acquired audio data is transmitted to a timbre identification module; the timbre recognition module sets different characteristic parameters for different musical instruments based on a convolutional neural network so as to judge whether the musical instruments are in a musical instrument practicing state or not; according to the recognition result of the tone recognition module, in combination with a preset time threshold value used for judging violin practice interruption, counting each section of violin practice time; according to the invention, the tone recognition module based on the neural network model is constructed, the audio is collected in real time to judge whether the user is in the piano practice state or not, then the piano practice time is counted, meanwhile, whether piano practice is interrupted or not is judged in combination with the time threshold value during counting, jamming caused by insufficient proficiency during piano practice of the user is avoided, and the user experience is improved. Therefore, the timing supervision of the piano practice condition of the children can be effectively and accurately realized.

Description

Technical Field

[0001] The invention relates to the field of piano practice timing, and in particular to a practice timing method, terminal and system based on timbre recognition. Background Art

[0002] In order to promote children's creativity and musical literacy, as well as to cultivate their elegant aesthetics and temperament, more and more parents are introducing their children to learn musical instruments, such as playing the piano, guzheng, and violin. However, due to lack of experience and other reasons, parents often find it difficult to effectively supervise their children's piano practice. This may lead to inefficiency in piano practice for children who lack self-discipline, affecting their learning progress. Therefore, there is an urgent need for a method and equipment that can help parents supervise their children's piano practice at any time and any place. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a practice timing method, terminal and system based on timbre recognition to realize the timing supervision of children's piano practice.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: A method for practicing timing based on timbre recognition comprises the following steps: S1, collect audio data in real time at a preset frequency, and transmit the collected audio data to the timbre recognition module; The timbre recognition module is based on a convolutional neural network and has different characteristic parameters for different musical instruments to determine whether the instrument is in a piano practice state; S2. According to the recognition result of the timbre recognition module and in combination with a preset time threshold for determining the interruption of piano practice, statistics are collected for each piano practice period.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A practice timing terminal based on timbre recognition includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: S1, collect audio data in real time at a preset frequency, and transmit the collected audio data to the timbre recognition module; The timbre recognition module is based on a convolutional neural network and has different characteristic parameters for different musical instruments to determine whether the instrument is in a piano practice state; S2. According to the recognition result of the timbre recognition module and in combination with a preset time threshold for determining the interruption of piano practice, statistics are collected for each piano practice period.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: A practice timing system based on timbre recognition, comprising a server, a client and a supervisor; The client implements the steps in the above-mentioned method for practicing timing based on timbre recognition, and uploads the statistical piano practicing time to the server for storage; The server receives the data access request from the supervisor, and returns the statistical piano practice time to the supervisor for display.

[0007] The beneficial effects of the present invention are as follows: the present invention provides a practice timing method, terminal and system based on timbre recognition, constructs a timbre recognition module based on a neural network model, collects audio in real time to determine whether the user is currently in a piano practice state, and then performs statistics on the piano practice time. At the same time, when performing statistics, it combines the time threshold to determine whether the piano practice is interrupted, thereby avoiding errors in piano practice time statistics caused by freezes due to insufficient proficiency of the user when practicing the piano, and can effectively and accurately realize the timing supervision of children's piano practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A flowchart of a method for practicing timing based on timbre recognition according to an embodiment of the present invention; Figure 2 A structural diagram of a practice timing terminal based on timbre recognition according to an embodiment of the present invention; Figure 3 A specific flow chart of a method for practicing timing based on timbre recognition according to an embodiment of the present invention; Figure 4 A structural diagram of a practice timing system based on timbre recognition according to an embodiment of the invention; Description of labels: 1. A practice timing terminal based on timbre recognition; 2. Processor; 3. Memory; 4. A practice timing system based on timbre recognition; 5. Server; 6. Client; 7. Supervisor. DETAILED DESCRIPTION

[0009] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0010] Please refer to Figure 1 as well as Figure 3 , a practice timing method based on timbre recognition, comprising the steps of: S1, collect audio data in real time at a preset frequency, and transmit the collected audio data to the timbre recognition module; The timbre recognition module is based on a convolutional neural network and has different characteristic parameters for different musical instruments to determine whether the instrument is in a piano practice state; S2. According to the recognition result of the timbre recognition module and in combination with a preset time threshold for determining the interruption of piano practice, statistics are collected for each piano practice period.

[0011] From the above description, it can be seen that the beneficial effects of the present invention are: a practice timing method based on timbre recognition of the present invention constructs a timbre recognition module based on a neural network model, collects audio in real time to determine whether the current piano practice state is in progress, and then performs statistics on the piano practice time. At the same time, when performing statistics, it combines the time threshold to determine whether the piano practice is interrupted, thereby avoiding the error in the piano practice time statistics caused by the user's lack of proficiency when practicing the piano, and can effectively and accurately realize the timing supervision of children's piano practice.

[0012] Further, the time threshold includes a first threshold and a second threshold; Step S2 comprises the steps of: Obtaining the recognition result of the timbre recognition module on the audio data collected in real time; If the recognition result is a piano practice state, the duration of the recognition result being the piano practice state and the recorded current state are judged: if the recognition result is that the duration of the piano practice state exceeds a first threshold, and the current state is a non-practice state, the current state is recorded as the piano practice state, and the piano practice start time is recorded, otherwise the state is not changed; If the recognition result is a non-practice state, then the duration of the recognition result as the practice state and the recorded current state are judged: if the duration of the recognition result as the non-practice state exceeds a second threshold, and the current state is the practice state, then the current state is recorded as the non-practice state, the practice end time is recorded, and the practice time of the current segment is calculated based on the recorded practice start time and practice end time, otherwise no state change is made.

[0013] From the above description, it can be seen that the piano practice time is accumulated according to the recognition results of the tone recognition module. According to the duration of the non-practice state based on the recognition result, combined with the time threshold, it is judged whether the user stops practicing the piano or the practice is stuck due to low proficiency. This can achieve more accurate accumulation of piano practice time.

[0014] Furthermore, the timbre recognition module is implemented including the steps of: S11, performing data framing on the collected real-time audio to obtain audio data frames; S12, extracting frequency domain feature vectors from the audio data frame; S13, the frequency domain feature vector belongs to a pre-trained convolutional neural network, and the convolutional neural network is pre-loaded with corresponding feature parameters according to the specified musical instrument.

[0015] From the above description, it can be seen that the timbre recognition module is based on a convolutional neural network, and different feature parameters are configured for different musical instrument training, which can adapt to the practice scenarios of different musical instruments.

[0016] Furthermore, the convolutional neural network adopts a binary classification method, and the output layer adopts a sigmoid function, with the probability value of the positive sample as the output; The timbre recognition module outputs a recognition result of piano practice or non-practice according to whether the output result of the convolutional neural network is within a preset value range.

[0017] From the above description, it can be seen that the convolutional neural network adopts a binary classification method, with the probability of being in a piano practicing state as the output, and the timbre recognition module determines whether the state is in a piano practicing state based on the output result of the convolutional neural network.

[0018] Furthermore, the preset frequency is greater than or equal to 16KHz.

[0019] From the above description, it can be seen that the acquisition frequency of real-time audio is greater than or equal to 16KHz to ensure the continuity of state judgment.

[0020] Please refer to Figure 2 , a practice timing terminal based on timbre recognition, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: S1, collect audio data in real time at a preset frequency, and transmit the collected audio data to the timbre recognition module; The timbre recognition module is based on a convolutional neural network and has different characteristic parameters for different musical instruments to determine whether the instrument is in a piano practice state; S2. According to the recognition result of the timbre recognition module and in combination with a preset time threshold for determining the interruption of piano practice, statistics are collected for each piano practice period.

[0021] From the above description, it can be seen that the beneficial effects of the present invention are: a practice timing method based on timbre recognition of the present invention constructs a timbre recognition module based on a neural network model, collects audio in real time to determine whether the current piano practice state is in progress, and then performs statistics on the piano practice time. At the same time, when performing statistics, it combines the time threshold to determine whether the piano practice is interrupted, thereby avoiding the error in the piano practice time statistics caused by the user's lack of proficiency when practicing the piano, and can effectively and accurately realize the timing supervision of children's piano practice.

[0022] Further, the time threshold includes a first threshold and a second threshold; Step S2 comprises the steps of: Obtaining the recognition result of the timbre recognition module on the audio data collected in real time; If the recognition result is a piano practice state, the duration of the recognition result being the piano practice state and the recorded current state are judged: if the recognition result is that the duration of the piano practice state exceeds a first threshold, and the current state is a non-practice state, the current state is recorded as the piano practice state, and the piano practice start time is recorded, otherwise the state is not changed; If the recognition result is a non-practice state, then the duration of the recognition result as the practice state and the recorded current state are judged: if the duration of the recognition result as the non-practice state exceeds a second threshold, and the current state is the practice state, then the current state is recorded as the non-practice state, the practice end time is recorded, and the practice time of the current segment is calculated based on the recorded practice start time and practice end time, otherwise no state change is made.

[0023] From the above description, it can be seen that the piano practice time is accumulated according to the recognition results of the tone recognition module. According to the duration of the non-practice state based on the recognition result, combined with the time threshold, it is judged whether the user stops practicing the piano or the practice is stuck due to low proficiency. This can achieve more accurate accumulation of piano practice time.

[0024] Furthermore, the timbre recognition module is implemented including the steps of: S11, performing data framing on the collected real-time audio to obtain audio data frames; S12, extracting frequency domain feature vectors from the audio data frame; S13, the frequency domain feature vector belongs to a pre-trained convolutional neural network, and the convolutional neural network is pre-loaded with corresponding feature parameters according to the specified musical instrument.

[0025] From the above description, it can be seen that the timbre recognition module is based on a convolutional neural network, and different feature parameters are configured for different musical instrument training, which can adapt to the practice scenarios of different musical instruments.

[0026] Furthermore, the convolutional neural network adopts a binary classification method, and the output layer adopts a sigmoid function, with the probability value of the positive sample as the output; The timbre recognition module outputs a recognition result of piano practice or non-practice according to whether the output result of the convolutional neural network is within a preset value range.

[0027] From the above description, it can be seen that the convolutional neural network adopts a binary classification method, with the probability of being in a piano practicing state as the output, and the timbre recognition module determines whether the state is in a piano practicing state based on the output result of the convolutional neural network.

[0028] Furthermore, the preset frequency is greater than or equal to 16KHz.

[0029] From the above description, it can be seen that the acquisition frequency of real-time audio is greater than or equal to 16KHz to ensure the continuity of state judgment.

[0030] Please refer to Figure 4 ,A practice timing system based on timbre recognition includes a server end, a client end and a supervisor end; The client implements the steps in the above-mentioned method for practicing timing based on timbre recognition, and uploads the statistical piano practicing time to the server for storage; The server receives the data access request from the supervisor, and returns the statistical piano practice time to the supervisor for display.

[0031] The invention discloses a practice timing method, terminal and system based on timbre recognition, which are suitable for duration statistics and supervision of musical instrument practice.

[0032] Please refer to Figure 1 and Figure 3 , Embodiment 1 of the present invention is: A method for practicing timing based on timbre recognition comprises the following steps: S1, collect audio data in real time at a preset frequency, and transmit the collected audio data to the timbre recognition module; The preset frequency is greater than or equal to 16KHz.

[0033] In this embodiment, after the user turns on the piano practice timing function, the timing terminal starts to collect audio data in real time through the microphone into the buffer of the terminal, and the collection rate is not less than 16KHz.

[0034] The timing terminal reads 4096 bytes of audio data from the buffer and sends it to the timbre recognition module. The timbre recognition module can output two results, "practice" and "not practice" according to the instrument type selected by the user. If the sampling rate is 16KHz, the data can be read once every 125ms, and the algorithm can be run once. If a higher sampling rate is selected, the interval between running the timing algorithm will be smaller.

[0035] The timbre recognition module is based on a convolutional neural network and has different characteristic parameters for different musical instruments to determine whether the instrument is in a piano practice state.

[0036] The convolutional neural network adopts a binary classification method, and the output layer adopts a sigmoid function, with the probability value of the positive sample as the output; The timbre recognition module outputs a recognition result of piano practice or non-practice according to whether the output result of the convolutional neural network is within a preset value range.

[0037] In the present embodiment, the timbre recognition module is a two-classification music signal recognition method based on a convolutional neural network. Because of the two-classification method adopted, each musical instrument has a corresponding set of characteristic parameters (each set of characteristic parameters needs to be trained in advance on the PC side using the same convolutional neural network model), and the corresponding characteristic parameters are loaded when the user selects which musical instrument. And the sigmod function used in the output layer of the convolutional neural network, so the result is the probability that the input audio signal is judged to be a positive sample (the value is between 0 and 1.0). According to a large number of test results, this design judges 0 to 0.85 as "not practicing the piano", and judges 0.85 to 1.0 as "practicing piano state". In this way, the timbre recognition module can give the current piano practicing state according to the type of musical instrument selected by the user.

[0038] The timbre recognition module comprises the following steps: S11, performing data framing on the collected real-time audio to obtain audio data frames; S12, extracting frequency domain feature vectors from the audio data frame; S13, the frequency domain feature vector belongs to a pre-trained convolutional neural network, and the convolutional neural network is pre-loaded with corresponding feature parameters according to the specified musical instrument.

[0039] In this embodiment, the binary classification music signal recognition method based on convolutional neural network mainly comprises the steps of data framing, frequency domain feature vector extraction, and convolutional neural network reasoning.

[0040] S2. According to the recognition result of the timbre recognition module and in combination with a preset time threshold for judging the interruption of piano practice, statistics are collected for each piano practice time; The time threshold includes a first threshold and a second threshold.

[0041] In this embodiment, the time threshold is determined by the sensitivity selected by the user. The sensitivity includes the time threshold (first threshold) T of the "piano practice" state. th1 and the time threshold of the “non-piano practice” state (the second threshold) T th2 , T th1 and T th2 The value can be a continuous value (such as all values ​​in the interval from 1s to 5s) or several discrete values ​​(such as setting the sensitivity to low, medium, and high levels, T th1 The three sensitivity values ​​are 1s, 1.5s, and 2s, respectively. th2 The three sensitivity values ​​are 4s, 3.5s, and 3s respectively).

[0042] Step S2 comprises the steps of: Obtaining the recognition result of the timbre recognition module on the audio data collected in real time; If the recognition result is a piano practice state, the duration of the recognition result being the piano practice state and the recorded current state are judged: if the recognition result is that the duration of the piano practice state exceeds a first threshold, and the current state is a non-practice state, the current state is recorded as the piano practice state, and the piano practice start time is recorded, otherwise the state is not changed; If the recognition result is a non-practice state, then the duration of the recognition result as the practice state and the recorded current state are judged: if the duration of the recognition result as the non-practice state exceeds a second threshold, and the current state is the practice state, then the current state is recorded as the non-practice state, the practice end time is recorded, and the practice time of the current segment is calculated based on the recorded practice start time and practice end time, otherwise no state change is made.

[0043] In this embodiment, a timing module is provided, which can calculate the total duration of current piano practice and the effective piano practice duration by the above formula. Figure 3 , the specific calculation is as follows: The first time the timing algorithm is run, a piano practice start time T0 is recorded. Each time the timing algorithm of the timing module is run, the current time T is obtained. t .

[0044] The default state of the timing algorithm is "Not Practicing". When it detects that the state of the algorithm output has changed, the state will change according to the sensitivity level selected by the user.

[0045] At the beginning of a piano practice period, the recognition result output by the timbre recognition module changes from the "no piano practice" state to the "piano practice" state, and the time T is recorded. ns , the time T from "practice piano" state to "no practice piano" state ne , so the effective piano practice time for the nth period is: T nt =T ne -T ns If you are still practicing piano, the practice is still being recorded, and the current effective time is: T nt =T t -T ns .

[0046] The total effective piano practice time is calculated as follows: T eff = , the total piano practice time is T total =T t -T0.

[0047] Please refer to Figure 2 , Embodiment 2 of the present invention is: A practice timing terminal 1 based on timbre recognition comprises a processor 2, a memory 3 and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, the steps of a practice timing method based on timbre recognition in the above embodiment 1 are implemented.

[0048] Please refer to Figure 4 , Embodiment 3 of the present invention is: A practice timing system 4 based on timbre recognition, comprising a server 5, a client 6 and a supervisor 7; The client 6 implements the steps of the practice timing method based on timbre recognition described in the first embodiment above, and uploads the statistical piano practice time to the server 5 for storage; The server 5 receives the data access request from the supervisor 7 and returns the statistical piano practice time to the supervisor 7 for display.

[0049] In this embodiment, the client 6 completes the piano practice audio data collection and piano practice timing functions and sends the relevant recorded data to the server 5.

[0050] The server 5 is used to store and manage piano practice result data (including but not limited to piano practice audio data, piano practice status records, total piano practice time and effective piano practice time), and can respond to requests from the client 6 and the supervisor 7 (such as querying data); The monitoring terminal 7 can check the historical and current piano practice data (including but not limited to piano practice audio data, piano practice status records, total piano practice time and effective piano practice time) through the APP. The APP is used by parents, so that the function of remotely supervising children's piano practice can be realized.

[0051] In summary, the present invention provides a practice timing method, terminal and system based on timbre recognition, which construct a timbre recognition module based on a neural network model, collect audio in real time to determine whether the user is currently in a piano practice state, and then perform statistics on the piano practice time. At the same time, when performing statistics, the time threshold is combined to determine whether the piano practice is interrupted, thereby avoiding the error in the piano practice time statistics caused by the user's lack of proficiency when practicing the piano, and can effectively and accurately realize the timing supervision of children's piano practice.

[0052] The timbre recognition algorithm based on a single instrument can quickly and accurately identify whether the input audio data is the sound of the selected instrument. However, for practitioners, especially beginners, the practice rhythm is relatively slow, and the interval between each note may be relatively long. Therefore, a timing method that dynamically adjusts the timing sensitivity according to the practitioner's current playing level can more accurately feedback the piano practice status, thereby better reflecting the practitioner's practice results each time.

[0053] Accompaniment apps implemented based on mobile phone systems require the mobile phone or tablet to be handed over to the child. The powerful functions of the mobile phone or tablet may induce the child to use the mobile phone or tablet for non-piano practice related purposes, such as playing electronic games. The terminal and system proposed in the present invention separate the terminals used by the supervised and the supervisor. The supervised can use a specially designed terminal with clear and single functions, which can be safely handed over to the child for use. The supervisor can realize the supervision function conveniently, quickly and at a low cost by installing an app on the existing terminal.

[0054] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A practice timing method based on timbre recognition, characterized in that: Includes steps: S1, collect audio data in real time at a preset frequency, and transmit the collected audio data to the timbre recognition module; The timbre recognition module is based on a convolutional neural network and has different characteristic parameters for different musical instruments to determine whether the instrument is in a piano practice state; S2. According to the recognition result of the timbre recognition module and in combination with a preset time threshold for determining the interruption of piano practice, statistics are collected for each piano practice period.

2. A method for practicing timing based on timbre recognition according to claim 1, characterized in that: The time threshold includes a first threshold and a second threshold; Step S2 comprises the steps of: Obtaining the recognition result of the timbre recognition module on the audio data collected in real time; If the recognition result is a piano practice state, the duration of the recognition result being the piano practice state and the recorded current state are judged: if the recognition result is that the duration of the piano practice state exceeds a first threshold, and the current state is a non-practice state, the current state is recorded as the piano practice state, and the piano practice start time is recorded, otherwise the state is not changed; If the recognition result is a non-practice state, then the duration of the recognition result as the practice state and the recorded current state are judged: if the duration of the recognition result as the non-practice state exceeds a second threshold, and the current state is the practice state, then the current state is recorded as a non-practice state, the practice end time is recorded, and the practice time of the current segment is calculated based on the recorded practice start time and practice end time, otherwise no state change is made.

3. A method for practicing timing based on timbre recognition according to claim 1, characterized in that: The timbre recognition module comprises the following steps: S11, performing data framing on the collected real-time audio to obtain audio data frames; S12, extracting frequency domain feature vectors from the audio data frame; S13, the frequency domain feature vector belongs to a pre-trained convolutional neural network, and the convolutional neural network is pre-loaded with corresponding feature parameters according to the specified musical instrument.

4. A method for practicing timing based on timbre recognition according to claim 1, characterized in that: The convolutional neural network adopts a binary classification method, and the output layer adopts a sigmoid function, with the probability value of the positive sample as the output; The timbre recognition module outputs a recognition result of piano practice or non-practice according to whether the output result of the convolutional neural network is within a preset value range.

5. A method for practicing timing based on timbre recognition according to claim 1, characterized in that: The preset frequency is greater than or equal to 16KHz.

6. A practice timing terminal based on timbre recognition, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: S1, collect audio data in real time at a preset frequency, and transmit the collected audio data to the timbre recognition module; The timbre recognition module is based on a convolutional neural network and has different characteristic parameters for different musical instruments to determine whether the instrument is in a piano practice state; S2. According to the recognition result of the timbre recognition module and in combination with a preset time threshold for determining the interruption of piano practice, statistics are collected for each piano practice period.

7. A practice timing terminal based on timbre recognition according to claim 6, characterized in that: The time threshold includes a first threshold and a second threshold; Step S2 comprises the steps of: Obtaining recognition results of the timbre recognition module on the audio data collected in real time; If the recognition result is a piano practice state, the duration of the recognition result being the piano practice state and the recorded current state are judged: if the recognition result is that the duration of the piano practice state exceeds a first threshold, and the current state is a non-practice state, the current state is recorded as the piano practice state, and the piano practice start time is recorded, otherwise the state is not changed; If the recognition result is a non-practice state, then the duration of the recognition result as the practice state and the recorded current state are judged: if the duration of the recognition result as the non-practice state exceeds a second threshold, and the current state is the practice state, then the current state is recorded as the non-practice state, the practice end time is recorded, and the practice time of the current segment is calculated based on the recorded practice start time and practice end time, otherwise no state change is made.

8. A practice timing terminal based on timbre recognition according to claim 6, characterized in that: The timbre recognition module comprises the following steps: S11, performing data framing on the collected real-time audio to obtain audio data frames; S12, extracting frequency domain feature vectors from the audio data frame; S13, the frequency domain feature vector belongs to a pre-trained convolutional neural network, and the convolutional neural network is pre-loaded with corresponding feature parameters according to the specified musical instrument.

9. A practice timing terminal based on timbre recognition according to claim 6, characterized in that: The convolutional neural network adopts a binary classification method, and the output layer adopts a sigmoid function, with the probability value of the positive sample as the output; The timbre recognition module outputs a recognition result of piano practice or non-practice according to whether the output result of the convolutional neural network is within a preset value range.

10. A practice timing system based on timbre recognition, characterized in that: Including server, client and supervisor; The client implements the steps of the practice timing method based on timbre recognition described in any one of claims 1 to 5 above, and uploads the statistical piano practice time to the server for storage; The server receives the data access request from the supervisor, and returns the statistical piano practice time to the supervisor for display.

Images