Music rhythm training method based on VAD and pronunciation detection

By using VAD and attack detection technology in the intelligent auxiliary music rhythm training system, the rhythm of user singing is analyzed in real time, and the existing system's problems of imperfect rhythm judgment logic, poor robustness and insufficient real-time performance are solved, and the latest development is achieved. Efficient rhythm training effects and user experience.

CN119068909APending Publication Date: 2024-12-03CHENGDU POTENTIAL ARTIFICIAL INTELLIGENCE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411090420.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-03

Smart Images

  • Figure CN119068909A_ABST
    Figure CN119068909A_ABST
Patent Text Reader

Abstract

A music rhythm training method based on VAD and pronunciation detection relates to the technical field of multimedia teaching, and comprises the following steps: firstly, determining an accompaniment for user training; audio stream data output by a microphone and accompaniment Midi data are acquired in real time; feature parameters of each audio frame in the audio stream data are extracted in real time; judging whether the current singing is finished or not based on the Midi data and the characteristic parameters of the audio frames; if the playing is not finished, returning to the previous step; if the playing is finished, performing voice activity detection and audio starting point detection on all the audio frames, and judging a voice activity frame and a voice starting frame in the audio stream data; and finally, based on the audio frame category data and the Midi data of the accompaniment, judging the difference between the audio stream data and the accompaniment. The method can effectively improve the training effect of students.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of multimedia teaching, and in particular to a music rhythm training method based on VAD and onset detection. Background Art

[0002] With the continuous development of the economy and the popularization of quality education, more and more parents have begun to pay attention to the cultivation of their children's extracurricular interests, especially in the field of music. This trend has attracted a large number of students and even adults to participate.

[0003] Through professional training, individuals can master a more precise sense of music and rhythm. However, finding a professional music teacher is still a big challenge, whether in a professional music school or a music training institution on the market. In addition, in addition to professional guidance, the lack of a convenient and efficient method and system for rhythm training in daily practice has also become a problem that needs to be solved urgently.

[0004] Existing music rhythm training auxiliary systems usually require the use of some special equipment for practice, such as wireless drum sets. Although these devices can provide certain training effects, due to their complexity and high cost, they often increase the difficulty of users in music rhythm training and may reduce users' interest in music. In addition, the portability and ease of use of these systems also have certain limitations and cannot meet the needs of users for music training anytime and anywhere.

[0005] However, intelligent auxiliary music rhythm training systems have significant advantages over traditional auxiliary systems. First, these systems can provide a more convenient training method, allowing users to perform effective music rhythm exercises without relying on complex and expensive equipment; second, intelligent systems use visual interfaces and real-time feedback to help users understand and master rhythm points more intuitively, thereby improving learning efficiency; in addition, these systems are usually portable, and users can use mobile devices to train anytime and anywhere, greatly improving the flexibility and convenience of use.

[0006] Through advanced technologies such as voice activity detection (VAD) and onset detection, the system can analyze the user's performance in real time and provide precise guidance, further enhancing the training effect; therefore, the intelligent assisted music rhythm training system not only lowers the user's learning threshold, but also significantly improves the user's learning experience and effect, meeting the modern user's demand for efficient and convenient music education.

[0007] However, the current intelligent auxiliary music rhythm training system still has the following defects:

[0008] 1. The rhythm judgment logic is not perfect. It only has two judgments: wrong and correct. The common situations of rushing the beat and dragging the beat in rhythm training are not subdivided.

[0009] 2. The system has poor robustness, as it only determines rhythm based on sound energy and does not consider the impact of noise on system performance.

[0010] 3. The real-time performance is poor, and it is impossible to judge each rhythm note in real time.

[0011] The Chinese patent (CN202310381576.6) discloses an interactive music rhythm practice system that can automatically identify the rhythm of students' singing based on a rhythm recognition module and adjust the speed of the played audio rhythm according to the rhythm of the singing, which has a good interactive effect; however, it lacks the effect of actively training students;

[0012] A Chinese patent (CN201910461070.X) discloses a guidance and training system for generating music rhythm training questions, which is used for guiding and training people on music rhythm. According to different question types, it can be provided to people who need to repeatedly train the basics of music rhythm and learn about music rhythm for guidance and training; this invention can increase the question types according to the continuous enrichment of rhythm knowledge in the field of music, so that people who learn music rhythm can receive a complete system of music rhythm training; people can conduct self-study efficiently and conveniently through WeChat on their mobile phones, but this patent mainly decomposes music question types and then guides students, which has poor training effect on students.

[0013] Therefore, we propose a method that can effectively improve students' training results. Summary of the Invention

[0014] In order to overcome the deficiencies in the background technology, the present invention discloses a music rhythm training method based on VAD and onset detection.

[0015] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0016] A music rhythm training method based on VAD and onset detection, the specific steps are as follows:

[0017] S1. Determine the accompaniment for user training;

[0018] S2. Get the audio stream data output by the microphone and the MIDI data of the accompaniment in real time;

[0019] S3 real-time extraction of characteristic parameters of each audio frame in the audio stream data;

[0020] S4. Based on the Midi data and the system timing, determine whether the playback is finished;

[0021] S5. If the playback is not finished, return to step S3;

[0022] S6. If playback is finished, voice activity detection and audio starting point detection are performed on all audio frames to determine the voice activity frame and voice start frame in the audio stream data;

[0023] S7. Based on the audio frame type data of step S6 and the MIDI data of the accompaniment, determine the difference between the audio stream data and the accompaniment.

[0024] Preferably, the specific steps of extracting audio frame feature parameters in step S3 are:

[0025] S31 preprocesses the real-time audio frame;

[0026] S32. Extract short-time energy, zero-crossing rate, and spectrum energy from the preprocessed audio frame data.

[0027] Preferably, the audio frame preprocessing step in step S31 is:

[0028] S311 pre-emphasizes the audio frame;

[0029] S312. Window the pre-emphasized audio frame.

[0030] Preferably, the calculation formula of step S311 is:

[0031] y[n]=x[n] - α·x[n-1]; (1)

[0032] Where y[n] is the pre-emphasized signal, x[n] is the original signal, and α is the pre-emphasis coefficient.

[0033] Preferably, the calculation formula of step S312 is:

[0034]

[0035] Where w[n] is the window function, N is the length of the window, and n is the index of the sample point of the current speech frame;

[0036] z[n]=w[n]·y[n](3)

[0037] Where z[n] is the windowed signal.

[0038] Preferably, the steps of extracting short-time energy, zero-crossing rate and spectrum energy in step S32 are:

[0039] S321. Calculate the short-time energy E of the audio frame. The calculation formula is as follows:

[0040]

[0041] Where E is the energy of each frame, and n is the index of the sample point of the current speech frame.

[0042] S322: Calculate the zero crossing rate ZCR of the audio frame. The calculation formula is as follows:

[0043]

[0044] Where N is the total number of sample point indices in a speech frame, n is the index of the nth sample point, and sgn(x) is a conforming function defined as follows:

[0045]

[0046] S323: Calculate the spectrum energy of the audio frame. The calculation formula is as follows:

[0047] X[k]=FFT(w[n]); (7)

[0048] Where FFT is fast Fourier transform, X[k] is the frequency domain signal;

[0049]

[0050] Where E is the spectrum energy and N is the length after conversion into the frequency domain signal.

[0051] Preferably, the method for voice activity detection in step S6 is specifically as follows:

[0052] S61. Set the thresholds of short-time energy, zero-crossing rate and spectral energy in each audio frame respectively;

[0053] S62 uses the following formula to determine whether the audio frame is a voice activity frame;

[0054]

[0055] Among them, VAD n =1 means the nth frame is a speech active frame, VAD n =0 means the nth frame is a non-speech activity frame, E m is the short-time energy of the speech frame, θ E is the short-time energy threshold; Z m is the zero-crossing rate, θ Z is the zero-crossing rate threshold; S m is the speech frame spectrum energy, θ S is the speech frame spectrum energy;

[0056] S63. Use the following formula to determine whether the audio frame is the voice start frame Onset:

[0057]

[0058] Among them, E k and Z k are the short-time energy and zero-crossing rate of the kth frame respectively, last_onset is the index of the last detected Onset frame, W is the length of the suppression window, E threshold is the short-time energy threshold, Z threshold is the zero-crossing rate threshold;

[0059] S64. When a frame is marked as VAD and Onset at the same time, it is determined that the frame is a speech start frame K start Or speech start frame K end .

[0060] Preferably, the speech start frame K in step S64 start and end frame K end The judgment formula is:

[0061] K start =min{t|V(t)=1and O(t)=1}; (11)

[0062] K end =max{t|V(t)=1and O(t)=1}; (12)

[0063] Among them, V(t)=1 means that the t-th audio frame is a speech activity frame, and O(t)=1 means that the t-th audio frame is a speech start frame.

[0064] Preferably, the method for determining the difference between the audio stream data and the accompaniment in step S7 is:

[0065] S71. The rest notes in the Midi data are divided into long-duration rests and short-duration rests, and different thresholds are set for notes of different durations;

[0066] S72 based on the voice activity frame VAD obtained in step S62, calculate the number of frames of voice activity frame VAD in the rest threshold;

[0067] S73. When the number of voice activity frames is less than the corresponding threshold, the judgment is correct, otherwise it is an error;

[0068] S74. The non-rest notes in the Midi data are divided into long-duration non-rest notes and short-duration non-rest notes, and different thresholds are set for notes of different durations;

[0069] S75. Based on the speech start frame Onset obtained in step S63, and the speech start frame K start Or speech start frame K end, combined with the non-rest note positions in the Midi data, to determine whether the user's singing is correct.

[0070] Preferably, the judgment formula in step S73 is:

[0071] Right=K vad / K≤VAD threshold (13)

[0072] Among them, K vad is the number of VAD frames, K is the number of total speech frames; VAD tresold is the VAD threshold;

[0073] The judgment formula in step S74 is:

[0074]

[0075] Among them, Note start Note is the start time of the rest note. end is the end time of the rest note, VAD threshold is the speech activity frame threshold.

[0076] Due to the adoption of the above-mentioned technical solution, the present invention has the following beneficial effects:

[0077] The present invention discloses a music rhythm training method based on VAD and onset detection,

[0078] 1. By designing more conditions, the rhythm evaluation is enriched. In addition, the rhythm evaluation algorithm is optimized from the perspective of music education based on the user's operating habits and reactions to short-duration notes, improving the user experience and accuracy during rhythm training.

[0079] 2. Calculate multiple audio features of the audio frame and determine whether the speech is an active frame by combining multiple features with a threshold, which improves the system's resistance to noise interference and makes the system suitable for different environments;

[0080] 3. Based on the start and end time of the voice and the number of valid voice activity frames, the user's grasp of the duration of a certain note can be judged in real time to determine whether the singer using the microphone is dragging the beat, rushing the beat, or keeping the beat, rather than judging by recording a piece of audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 A schematic diagram of a process of the present invention;

[0082] Figure 2 The result of the singing is a schematic diagram of the timing situation;

[0083] Figure 3The result of the singing is a schematic diagram of the dragging situation;

[0084] Figure 4 The result of the performance is a schematic diagram of the situation of snapping shots;

[0085] Figure 5 Schematic diagram of the audio waveform of the human voice for rhythm training;

[0086] Figure 6 Schematic diagram of the spectrogram of a human voice trained for rhythm. DETAILED DESCRIPTION

[0087] The technical solutions of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention. In the description, it should be understood that if there are terms such as "upper", "lower", "front", "rear", "left", "right" and the like indicating directions or positional relationships, these are merely for the purpose of corresponding to the drawings of the present invention and for the convenience of describing the present invention. They do not indicate or imply that the devices or components referred to must have a specific direction.

[0088] Example 1 is:

[0089] Combined with attachment Figure 1 The music rhythm training method based on VAD and onset detection comprises the following steps:

[0090] S1. Determine the accompaniment for user training;

[0091] Among them, multiple accompaniments will be concentrated in the accompaniment library, and then users can choose the appropriate specific accompaniment according to actual needs;

[0092] S2. Get the audio stream data output by the microphone and the MIDI data of the accompaniment in real time;

[0093] The number of sampling points in each frame of audio stream is 1024, that is, one frame of voice contains 1024 voice data points, and Midi data mainly includes information such as the start time and end time of each note and whether the note is a rest.

[0094] S3 real-time extraction of characteristic parameters of each audio frame in the audio stream data;

[0095] The audio frame is the sampled data within a certain time period in the audio stream data, and the characteristic parameters of each audio frame are different according to the different sampling time periods;

[0096] S4. Based on the Midi data and the system timing, determine whether the playback is finished;

[0097] The MIDI data contains the total duration of the accompaniment notes, as well as the timestamps of the total duration of various notes, such as the start note, end note, and rest note. Then, combined with the system timing, when the system starts playing the accompaniment, the system timing is started synchronously, and then the playing duration between the accompaniment note and the end note is determined. If the system timing is greater than the playing duration, it is defined as the end of the playback, otherwise the playback is not over.

[0098] S5. If playback is not finished, return to step S3 and continue to extract the next audio frame of the audio stream data output by the microphone;

[0099] S6. If playback is finished, voice activity detection and audio starting point detection are performed on all audio frames to determine the voice activity frame and voice start frame in the audio stream data;

[0100] All audio frames, that is, all audio frames output by the microphone during the playback period, represent the audio stream data of the user singing with the microphone during the accompaniment playback period;

[0101] In addition, the speech activity frame represents the frame number when the user sings, and the speech start frame represents the specific audio frame when the user starts singing and stops singing;

[0102] S7. Based on the audio frame category data of step S6 and the Midi data of the accompaniment, determine the difference between the audio stream data and the accompaniment, that is, determine whether the user's singing is in beat, dragging the beat, or rushing the beat based on whether the user's singing time is consistent with the time that should be sung in the accompaniment.

[0103] Example 2 is:

[0104] On the basis of Example 1, the step S3 is further limited, that is, the specific steps of extracting the audio frame feature parameters in step S3 are:

[0105] S31 pre-processes the real-time audio frames to remove noise, blank sounds, etc. in the audio stream data that may interfere with subsequent judgments;

[0106] S32. Extract the short-time energy, zero-crossing rate and spectral energy from the pre-processed audio frame data. These characteristic parameters can reflect the vocal part in the audio frame, thereby facilitating the subsequent judgment of whether the user's singing is in tune.

[0107] Example 3 is:

[0108] Based on Example 2, the audio frame preprocessing step is further limited. The audio frame preprocessing step in step S31 is as follows:

[0109] S311 pre-emphasizes the audio frame to compensate for the loss of high-frequency audio energy;

[0110] S312. Window the pre-emphasized audio frame to reduce discontinuity between frames and reduce spectrum leakage.

[0111] In addition, the calculation formula of step S311 is:

[0112] y[n]=x[n] - α·x[n-1]; (1)

[0113] Where y[n] is the pre-emphasized signal, x[n] is the original signal, and α is the pre-emphasis coefficient.

[0114] In addition, the calculation formula of step S312 is:

[0115]

[0116] Where w[n] is the window function, N is the length of the window, and n is the index of the sample point of the current speech frame;

[0117] z[n]=w[n]·y[n](3)

[0118] Where z[n] is the windowed signal.

[0119] In addition, the steps of extracting short-time energy, zero-crossing rate and spectrum energy in step S32 are:

[0120] S321. Calculate the short-time energy E of the audio frame. The calculation formula is as follows:

[0121]

[0122] Where E is the energy of each frame, and n is the index of the sample point of the current speech frame.

[0123] S322: Calculate the zero crossing rate ZCR of the audio frame. The calculation formula is as follows:

[0124]

[0125] Where N is the total number of sample point indices in a speech frame, n is the index of the nth sample point, and sgn(x) is a conforming function defined as follows:

[0126]

[0127] S323: Calculate the spectrum energy of the audio frame. The calculation formula is as follows:

[0128] X[k]=FFT(w[n]); (7)

[0129] Where FFT is fast Fourier transform, X[k] is the frequency domain signal;

[0130]

[0131] Where E is the spectrum energy and N is the length after conversion into the frequency domain signal.

[0132] Example 4 is:

[0133] Based on Example 1, step S6 is further defined. The method for voice activity detection in step S6 is specifically as follows:

[0134] S61. Set the thresholds of short-time energy, zero-crossing rate and spectral energy in each audio frame respectively;

[0135] S62 uses the following formula to determine whether the audio frame is a voice activity frame VAD;

[0136]

[0137] Among them, VAD n =1 means the nth frame is a speech active frame, VAD n =0 means the nth frame is a non-speech activity frame, E m is the short-time energy of the speech frame, θ E is the short-time energy threshold; Z m is the zero-crossing rate, θ Z is the zero-crossing rate threshold; S m is the speech frame spectrum energy, θ S is the speech frame spectrum energy;

[0138] S63. Use the following formula to determine whether the audio frame is the voice start frame Onset:

[0139]

[0140] Among them, E k and Z k are the short-time energy and zero-crossing rate of the kth frame respectively, last_onset is the index of the last detected Onset frame, W is the length of the suppression window, E threshold is the short-time energy threshold, Z threshold is the zero-crossing rate threshold;

[0141] S64. When a frame is marked as VAD and Onset at the same time, it is determined that the frame is a speech start frame K start Or speech start frame K end .

[0142] In addition, the speech start frame K in step S64 start and end frame Kend The judgment formula is:

[0143] K start =min{t|V(t)=1and O(t)=1}; (11)

[0144] K end =max{t|V(t)=1and O(t)=1}; (12)

[0145] Among them, V(t)=1 means that the t-th audio frame is a voice activity frame, and O(t)=1 means that the t-th audio frame is an Onset frame.

[0146] Example 5 is:

[0147] Based on Example 1, the step S7 is further limited. The method for determining the difference between the audio stream data and the accompaniment in the step S7 is as follows:

[0148] S71. The rest notes in the Midi data are divided into long-duration rests and short-duration rests, and different thresholds are set for notes of different durations;

[0149] S72. Based on the VAD obtained in step S62, the number of frames of the VAD in the rest threshold is calculated;

[0150] S73. When the number of voice activity frames is less than the corresponding threshold, the judgment is correct, otherwise it is an error;

[0151] S74. The non-rest notes in the Midi data are divided into long-duration non-rest notes and short-duration non-rest notes, and different thresholds are set for notes of different durations;

[0152] S75. Based on the speech start frame Onset obtained in step S63, and the speech start frame K start Or speech start frame K end , combined with the non-rest note positions in the Midi data, to determine whether the user's singing is correct;

[0153] In particular, the judgment formula in step S73 is:

[0154] Right=K vad / K≤VAD threshold (13)

[0155] Among them, K vad is the number of VAD frames, K is the number of total speech frames; VAD tresold is the VAD threshold;

[0156] The judgment formula in step S75 is:

[0157]

[0158] Among them, Note start Note is the start time of the rest note. end is the end time of the rest note, VAD threshold is the speech activity frame threshold;

[0159] If the user's singing is correct at both rest notes and non-rest notes, it means that the user's singing process is in time, as shown in the attached figure. Figure 2 As shown;

[0160] If the user's singing is correct at the rest notes, but wrong at the non-rest notes, it means that the user's singing is in a rush, as shown in the attached figure. Figure 3 As shown;

[0161] If the user's singing is wrong at the rest notes, but correct at the non-rest notes, it means that the user's singing is dragging the beat, as shown in the attached figure. Figure 4 shown.

[0162] In addition, as attached Figure 5-6 As shown, after the actual singing is simulated using this method, the audio waveform and spectrum diagram of the synchronization situation can be used to intuitively discover the user's singing situation.

[0163] Example 6 is:

[0164] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-mentioned training method.

[0165] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, a computer-readable storage medium stores a computer program that is used by a processor to execute the aforementioned training method. When the computer program is executed by the processor, the aforementioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0166] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0167] In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0168] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wire, optical cable, RF (radio frequency), etc., or any suitable combination thereof.

[0169] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0170] The parts of the present invention that are not described in detail are prior art. It is obvious to those skilled in the art that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is limited by the appended claims rather than the above description. Therefore, it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any figure marks in the claims should not be regarded as limiting the claims involved.

Claims

1. A music rhythm training method based on VAD and onset detection, characterized by: Specific steps for: S1. Determine the accompaniment for user training; S2. Obtain the audio stream data output by the microphone and the Midi data of the accompaniment in real time; S3 real-time extraction of characteristic parameters of each audio frame in the audio stream data; S4. Based on the Midi data and the timing of the system, determine whether the playback is finished; S5. If the playback is not finished, return to step S3; S6. If playback is finished, voice activity detection and audio starting point detection are performed on all audio frames to determine the voice activity frame and voice start frame in the audio stream data; S7. Based on the audio frame type data of step S6 and the Midi data of the accompaniment, determine the difference between the audio stream data and the accompaniment.

2. The music rhythm training method based on VAD and onset detection as claimed in claim 1, characterized in that: The specific steps of extracting audio frame feature parameters in step S3 are: S31 preprocesses the real-time audio frame; S32. Extract short-time energy, zero-crossing rate and spectrum energy from the preprocessed audio frame data.

3. The music rhythm training method based on VAD and onset detection as claimed in claim 2, characterized in that: The pre-processing steps of the audio frame in step S31 are: S311. Pre-emphasize the audio frame; S312. Window the pre-emphasized audio frame.

4. The music rhythm training method based on VAD and onset detection as claimed in claim 3, characterized in that: The calculation formula of step S311 is: y[n]=x[n] - α·x[n-1]; (1) Among them, y[n] is the pre-emphasized signal, x[n] is the original signal, and α is the pre-emphasis coefficient.

5. The music rhythm training method based on VAD and onset detection as claimed in claim 3, characterized in that: The calculation formula of step S312 is: Where w[n] is the window function, N is the length of the window, and n is the index of the sample point of the current speech frame; z[n]=w[n]·y[n](3) Among them, z[n] is the windowed signal.

6. The music rhythm training method based on VAD and onset detection as claimed in claim 2, characterized in that: The steps of extracting short-time energy, zero-crossing rate and spectrum energy in step S32 are: S321. Calculate the short-time energy E of the audio frame. The calculation formula is as follows: Where E is the energy of each frame, and n is the index of the sample point of the current speech frame. S322: Calculate the zero crossing rate ZCR of the audio frame. The calculation formula is as follows: Where N is the total number of sample point indexes in a speech frame, n is the index of the nth sample point, and sgn(x) is a conforming function defined as follows: S323: Calculate the spectrum energy of the audio frame. The calculation formula is as follows: X[k]=FFT(w[n]); (7) Wherein, FFT is fast Fourier transform, X[k] is the frequency domain signal; Wherein, E is the spectrum energy and N is the length after conversion into frequency domain signal.

7. The music rhythm training method based on VAD and onset detection as claimed in claim 1, characterized in that: The method for voice activity detection in step S6 is specifically as follows: S61. Set the thresholds of short-time energy, zero-crossing rate and spectral energy in each audio frame respectively; S62. Use the following formula to determine whether the audio frame is a voice activity frame; Among them, VAD n =1 means the nth frame is a speech active frame, VAD n =0 means the nth frame is a non-speech activity frame, E m is the short-time energy of speech frame, θ E is the short-time energy threshold; Z m is the zero-crossing rate, θ Z is the zero-crossing rate threshold; S m is the speech frame spectrum energy, θ S is the speech frame spectrum energy; S63. Use the following formula to determine whether the audio frame is a speech start frame Onset: Among them, E k and Z k are the short-time energy and zero-crossing rate of the kth frame, respectively. last_onset is the index of the last detected Onset frame. W is the length of the suppression window. E threshold is the short-time energy threshold, Z threshold is the zero-crossing rate threshold; S64. When a frame is marked as VAD and Onset at the same time, it is determined that the frame is a speech start frame K start Or speech start frame K end .

8. The music rhythm training method based on VAD and onset detection as claimed in claim 7, characterized in that: In step S64, the speech start frame K start and end frame K end The judgment formula is: K start =min{t|V(t)=1and O(t)=1}; (11) K end =max{t|V(t)=1and O(t)=1}; (12) Among them, V(t)=1 means that the t-th audio frame is a speech activity frame, and O(t)=1 means that the t-th audio frame is a speech start frame.

9. The music rhythm training method based on VAD and onset detection as claimed in claim 1, characterized in that: The method for determining the difference between the audio stream data and the accompaniment in step S7 is: S71. The rest notes in the Midi data are divided into long-duration rests and short-duration rests, and different thresholds are set for notes of different durations; S72. Based on the voice activity frame VAD obtained in step S62, the number of frames of the voice activity frame VAD in the rest threshold is calculated; S73. When the number of voice activity frames is less than the corresponding threshold, the judgment is correct, otherwise it is an error; S74. The non-rest notes in the Midi data are divided into long-duration non-rest notes and short-duration non-rest notes, and different thresholds are set for notes of different durations; S75. Based on the speech start frame Onset obtained in step S63, and the speech start frame K start Or speech start frame K end , combined with the non-rest note positions in the Midi data, to determine whether the user's singing is correct.

10. The music rhythm training method based on VAD and onset detection as claimed in claim 9, characterized in that: The judgment formula in step S73 is: Right=K vad / K≤VAD threshold (13) Among them, K vad is the number of VAD frames, K is the number of total speech frames; VAD tresold is the VAD threshold; The judgment formula in step S74 is: Among them, Note start Note is the start time of the rest note. end is the end time of the rest note, VAD thresjold is the speech activity frame threshold.