An interactive music rhythm practicing system

Abstract

The application relates to an interactive music rhythm practicing system, which comprises an audio acquisition module, a storage module, a rhythm recognition module and a player; the audio acquisition module is used for collecting real-time audio streams from a user end and saving the real-time audio streams as time domain data; the rhythm recognition module is used for recognizing rhythm characteristics from the time domain data and comparing rhythm characteristics of different audio streams; and the player is used for adjusting the rhythm speed of audio playing according to a user end instruction or the rhythm characteristics. The technical scheme of the application can automatically recognize the rhythm of student playing and adjust the rhythm speed of audio playing according to the rhythm of playing, and has a good interactive effect.

Description

Technical Field

[0001] This invention belongs to the field of audio temporal feature analysis technology, specifically relating to an interactive music rhythm practice system. Background Technology

[0002] In existing music teaching software, interactive music applications can display interactive prompts to users based on the rhythm points of the music. Users can then input interactive operations according to the prompts, which is very helpful for developing a sense of rhythm. However, many beginners struggle with rhythm when practicing instruments, often deviating from the rhythm of the music. Furthermore, existing music teaching software does not provide effective interaction for students in rhythm practice. Summary of the Invention

[0003] To address the above technical problems, this invention proposes an interactive music rhythm practice system, including an audio acquisition module, a storage module, a rhythm recognition module, and a player;

[0004] The audio acquisition module is used to acquire audio streams from the user terminal in real time;

[0005] The storage module saves the audio stream as time-domain data, and the audio stream includes a first audio stream acquired from the user terminal and a second audio stream obtained from the network interface.

[0006] The rhythm recognition module identifies and extracts rhythm features from time-domain data;

[0007] The rhythm recognition module compares rhythm features extracted from time-domain data of different audio streams.

[0008] The player is used to adjust the speed of audio playback according to user instructions or the rhythmic characteristics.

[0009] The technical solution of this invention can automatically identify the rhythm of the student's performance and adjust the speed of the played audio rhythm according to the performance rhythm, which has a good interactive effect. Attached Figure Description

[0010] Figure 1 One implementation involves a system architecture;

[0011] Figure 2 A flowchart of one implementation method;

[0012] Figure 3 Audio stream time-domain data in one implementation. Detailed Implementation

[0013] The following embodiments further illustrate the content of the present invention, but should not be construed as limiting the present invention. Any modifications or substitutions made to the methods, steps, or conditions of the present invention without departing from the spirit and essence of the invention are within the scope of the present invention.

[0014] Some implementations of the system have Figure 1 The functional modules shown include an audio acquisition module, a storage module, a rhythm recognition module, and a player. The audio acquisition module connects to an audio acquisition device with a microphone at the user end. The audio acquisition module processes the acquired audio and stores it in the buffer module for use by the rhythm recognition module. Each functional module can implement… Figure 2 Steps:

[0015] The audio acquisition module is used to acquire audio streams from the user terminal in real time;

[0016] The storage module saves the audio stream as time-domain data, and the audio stream includes a first audio stream acquired from the user terminal and a second audio stream obtained from the network interface.

[0017] The rhythm recognition module identifies and extracts rhythm features from time-domain data;

[0018] The rhythm recognition module compares rhythm features extracted from time-domain data of different audio streams.

[0019] The player is used to adjust the speed of audio playback according to user instructions or the rhythmic characteristics.

[0020] The steps of rhythm recognition modules in some implementations to identify and extract rhythm features from time-domain data specifically include:

[0021] Acquire the time-domain data within the window time period, and calculate the time-domain analysis data of the time-domain data including the start time, duration, and volume peak value. Preferably, the time-domain analysis data also includes peak factor, impulse factor, margin factor, skewness, and kurtosis.

[0022] The time-domain analysis data of the time-domain data is saved as a rhythm feature matrix.

[0023] The above terms, based on common knowledge in the field, should be understood as follows:

[0024] Peak volume includes peak volume level, measured in decibels (dB); crazing factor is the ratio of the peak value to the effective mean square (RMS) value of the audio signal, representing the extreme degree of the peak in the waveform; impulse factor is the ratio of the peak value to the rectified average value (average of absolute values) of the audio signal; margin factor is the ratio of the peak value to the root square amplitude of the audio signal. Skewness, also called skewness or skewness distribution, is the ratio of the third central moment to the cube of the standard deviation; kurtosis is the ratio of the fourth central moment to the fourth power of the standard deviation; skewness and kurtosis describe the distribution of the audio signal.

[0025] Specifically, the window time period can be set according to user instructions.

[0026] The steps of the rhythm recognition module in some implementations to identify and extract rhythm features from time-domain data further include:

[0027] The second audio stream is divided into several time intervals, and the time domain data of the second audio stream in different time intervals are traversed in chronological order.

[0028] Match the rhythm feature matrix of the first audio stream with the rhythm feature matrix of the second audio stream. If the match is successful, mark the time interval of the window time period of the rhythm feature matrix of the first audio stream in the corresponding time interval of the second audio stream.

[0029] In some implementations, based on the above implementations, the player adjusts the playback speed of the second audio stream according to the time interval and the time window.

[0030] In some implementations, the rhythm recognition module's steps for identifying and extracting rhythm features from time-domain data specifically include noise reduction, amplitude envelope analysis, calculation of short-time average energy, calculation of root mean square energy, calculation of zero-crossing rate, and calculation of short-time autocorrelation function.

[0031] An amplitude envelope is a curve formed by connecting the highest points of amplitude at different frequencies.

[0032] Short-time average energy can be used to distinguish voiced sounds from unvoiced sounds (the en in voiced sounds is much larger than that in unvoiced sounds), to determine the boundaries between initials and finals, voiceless and voiced sounds, ligatures, etc., and can also be used as suprasegmental information for speech recognition.

[0033] The steps involved in calculating the short-time average energy are as follows: first, calculate the square of the signal grid sample value; then, use a moving window h (nm) to select a series of short-time squared sequences and sum the squared values ​​of each segment to obtain the short-time energy sequence.

[0034] The root mean square energy (RMS) represents a comprehensive set of information about all sample points within a frame, including loudness information. It can be applied to audio segmentation and music genre classification.

[0035] During voice recording, due to various factors including static effects, microphone breathing, and 50Hz AC voltage signals, the recorded voice signal is likely to oscillate (shift) near non-zero time-varying values. To avoid this drift within a frame, a simple approach is to identify time-varying zero curves through polynomial fitting (meaning there are signals that are time-varying near zero even in the absence of sound) and eliminate the drift by removing noisy sub-tracks from the curve in the original frame.

[0036] For a speech sequence x(n), and the i-th frame of speech signal x_{i}(m), the short-time autocorrelation function is defined as follows:

[0037] R_{i}(k)=\sum_{m=1}^{Nm}x_{i}(m)x_{i}(m+k) where k is the time delay. The correlation function is used to determine the time-domain similarity between two signals.

[0038] In some implementations of the system, the step of the rhythm recognition module identifying and extracting rhythm features from time-domain data further includes: identifying rhythmic features in the rhythm feature matrix of the audio stream, wherein the rhythmic features include the distribution of volume peaks within the time window, and the rhythmic features specifically include equal-value distribution, alternating long and short values, dotted rhythm, syncopated rhythm, and rhythm containing rests.

[0039] In one application scenario suitable for this invention, the user uses the system for rhythm practice on a drum kit. During the practice, the user plays standard sheet music data obtained from the network as a practice template (i.e., the second audio stream), while simultaneously opening the microphone to capture the music played on the drum kit (i.e., the first audio stream).

[0040] In this scenario, the time-domain data of the music played by the drum kit is collected and saved. A segment of time-domain data within an 8-beat time window T is then processed as follows:

[0041] Divide the time window T into 8 equal segments, each segment corresponding to one beat out of 8 beats, and calculate as follows: Figure 3 The number of volume peaks (or other types of time-domain analysis data) occurring in each time period is represented as an 8-dimensional vector: V1 = [0,2,1,0,0,1,1,0]; this vector can be used as a vector in the rhythm feature matrix to identify the rhythmic patterns appearing in the time-domain data.

[0042] In some implementations, after receiving a user instruction, the player performs the following steps:

[0043] Determine whether the playback object type in the user terminal instruction is either audio or video media type;

[0044] If the playback object type is audio or video, determine whether the network resource address of the media currently being played by the player is the same as the network resource address of the playback object. If not, stop the current playback of the media and set the value of the media currently being played by the player to empty.

[0045] The value of the playback object in the user-side command is assigned to the media currently being played by the player.

[0046] The player performs the following steps:

[0047] Switch the currently playing media from the playing state to the stopped state;

[0048] Switch the currently playing media from a stopped state to a playing state.

[0049] In some of the above implementations, the second audio stream is the audio stream of the media currently being played by the player.

[0050] The embodiments and functional operations of the subject matter described in this specification can be implemented in the following ways: digital electronic circuits, tangibly implemented computer software or firmware, computer hardware, including the structures disclosed in this specification and their equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on one or more tangible, non-transitory program carriers, for execution by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated as encoded information to be transmitted to an appropriate receiver device executed by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof.

[0051] The term "processor" encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiprocessor systems or multiple computers. Devices may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, devices may include code that creates the execution environment for associated computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0052] To transmit interactions with a user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user; and a keyboard and a positioning device, such as a mouse or trackball, which the user can use to send input to the computer. Other types of devices can also be used to transmit interactions with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound input, voice input, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a received request from a web browser.

[0053] Specific implementations of the subject matter have been described. Other implementations are within the scope of the following claims. For example, the activities described in the claims can be performed in a different order and still achieve the desired result. As an example, the processes described in the drawings do not necessarily require a specific order or sequence to be shown in order to achieve the desired result. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. A drum kit rhythm practice system, characterized in that, The system includes an audio acquisition module, a storage module, a rhythm recognition module, and a player; The audio acquisition module is used to acquire a first audio stream from the user terminal in real time, the first audio stream being the audio stream of music played by a drum kit; The user client plays standard sheet music data obtained from the network as a practice template, and the standard sheet music data is a second audio stream; the second audio stream is the audio stream of the media currently being played by the player; The storage module is used to save the first audio stream and the second audio stream as time-domain data; The rhythm recognition module is used to identify and extract rhythm features from the time-domain data, and compare the rhythm feature matrix of the first audio stream with the rhythm feature matrix of the second audio stream. The rhythm recognition module obtains a segment of time-domain data within an 8-beat time window T from the time-domain data of the first audio stream, divides the time window T into 8 equal segments, each segment corresponding to one beat in the 8 beats, calculates the number of volume peaks that occur in each time segment, and represents the calculation result as an 8-dimensional vector. The 8-dimensional vector is used as a vector in the rhythm feature matrix of the first audio stream to identify the rhythm patterns that occur in the time-domain data of the first audio stream. The rhythm patterns include equal division of pitch values, alternation of long and short pitch values, dotted rhythm patterns, syncopated rhythm patterns, and rhythm patterns containing rests. The rhythm recognition module is further configured to divide the second audio stream into several time intervals, traverse the time domain data of the second audio stream in different time intervals in chronological order, match the rhythm feature matrix of the first audio stream with the rhythm feature matrix of the second audio stream, and if the match is successful, mark the time interval in the second audio stream corresponding to the window time period of the rhythm feature matrix of the first audio stream. The player is used to adjust the speed of audio playback according to the rhythmic characteristics.

2. The system as described in claim 1, characterized in that, After receiving a user instruction, the player performs the following steps: Determine whether the playback object type in the user terminal instruction is either audio or video media type; If the playback object type is audio or video, determine whether the network resource address of the media currently being played by the player is the same as the network resource address of the playback object. If not, stop the current playback of the media and set the value of the media currently being played by the player to empty. The value of the playback object in the user-side command is assigned to the media currently being played by the player.

3. The system as described in claim 2, characterized in that, The player performs the following steps: Switch the currently playing media from the playing state to the stopped state; Switch the currently playing media from a stopped state to a playing state.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements all the steps of the interactive music rhythm practice system for the drum kit as described in any one of claims 1-3.

Images