Real-time music score following method, system and computer-readable storage medium

Abstract

The present invention discloses a real-time music score tracking method, system, and computer-readable storage medium. The method comprises the following steps: constructing a mathematical model of the music score following system; establishing a state equation and observation equation for the music score following system based on the mathematical model; and utilizing the constructed system state equation and observation equation to perform real-time music score tracking. By constructing a system state equation and observation equation suitable for nonlinear, non-Gaussian systems and employing Monte Carlo sampling to approximate posterior probabilities, the present invention employs a particle filter method to calculate the current filter output, thereby improving the accuracy and stability of real-time music score tracking.

Description

Technical Field

[0001] The present invention relates to the field of musical sound processing technology, and more particularly, to a real-time music score following method, system, and computer-readable storage medium. Background Art

[0002] In recent years, the continuous application of signal processing and machine learning methods in the music field has led to the emergence of a wide range of new software, such as AI-powered practice, automatic accompaniment playback, solo accompaniment, and interactive audio and video demonstrations with sheet music. These features have greatly benefited music enthusiasts, especially instrumentalists. These features rely on audio-score alignment, or sheet music following technology.

[0003] The existing real-time music score following method has unsatisfactory following accuracy, poor following path stability, and is prone to fluctuations, making it unsuitable for nonlinear non-Gaussian systems.

[0004] Therefore, there is an urgent need for a real-time music score following method, system and computer readable storage medium Summary of the Invention

[0005] In order to overcome at least one of the above-mentioned defects of the prior art, the present invention provides a real-time music score following method, system and computer-readable storage medium.

[0006] The primary purpose of the present invention is to solve the above technical problems, and the technical solutions of the present invention are as follows:

[0007] A first aspect of the present invention provides a real-time music score following method, comprising the following steps:

[0008] Construct a mathematical model of the score-following system;

[0009] Based on the mathematical model of the music score following system, the state equation and observation equation of the music score following system are established;

[0010] Real-time music score following is performed using the constructed system state equations and observation equations.

[0011] In this solution, the specific process of constructing the mathematical model of the music score following system is as follows:

[0012] Get the real-time audio of the performance, and perform frame processing on the obtained performance audio. Then the audio y of the nth frame n The corresponding two-dimensional hidden state vector is recorded as: s n =(x n ,v n ) T , where x n Indicates the position of the musical score, the unit is beat; v n Indicates the playing speed corresponding to the current audio frame, in BPM, where T stands for transpose.

[0013] In this solution, a state equation of the music score following system is established based on the mathematical model of the music score following system. The system state equation is expressed as:

[0014] x n =x n-1 +λ·v n-1

[0015]

[0016] Where n represents the nth frame of the audio, and x in the first equation is n Indicates the music score playing position corresponding to the current audio frame n, x n-1 Indicates the position of the music score corresponding to the previous audio frame, λ indicates the frame jump, v n-1 Indicates the playing speed corresponding to the previous audio frame;

[0017] v n Represents the current audio frame y n The corresponding playing speed, when the position of the music score corresponding to the nth audio frame just passes through a note, it is considered that there is a possibility of speed change, at the speed v of the previous frame n-1 On the basis of v To simulate the possible acceleration or deceleration in the performance; when the current audio frame is within the duration of a note, it is considered that the performance speed v n and the playing speed v of the previous frame n-1 The same means that there is no speed change operation within the default note duration.

[0018] In this scheme, the observation equation of the music score following system is established based on the mathematical model of the music score following system:

[0019] The observation equation is defined by combining pitch features and Chroma features:

[0020] In the current audio frame signal y n and its corresponding musical score position x n When the frequency distribution of all notes is θ, calculate their pitch feature probability values respectively and Chroma characteristic probability value p chroma , where the probability formula of pitch feature is:

[0021]

[0022] Among them, θ represents the current score position x n The frequency distribution of all notes at n|θ) represents the observation value y of the current audio frame under the premise of a given note frequency distribution n Likelihood, A is the normalization constant;

[0023] Chroma feature probability formula:

[0024]

[0025]

[0026] Convert the spectrum of a frame signal into a Chroma vector with a dimension of 12. Represents the Chroma feature of the audio signal, c m The Chroma feature corresponding to the MIDI number of the pitch on the musical score is represented by α, which represents the cosine angle between the two Chroma features of the MIDI corresponding to the musical score, where B is a normalization constant.

[0027] The final observation equation, that is, the emission probability p(y n |s n ) is expressed as:

[0028] p(y n |s n )=p pitch ·p chroma .

[0029] In this solution, the constructed system state equation and observation equation are used to perform real-time music score following. The specific process is as follows:

[0030] (1) Initialize M particles in a two-dimensional space and record the current position x of the i-th particle (i) and its corresponding playing speed v (i) , M particles in two-dimensional space are expressed as: (x (1) ,v (1) ),…,(x (M) ,v (M) ), where x (i) = 1, indicating that all particles are on the first beat of the score at the beginning; and the speeds of the M particles obey a uniform distribution, with upper and lower limits at the default fastest playing speed v h and the slowest playing tempo v l Between, that is, v (i) ~U[v l ,v h ];

[0031] (2) For the 0th frame of the performance audio, let the initial score position x0 = 1, v0 = initial score velocity;

[0032] (3) For n=1 to the Nth audio frame, perform the following processing on each particle:

[0033] x (i) =x (i) +l·v (i)

[0034] l represents the number of minutes that the frame jump length of each audio frame can be converted into. If x n-2 ≤z k ≤x n-1 , where z k Indicates the beat position of the kth note on the musical score:

[0035] Then v (i) =v n-1 +n v , otherwise v (i) =v (i) ;

[0036] (4) The observation equation of the system is rewritten as: (i) =p(y n |x (i) ,v (i) );

[0037] (5) Weight w for all particles (i) Perform sum normalization:

[0038] (6) According to w (i) Replace all particles of this iteration to obtain a new set of particles (x (1) ,v (1) ),…,(x (M) ,v (M) );

[0039] (7) Output the music score position corresponding to the current audio frame and the playing speed of the current audio frame

[0040] In this solution, the real-time music score following method further includes: when a pause occurs in the performance, the RMS energy value of each audio input frame is first detected. When the RMS energy values of 50 consecutive frames of signals are all lower than the threshold and the current score position is in a non-rest paragraph, it is considered that the performance has been interrupted. At this time, the state x of all particles is changed to (i) ,v (i) The current position of the music score is kept unchanged and the interrupted beat is recorded until the RMS energy of the latest audio frame is detected to be greater than the threshold. Then the performance is considered to have restarted and the particle filter algorithm steps (1) to (7) are resumed.

[0041] In this solution, the real-time music score following method further includes: when the performance pauses and returns to play, all particles are evenly distributed between the interrupted beat and the four beats before it, and the performance then restarts from any note between these four beats.

[0042] A second aspect of the present invention provides a real-time music score following system, comprising: a memory and a processor, wherein the memory includes a real-time music score following method program, and when the real-time music score following method program is executed by the processor, the following steps are implemented:

[0043] Construct a mathematical model of the score-following system;

[0044] Based on the mathematical model of the music score following system, the state equation and observation equation of the music score following system are established;

[0045] Real-time music score following is performed using the constructed system state equations and observation equations.

[0046] In this solution, the specific process of constructing the mathematical model of the music score following system is as follows:

[0047] Get the real-time audio of the performance, and perform frame processing on the obtained performance audio. Then the audio y of the nth frame n The corresponding two-dimensional hidden state vector is recorded as: s n =(x n ,v n ) T , where x n Indicates the position of the musical score, the unit is beat; v n Indicates the playing speed corresponding to the current audio frame, in BPM, where T stands for transpose.

[0048] A third aspect of the present invention provides a computer-readable storage medium, which includes a real-time music score following method program. When the real-time music score following method program is executed by a processor, the steps of the real-time music score following method are implemented.

[0049] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0050] The present invention constructs a system state equation and observation equation suitable for nonlinear and non-Gaussian systems, adopts Monte Carlo sampling to approximate the posterior probability, that is, a particle filtering method to calculate the current filter output, which can improve the accuracy and stability of real-time music score tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of the real-time music score following method of the present invention.

[0052] Figure 2This is a schematic diagram of the HMM model principle of the music score following system according to an embodiment of the present invention.

[0053] Figure 3 This is a flowchart of performing real-time music score following using the constructed system state equation and observation equation according to an embodiment of the present invention.

[0054] Figure 4 This is a flowchart of the scene processing of stopping performance and returning to play again in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0057] Example 1

[0058] like Figure 1 As shown, the first aspect of the present invention provides a real-time music score following method, comprising the following steps:

[0059] S1 builds a mathematical model of the music score following system;

[0060] It should be noted that the present invention uses HMM (Hidden Markov Model) to perform mathematical modeling on the music score following, such as Figure 2 As shown, the real-time audio of the performance is obtained and the obtained performance audio is framed. The audio y of the nth frame is n The corresponding two-dimensional hidden state vector is recorded as: s n =(x n ,v n ) T , where x n Indicates the position of the musical score, the unit is beat; v n Indicates the playing speed in BPM, and T stands for transpose.

[0061] It should be noted that the purpose of constructing the above mathematical model is to obtain the current and historical observation signals y1,…,y n Estimate the position x of the current audio frame on the score n And the corresponding playing speed v n .

[0062] S2 establishes the state equation and observation equation of the music score following system based on the mathematical model of the music score following system;

[0063] It should be noted that after constructing the mathematical model of the score-following system, it is necessary to further establish the system state equation and observation equation. Given an audio frame input, the current score position can be inferred and a real-time method can be found to implement the entire process.

[0064] Based on the mathematical model of the music score following system, a state equation of the music score following system is established. The system state equation is expressed as follows:

[0065] x n =x n-1 +λ·v n-1

[0066]

[0067] Where n represents the nth frame of the audio, and x in the first equation is n Indicates the music score playing position corresponding to the current audio frame n, x n-1 Indicates the position of the music score corresponding to the previous audio frame, λ indicates the frame jump, v n-1 Indicates the playing speed corresponding to the previous audio frame;

[0068] v n Represents the current audio frame y n The corresponding playing speed, when the position of the music score corresponding to the nth audio frame just passes through a note, it is considered that there is a possibility of speed change, at the speed v of the previous frame n-1 On the basis of v To simulate the possible acceleration or deceleration in the performance; when the current audio frame is within the duration of a note, it is considered that the performance speed v n and the playing speed v of the previous frame n-1 The same means that there is no speed change operation within the default note duration.

[0069] It should be noted that the role of the observation model (i.e., observation equation) is to describe whether the current inferred state s n Can explain the observed value y n , that is, calculate p({y n |s n In this step, different audio features can be used, such as power spectrum features, Chroma features, pitch features, onset features, and other commonly used audio features. The present invention uses the combination of pitch features and Chroma features as an example to define the observation model.

[0070] The observation equation of the music score following system is established based on the mathematical model of the music score following system:

[0071] The observation equation is defined by combining pitch features and Chroma features:

[0072] In the current audio frame signal y n and its corresponding musical score position x n When the frequency distribution of all notes at is θ, calculate the pitch feature probability value p respectively pitch and Chroma characteristic probability value p chroma , where the probability formula of pitch feature is:

[0073]

[0074] Among them, θ represents the current score position x n The frequency distribution of all notes at n |θ) represents the observation value y of the current audio frame under the premise of a given note frequency distribution n Likelihood, A is the normalization constant;

[0075] Chroma feature probability formula:

[0076]

[0077]

[0078] Convert the spectrum of a frame signal into a Chroma vector with a dimension of 12. Represents the Chroma feature of the audio signal, c m The Chroma feature corresponding to the MIDI number of the pitch on the musical score is represented by α, which represents the cosine angle between the two Chroma features of the MIDI corresponding to the musical score, where B is a normalization constant.

[0079] The final observation equation, that is, the emission probability p(y n |s n ) is expressed as:

[0080] p(y n |s n )=p pitch ·p chroma .

[0081] S3 uses the constructed system state equations and observation equations to perform real-time music score following.

[0082] It should be noted that the system to which this invention applies, based on the constructed state equation and observation equation, is a nonlinear, non-Gaussian system. Bayesian filtering theory cannot derive an analytical solution, requiring an approximate approach. Monte Carlo sampling is a suitable approach for this problem. Specifically, a particle filter algorithm is used to approximate the output of the score-following system.

[0083] Use the constructed system state equation and observation equation to perform real-time music score following, such as Figure 3 As shown, the specific process is:

[0084] (1) Initialize M particles in a two-dimensional space and record the current position x of the i-th particle (i) and its corresponding playing speed v (i) , M particles in two-dimensional space are expressed as: (x (1) ,v (1) ),…,(x (M) ,v (M) ), where x (i) = 1, indicating that all particles are on the first beat of the score at the beginning; and the speeds of the M particles obey a uniform distribution, with upper and lower limits at the default fastest playing speed v h and the slowest playing tempo v l Between, that is, v (i) ~U[v l ,v h ];

[0085] (2) For the 0th frame of the performance audio, let the initial score position x0 = 1, v0 = initial score velocity;

[0086] (3) For n=1 to the Nth audio frame, perform the following processing on each particle:

[0087] x (i) =x (i) +l·v (i)

[0088] l represents the number of minutes that the frame jump length of each audio frame can be converted into, that is, how many minutes, the unit is minutes, which is a constant. For example: if the audio sampling rate is 44100Hz and the frame jump is 441 sampling points, then the interval between two audio frames is 0.01 seconds, and the unit converted into minutes is 0.01 / 60 minutes. At this time, l = 0.01 / 60, and v in the formula (i) The unit is beat / minute, and after multiplying it with l, we get the current position x of particle i. (i) The number of beats.

[0089] If x n-2 ≤z k ≤xn-1 , where z k Indicates the beat position of the onset / offset of the kth note on the musical score:

[0090] Then v (i) =v n-1 +n v , otherwise v (i) =v (i) ;

[0091] (4) The observation equation of the system is rewritten as: (i) =p(y n |x (i) ,v (i) );

[0092] (5) Weight w for all particles (i) Perform sum normalization:

[0093] (6) According to w i) Replace all particles of this iteration to obtain a new set of particles (x (1) ,v (1) ),…,(x (M) ,v (M) ), the basic idea of replacement is to copy particles with larger weights several times and ignore particles with smaller weights;

[0094] (7) Output the music score position corresponding to the current audio frame and the playing speed of the current audio frame

[0095] like Figure 4 As shown, in this solution, the real-time music score following method further includes: when a pause occurs in the performance, the RMS energy value of each audio input frame is first detected. When the RMS energy values of 50 consecutive frames of signals are all lower than the threshold value and the current score position is in a non-rest paragraph, it is considered that the performance has been interrupted. At this time, the state x of all particles is reset. (i) ,v (i) The current position of the music score is kept unchanged and the interrupted beat is recorded until the RMS energy of the latest audio frame is detected to be greater than the threshold. Then the performance is considered to have restarted and the particle filter algorithm steps (1) to (7) are resumed.

[0096] In this solution, the real-time music score following method further includes: when the performance pauses and returns to play, all particles are evenly distributed between the interrupted beat and the four beats before it, and the performance then restarts from any note between these four beats.

[0097] A second aspect of the present invention provides a real-time music score following system, comprising: a memory and a processor, wherein the memory includes a real-time music score following method program, and when the real-time music score following method program is executed by the processor, the following steps are implemented:

[0098] Construct a mathematical model of the score-following system;

[0099] Based on the mathematical model of the music score following system, the state equation and observation equation of the music score following system are established;

[0100] Real-time music score following is performed using the constructed system state equations and observation equations.

[0101] In this solution, the specific process of constructing the mathematical model of the music score following system is as follows:

[0102] Get the real-time audio of the performance, and perform frame processing on the acquired performance audio. The corresponding two-dimensional hidden state vector of the audio of the nth frame is recorded as: n =(x n ,v n ) T , where x n Indicates the position of the musical score, the unit is beat; v n Indicates the playing speed in BPM, and T stands for transpose.

[0103] A third aspect of the present invention provides a computer-readable storage medium, which includes a real-time music score following method program. When the real-time music score following method program is executed by a processor, the steps of the real-time music score following method are implemented.

[0104] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A real-time music score following method, characterized in that: The following steps are involved: Construct a mathematical model of the score-following system; Based on the mathematical model of the music score following system, the state equation and observation equation of the music score following system are established; The specific process of constructing the mathematical model of the music score following system is as follows: Get the real-time audio of the performance, and perform frame processing on the obtained performance audio. The audio of the nth frame The corresponding two-dimensional hidden state vector is recorded as: ,in Indicates the position of the musical score, the unit is beat; Indicates the playing speed corresponding to the current audio frame, the unit is BPM, that is, beats per minute, T represents transpose; The state equation of the music score following system is established based on the mathematical model of the music score following system. The system state equation is expressed as follows: ； ； Where n represents the nth frame of the audio, and in the first equation Indicates the music score playing position corresponding to the current audio frame n. Indicates the musical score position corresponding to the previous audio frame. Indicates frame skipping, Indicates the playing speed corresponding to the previous audio frame; Indicates the current audio frame The corresponding playing speed, when the position of the music score corresponding to the nth audio frame just passes through a note, it is considered that there is a possibility of speed change, at the speed of the previous frame Based on this, noise is added To simulate the possible acceleration or deceleration in the performance; when the current audio frame is within the duration of a note, it is considered that the performance speed at this time and the playing speed of the previous frame The same, that is, there is no operation of changing the playing speed within the default note's effective time; The observation equation of the music score following system is established based on the mathematical model of the music score following system: The observation equation is defined by combining pitch features and Chroma features: In the current audio frame signal and its corresponding musical score position The frequency distribution of all notes at is In the case of, calculate the pitch feature probability value respectively and Chroma characteristic probability value , where the probability formula of pitch feature is: ； in, Indicates the current score position The frequency distribution of all notes at Represents the observed value of the current audio frame given the frequency distribution of the given note Likelihood, A is the normalization constant; Chroma feature probability formula: ； ； Convert the spectrum of a frame signal into a Chroma vector with a dimension of 12. Represents the Chroma characteristics of the audio signal, The Chroma feature corresponding to the MIDI number of the pitch on the musical score, Indicates the calculation of the cosine angle of the two Chroma features of the audio corresponding to the MIDI score, where B is the normalization constant; The final observation equation, that is, the emission probability in HMM Expressed as: ； The constructed system state equation and observation equation are used to perform real-time music score following. When the performance pauses or when the performance pauses and then returns to play, different music score following processing is performed.

2. A real-time music score following method according to claim 1, characterized in that: The constructed system state equation and observation equation are used to perform real-time music score following. The specific process is as follows: (1) Initialize M particles in a two-dimensional space, record the first and its corresponding playing speed , M particles in two-dimensional space are expressed as: ,in, Indicates that at the beginning all particles are on the first beat of the score; and the speeds of the M particles obey a uniform distribution, with the upper and lower limits being the default fastest playing speed. and the slowest playing speed Between, that is ; (2) For the 0th frame of the performance audio, let the initial score position , ; (3) For The following processing is performed on each particle until the Nth audio frame: ； Indicates the number of minutes that the frame jump length of each audio frame can be converted into. If ,in Indicates the first The beat position of the note head: but ,otherwise ; (4) The observation equation of the system is rewritten as: ; (5) Weights of all particles Perform sum normalization: ; (6) According to Replace all particles of this iteration to get a new set of particles ; (7) Output the music score position corresponding to the current audio frame and the playing speed of the current audio frame .

3. A real-time music score following method according to claim 2, characterized in that: The real-time score following method also includes: when the performance pauses, first detect the RMS energy value of each audio input frame, when the RMS energy values of 50 consecutive frames of signals are all lower than the threshold value, and the current score position is in a non-rest paragraph, it is considered that the performance is interrupted, at this time the state of all particles is reset. The current position of the music score is kept unchanged and the interrupted beat is recorded until the RMS energy of the latest audio frame is detected to be greater than the threshold. Then the performance is considered to have restarted and the particle filter algorithm steps (1) to (7) are resumed.

4. A real-time music score following method according to claim 2, characterized in that: The real-time score following method also includes: when the performance pauses and plays back, all particles are evenly distributed between the interrupted beat and the four beats before it, and the performance is restarted from any note between the four beats.

5. A real-time music score following system, characterized in that: The system includes: a memory and a processor. The memory includes a real-time music score following method program. When the real-time music score following method program is executed by the processor, the following steps are implemented: Construct a mathematical model of the score-following system; Based on the mathematical model of the music score following system, the state equation and observation equation of the music score following system are established; The specific process of constructing the mathematical model of the music score following system is as follows: Get the real-time audio of the performance, and perform frame processing on the obtained performance audio. The audio of the nth frame The corresponding two-dimensional hidden state vector is recorded as: ,in Indicates the position of the musical score, the unit is beat; Indicates the playing speed corresponding to the current audio frame, the unit is BPM, that is, beats per minute, T represents transpose; The state equation of the music score following system is established based on the mathematical model of the music score following system. The system state equation is expressed as follows: ； ； Where n represents the nth frame of the audio, and in the first equation Indicates the music score playing position corresponding to the current audio frame n. Indicates the musical score position corresponding to the previous audio frame. Indicates frame skipping, Indicates the playing speed corresponding to the previous audio frame; Indicates the current audio frame The corresponding playing speed, when the position of the music score corresponding to the nth audio frame just passes through a note, it is considered that there is a possibility of speed change, at the speed of the previous frame Based on this, noise is added To simulate the possible acceleration or deceleration in the performance; when the current audio frame is within the duration of a note, it is considered that the performance speed at this time and the playing speed of the previous frame The same, that is, there is no operation of changing the playing speed within the default note's effective time; The observation equation of the music score following system is established based on the mathematical model of the music score following system: The observation equation is defined by combining pitch features and Chroma features: In the current audio frame signal and its corresponding musical score position The frequency distribution of all notes at is In the case of, calculate the pitch feature probability value respectively and Chroma characteristic probability value , where the probability formula of pitch feature is: ； in, Indicates the current score position The frequency distribution of all notes at Represents the observed value of the current audio frame given the frequency distribution of the given note Likelihood, A is the normalization constant; Chroma feature probability formula: ； ； Convert the spectrum of a frame signal into a Chroma vector with a dimension of 12. Represents the Chroma characteristics of the audio signal, The Chroma feature corresponding to the MIDI number of the pitch on the musical score, Indicates the calculation of the cosine angle of the two Chroma features of the audio corresponding to the MIDI score, where B is the normalization constant; The final observation equation, that is, the emission probability in HMM Expressed as: ； The constructed system state equation and observation equation are used to perform real-time music score following. When the performance pauses or when the performance pauses and then returns to play, different music score following processing is performed.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a real-time music score following method program, and when the real-time music score following method program is executed by a processor, the steps of a real-time music score following method according to any one of claims 1 to 4 are implemented.

Images