A method for beat detection and velocity estimation based on rhythm state space graph

Abstract

This invention discloses a beat detection and tempo estimation method based on a rhythm state space graph. The method includes acquiring the beat and tempo detection results of music; constructing a two-dimensional rhythm state space graph of the music, explicitly combining the note intensity feature function with the music tempo spectrum; and, based on this state space graph and a Bayesian optimization method, achieving joint tracking of beat and local tempo under complex musical styles through maximum a posteriori estimation. This method uses a Bayesian approach to jointly analyze the beat and tempo detection results, enabling stable tracking in dynamically changing and noisy musical environments. It is particularly suitable for styles with complex rhythmic variations, such as classical music and jazz, effectively reducing tempo estimation errors and improving the accuracy and stability of beat detection.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for beat detection and velocity estimation based on rhythm state space graphs. Background Technology

[0002] Currently, music beat and tempo detection are widely used in music recommendation, audio editing, music composition, and musicological analysis. Software-based methods enable functions such as classification and retrieval, style analysis, track alignment, and automatic music composition, enhancing the user experience. Beat is a periodic, perceptible rhythmic position in music; tempo is the frequency of beats, usually expressed in beats per minute (BPM), representing the tempo of the music. While beat detection and tempo estimation are relatively easy to implement in simple musical environments, they face greater challenges in complex settings such as classical or jazz music. These styles often lack prominent percussion, making beats difficult to identify. The natural fluctuations in human performance and the dynamic changes in musical rhythm (acceleration or deceleration) further complicate beat detection and tempo estimation. Furthermore, the presence of rests and syncopation can cause notes to shift or disappear. Mainstream processing methods rely primarily on note intensity characteristics, often treating tempo estimation as a subsequent task. In related technologies, after obtaining the note intensity feature function from the signal processing module or neural network, assuming the tempo of the entire piece is stable, a peak selection algorithm is used to select significant positions in this feature function as beats, and the tempo is calculated by statistically analyzing all beat intervals. The drawback of this type of post-processing method is that it requires a pre-assertion of a correct rhythmic variation range, and it cannot accurately track large jumps or drifts in local performance tempo. Some techniques treat tempo estimation as a separate problem, using neural networks to predict the local tempo at the current moment, obtaining a tempo spectrum showing the probability distribution of the local tempo over time, and extracting the local tempo through local maxima. The drawback of this method is that it ignores the implicit correlation of tempo changes, is prone to rhythmic instability, and cannot simultaneously obtain the beat position. Therefore, it is necessary to fuse the note intensity features and the tempo spectrum to provide a joint post-processing method for beats and tempo. Summary of the Invention

[0003] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of this invention is to provide a beat detection and velocity estimation method based on rhythm state space diagram.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A beat detection and velocity estimation method based on rhythm state space graph includes:

[0006] Acquire audio data, divide the audio data into frames to obtain multiple data frames and the corresponding frame timestamps, and extract the spectral features of the audio data.

[0007] The note intensity feature function is obtained from the spectral features;

[0008] The velocity spectrum of the music is obtained from the spectral features;

[0009] By modeling the joint probability of the note intensity feature function and the tempo spectrum, a two-dimensional rhythm state space diagram is obtained, wherein the state variables of the two-dimensional rhythm state space diagram are the musical beat position and the musical tempo.

[0010] Hidden Markov Models are introduced to estimate the state variables in the two-dimensional rhythm state space, obtaining the optimal estimated sequence of music beat and tempo, which is used to describe music rhythm.

[0011] Furthermore, the audio data is a mono audio sampling point sequence, and the spectral features are obtained by performing a short-time Fourier transform on the data frames.

[0012] Furthermore, the acquisition of the note intensity feature function specifically employs signal processing methods or neural network models to predict the probability of musical beats, thereby obtaining the note intensity feature function. The note intensity feature function is specifically a one-dimensional vector O of length N, where each element of the one-dimensional vector corresponds to the probability value of a frame of data being a beat.

[0013] Furthermore, the spectral features are used to obtain the velocity spectrum of the music, specifically by using signal processing or neural network models to predict the probability distribution of the velocity at all the frame moments.

[0014] Furthermore, the two-dimensional rhythm state space diagram is a fusion of the beat probability and speed probability distribution of music, specifically, the two probability distributions are multiplied to obtain a matrix of size N×M.

[0015] Furthermore, a Hidden Markov Model is introduced to estimate the state variables in the two-dimensional rhythm state space graph, obtaining the optimal estimated sequence of musical beats and tempo, which is used to describe musical rhythm, specifically:

[0016] Hidden Markov models are introduced to describe the relationship between state variables and observations, transforming the problem of maximizing the joint probability distribution into a recursive Bayesian estimation problem;

[0017] Based on the dynamic model of rhythm, assumptions are made about the prior distribution of state variables. Based on the prior, observation information is obtained from the two-dimensional rhythm state space. Under the Bayesian framework, the maximum a posteriori estimate of the state variables is solved sequentially to obtain the optimal state estimate sequence. Furthermore, the optimal estimate sequences of music beat and music tempo are obtained.

[0018] Furthermore, specifically:

[0019] The state variables include random variables τ and v, where τ represents the clock time and v represents the velocity;

[0020] Based on existing observations y 1:K Find a sequence of state variables x under the given conditions 1:K The joint distribution p(x) is such that 1:K ,y 1:K The maximum value is obtained, where K is the number of state variables, and K is a positive integer;

[0021] Based on the first-order Markov property assumption and the observation independence assumption, the state at time k depends only on the state at time k-1, and the observation y... k Depends only on the state x at time k k The joint distribution can be factored as follows:

[0022]

[0023] Where p(x1) is the prior distribution of the initial state, p(x k |x k-1 p(y) represents the state transition probability. k |x k () is a two-dimensional rhythmic state space diagram;

[0024] The optimal estimated sequence of beats is obtained by maximizing the joint distribution.

[0025] An apparatus based on the aforementioned beat detection and velocity estimation method includes:

[0026] Music feature extraction module: used to extract spectral features from audio data;

[0027] Beat probability extraction module: used to extract beat detection results and obtain note intensity feature function;

[0028] Velocity probability extraction module: used to extract the probability distribution of velocity to obtain the velocity spectrum;

[0029] Rhythm State Space Modeling Module: Used to create a two-dimensional rhythm state space diagram;

[0030] State variable estimation module: Used to estimate the optimal sequence of music beats and tempos to describe the music rhythm.

[0031] A computer-readable storage medium storing a computer program for execution by a processor to implement the method.

[0032] A computer program product comprising a computer program that is loaded and executed by a processor to implement the method.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] (1) By constructing a two-dimensional rhythm state space graph, the note intensity feature function and the velocity spectrum are explicitly combined and modeled in a unified manner to achieve synchronous optimization of beat position and velocity, thereby improving the detection accuracy in complex music environments.

[0035] (2) By introducing the Bayesian method, the beat and speed are jointly tracked based on the maximum a posteriori estimation, which can provide more stable detection results in dynamic and noisy music environments.

[0036] (3) This method is designed for styles with complex rhythmic changes, such as classical music and jazz. It can cope with challenges such as natural fluctuations, speed drift, and lack of percussion instruments. It is especially suitable for music styles with irregular rhythms and large speed changes. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the implementation environment of the method of the present invention;

[0038] Figure 2 This is a schematic diagram of the structure of the method of the present invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.

[0040] like Figure 1 As shown, one embodiment of the present invention provides a solution implementation environment based on computer device 10. This environment includes electronic devices capable of data computation, processing, and storage. Computer device 10 can be a terminal device (such as an edge device, mobile phone, computer, intelligent voice interaction device, smart home appliance, vehicle terminal, aircraft, game console, wearable device, multimedia playback device, augmented reality device, virtual reality device, etc.) or a server (such as an independent physical server, server cluster or distributed system, and cloud server that provides basic cloud computing services such as edge computing, cloud services, cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network, big data and artificial intelligence platform, etc.).

[0041] By implementing the method provided in this embodiment, the computer device 10 can perform joint analysis of the beat and speed detection results, and can stably track in dynamic and noisy music environments. It is particularly suitable for styles with complex rhythmic changes, such as classical music and jazz. The embodiments of this application can be widely used in scenarios such as music creation, music editing, virtual instrument performance, and online singing platforms.

[0042] like Figure 2 As shown, this embodiment provides a beat detection and velocity estimation device based on a rhythm state space graph, suitable for styles with complex rhythmic variations such as classical music and jazz. This device has the functionality to implement the method example described above. This functionality can be implemented in hardware or by hardware executing corresponding software. This device can be the computer device 10 described above, or it can be installed within the computer device 10. For example... Figure 2 As shown, the device 200 may include an audio feature extraction module 210, a beat probability extraction module 220, a velocity probability extraction module 230, a rhythm state space modeling module 240, and a state variable estimation module 250.

[0043] include:

[0044] The audio feature extraction module 210 is used to extract the spectral features of the audio, wherein the audio data is input to the system as a mono signal with a sampling rate of 44.1kHz. If it is multi-channel audio data, the mono data is obtained by averaging all channels; if it is audio data other than 44.1kHz, it is obtained by upsampling / downsampling to 44.1kHz. Assuming that the audio data is divided into frames with a window length of 882 and a step size of 441, the interval between each data frame is 10ms, and the time of each frame is t = 10, 20, 30, ... ms. A Hanning window is applied to the data frames, and a short-time Fourier transform is performed to obtain the time spectrum. Optionally, the time spectrum is subjected to Mel scaling to obtain the Mel cepstrum.

[0045] The beat probability extraction module 220 is used to extract the beat detection results and obtain the note intensity feature function, which can be used to predict the beat probability of music through signal processing methods or neural network models. Optionally, the above music beat extraction result can be a single vector. For example, if N is the number of frames and N is a positive integer, the prediction result is a one-dimensional vector O of length N, where each element of the one-dimensional vector corresponds to the probability value of a frame being a beat. In some embodiments, the beat probability detection process can be obtained by performing time-domain differential summation, half-wave rectification, and normalization on the spectrum. Beat probability detection can also be implemented using one of the following neural network models: recurrent neural network, convolutional neural network, long short-term memory recurrent neural network, etc., or other models capable of predicting the probability value of each frame of music being a beat.

[0046] The velocity probability extraction module 230 is used to extract the probability distribution of velocity to obtain a velocity spectrum. It can predict the probability distribution of velocity at all frame moments using signal processing or a neural network model. For example, assuming the number of possible discrete velocity values ​​is M, where M is a positive integer, the prediction result is an N×M matrix R(n,m) representing the probability that the velocity is valued as m at time n in the discrete space. Optionally, the velocity probability extraction module can be implemented using autocorrelation methods, wavelet transforms, template similarity matching, and convolution, or it can be implemented using one of the following neural network models: recurrent neural networks, convolutional neural networks, long short-term memory recurrent neural networks, etc., or other models capable of predicting the probability distribution of velocity in each frame of music.

[0047] The rhythm state space modeling module 240 is used to establish a two-dimensional rhythm state space diagram. Based on the beat probability vector O and the tempo probability distribution matrix R of the music, the joint probability distribution of beat and tempo is modeled as a two-dimensional rhythm state space diagram Tg, which is a matrix of size N×M. The state variables include beat position and tempo. The two-dimensional rhythm state space diagram is a fusion of the probability distributions of the beat and tempo of the music. Mathematically, it can be...

[0048] T g (n,m)=O(n)·R(n,m)

[0049] Where n = 1, 2, ..., N, m = 1, 2, ..., M. n and m correspond to beat position and velocity, respectively. This method describes the joint characteristics of rhythm by using the product of two probability distributions as the value of the state-space diagram.

[0050] To further explain, the fusion method of the beat probability vector O and the velocity probability distribution matrix R is not limited to multiplication. Other mathematical methods such as linear combination, logical functions, power functions, or polynomial functions can also be used to fuse them in a more flexible way to capture the relationship between rhythm and velocity.

[0051] To further explain, in order to reduce the impact of outliers on the calculation results, smoothing techniques such as Gaussian smoothing, exponential smoothing, or Laplace smoothing can be applied to the obtained state space graph to improve the robustness and accuracy of the model.

[0052] To further explain, the discretization of the rhythm state space diagram can be combined with multi-resolution modeling methods to describe the multi-level characteristics of rhythm by constructing state diagrams at different time and velocity scales.

[0053] The state variable estimation module 250 is used to provide the optimal estimate of the state variables, using a Hidden Markov Model (HMM) to describe the music rhythm.

[0054] Specifically, the state variable x includes random variables τ and v, where τ represents the beat time and v represents the speed, which can be represented by the time interval between adjacent beats or by the number of beats per minute (BPM). There is no restriction here.

[0055] In order to find the best prediction of beat and velocity, based on existing observations y 1:K Find a state sequence x under the condition 1:K The joint distribution p(x) is such that 1:K ,y 1:K The state at time k reaches its maximum value, where K is the number of state variables, and K is a positive integer. Based on the first-order Markov property assumption and the observation independence assumption, the state at time k depends only on the state at time k-1, and the observation y... k Depends only on the state x at time k k The joint distribution can be factored as follows:

[0056]

[0057] Where p(x1) is the prior distribution of the initial state, p(x k |x k-1 p(y) represents the state transition probability, given by the system dynamic model; k |x k ) is the observation model, derived from the state-space graph T g Provided.

[0058] By maximizing the joint distribution p(x) 1:K ,y 1:K Solving for the optimal estimate of the beat is transformed into recursively solving for the maximum a posteriori probability p(x) at each time step. k |y k ,x k-1 By Bayes' theorem, this is equivalent to maximizing the product p(y) of the transition probability and the observation model step by step. k |x k )p(x k |x k-1 The derivation process is as follows:

[0059]

[0060] Where p(x1) is the prior distribution of the initial state, p(x k |x k-1 p(y) represents the state transition probability, given by the system dynamic model; k|x k ) is the observation model, derived from the state-space graph T g Provided.

[0061] Due to the nature of the model, the observed y 1:K In fact, it is implicit in the state sequence x 1:K In this process, the observation process can be viewed as a state-driven generative process, gradually generating y. 1:K First, initialize an initial state x1 and generate initial observation y1 using the model p(y1|x1). Then, for subsequent times k = 2, 3, ..., K, according to the state transition model p(x1|x1),... k |x k-1 Predict the next state Then, based on the observation model p(y) k |x k ) and predicted state Generate observation y k .

[0062] To further explain, the observation model p(y) k |x k From the two-dimensional rhythmic state space diagram T g express:

[0063] p(y k |x k ) = T g (x k )

[0064] Where, x k The cycle time n and velocity m are represented in the discretized state space, where n∈[1,N], m∈[1,M]. To reduce computational cost and avoid the influence of certain anomalous observations, the possible observation range χ is limited. k And renormalize the conditional probability p(y) k |x k This mechanism combines geometric constraints and statistical probability, proving highly effective in applications with strong spatiotemporal correlations, such as music rhythm analysis. One possible implementation is within a window. Inside,

[0065]

[0066] Among them, the normalization factor For T g Normalization is performed to ensure that in χ² k The probability of generation within the window is 1, and r is the window radius, which defines the range of observations allowed.

[0067] To further explain, assume that the state variables follow a Gaussian distribution:

[0068]

[0069] Within the framework of a linear Gaussian model, the transition probability of the current state is obtained from a recursive relation:

[0070]

[0071] μ k =Fμ k-1 ,Σ k =FΣ k-1 F T +Q

[0072] Where F is the state transition matrix, describing the linear dynamic characteristics of the system, Q is the process noise covariance matrix, representing the uncertainty introduced during state transition, and μ k-1 ,Σ k-1 These are the mean and covariance of the state at the previous time step. In relevant embodiments, the state transition matrix F, the process noise covariance matrix Q, and the mean and covariance μ0, Σ0 of the initial state are not specified. After the above decoding process, the optimal estimated state sequence can be obtained. This leads to the optimal estimation sequence of beat and velocity.

[0073] To further explain, the state variables can be extended to include not only the existing clock cycle time τ and velocity v, but also their higher-order derivatives, such as acceleration. This allows the model to more accurately capture the dynamics of rhythm changes. At this point, the Hidden Markov Model can be used to describe the expanded state variables, where the state transition probabilities include τ, v, and ... The relationship between them.

[0074] To further clarify, the model structure is not limited to Markov chains; other types of probabilistic graphical models, such as Bayesian networks, can be used. These models represent the causal relationships between state variables through directed acyclic graphs (DAGs) or use dynamic Bayesian networks (DBNs) to process time-series data, thereby achieving more flexible state prediction and estimation.

[0075] To further illustrate, the dynamic transit relationships of state variables can be described using linear or nonlinear models. For example, the state transition equations can be first-order linear dynamic equations, or nonlinear dynamic equations can be introduced to model complex rhythmic changes, or even higher-order equations to capture more subtle patterns of change.

[0076] To further explain, the optimization objective can be maximum likelihood estimation (MLE) to maximize the conditional probability of the observed data; or other objective functions can be used, such as minimizing the observation error, to optimize the state estimation process. Furthermore, optimization methods within a non-Bayesian framework can also serve as alternatives to meet the needs of different application scenarios.

[0077] This embodiment also provides a beat detection and velocity estimation method based on a rhythm state space graph, including:

[0078] Acquire audio data, divide the audio data into frames to obtain multiple data frames and the corresponding frame timestamps, and extract the spectral features of the audio data.

[0079] The note intensity feature function is obtained from the spectral features;

[0080] The velocity spectrum of the music is obtained from the spectral features;

[0081] By modeling the joint probability of the note intensity feature function and the tempo spectrum, a two-dimensional rhythm state space diagram is obtained, wherein the state variables of the two-dimensional rhythm state space diagram are the musical beat position and the musical tempo.

[0082] By introducing a Hidden Markov Model (HMM) to estimate the state variables in the two-dimensional rhythmic state space, the optimal estimated sequences of musical beat and tempo are obtained to describe the musical rhythm, specifically:

[0083] In the process of beat detection and velocity estimation, a two-dimensional rhythm state space diagram is established, and the state variables are recursively estimated by combining the dynamic transfer equation. The recursive estimation combines the joint probability distribution of the musical beat position and velocity.

[0084] Using dynamic modeling techniques, the transfer equation can be a linear or nonlinear model, and can be modeled using the dynamic relationship of first-order or higher-order derivatives;

[0085] During the state estimation process, the state space graph is smoothed to eliminate outliers. The smoothing process includes, but is not limited to, Gaussian smoothing, exponential smoothing, or Laplace smoothing.

[0086] By optimizing the objective function, such as maximum a posteriori estimation, maximum likelihood estimation, or other error minimization methods, the beat and velocity state sequences are decoded, and the optimal beat detection results and velocity estimates are output.

[0087] This embodiment also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement a beat detection and speed estimation method.

[0088] This embodiment also provides a computer program product, which includes a computer program that is loaded and executed by a processor to implement a beat detection and speed estimation method.

[0089] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for beat detection and velocity estimation based on rhythm state space graphs, characterized in that, include: Acquire audio data, divide the audio data into frames to obtain multiple data frames and the corresponding frame timestamps, and extract the spectral features of the audio data. The note intensity feature function is obtained from the spectral features; The velocity spectrum of the music is obtained from the spectral features; By modeling the joint probability of the note intensity feature function and the tempo spectrum, a two-dimensional rhythm state space diagram is obtained, wherein the state variables of the two-dimensional rhythm state space diagram are the musical beat position and the musical tempo. By introducing a Hidden Markov Model (HMM) to estimate the state variables in the two-dimensional rhythmic state space, the optimal estimated sequences of musical beat and tempo are obtained, which are used to describe musical rhythm. Specifically: Hidden Markov models are introduced to describe the relationship between state variables and observations, transforming the problem of maximizing the joint probability distribution into a recursive Bayesian estimation problem; Based on the dynamic model of rhythm, assumptions are made about the prior distribution of state variables. Based on the prior, observation information is obtained from the two-dimensional rhythm state space. The maximum a posteriori estimate of the state variables is solved sequentially under the Bayesian framework to obtain the optimal state estimation sequence. Furthermore, the optimal estimation sequences of music beat and music tempo are obtained. The state variables include random variables τ and v, where τ represents the clock time and v represents the velocity; Based on existing observations y 1:K Find a sequence of state variables x under the given conditions 1:K The joint distribution p(x) is such that 1:K ,y 1:K The maximum value is obtained, where K is the number of state variables, and K is a positive integer; Based on the first-order Markov property assumption and the observation independence assumption, the state at time k depends only on the state at time k-1, and the observation y... k Depends only on the state x at time k k The joint distribution can be factored as follows: Where p(x1) is the prior distribution of the initial state, p(x k |x k-1 p(y) represents the state transition probability. k |x k () is a two-dimensional rhythmic state space diagram; The optimal estimated sequence of beats is obtained by maximizing the joint distribution.

2. The beat detection and velocity estimation method according to claim 1, characterized in that, The audio data is a mono audio sampling point sequence, and the spectral features are obtained by performing a short-time Fourier transform on the data frames.

3. The beat detection and velocity estimation method according to claim 1, characterized in that, The acquisition of the note intensity feature function specifically employs signal processing methods or neural network models to predict the probability of musical beats, thereby obtaining the note intensity feature function. The note intensity feature function is specifically a one-dimensional vector of length N, where each element corresponds to the probability value of a frame of data being a beat.

4. The beat detection and velocity estimation method according to claim 1, characterized in that, The spectral features are used to obtain the velocity spectrum of the music, and signal processing or neural network models are used to predict the probability distribution of the velocity at all the frame moments.

5. The beat detection and velocity estimation method according to claim 1, characterized in that, The two-dimensional rhythm state space diagram is a fusion of the beat probability and tempo probability distribution of music. Specifically, it is the product of the two probability distributions to obtain a matrix of size N×M.

6. An apparatus based on the beat detection and velocity estimation method according to any one of claims 1-5, characterized in that, include: Music feature extraction module: used to extract spectral features from audio data; Beat probability extraction module: used to extract beat detection results and obtain note intensity feature function; Velocity probability extraction module: used to extract the probability distribution of velocity to obtain the velocity spectrum; Rhythm State Space Modeling Module: Used to create a two-dimensional rhythm state space diagram; State variable estimation module: Used to estimate the optimal sequence of music beats and tempos to describe the music rhythm.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1 to 5.

8. A computer program product, characterized in that, The computer program product includes a computer program that is loaded and executed by a processor to implement the method as described in any one of claims 1 to 5.

Images