Training of beat-repeat joint detection model and beat-repeat joint detection method
By including the annotation information of beat, remake and non-beat information in the beat detection model and correlate it with beat count per minute, the problem of low accuracy of overfitting and remake detection in the prior art is solved, and the beat is improved and accuracy of reshoot detection.
Patent Information
- Application Number
- CN202210470363.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-04-28
AI Technical Summary
The prior art has problems with overfitting in beat detection, especially in Western music, where the accuracy of remapping detection is low, and the model is prone to overfitting during training, and it is impossible to effectively distinguish between beats and non-beats.
A training method for joint detection model of beat remapping is proposed. By obtaining sample audio signals and corresponding beat remapping annotation information, the training loss is calculated, and the model parameters are adjusted based on dynamic marking information, including beat, remapping and non-beat information. Avoid overfitting and correlate the label information with the beats per minute of the sample audio signal.
It effectively avoids model overfitting, improves the accuracy of beat and remapping detection, and considers the perceived characteristics of the human ear to rhythm, improves the convergence and detection effect of the model.
Smart Images

Figure CN114897157B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of audio processing, and in particular to a training of a beat-repeat joint detection model and a beat-repeat joint detection method. Background Art
[0002] In music, beat is an organizational form that represents fixed unit duration and dynamics. It defines the rhythmic structure of a musical work, and rhythm is characterized by a repeated sequence of downbeats and non-downbeats. Downbeats refer to strong beats in music. There has been a considerable amount of research focused on finding the beats of music, but much less effort has been made to track downbeats. In Western music, downbeats usually coincide with chord changes or harmony changes, so the joint detection of beats and downbeats is crucial for music analysis, music structure delineation, automatic music matching, and even some creative card point video production, beat-dominated light shows and other practical application scenarios.
[0003] At present, the beat detection algorithm based on deep learning has solved some of the problems of traditional algorithms. It does not need to estimate the position of the starting point of the note, relies on less prior knowledge, and improves the accuracy of beat detection to a certain extent. However, the number of beats, beats, and non-beats in music is seriously unbalanced, and the number of beats is far less than that of non-beats. If only beats and beats are annotated in the annotation information used for training, the model is prone to overfitting during the training process. Summary of the invention
[0004] The present disclosure provides a training method for a beat-repeat joint detection model and a beat-repeat joint detection method to at least solve the problems in the above-mentioned related technologies, or may not solve any of the above-mentioned problems.
[0005] According to a first aspect of an embodiment of the present disclosure, a training method for a beat-rebeat joint detection model is provided, the training method comprising: obtaining a sample audio signal and beat-rebeat labeling information corresponding to the sample audio signal, the beat-rebeat labeling information comprising beat labeling information, rebeat labeling information and non-beat labeling information; inputting the audio features of the sample audio signal into the beat-rebeat joint detection model to obtain a beat-rebeat point prediction probability for each frame of data in the sample audio signal, wherein the beat-rebeat point prediction probability represents the prediction probability that each frame of data in the sample audio signal belongs to a beat point, a rebeat point and a non-beat point respectively; mapping the beat-rebeat labeling information into dynamic labeling information, and calculating a training loss based on the beat-rebeat point prediction probability and the corresponding dynamic labeling information, wherein the dynamic labeling information is associated with the number of beats per minute of the sample audio signal; and adjusting the parameters of the beat-rebeat joint detection model according to the training loss.
[0006] Optionally, mapping the beat and rebeat marking information into dynamic marking information includes: performing windowing processing on the beat and rebeat marking information based on a preset window function to obtain the dynamic marking information.
[0007] Optionally, the window function includes a hyperparameter for determining the curve width of the window function, wherein the beat and rebeat marking information is windowed based on a pre-set window function to obtain the dynamic marking information, including: determining the number of beats per minute of the sample audio signal based on the beat and rebeat marking information; determining the value of the hyperparameter based on the determined number of beats per minute of the sample audio signal; and obtaining the dynamic marking information by windowing the beat and rebeat marking information through the window function based on the determined value of the hyperparameter.
[0008] Optionally, the value of the hyperparameter is negatively correlated with the beats per minute of the sample audio signal.
[0009] Optionally, performing windowing processing on the beat and rebeat marking information by using the window function to obtain the dynamic marking information includes: extracting information from the beat and rebeat marking information based on the window function to obtain the dynamic marking information.
[0010] Optionally, the audio feature of the sample audio signal is acquired by the following steps: converting the sample audio signal into a time-frequency domain signal, and using the time-frequency domain signal as the audio feature of the sample audio signal.
[0011] Optionally, the using the time-frequency domain signal as the audio feature of the sample audio signal includes: filtering the time-frequency domain signal, and using the filtered time-frequency domain signal as the audio feature of the sample audio signal.
[0012] Optionally, the calculating of training loss based on the predicted probability of the beat and the re-beat points and the corresponding dynamic annotation information includes: calculating, according to the predicted probabilities of the beat points, the re-beat points and the non-beat points and the corresponding dynamic annotation information, a first loss corresponding to the predicted probability of the beat points, a second loss corresponding to the predicted probability of the re-beat points and a third loss corresponding to the predicted probability of the non-beat points; and determining the training loss according to the first loss, the second loss and the third loss.
[0013] According to a second aspect of an embodiment of the present disclosure, a beat-repeat joint detection method is provided, and the beat-repeat joint detection method comprises: obtaining an audio signal to be processed; inputting audio features of the audio signal to be processed into a beat-repeat joint detection model trained by the training method of the beat-repeat joint detection model as described above, to obtain the beat-repeat point probability of each frame of data in the audio signal to be processed, wherein the beat-repeat point probability represents the probability that each frame of data in the audio signal to be processed belongs to a beat point, a beat point, and a non-beat point, respectively; and determining the beat-repeat joint detection result of the audio signal to be processed according to the beat-repeat point probability of each frame of data in the audio signal to be processed.
[0014] Optionally, the audio feature of the audio signal to be processed is acquired by the following steps: converting the audio signal to be processed into a time-frequency domain signal, and using the time-frequency domain signal as the audio feature of the audio signal to be processed.
[0015] Optionally, taking the time-frequency domain signal as the audio feature of the audio signal to be processed includes: filtering the time-frequency domain signal, and taking the filtered time-frequency domain signal as the audio feature of the audio signal to be processed.
[0016] Optionally, determining the beat-repeat joint detection result of the audio signal to be processed based on the beat-repeat point probability of each frame of data in the audio signal to be processed includes: decoding the beat-repeat point probability of each frame of data in the audio signal to be processed to obtain a beat-repeat time series, and using the beat-repeat time series as the beat-repeat joint detection result of the audio signal to be processed.
[0017] According to a third aspect of an embodiment of the present disclosure, a training device for a beat-rebeat joint detection model is provided, the training device comprising: a sample acquisition unit, configured to acquire a sample audio signal and beat-rebeat labeling information corresponding to the sample audio signal, the beat-rebeat labeling information comprising beat labeling information, rebeat labeling information and non-beat labeling information; a probability prediction unit, configured to input the audio features of the sample audio signal into the beat-rebeat joint detection model, and obtain the beat-rebeat point prediction probability of each frame of data in the sample audio signal, wherein the beat-rebeat point prediction probability represents the prediction probability that each frame of data in the sample audio signal belongs to a beat point, a rebeat point and a non-beat point respectively; a loss determination unit, configured to map the beat-rebeat labeling information into dynamic labeling information, and calculate the training loss based on the beat-rebeat point prediction probability and the corresponding dynamic labeling information, wherein the dynamic labeling information is associated with the number of beats per minute of the sample audio signal; and a model training unit, configured to adjust the parameters of the beat-rebeat joint detection model according to the training loss.
[0018] Optionally, the loss determination unit is further configured to: perform windowing processing on the beat and rebeat marking information based on a preset window function to obtain the dynamic marking information.
[0019] Optionally, the window function includes a hyperparameter for determining a curve width of the window function, wherein the loss determination unit is further configured to: determine the number of beats per minute of the sample audio signal based on the beat and rebeat labeling information; determine the value of the hyperparameter based on the determined number of beats per minute of the sample audio signal; and perform windowing processing on the beat and rebeat labeling information through the window function based on the determined value of the hyperparameter to obtain the dynamic labeling information.
[0020] Optionally, the value of the hyperparameter is negatively correlated with the beats per minute of the sample audio signal.
[0021] Optionally, the loss determination unit is further configured to: extract information from the beat and rebeat marking information based on the window function to obtain the dynamic marking information.
[0022] Optionally, the sample acquisition unit is further configured to: convert the sample audio signal into a time-frequency domain signal, and use the time-frequency domain signal as an audio feature of the sample audio signal.
[0023] Optionally, the sample acquisition unit is further configured to: perform filtering processing on the time-frequency domain signal, and use the filtered time-frequency domain signal as the audio feature of the sample audio signal.
[0024] Optionally, the loss determination unit is further configured to: calculate a first loss corresponding to the predicted probability of the beat point, a second loss corresponding to the predicted probability of the beat point, and a third loss corresponding to the predicted probability of the non-beat point, respectively, based on the predicted probabilities of the beat points, re-beat points, and non-beat points and the corresponding dynamic annotation information; determine the training loss based on the first loss, the second loss, and the third loss.
[0025] According to a fourth aspect of an embodiment of the present disclosure, a beat-repeat joint detection device is provided, and the beat-repeat joint detection device comprises: an audio acquisition unit, configured to acquire an audio signal to be processed; a probability determination unit, configured to input the audio features of the audio signal to be processed into a beat-repeat joint detection model trained by the training method of the beat-repeat joint detection model as described above, to obtain the beat-repeat point probability of each frame of data in the audio signal to be processed, wherein the beat-repeat point probability represents the probability that each frame of data in the audio signal to be processed belongs to a beat point, a beat point and a non-beat point respectively; and a result determination unit, configured to determine the beat-repeat joint detection result of the audio signal to be processed according to the beat-repeat point probability of each frame of data in the audio signal to be processed.
[0026] Optionally, the audio acquisition unit is further configured to: convert the audio signal to be processed into a time-frequency domain signal, and use the time-frequency domain signal as the audio feature of the audio signal to be processed.
[0027] Optionally, the audio acquisition unit is further configured to: perform filtering processing on the time-frequency domain signal, and use the filtered time-frequency domain signal as the audio feature of the audio signal to be processed.
[0028] Optionally, the result determination unit is further configured to: obtain a beat-rebeat time series by decoding the beat-rebeat point probability of each frame of data in the audio signal to be processed, and use the beat-rebeat time series as the beat-rebeat joint detection result of the audio signal to be processed.
[0029] According to the fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute instructions to implement a training method for a beat-rebeat joint detection model or a beat-rebeat joint detection method according to the present disclosure.
[0030] According to the fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the beat-rebeat joint detection model or the beat-rebeat joint detection method according to the present disclosure.
[0031] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement a training method for a beat-rebeat joint detection model or a beat-rebeat joint detection method according to the present disclosure.
[0032] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0033] According to the training of the beat-rebeat joint detection model and the beat-rebeat joint detection method disclosed in the present invention, during the training process, the loss is calculated by using the labeled information containing beat information, beat information and non-beat information, and the parameters of the beat-rebeat joint detection model are adjusted on this basis, which can effectively avoid overfitting of the model. Taking into account that the human ear's perception of rhythm changes linearly, the labeled information is also associated with the number of beats per minute of the sample audio signal, which helps the model converge and improves the accuracy of beat and beat detection.
[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0036] Figure 1 is a logic diagram showing a beat detection method of the related art;
[0037] Figure 2 is a logic diagram showing another related art beat detection method;
[0038] Figure 3 is a schematic diagram showing an implementation scenario of a training method for a beat-rebeat joint detection model according to an exemplary embodiment of the present disclosure;
[0039] Figure 4 is a flow chart illustrating a method for training a beat-rebeat joint detection model according to an exemplary embodiment of the present disclosure;
[0040] Figure 5 is a curve diagram showing a window function according to an exemplary embodiment of the present disclosure;
[0041] Figure 6 is a flow chart illustrating a beat-rebeat joint detection method according to an exemplary embodiment of the present disclosure;
[0042] Figure 7 is a logic diagram illustrating a beat-rebeat joint detection method according to an exemplary embodiment of the present disclosure;
[0043] Figure 8 is a schematic diagram showing a detection result according to an exemplary embodiment of the present disclosure;
[0044] Fig. 9 is a block diagram illustrating a training apparatus for a beat-rebeat joint detection model according to an exemplary embodiment of the present disclosure;
[0045] Fig.10 is a block diagram illustrating a beat-rebeat joint detection apparatus according to an exemplary embodiment of the present disclosure;
[0046] Fig.11 is a block diagram illustrating an electronic device 1100 according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0047] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.
[0049] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.
[0050] Figure 1 is a logic diagram showing a related art beat detection method.
[0051] Reference Figure 1 The beat detection method of the related art generally first estimates the note starting point (onset) by using the changes in energy / pitch / timbre / phase, and then estimates the BPM (Beats per Minute) by using the correlation between the note starting points based on the prior knowledge that the music speed is basically stable and the range of the music speed. Finally, dynamic programming is used to find a set of beat sequences that can match the estimated BPM to the greatest extent, and the position of the note starting point with the greatest intensity is taken as the optimal beat position.
[0052] The beat detection method of the above-mentioned related technology generally combines the prior knowledge of signal processing and music theory. When the beat is stable, it can obtain relatively accurate results. However, it is difficult to obtain accurate results for music types with varying beats or weak sense of beat, such as classical music and jazz. In addition, due to the small difference between re-beats and non-re-beats, the beat detection method of the above-mentioned related technology has never made a big breakthrough in re-beat detection. Specifically, the beat detection method of the above-mentioned related technology needs to rely on a lot of manual experience and prior knowledge, but these cannot be adapted to all data. This results in that in some music types, the beat detection method of the above-mentioned related technology can obtain relatively accurate detection results, but in some cases that do not match the prior knowledge, the detection results will be very poor. For example, the beat detection method of the above-mentioned related technology takes the BPM of the music as a priori condition that it always remains stable, and is not applicable to music with large BPM changes; and the beat detection method of the above-mentioned related technology generally needs to calculate the starting point of the note first, which can estimate the starting point of the note relatively well for percussion instruments, but cannot estimate the starting point of the note well for string instruments or vocals with strong beats and weak beats, resulting in inaccurate beat detection.
[0053] Figure 2 is a logic diagram showing another related art beat detection method.
[0054] Reference Figure 2 Another related beat detection method is a deep learning-based detection method, which is generally divided into three main steps: feature extraction, beat probability prediction of deep models, and global beat position estimation. First, feature extraction usually uses frequency domain features; second, deep models are generally built based on networks such as BiLSTM (Bi-directional Long Short-Term Memory), and the probability of each frame of data belonging to a beat point can be calculated through deep models; finally, global beat position estimation will be combined with dynamic programming, such as DBN (Dynamic Bayesian Network), to obtain a set of globally optimal beat positions.
[0055] The beat detection method of another related technology mentioned above does not need to estimate the position of the starting point of the note, relies less on prior knowledge, and improves the accuracy of beat detection to a certain extent. However, RNN (Recurrent Neural Network) such as BiLSTM and BiGRU (Bi-directional Gate Recurrent Unit) mainly performs sequential processing over time, and there are potential problems such as gradient explosion and disappearance, and it is impossible to process longer sequence information. In addition, although replacing RNN with TCN (Temporal convolutional network) can change the receptive field by increasing the number of layers, changing the expansion coefficient and the size of the filter, making the length of historical information more flexible, in testing, RNN only needs to maintain a hidden state and accept the current input to generate a prediction, while TCN still requires a complete sequence to make a prediction.
[0056] In addition, beat detection is different from beat detection. The perception of beats depends more on the changes in chords and musical textures. However, since beats are included in beats and together with beats determine the rhythm of music, there are some problems in predicting beats alone. And because the number of beats, beats, and non-beats is seriously unbalanced, the number of beats is far less than that of non-beats. Therefore, if only beats and beats are annotated in the annotation information, the model is prone to overfitting.
[0057] In response to the above problems, the present disclosure provides a training of a beat-rebeat joint detection model and a beat-rebeat joint detection method, which can build a model based on CRNN (Convolutional Recurrent Neural Network) to combine the advantages of CNN (Convolutional Neural Networks) and RNN, and can not only directly learn local features with fewer dimensions from input features, but also learn long-term dependent timing information. In addition, during the training process, the loss can be calculated through the annotation information containing beat information, beat information and non-beat information, and the parameters of the beat-rebeat joint detection model can be adjusted on this basis, which can effectively avoid model overfitting, and considering that the human ear's perception of rhythm is a linear change, the annotation information is also associated with the number of beats per minute of the sample audio signal, which helps the model converge and improves the accuracy of beat and rebeat detection.
[0058] Figure 3 2 is a schematic diagram showing an implementation scenario of a method for training a beat-rebeat joint detection model according to an exemplary embodiment of the present disclosure.
[0059] Reference Figure 3 The implementation scenario includes a server 300, a user terminal 310 and a user terminal 320, wherein the user terminals are not limited to two, and include but are not limited to mobile phones, personal computers and other devices. The user terminal can install an application for obtaining music. The server can be a single server, or a server cluster composed of several servers, or a cloud computing platform or a virtualization center.
[0060] After receiving the request for training the beat-re-beat joint detection model sent by the user terminals 310 and 320, the server 300 counts the music clips received from the user terminals 310 and 320 in the past and respectively marks the beats, re-beats and non-beats in the counted music clips, and merges the marked music clips together as a training sample set, wherein the training sample set includes multiple music clips and the beat marking information, re-beat marking information and non-beat marking information corresponding to each music clip. After acquiring the training sample set, the server 300 records the audio of the music clips therein. The features are input into the beat detection network to obtain the predicted probabilities of the beat points, beat points and non-beat points of each frame of data in the music clip. Then, based on the predicted probabilities of the beat points, beat points and non-beat points, and the corresponding beat annotation information, beat annotation information and non-beat annotation information, the loss is calculated. The parameters of the beat-beat joint detection model are adjusted according to the loss to train the beat-beat joint detection model to obtain the optimal beat-beat joint detection model. The trained model can accurately process the input music clip to be processed, that is, the beat and beat of the music to be processed are obtained.
[0061] Next, we will refer to Figures 4 to 11 The training of a beat-rebeat joint detection model and a beat-rebeat joint detection method and device according to an exemplary embodiment of the present disclosure are described in detail.
[0062] Figure 4 is a flowchart illustrating a training method of a beat-rebeat joint detection model according to an exemplary embodiment of the present disclosure.
[0063] Reference Figure 4, in step S401, a sample audio signal and the beat and downbeat annotation information corresponding to the sample audio signal can be obtained. Here, the beat and downbeat annotation information may include beat annotation information, downbeat annotation information, and non-beat annotation information. Further, the time positions of the beat annotation information, downbeat annotation information, and non-beat annotation information of the sample audio signal can be processed to obtain the basic label for each frame. For example, non-beat can be represented as [1, 0, 0], beat can be represented as [0, 1, 0], and downbeat can be represented as [0, 0, 1], but not limited thereto. By adding the annotation of non-beat, overfitting of the model can be avoided in the case of unbalanced numbers of beats, downbeats, and non-beats. In addition, the beat and downbeat annotation information corresponding to each of the above sample audio signals can be manually annotated or through other annotation methods, and the present disclosure does not limit this here.
[0064] Next, in step S402, the audio features of the sample audio signal can be input into the beat and downbeat joint detection model to obtain the beat and downbeat point prediction probabilities for each frame of data in the sample audio signal. Here, the beat and downbeat point prediction probabilities represent the prediction probabilities that each frame of data in the sample audio signal belongs to a beat point, a downbeat point, and a non-beat point, respectively.
[0065] According to an exemplary embodiment of the present disclosure, the audio features of the sample audio signal can be obtained through the following steps: converting the sample audio signal into a time-frequency domain signal and using the time-frequency domain signal as the audio features of the sample audio signal. By converting the sample audio signal into a time-frequency domain signal, the model can better extract the temporal features and frequency features of the sample audio signal during training. Further, the time-frequency domain signal can be filtered, and the filtered time-frequency domain signal can be used as the audio features of the sample audio signal. Through the above filtering process, data dimensionality reduction of the time-frequency domain signal can be performed, reducing the resource occupation during model training, thereby improving the model performance.
[0066] As an example, first, the sample audio signal can be converted into a time-frequency domain signal, for example, but not limited to, by Short-Time Fourier Transform (STFT); then, the time-frequency domain signal can be converted into logarithmic spectrum features, for example, but not limited to, by logarithmic calculation, so as to filter the time-frequency domain signal and use the obtained logarithmic spectrum features as the audio features of the sample audio signal. Specifically, the audio features of the sample audio signal can be obtained in the following manner:
[0067] 1) Convert the original audio signal into the time-frequency domain through the short-time Fourier transform. If the original audio signal x with a length of T is x(t) in the time domain, where t represents time and 0 < t ≤ T, then after the short-time Fourier transform, x(t) can be represented as X(n, k) in the time-frequency domain:
[0068] X(n,k) = STFT(x(t)) (1)
[0069] Here, STFT(x(t)) represents performing a short-time Fourier transform on x(t); n is the frame sequence, 0 < n ≤ Ns, (Ns is the total number of frames); k is the center frequency sequence, 0 < k ≤ K; (K is the total number of frequency points).
[0070] 2) Calculating the logarithmic spectral features (i.e., the above-mentioned audio features) can be achieved through the following formula:
[0071] Log_Spec(n,k‘) = Filter(X(n,k)) (2)
[0072] Here, Log_Spec(n,k‘) represents the logarithmic spectral features; Filter(X(n,k)) represents performing a filtering process on X(n,k), and k‘ is the frequency sequence after logarithmic spectral conversion.
[0073] After obtaining the logarithmic spectral features, the obtained logarithmic spectral features can be input into the beat and downbeat joint detection model to obtain the probabilities corresponding to each frame of data being a beat point, a downbeat point, and a non-beat point, which can be specifically expressed as follows:
[0074] P(n) = CRNN(Log_Spec(n,k‘)) (3)
[0075] Here, P(n) represents the prediction probability of beat and downbeat points, and CRNN(Log_Spec(n,k‘)) represents inputting the logarithmic spectral features into the model constructed by CRNN.
[0076] Next, in step S403, the beat and downbeat annotation information can be mapped to dynamic annotation information, and based on the prediction probability of beat and downbeat points and the corresponding dynamic annotation information, the training loss is calculated. Here, the dynamic annotation information can be associated with the beats per minute of the sample audio signal. Calculating the loss through annotation information containing beat information, downbeat information, and non-beat information can effectively avoid model overfitting, and associating the annotation information with the beats per minute of the sample audio signal can consider that the human ear's perception of rhythm is linearly changing, thus contributing to model convergence.
[0077] According to an exemplary embodiment of the present disclosure, the beat-rebeat marking information can be windowed based on a pre-set window function to obtain dynamic marking information. Here, the window function may include a hyperparameter for determining the curve width of the window function. Further, the number of beats per minute of the sample audio signal can be determined based on the beat-rebeat marking information; then, the value of the hyperparameter can be determined based on the determined number of beats per minute of the sample audio signal; then, the beat-rebeat marking information can be windowed through the window function based on the determined value of the hyperparameter to obtain dynamic marking information. Here, information extraction can be performed on the beat-rebeat marking information based on the window function to obtain dynamic marking information, such as convolving the beat-rebeat marking information with the window function, but the present disclosure is not limited to this, and those skilled in the art can determine the calculation method for information extraction according to actual conditions. Furthermore, the value of the hyperparameter is negatively correlated with the number of beats per minute of the sample audio signal, such as inversely proportional, but the present disclosure is not limited to this. By adding a window function to the annotation information, we can combine the fact that human ears perceive rhythm in a linear manner and associate the annotation information with the number of beats per minute of the sample audio signal, which helps the model converge and improves the accuracy of beat and re-beat detection. In addition, setting the value of the hyperparameter to be negatively correlated with the number of beats per minute of the sample audio signal can take into account that the faster the song, the shorter the beat duration, and the slower the song, the longer the beat duration, so that the results after windowing are more in line with the characteristics of human ear perception.
[0078] Figure 5 is a graph diagram showing a window function according to an exemplary embodiment of the present disclosure.
[0079] Reference Figure 5 , the window function can be a Gaussian kernel function (Gaussian Kernel Function), and the Gaussian kernel function g(x) can be specifically expressed as follows:
[0080]
[0081] Here, σ is a hyperparameter used to determine the curve width of the window function. Figure 5 The curve of the window function shown can be the curve of the Gaussian kernel function when σ takes the value of 1. By adjusting the size of σ, the value of the Gaussian kernel function can be adjusted. Since the human ear's perception of speed is basically linear, it is assumed that the faster the song, the shorter the beat duration, and a smaller σ is required. The slower the song, the longer the beat duration, and a larger σ is required. In addition, the BPM of a song usually ranges from 50 to 200, so the beat interval ranges from 0.3s to 1.2s. On this basis, the σ of the Gaussian kernel function can be controlled to be inversely proportional to the BPM of the song, that is, proportional to the beat interval. As an example, the ratio of σ to the beat interval can be selected as 2, and the hyperparameter σ can be specifically expressed as follows:
[0082] σ=2×interval (5)
[0083] In addition, according to an exemplary embodiment of the present disclosure, after obtaining the dynamic annotation information, the training loss l(n) may be calculated based on the beat-re-beat point prediction probability P(n) and the corresponding dynamic annotation information B(n):
[0084] l(n)=Loss(P(n),B(n)) (6)
[0085] Here, according to the predicted probabilities of beat points, re-beat points and non-beat points and the corresponding dynamic annotation information, the first loss corresponding to the predicted probability of beat points, the second loss corresponding to the predicted probability of re-beat points and the third loss corresponding to the predicted probability of non-beat points can be calculated respectively; then, the training loss can be determined according to the first loss, the second loss and the third loss. Further, as an example, the first loss, the second loss and the third loss can be weighted and summed to determine the training loss, but the present disclosure is not limited to this. Here, when determining the training loss by the above-mentioned weighted summation, the weights of the first loss, the second loss and the third loss can be made 19:59:1, but the present disclosure is not limited to this, and those skilled in the art can adjust the weights of the first loss, the second loss and the third loss according to the actual situation, so as to optimize the model training. Furthermore, the loss can be calculated using a weighted cross entropy loss function, that is, the Loss loss function in formula (6) can be a weighted cross entropy loss function, but the present disclosure is not limited to this.
[0086] Return to reference Figure 4 In step S404, the parameters of the beat-repeat joint detection model can be adjusted according to the training loss to train the beat-repeat joint detection model. Here, the parameters corresponding to the model can be updated by reducing the loss function through the stochastic gradient descent method, but not limited to, and the Adam (adaptive moment estimation) optimization is used to accelerate the iterative update of the model training.
[0087] According to an exemplary embodiment of the present disclosure, the beat-repeat joint detection model can be constructed by a convolutional recurrent neural network. Here, the CRNN may include two CNN layers and two RNN layers, and the RNN may use BiLSTM. By constructing the model by CRNN, the advantages of CNN and RNN can be combined, and local features with fewer dimensions can be learned directly from the input features, and long-term dependent temporal information can also be learned.
[0088] Figure 6 is a flowchart illustrating a beat-rebeat joint detection method according to an exemplary embodiment of the present disclosure.
[0089] Reference Figure 6 In step S601, an audio signal to be processed may be obtained.
[0090] Next, in step S602, the audio features of the audio signal to be processed may be input into the beat-re-beat joint detection model trained according to the training method of the beat-re-beat joint detection model disclosed in the present invention, and the beat-re-beat point probability of each frame of data in the audio signal to be processed is obtained. Here, the beat-re-beat point probability represents the probability that each frame of data in the audio signal to be processed is a beat point, a re-beat point, and a non-beat point.
[0091] According to an exemplary embodiment of the present disclosure, the audio features of the audio signal to be processed can be obtained through the following steps: converting the audio signal to be processed into a time-frequency domain signal, and using the time-frequency domain signal as the audio feature of the audio signal to be processed. By converting the audio signal to be processed into a time-frequency domain signal, the model can better extract the timing features and frequency features of the audio signal to be processed during detection. Furthermore, the time-frequency domain signal can be filtered, and the filtered time-frequency domain signal can be used as the audio feature of the audio signal to be processed. Through the above-mentioned filtering process, the time-frequency domain signal can be subjected to data dimensionality reduction, reducing the resource usage during model detection, thereby improving the model detection performance.
[0092] As an example, first, the audio signal to be processed can be mapped into a time-frequency domain signal through short-time Fourier transform, but is not limited to it; then, the time-frequency domain signal can be converted into a logarithmic spectrum feature through logarithmic calculation, but is not limited to it, so as to filter the time-frequency domain signal, and use the logarithmic spectrum feature obtained by filtering as the audio feature of the audio signal to be processed.
[0093] Next, in step S603, the beat-repeat joint detection result of the audio signal to be processed can be determined according to the beat-repeat point probability of each frame of data in the audio signal to be processed. Here, the beat-repeat point probability of each frame of data in the audio signal to be processed can be decoded to obtain a beat-repeat time series, and the beat-repeat time series can be used as the beat-repeat joint detection result of the audio signal to be processed. As an example, the beat-repeat point probability can be decoded using, but not limited to, DBN / HMM (Hidden Markov Model) / viterbi (Viterbi algorithm) and other methods to obtain a global beat-repeat time series.
[0094] In order to facilitate the understanding of the above embodiments, Figure 7 and Figure 8 Provide a description of the system.
[0095] Figure 7is a logic diagram illustrating a beat-rebeat joint detection method according to an exemplary embodiment of the present disclosure, Figure 8 is a schematic diagram illustrating a detection result according to an exemplary embodiment of the present disclosure.
[0096] Reference Figure 7 ,The beat-repeat joint detection model is constructed by CRNN. First, the audio signal to be processed is converted into audio features in the time-frequency domain, and then the audio features are input into CRNN, and the local features of the audio features are extracted through two layers of convolutional neural network layers (CNN layers), and then through two layers of recurrent neural network layers (RNN layers), namely Figure 7 The BiGRU layer shown in the figure extracts the time series features, and then passes through a fully connected layer (Dense layer) to obtain the probability of each frame data being a beat, a beat, or a non-beat. Finally, in order to obtain global information, a post-processing module can be used, such as the DBNDownBeatTracking module in Madmom (a Python development kit for music information extraction), to decode the probability output by the above Dense layer to obtain the final beat-beat time series. Common music time signature types include 2 / 4, 3 / 4, and 4 / 4. Figure 8 The corresponding beat frames, weak beat frames in the beat, and non-beat frames can be used as examples of the beat-and-beat joint detection results of the audio signal to be processed.
[0097] According to the training of the beat-rebeat joint detection model and the beat-rebeat joint detection method disclosed in the present invention, a model can be constructed based on CRNN to combine the advantages of CNN and RNN, and can not only directly learn local features with fewer dimensions from input features, but also learn long-term dependent timing information. In addition, according to the training of the beat-rebeat joint detection model and the beat-rebeat joint detection method disclosed in the present invention, during the training process, the loss is calculated by labeling information including beat information, beat information and non-beat information, and the parameters of the beat-rebeat joint detection model are adjusted on this basis, which can effectively avoid overfitting of the model, and considering that the human ear's perception of rhythm is a linear change, the labeling information is also associated with the number of beats per minute of the sample audio signal, thereby facilitating model convergence and improving the accuracy of beat and beat detection.
[0098] Fig. 9 is a block diagram showing a training apparatus for a beat-rebeat joint detection model according to an exemplary embodiment of the present disclosure. Fig. 9 The training device 900 of the beat-rebeat joint detection model may include: a sample acquisition unit 910, a probability prediction unit 920, a loss determination unit 930 and a model training unit 940.
[0099] The sample acquisition unit 910 may acquire the sample audio signal and the beat and re-beat marking information corresponding to the sample audio signal. Here, the beat and re-beat marking information may include beat marking information, re-beat marking information and non-beat marking information.
[0100] The probability prediction unit 920 may input the audio features of the sample audio signal into the beat-re-beat joint detection model to obtain the beat-re-beat point prediction probability of each frame of data in the sample audio signal. Here, the beat-re-beat point prediction probability may represent the prediction probability of each frame of data in the sample audio signal being a beat point, a beat point, and a non-beat point.
[0101] The loss determination unit 930 may map the beat-re-beat annotation information to dynamic annotation information, and calculate the training loss based on the beat-re-beat point prediction probability and the corresponding dynamic annotation information. Here, the dynamic annotation information is associated with the number of beats per minute of the sample audio signal.
[0102] The model training unit 940 may adjust the parameters of the beat-rebeat joint detection model according to the training loss to train the beat-rebeat joint detection model.
[0103] According to an exemplary embodiment of the present disclosure, the loss determination unit 930 may also perform windowing processing on the beat-rebeat marking information based on a preset window function to obtain dynamic marking information. Here, the window function may include a hyperparameter for determining a curve width of the window function.
[0104] According to an exemplary embodiment of the present disclosure, the loss determination unit 930 may also determine the number of beats per minute of the sample audio signal based on the beat-rebeat labeling information; then, the value of the hyperparameter may be determined based on the determined number of beats per minute of the sample audio signal; then, the beat-rebeat labeling information may be windowed by a window function based on the determined value of the hyperparameter to obtain dynamic labeling information. Here, the value of the hyperparameter may be negatively correlated with the number of beats per minute of the sample audio signal.
[0105] According to an exemplary embodiment of the present disclosure, the loss determination unit 930 may further extract information from the beat and rebeat marking information based on a window function to obtain dynamic marking information.
[0106] According to an exemplary embodiment of the present disclosure, the loss determination unit 930 may also calculate, based on the prediction probabilities of the beat points, the beat points and the non-beat points and the corresponding dynamic annotation information, a first loss corresponding to the prediction probability of the beat points, a second loss corresponding to the prediction probability of the beat points and a third loss corresponding to the prediction probability of the non-beat points; then, the training loss may be determined based on the first loss, the second loss and the third loss.
[0107] According to an exemplary embodiment of the present disclosure, the sample acquisition unit 910 may further convert the sample audio signal into a time-frequency domain signal, and use the time-frequency domain signal as an audio feature of the sample audio signal.
[0108] According to an exemplary embodiment of the present disclosure, the sample acquisition unit 910 may further perform filtering processing on the time-frequency domain signal, and use the filtered time-frequency domain signal as the audio feature of the sample audio signal.
[0109] According to an exemplary embodiment of the present disclosure, as described above, the beat-rebeat joint detection model can be constructed by a convolutional recurrent neural network.
[0110] Fig.10 is a block diagram showing a beat-rebeat joint detection apparatus according to an exemplary embodiment of the present disclosure. Fig.10 The beat-rebeat joint detection device 1000 may include an audio acquisition unit 1010 , a probability determination unit 1020 and a result determination unit 1030 .
[0111] The audio acquisition unit 1010 can acquire an audio signal to be processed.
[0112] The probability determination unit 1020 may input the audio features of the audio signal to be processed into the beat-re-beat joint detection model trained according to the training method of the beat-re-beat joint detection model disclosed in the present invention, and obtain the beat-re-beat point probability of each frame of data in the audio signal to be processed. Here, the beat-re-beat point probability may represent the probability that each frame of data in the audio signal to be processed belongs to a beat point, a beat point, and a non-beat point, respectively.
[0113] The result determination unit 1030 may determine the beat-rebeat joint detection result of the audio signal to be processed according to the beat-rebeat point probability of each frame of data in the audio signal to be processed.
[0114] According to an exemplary embodiment of the present disclosure, the audio acquisition unit 1010 may further convert the audio signal to be processed into a time-frequency domain signal, and use the time-frequency domain signal as the audio feature of the audio signal to be processed.
[0115] According to an exemplary embodiment of the present disclosure, the audio acquisition unit 1010 may further perform filtering processing on the time-frequency domain signal, and use the filtered time-frequency domain signal as the audio feature of the audio signal to be processed.
[0116] According to an exemplary embodiment of the present disclosure, the result determination unit can also decode the beat-rebeat point probability of each frame data in the audio signal to be processed to obtain a beat-rebeat time series, and use the beat-rebeat time series as the beat-rebeat joint detection result of the audio signal to be processed.
[0117] According to the training of the beat-rebeat joint detection model and the beat-rebeat joint detection method and device disclosed in the present invention, a model can be constructed based on CRNN to combine the advantages of CNN and RNN, and can not only directly learn local features with fewer dimensions from input features, but also learn long-term dependent timing information. In addition, according to the training of the beat-rebeat joint detection model and the beat-rebeat joint detection method and device disclosed in the present invention, during the training process, the loss is calculated by labeling information including beat information, rebeat information and non-beat information, and the parameters of the beat-rebeat joint detection model are adjusted on this basis, which can effectively avoid overfitting of the model, and considering that the human ear's perception of rhythm is a linear change, the labeling information is also associated with the number of beats per minute of the sample audio signal, thereby facilitating model convergence and improving the accuracy of beat and rebeat detection.
[0118] According to an embodiment of the present disclosure, an electronic device may be provided. Fig.11 It is a block diagram of an electronic device 1100 according to an embodiment of the present disclosure, and the electronic device includes at least one memory 1110 and at least one processor 1120, wherein the at least one memory stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor, a training method of a beat-repeat joint detection model or a beat-repeat joint detection method according to an embodiment of the present disclosure is executed.
[0119] As an example, the electronic device 1100 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1100 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 1100 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.
[0120] In the electronic device 1100, the processor 1120 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 1120 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.
[0121] The processor 1120 may execute instructions or codes stored in the memory, wherein the memory 1110 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.
[0122] The memory 1110 may be integrated with the processor 1120, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1110 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1110 and the processor 1120 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1120 can read files stored in the memory 1110.
[0123] In addition, the electronic device 1100 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.
[0124] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the beat-rebeat joint detection model or the beat-rebeat joint detection method of the embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.
[0125] According to an embodiment of the present disclosure, a computer program product is provided, including computer instructions, which, when executed by a processor, implement the training method of the beat-repeat joint detection model or the beat-repeat joint detection method of the embodiment of the present disclosure.
[0126] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0127] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A training method for a beat-repeat joint detection model, characterized in that: The training method comprises: Acquire a sample audio signal and beat and re-beat marking information corresponding to the sample audio signal, wherein the beat and re-beat marking information includes beat marking information, re-beat marking information and non-beat marking information; Input the audio features of the sample audio signal into a beat-re-beat joint detection model to obtain a beat-re-beat point prediction probability for each frame of data in the sample audio signal, wherein the beat-re-beat point prediction probability represents the prediction probability that each frame of data in the sample audio signal belongs to a beat point, a beat point, and a non-beat point respectively; Mapping the beat-re-beat annotation information into dynamic annotation information, and calculating a training loss based on the beat-re-beat point prediction probability and the corresponding dynamic annotation information, wherein the dynamic annotation information is associated with the number of beats per minute of the sample audio signal; adjusting parameters of the beat-rebeat joint detection model according to the training loss; The step of mapping the beat and rebeat marking information into dynamic marking information includes: Determining the number of beats per minute of the sample audio signal based on the beat and rebeat marking information; Determining a value of a hyperparameter based on the determined number of beats per minute of the sample audio signal; Based on the determined value of the hyperparameter, the beat and rebeat marking information is subjected to windowing processing by a preset window function to obtain the dynamic marking information; The window function includes the hyperparameter for determining the curve width of the window function, and the value of the hyperparameter is negatively correlated with the number of beats per minute of the sample audio signal.
2. The training method according to claim 1, characterized in that: The step of performing windowing processing on the beat and rebeat marking information by using a preset window function to obtain the dynamic marking information includes: The beat and rebeat marking information is extracted based on the window function to obtain the dynamic marking information.
3. The training method according to claim 1, characterized in that: The audio features of the sample audio signal are obtained by the following steps: The sample audio signal is converted into a time-frequency domain signal, and the time-frequency domain signal is used as an audio feature of the sample audio signal.
4. The training method according to claim 3, characterized in that: The using the time-frequency domain signal as the audio feature of the sample audio signal comprises: The time-frequency domain signal is filtered, and the filtered time-frequency domain signal is used as the audio feature of the sample audio signal.
5. The training method according to claim 1, characterized in that: The calculating of the training loss based on the beat-re-beat point prediction probability and the corresponding dynamic annotation information includes: According to the predicted probabilities of the beat points, the beat points and the non-beat points and the corresponding dynamic annotation information, respectively calculating a first loss corresponding to the predicted probability of the beat points, a second loss corresponding to the predicted probability of the beat points and a third loss corresponding to the predicted probability of the non-beat points; The training loss is determined according to the first loss, the second loss and the third loss.
6. A beat-repeat joint detection method, characterized in that: The beat-rebeat joint detection method comprises: Obtaining the audio signal to be processed; Input the audio features of the audio signal to be processed into the beat-rebeat joint detection model trained by the training method of the beat-rebeat joint detection model as described in any one of claims 1 to 5, and obtain the beat-rebeat point probability of each frame of data in the audio signal to be processed, wherein the beat-rebeat point probability represents the probability that each frame of data in the audio signal to be processed belongs to a beat point, a beat point, and a non-beat point respectively; According to the beat-rebeat point probability of each frame of data in the audio signal to be processed, a beat-rebeat joint detection result of the audio signal to be processed is determined.
7. The beat-repeat joint detection method according to claim 6, characterized in that: The audio features of the audio signal to be processed are obtained by the following steps: The audio signal to be processed is converted into a time-frequency domain signal, and the time-frequency domain signal is used as an audio feature of the audio signal to be processed.
8. The beat-repeat joint detection method according to claim 7, characterized in that: The step of using the time-frequency domain signal as the audio feature of the audio signal to be processed includes: The time-frequency domain signal is filtered, and the filtered time-frequency domain signal is used as the audio feature of the audio signal to be processed.
9. The beat-repeat joint detection method according to claim 6, characterized in that: The step of determining the beat-rebeat joint detection result of the audio signal to be processed according to the beat-rebeat point probability of each frame of data in the audio signal to be processed comprises: The beat-rebeat point probability of each frame of data in the audio signal to be processed is decoded to obtain a beat-rebeat time series, and the beat-rebeat time series is used as the beat-rebeat joint detection result of the audio signal to be processed.
10. A training device for a beat-repeat joint detection model, characterized in that: The training device comprises: A sample acquisition unit is configured to acquire a sample audio signal and beat and re-beat marking information corresponding to the sample audio signal, wherein the beat and re-beat marking information includes beat marking information, re-beat marking information and non-beat marking information; A probability prediction unit is configured to input the audio features of the sample audio signal into a beat-re-beat joint detection model to obtain a beat-re-beat point prediction probability of each frame of data in the sample audio signal, wherein the beat-re-beat point prediction probability represents the prediction probability that each frame of data in the sample audio signal belongs to a beat point, a beat point, and a non-beat point respectively; a loss determination unit configured to map the beat-re-beat annotation information into dynamic annotation information, and calculate a training loss based on the beat-re-beat point prediction probability and the corresponding dynamic annotation information, wherein the dynamic annotation information is associated with the number of beats per minute of the sample audio signal; A model training unit, configured to adjust parameters of the beat-rebeat joint detection model according to the training loss; Wherein, the loss determination unit is further configured to: Determining the number of beats per minute of the sample audio signal based on the beat and rebeat marking information; Determining a value of a hyperparameter based on the determined number of beats per minute of the sample audio signal; Based on the determined value of the hyperparameter, the beat and rebeat marking information is subjected to windowing processing by a preset window function to obtain the dynamic marking information; The window function includes the hyperparameter for determining the curve width of the window function, and the value of the hyperparameter is negatively correlated with the number of beats per minute of the sample audio signal.
11. The training device according to claim 10, characterized in that The loss determination unit is further configured to: The beat and rebeat marking information is extracted based on the window function to obtain the dynamic marking information.
12. The training device according to claim 10, characterized in that The sample acquisition unit is further configured to: The sample audio signal is converted into a time-frequency domain signal, and the time-frequency domain signal is used as an audio feature of the sample audio signal.
13. The training device according to claim 12, characterized in that The sample acquisition unit is further configured to: The time-frequency domain signal is filtered, and the filtered time-frequency domain signal is used as the audio feature of the sample audio signal.
14. The training device according to claim 10, characterized in that The loss determination unit is further configured to: According to the predicted probabilities of the beat points, the beat points and the non-beat points and the corresponding dynamic annotation information, respectively calculating a first loss corresponding to the predicted probability of the beat points, a second loss corresponding to the predicted probability of the beat points and a third loss corresponding to the predicted probability of the non-beat points; The training loss is determined according to the first loss, the second loss and the third loss.
15. A beat and re-beat joint detection device, characterized in that: The beat-rebeat joint detection device comprises: An audio acquisition unit, configured to acquire an audio signal to be processed; A probability determination unit, configured to input the audio features of the audio signal to be processed into a beat-rebeat joint detection model trained by the training method of the beat-rebeat joint detection model according to any one of claims 1 to 5, to obtain the beat-rebeat point probability of each frame of data in the audio signal to be processed, wherein the beat-rebeat point probability represents the probability that each frame of data in the audio signal to be processed belongs to a beat point, a beat point, and a non-beat point respectively; The result determination unit is configured to determine the beat-rebeat joint detection result of the audio signal to be processed according to the beat-rebeat point probability of each frame of data in the audio signal to be processed.
16. The beat and re-beat joint detection device according to claim 15, characterized in that: The audio acquisition unit is further configured to: The audio signal to be processed is converted into a time-frequency domain signal, and the time-frequency domain signal is used as an audio feature of the audio signal to be processed.
17. The beat and rebeat joint detection device according to claim 16, characterized in that: The audio acquisition unit is further configured to: The time-frequency domain signal is filtered, and the filtered time-frequency domain signal is used as the audio feature of the audio signal to be processed.
18. The beat and re-beat joint detection device according to claim 15, characterized in that: The result determination unit is further configured to obtain a beat-rebeat time series by decoding the beat-rebeat point probability of each frame of data in the audio signal to be processed, and use the beat-rebeat time series as the beat-rebeat joint detection result of the audio signal to be processed.
19. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the training method of the beat-rebeat joint detection model as described in any one of claims 1 to 5 or the beat-rebeat joint detection method as described in any one of claims 6 to 9.
20. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the beat-rebeat joint detection model as described in any one of claims 1 to 5 or the beat-rebeat joint detection method as described in any one of claims 6 to 9.
21. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by the processor, the training method of the beat-rebeat joint detection model as described in any one of claims 1 to 5 or the beat-rebeat joint detection method as described in any one of claims 6 to 9 is implemented.
