Training of beat-repeat joint detection model and beat-repeat joint detection method

By jointly training the beat and remapping detection network, the model parameters are optimized using the target loss function, the problem of insufficient accuracy of remapping detection in the existing technology is solved, and the high accuracy of beat and remapping detection is achieved.

CN114154574BActive Publication Date: 2024-12-03BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111465767.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-03
Publication Date
2024-12-03
Estimated Expiration
2041-12-03

AI Technical Summary

Technical Problem

In the prior art, the accuracy of beat detection is high, but the accuracy of remapping detection is low, resulting in insufficient accuracy of remapping detection in music analysis and other application scenarios.

Method used

A training method for the joint detection model of beat remapping is proposed. By obtaining the beat and remapping label information in the training sample set, combining the beat detection network and the remapping detection network for joint training, the target loss function is used to adjust the network parameters and improve the remapping. The accuracy of the detection and introduce remake information into beat detection to improve the accuracy of beat detection.

Benefits of technology

It effectively improves the accuracy of remapping detection, and at the same time improves the accuracy of beat detection, solving the problem of low accuracy of remapping detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114154574B_ABST
    Figure CN114154574B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a training of a beat-repeat joint detection model and a beat-repeat joint detection method. The training method includes: obtaining a training sample set, wherein the training sample set includes a plurality of sample audio signals and beat labeling information and rebeat labeling information corresponding to each sample audio signal; inputting the audio features of the sample audio signal into a beat detection network to obtain a beat point prediction probability for each frame of data in the sample audio signal; inputting the audio features of the sample audio signal into a rebeat detection network to obtain a rebeat point prediction probability for each frame of data in the sample audio signal; calculating the value of a target loss function based on the beat point prediction probability, the rebeat point prediction probability, the corresponding beat labeling information and the rebeat labeling information; and according to the value of the target loss function, jointly training the beat detection network and the rebeat detection network to obtain a trained beat-repeat joint detection model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of audio processing, and in particular to a training of a beat-repeat joint detection model and a beat-repeat joint detection method. Background Art

[0002] In music, beat is an organizational form that represents fixed unit duration and rhythmic patterns. It defines the rhythmic structure of a musical work, and rhythm is characterized by a repeated sequence of beats and non-beats. Downbeats refer to strong beats in music. There has been a considerable amount of research focused on finding the beats of music, but much less effort has been made to track downbeats. In Western music, downbeats usually coincide with chord changes or harmony changes, so the joint detection of beats and downbeats is crucial for music analysis, music structure delineation, automatic music matching, and even some creative card point video production, beat-driven light shows and other practical application scenarios.

[0003] At present, the beat detection algorithm based on deep learning has solved some problems of traditional algorithms. It does not need to estimate the position of the starting point of the note, relies on less prior knowledge, and improves the accuracy of beat detection to a certain extent. However, due to the large number of music types, the beat characteristics of various types of music are inconsistent, so the accuracy of beat detection is still much lower than the accuracy of beat detection, resulting in a low accuracy of beat detection. Summary of the invention

[0004] The present disclosure provides a training method for a beat-rebeat joint detection model and a beat-rebeat joint detection method, so as to at least solve the problem of low accuracy of rebeat detection in related technologies.

[0005] According to a first aspect of an embodiment of the present disclosure, a training method for a beat-rebeat joint detection model is provided, wherein the beat-rebeat joint detection model includes a beat detection network and a beat detection network, and the training method includes: obtaining a training sample set, wherein the training sample set includes multiple sample audio signals and beat annotation information and beat annotation information corresponding to each sample audio signal; inputting audio features of the sample audio signal into the beat detection network to obtain a beat point prediction probability for each frame of data in the sample audio signal, wherein the beat point prediction probability represents a prediction probability that each frame of data in the sample audio signal is a beat point; inputting audio features of the sample audio signal into the beat detection network to obtain a beat point prediction probability for each frame of data in the sample audio signal, wherein the beat point prediction probability represents a prediction probability that each frame of data in the sample audio signal is a beat point; calculating a value of a target loss function based on the beat point prediction probability, the beat point prediction probability, the corresponding beat annotation information and the beat annotation information; and according to the value of the target loss function, jointly training the beat detection network and the beat detection network to obtain a trained beat-rebeat joint detection model.

[0006] Optionally, the audio features of the sample audio signal are input into the beat detection network to obtain the beat point prediction probability of each frame of data in the sample audio signal, including: inputting the audio features of the sample audio signal into the beat detection network to obtain the global features output by at least one intermediate layer of the beat detection network and the beat point prediction probability of each frame of data in the sample audio signal; wherein, the audio features of the sample audio signal are input into the rebeat detection network to obtain the rebeat point prediction probability of each frame of data in the sample audio signal, including: inputting the audio features and global features of the sample audio signal into the rebeat detection network to obtain the rebeat point prediction probability of each frame of data in the sample audio signal.

[0007] Optionally, the audio features and global features of the sample audio signal are input into the beat detection network, including: inputting the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, splicing the first global feature output by the current intermediate layer and the second global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network to obtain a first spliced ​​global feature, and inputting the first spliced ​​global feature into the next intermediate layer of the current intermediate layer.

[0008] Optionally, the beat-rebeat joint detection model also includes a pre-trained style classification network, which is used to obtain the music type of the sample audio signal; wherein, the audio features of the sample audio signal are input into the beat detection network, including: inputting the audio features of the sample audio signal into the style classification network to obtain a first intermediate feature output by at least one intermediate layer in the style classification network; and inputting the audio features of the sample audio signal and the first intermediate feature into the beat detection network.

[0009] Optionally, the audio features and the first intermediate features of the sample audio signal are input into the beat detection network, including: inputting the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer in the beat detection network, concatenating the third global feature output by the current intermediate layer and the first intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the style classification network to obtain a second concatenated global feature, and inputting the second concatenated global feature into the next intermediate layer of the current intermediate layer.

[0010] Optionally, the beat-rebeat joint detection model also includes a pre-trained sway detection network, which is used to obtain the relationship between the beat and volume of the sample audio signal; wherein, inputting the audio features and global features of the sample audio signal into the beat detection network includes: inputting the audio features of the sample audio signal into the sway detection network to obtain a second intermediate feature output by at least one intermediate layer in the sway detection network; and inputting the audio features, global features and second intermediate features of the sample audio signal into the beat detection network.

[0011] Optionally, the audio features, global features and second intermediate features of the sample audio signal are input into the beat detection network, including: inputting the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, concatenating the fourth global feature output by the current intermediate layer, the second intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the sway detection network, and the fifth global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network to obtain a third concatenated global feature, and inputting the third concatenated global feature into the next intermediate layer of the current intermediate layer.

[0012] Optionally, based on the beat point prediction probability, the re-beat point prediction probability, the corresponding beat marking information and the re-beat marking information, the value of the target loss function is calculated, including: calculating the value of the first target loss function based on the beat point prediction probability and the corresponding beat marking information; calculating the value of the second target loss function based on the re-beat point prediction probability and the corresponding re-beat marking information; adding the value of the first target loss function and the value of the second target loss function to obtain the value of the target loss function.

[0013] According to a second aspect of an embodiment of the present disclosure, a beat-rebeat joint detection method is provided. The beat-rebeat joint detection method performs beat-rebeat joint detection using a pre-trained beat-rebeat joint detection model. The beat-rebeat joint detection model includes a beat detection network and a rebeat detection network. The beat-rebeat joint detection method includes: acquiring an audio signal to be processed; inputting audio features of the audio signal to be processed into the beat detection network to obtain a beat point probability for each frame of data in the audio signal to be processed, wherein the beat point probability represents the probability that each frame of data in the audio signal to be processed is a beat point; inputting the audio features of the audio signal to be processed into the rebeat detection network to obtain a rebeat point probability for each frame of data in the audio signal to be processed, wherein the rebeat point probability represents the probability that each frame of data in the audio signal to be processed is a rebeat point.

[0014] Optionally, the audio features of the audio signal to be processed are input into a beat detection network to obtain the beat point probability of each frame of data in the audio signal to be processed, including: inputting the audio features of the audio signal to be processed into the beat detection network to obtain the global features output by at least one intermediate layer of the beat detection network and the beat point probability of each frame of data in the audio signal to be processed; wherein, the audio features of the audio signal to be processed are input into the rebeat detection network to obtain the rebeat point prediction probability of each frame of data in the audio signal to be processed, including: inputting the audio features and global features of the audio signal to be processed into the rebeat detection network to obtain the rebeat point prediction probability of each frame of data in the audio signal to be processed.

[0015] Optionally, the audio features and global features of the audio signal to be processed are input into the beat detection network, including: inputting the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, concatenating the first global feature output by the current intermediate layer and the second global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network to obtain a first concatenated global feature, and inputting the first concatenated global feature into the next intermediate layer of the current intermediate layer.

[0016] Optionally, the beat-rebeat joint detection model also includes a pre-trained style classification network; wherein, the audio features of the audio signal to be processed are input into the beat detection network, and the style classification network is used to obtain the music type of the audio signal, including: inputting the audio features of the audio signal to be processed into the style classification network to obtain a first intermediate feature output by at least one intermediate layer in the style classification network; and inputting the audio features of the audio signal to be processed and the first intermediate feature into the beat detection network.

[0017] Optionally, the audio features of the audio signal to be processed and the first intermediate features are input into the beat detection network, including: inputting the audio features of the audio signal to be processed into the input layer features of the beat detection network; for each intermediate layer of at least one intermediate layer in the beat detection network, concatenating the third global feature output by the current intermediate layer and the first intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the style classification network to obtain a second concatenated global feature, and inputting the second concatenated global feature into the next intermediate layer of the current intermediate layer.

[0018] Optionally, the beat-rebeat joint detection model also includes a pre-trained sway detection network; wherein, the audio features and global features of the audio signal to be processed are input into the beat detection network, and the sway detection network is used to obtain the beat and volume relationship of the audio signal, including: inputting the audio features of the audio signal to be processed into the sway detection network to obtain a second intermediate feature output by at least one intermediate layer in the sway detection network; inputting the audio features, global features and second intermediate features of the audio signal to be processed into the beat detection network.

[0019] Optionally, the audio features, global features and second intermediate features of the audio signal to be processed are input into the beat detection network, including: inputting the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, concatenating the fourth global feature output by the current intermediate layer, the second intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the sway detection network, and the fifth global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network to obtain a third concatenated global feature, and inputting the third concatenated global feature into the next intermediate layer of the current intermediate layer.

[0020] Optionally, the beat-repeat joint detection model is trained based on the above-mentioned training method of the beat-repeat joint detection model.

[0021] According to a third aspect of an embodiment of the present disclosure, a training device for a beat-repeat joint detection model is provided, wherein the beat-repeat joint detection model includes a beat detection network and a beat detection network, and the training device includes: a sample set acquisition unit, configured to acquire a training sample set, wherein the training sample set includes a plurality of sample audio signals and beat labeling information and beat labeling information corresponding to each sample audio signal; a beat point probability acquisition unit, configured to input the audio features of the sample audio signal into the beat detection network, and obtain the beat point prediction probability of each frame of data in the sample audio signal, wherein the beat point prediction probability indicates whether each frame of data in the sample audio signal is a beat point. a beat point probability acquisition unit, configured to input the audio features of the sample audio signal into the beat detection network, and obtain the beat point prediction probability of each frame of data in the sample audio signal, wherein the beat point prediction probability represents the prediction probability that each frame of data in the sample audio signal is a beat point; a target loss function determination unit, configured to calculate the value of the target loss function based on the beat point prediction probability, the beat point prediction probability, the corresponding beat annotation information and the beat annotation information; a training unit, configured to jointly train the beat detection network and the beat detection network according to the value of the target loss function, and obtain a trained beat-beat joint detection model.

[0022] Optionally, the beat point probability acquisition unit is further configured to input the audio features of the sample audio signal into the beat detection network to obtain the global features output by at least one intermediate layer of the beat detection network and the beat point prediction probability of each frame data in the sample audio signal; the rebeat point probability acquisition unit is further configured to input the audio features and global features of the sample audio signal into the rebeat detection network to obtain the rebeat point prediction probability of each frame data in the sample audio signal.

[0023] Optionally, the beat point probability acquisition unit is also configured to input the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, concatenate the first global feature output by the current intermediate layer and the second global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network to obtain a first concatenated global feature, and input the first concatenated global feature to the next intermediate layer of the current intermediate layer.

[0024] Optionally, the beat-rebeat joint detection model also includes a pre-trained style classification network, which is used to obtain the music type of the sample audio signal; wherein the beat point probability acquisition unit is also configured to input the audio features of the sample audio signal into the style classification network to obtain a first intermediate feature output by at least one intermediate layer in the style classification network; and input the audio features of the sample audio signal and the first intermediate feature into the beat detection network.

[0025] Optionally, the beat point probability acquisition unit is also configured to input the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer in the beat detection network, concatenate the third global feature output by the current intermediate layer and the first intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the style classification network to obtain a second concatenated global feature, and input the second concatenated global feature to the next intermediate layer of the current intermediate layer.

[0026] Optionally, the beat-rebeat joint detection model also includes a pre-trained sway detection network, which is used to obtain the relationship between the beat and volume of the sample audio signal; wherein the beat point probability acquisition unit is also configured to input the audio features of the sample audio signal into the sway detection network to obtain a second intermediate feature output by at least one intermediate layer in the sway detection network; and input the audio features, global features and second intermediate features of the sample audio signal into the beat detection network.

[0027] Optionally, the beat point probability acquisition unit is also configured to input the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, the fourth global feature output by the current intermediate layer, the second intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the sway detection network, and the fifth global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network are spliced ​​to obtain a third spliced ​​global feature, and the third spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

[0028] Optionally, the target loss function determination unit is also configured to calculate the value of the first target loss function based on the beat point prediction probability and the corresponding beat marking information; calculate the value of the second target loss function based on the re-beat point prediction probability and the corresponding re-beat marking information; add the value of the first target loss function and the value of the second target loss function to obtain the value of the target loss function.

[0029] According to a fourth aspect of an embodiment of the present disclosure, a beat-repeat joint detection device is provided, characterized in that the beat-repeat joint detection device uses a pre-trained beat-repeat joint detection model to perform beat-repeat joint detection, the beat-repeat joint detection model includes a beat detection network and a beat detection network, and the beat-repeat joint detection device includes: an audio signal acquisition unit, configured to acquire an audio signal to be processed; a beat point probability acquisition unit, configured to input an audio feature of the audio signal to be processed into the beat detection network, and obtain the beat point probability of each frame of data in the audio signal to be processed, wherein , the beat point probability represents the probability that each frame of data in the audio signal to be processed is a beat point; the re-beat point probability acquisition unit is configured to input the audio features of the audio signal to be processed into the re-beat detection network to obtain the re-beat point probability of each frame of data in the audio signal to be processed, wherein the re-beat point probability represents the probability that each frame of data in the audio signal to be processed is a re-beat point; the determination unit is configured to determine the beat-re-beat joint detection result of the audio signal to be processed according to the probability that each frame of data in the audio signal to be processed is a beat point and the probability that each frame of data in the audio signal to be processed is a re-beat point.

[0030] Optionally, the beat point probability acquisition unit is configured to input the audio features of the audio signal to be processed into the beat detection network to obtain the global features output by at least one intermediate layer of the beat detection network and the beat point probability of each frame of data in the audio signal to be processed; the rebeat point probability acquisition unit is also configured to input the audio features and global features of the audio signal to be processed into the rebeat detection network to obtain the probability of the rebeat point of each frame of data in the audio signal to be processed.

[0031] Optionally, the beat point probability acquisition unit is also configured to input the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, concatenate the first global feature output by the current intermediate layer and the second global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network to obtain a first concatenated global feature, and input the first concatenated global feature to the next intermediate layer of the current intermediate layer.

[0032] Optionally, the beat-rebeat joint detection model also includes a pre-trained style classification network, which is used to obtain the music type of the audio signal; wherein the beat point probability acquisition unit is also configured to input the audio features of the audio signal to be processed into the style classification network to obtain a first intermediate feature output by at least one intermediate layer in the style classification network; and input the audio features of the audio signal to be processed and the first intermediate feature into the beat detection network.

[0033] Optionally, the beat point probability acquisition unit is also configured to input the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer in the beat detection network, concatenate the third global feature output by the current intermediate layer and the first intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the style classification network to obtain a second concatenated global feature, and input the second concatenated global feature to the next intermediate layer of the current intermediate layer.

[0034] Optionally, the beat-rebeat joint detection model also includes a pre-trained sway detection network, which is used to obtain the relationship between the beat and volume of the audio signal; wherein the beat point probability acquisition unit is further configured to input the audio features of the audio signal to be processed into the sway detection network to obtain a second intermediate feature output by at least one intermediate layer in the sway detection network; and input the audio features, global features and second intermediate features of the audio signal to be processed into the beat detection network.

[0035] Optionally, the beat point probability acquisition unit is also configured to input the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, the fourth global feature output by the current intermediate layer, the second intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the sway detection network, and the fifth global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network are spliced ​​to obtain a third spliced ​​global feature, and the third spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

[0036] Optionally, the beat-repeat joint detection model is trained based on the above-mentioned training method of the beat-repeat joint detection model.

[0037] According to the fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute instructions to implement a training method for a beat-rebeat joint detection model or a beat-rebeat joint detection method according to the present disclosure.

[0038] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the beat-rebeat joint detection model or the beat-rebeat joint detection method as described above according to the present disclosure.

[0039] According to the seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the training method of the beat-rebeat joint detection model or the beat-rebeat joint detection method according to the present disclosure.

[0040] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0041] According to the training of the beat-rebeat joint detection model and the beat-rebeat joint detection method disclosed in the present invention, during the training process, the beat detection network and the rebeat detection network are adjusted by a loss function that includes beat information and rebeat information, that is, the beat information is added to the training process of the rebeat detection network, which can reduce the error rate of rebeat detection and improve the accuracy of rebeat detection. In addition, the rebeat information is also introduced into the beat detection training process, which further improves the accuracy of beat detection.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0044] Figure 1 is a schematic diagram showing an implementation scenario of a training method for a beat-rebeat joint detection model according to an exemplary embodiment of the present disclosure;

[0045] Figure 2 is a flow chart of a method for training a beat-rebeat joint detection model according to an exemplary embodiment;

[0046] Figure 3 is a logic diagram of a training method for a beat-rebeat joint detection model according to an exemplary embodiment;

[0047] Figure 4 is a schematic diagram of an optional training method of a beat-rebeat joint detection model according to an exemplary embodiment;

[0048] Figure 5 is a schematic diagram of a detection result according to an exemplary embodiment;

[0049] Figure 6 is a schematic diagram of another optional training method of a beat-rebeat joint detection model according to an exemplary embodiment;

[0050] Figure 7 is a flow chart showing a beat-rebeat joint detection method according to an exemplary embodiment;

[0051] Figure 8 is a block diagram of a training device for a beat-rebeat joint detection model according to an exemplary embodiment;

[0052] Fig. 9 is a block diagram of a beat-rebeat joint detection device according to an exemplary embodiment;

[0053] Fig.10 is a block diagram of an electronic device 1000 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0055] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0056] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0057] In response to the above problems, the present disclosure provides a beat and rebeat joint detection model training and a beat and rebeat joint detection method, which can ensure the accuracy of beat detection while improving the accuracy of rebeat detection. The following is an example of detecting the beat and rebeat of a piece of music.

[0058] Figure 1 is a schematic diagram showing an implementation scenario of a training method for a beat-rebeat joint detection model according to an exemplary embodiment of the present disclosure, such as Figure 1 The implementation scenario includes a server 100, a user terminal 110 and a user terminal 120, wherein the user terminals are not limited to two, and include but are not limited to mobile phones, personal computers and other devices. The user terminal can install an application for obtaining music. The server can be a single server, or a server cluster composed of several servers, or a cloud computing platform or a virtualization center.

[0059] After receiving the request for training the beat-re-beat joint detection model sent by the user terminals 110 and 120, the server 100 counts the music clips received from the user terminals 110 and 120 in the past and respectively marks the beats and re-beats in the counted music clips, and merges the marked music clips together as a training sample set, wherein the training sample set includes multiple music clips and the beat marking information and re-beat marking information corresponding to each music clip. After obtaining the training sample set, the server 100 inputs the Mel spectrum features of the music clips therein into the beat detection network, and obtains the beat point prediction probability of each frame of data in the music clip and at least The global features of an intermediate layer output; the Mel spectrum features and global features of the audio signal are then input into the beat detection network to obtain the beat point prediction probability of each frame of the audio signal, and then the value of the target loss function is calculated based on the beat point prediction probability, the beat point prediction probability, the corresponding beat annotation information and the beat annotation information, and then the parameters of the beat detection network and the beat detection network are adjusted by adjusting the value of the target loss function, and the beat and beat joint detection model is trained to obtain the optimal beat and beat joint detection model. The trained transfer learning model can accurately process the input music clip to be processed, that is, the beat and beat of the music to be processed are obtained.

[0060] Next, we will refer to Figures 2 to 9 The training of the beat-rebeat joint detection model and the beat-rebeat joint detection method according to an exemplary embodiment of the present disclosure are described in detail.

[0061] Figure 2 is a flowchart of a training method for a beat-rebeat joint detection model according to an exemplary embodiment. Figure 2 As shown, the beat-rebeat joint detection model includes a beat detection network and a beat-rebeat detection network, and the training method includes the following steps:

[0062] In step S201, a training sample set is obtained, wherein the training sample set includes a plurality of sample audio signals and beat labeling information and re-beat labeling information corresponding to each sample audio signal. The beat labeling information and re-beat labeling information corresponding to each sample audio signal may be manually labeled or by other labeling methods, which are not disclosed in the present disclosure.

[0063] In step S202, the audio features of the sample audio signal are input into the beat detection network to obtain the beat point prediction probability of each frame of data in the sample audio signal, wherein the beat point prediction probability represents the prediction probability that each frame of data in the audio signal is a beat point. For example, the above-mentioned Mel spectrum features of the audio signal can be obtained by the following method:

[0064] 1) Transform the original audio signal to the time-frequency domain through short-time Fourier transform. If the original audio signal x with length T is x(t) in the time domain, where t represents time and 0 < t ≤ T, then after short-time Fourier transform, x(t) can be expressed in the time-frequency domain as follows:

[0065] X(n,k) = STFT(x(t)) (1)

[0066] Where n is the frame sequence, 0 < n ≤ Ns, (Ns is the total number of frames); k is the center frequency sequence, 0 < k ≤ K; (K is the total number of frequency points).

[0067] 2) Calculate the Mel-spectrum features (i.e., the above audio features) can be achieved through the following formula:

[0068] M(n,k) = Mel(X(n,k‘)) (2)

[0069] Where k‘ is the frequency sequence after Mel-spectrum mapping.

[0070] After obtaining the Mel-spectrum features, the obtained Mel-spectrum features can be input into the beat detection network to obtain the probability corresponding to whether each frame of data is a beat point, which can be specifically expressed as follows:

[0071] P(n) = Model(M(n,k)) (3)

[0072] According to the exemplary embodiments of the present disclosure, the beat and downbeat joint detection model further includes a pre-trained style classification network, and the style classification network is used to obtain the music type of the sample audio signal; wherein, inputting the audio features of the sample audio signal into the beat detection network may include: inputting the audio features of the sample audio signal into the style classification network to obtain the first intermediate features output by at least one intermediate layer in the style classification network; inputting the audio features of the sample audio signal and the first intermediate features into the beat detection network. Through this embodiment, the style classification information is used as a feature and input into the beat detection network, which improves the learning ability and recognition ability of the network, as well as the detection accuracy of the model. For example, the audio signal is first passed through an audio style classification network. Common music types include: rock, pop, classical, blues, reggae, electric, metal, jazz, etc. Different styles will correspond to different drumbeat features, forming a specific sense of rhythm for that style. Sending the style information to the beat detection network will improve the detection accuracy of the beat to a certain extent. The above first intermediate features may be the output features of the convolutional neural network (Convolutional Neural Networks, abbreviated as CNN) in the style classification network as shown below Figure 6 the output features of the convolutional neural network (CNN) in the style classification network as shown.

[0073] According to an exemplary embodiment of the present disclosure, inputting the audio features and the first intermediate features of the sample audio signal into the beat detection network may include: inputting the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer in the beat detection network, concatenating the third global feature output by the current intermediate layer and the first intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the style classification network to obtain a second concatenated global feature, and inputting the second concatenated global feature into the next intermediate layer of the current intermediate layer. Through this embodiment, the intermediate layer output of the style classification network is used as the input of the intermediate layer of the beat detection network and introduced into the beat detection, so that a beat detection result with a higher accuracy can be obtained. The above-mentioned third global feature can be as follows Figure 6 The output features of the Bilateral Long Short-Term Memory (BLSTM) network in the beat detection network shown in FIG. The second concatenated global feature can be as follows: Figure 6 Output features of the concatenation network CONCAT in the beat detection network shown.

[0074] return Figure 2 In step S203, the audio features of the sample audio signal are input into the beat detection network to obtain the beat point prediction probability of each frame of data in the audio signal, wherein the beat point prediction probability represents the prediction rate of each frame of data in the audio signal being a beat point.

[0075] According to an exemplary embodiment of the present disclosure, the audio features of the sample audio signal are input into the beat detection network to obtain the beat point prediction probability of each frame of data in the audio signal, including: inputting the audio features of the sample audio signal into the beat detection network to obtain the global features output by at least one intermediate layer of the beat detection network and the beat point prediction probability of each frame of data in the sample audio signal; wherein, inputting the audio features of the sample audio signal into the rebeat detection network may include: inputting the audio features and global features of the sample audio signal into the rebeat detection network to obtain the beat point prediction probability of each frame of data in the sample audio signal. Through this embodiment, the information output by the intermediate layer of the beat detection network is introduced into the rebeat detection network, that is, the beat information is added to the rebeat detection network, which can reduce the error rate of rebeat detection and further improve the accuracy of rebeat detection.

[0076] According to an exemplary embodiment of the present disclosure, inputting the audio features and global features of the sample audio signal into the beat detection network may include: inputting the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, splicing the first global feature output by the current intermediate layer and the second global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network to obtain a first spliced ​​global feature, and inputting the first spliced ​​global feature into the next intermediate layer of the current intermediate layer. Through this embodiment, the intermediate layer output of the beat detection network is used as the input of the intermediate layer of the beat detection network and introduced into the beat detection, so that a beat detection result with a higher accuracy can be obtained.

[0077] For example, in one implementation, the neural network can be composed of a layer of convolutional neural networks (CNN) and two layers of bidirectional long short-term memory networks (BLSTM). The beat detection network and the re-beat detection network can use the same network, and the output results of each layer of BLSM in the beat detection network are connected in series with the output of the corresponding BLSTM layer of the re-beat detection network, and then passed to the next layer of the network. In this way, the re-beat detection contains the beat information, which can further improve the accuracy of the model. It should be noted that the above-mentioned first global feature can be as follows Figure 4 The features output by the BLSTM layer in the retake detection network shown above, the second global feature can be as follows Figure 4 The features output by the BLSTM layer in the beat detection network shown in the figure, the first concatenated global features can be as follows Figure 4 Features output from the concatenation layer CONCAT in the retake detection network shown.

[0078] According to an exemplary embodiment of the present disclosure, the beat-re-beat joint detection model also includes a pre-trained swing detection network, and the swing detection network is used to obtain the beat and volume relationship of the sample audio signal; wherein, inputting the audio features and global features of the sample audio signal into the beat detection network may include: inputting the audio features of the sample audio signal into the swing detection network to obtain a second intermediate feature output by at least one intermediate layer in the swing detection network; and inputting the audio features, global features, and second intermediate features of the sample audio signal into the beat detection network. Through this embodiment, the swing classification information is input as a feature into the beat detection network, which improves the learning ability and recognition ability of the network, reduces confusion, and improves the detection accuracy of the model. For example, in addition to the above-mentioned style classification network, the music can also be passed through a swing detection network. Swing music usually gives people a sense of volume that is opposite to the actual strong and weak beats, that is, the volume of the beats is usually weaker, while the volume of non-beats is higher than the beats, so a swinging feeling is formed. This network helps the beat detection network to identify the beats and reduce confusion. The above-mentioned second intermediate feature can be as follows Figure 6 Output features of the Convolutional Neural Networks (CNN) in the sway detection network shown.

[0079] According to an exemplary embodiment of the present disclosure, inputting the audio features, global features and second intermediate features of a sample audio signal into a beat detection network may include: inputting the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, concatenating the fourth global feature output by the current intermediate layer, the second intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the sway detection network and the fifth global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network to obtain a third concatenated global feature, and inputting the third concatenated global feature into the next intermediate layer of the current intermediate layer. Through this embodiment, the intermediate layer output of the sway detection network is used as the input of the intermediate layer of the beat detection network and introduced into the beat detection, so as to obtain a beat detection result with a higher accuracy. The above-mentioned fourth global feature may be as follows Figure 6 The output features of BLSTM in the retake detection network shown in FIG. , the fifth global feature can be as follows Figure 6 The output features of BLSTM in the beat detection network shown, the third concatenated global features can be as follows Figure 6 Output features of the concatenation network CONCAT in the retake detection network shown.

[0080] In step S204, the value of the target loss function is calculated based on the beat point prediction probability, the re-beat point prediction probability, the corresponding beat marking information and the re-beat marking information.

[0081] According to an exemplary embodiment of the present disclosure, determining a target loss function based on a beat point prediction probability, a re-beat point prediction probability, corresponding beat labeling information, and re-beat labeling information may include: calculating a value of a first target loss function based on the beat point prediction probability and the corresponding beat labeling information; calculating a value of a second target loss function based on the re-beat point prediction probability and the corresponding re-beat labeling information; adding the value of the first target loss function to the value of the second target loss function to obtain the value of the target loss function. Through this embodiment, the beat detection and re-beat detection loss functions can be combined to jointly train the model to obtain better training results.

[0082] For example, the beat point probability P1(n), the re-beat point probability P2(n) and the beat annotation information B1(n) and the re-beat annotation information B2(n) can be obtained by two networks respectively, and the two can be added together to calculate the final target loss function l(n):

[0083] l1(n)=Loss(P1(n),B1(n)) (4)

[0084] l2(n)=Loss(P2(n),B2(n)) (5)

[0085] l(n)=l1(n)+l2(n) (6)

[0086] Among them, the Loss loss function can be a weighted cross entropy loss function.

[0087] In step S205, the beat detection network and the repetition detection network are jointly trained according to the value of the target loss function to obtain a trained beat repetition joint detection model. For example, the stochastic gradient descent method can be used to reduce the loss function to update the corresponding parameters of the model, and Adam optimization can be used to accelerate the iterative update of model training.

[0088] In order to facilitate understanding of the above embodiments, a systematic discussion is given below. In general, unlike beat detection, the perception of beats is more dependent on changes in chords and musical textures. However, since beats are included in the beats and together with the beats determine the musical rhythm of the music, there are some problems in predicting beats alone. In addition, inconsistencies in musical styles will also lead to large differences in the characteristics of different musical rhythms and beats. Therefore, if there is not enough data support, it will cause problems such as network failure to converge. In response to the above problems, the present disclosure proposes a training method for a beat and beat joint detection model. Figure 3 is a logic diagram of a training method for a beat-rebeat joint detection model according to an exemplary embodiment, such as Figure 3 As shown in the figure, the audio signal to be trained is first input into the beat detection network and the beat detection network respectively, and the probabilities of each frame data being non-beat, beat and beat are obtained. The target loss functions of the two are added together, that is, the deviation between the total target loss function and the beat labeling information and the beat labeling information is calculated, so as to optimize the target loss function by adjusting the parameters of the two networks, and finally the probabilities of the beat and beat of each frame are obtained at the same time. Finally, in order to obtain global information, post-processing such as Deep Belief Network (DBN) / Hidden Markov Model (HMM) / viterbi can be used to decode the output probability to obtain the final beat and downbeat time series.

[0089] The specific process of the above training can be combined with Figure 4 , Figure 5 and Figure 6 To understand.

[0090] like Figure 4 As shown in the figure, the beat detection network includes a layer of convolutional neural network (CNN), two layers of bidirectional long short-term memory network (BLSTM) and a layer of fully connected network DENSE, and the beat detection network includes a layer of CNN, a layer of BLSTM, a layer of concatenation network CONCAT, a layer of BLSTM, a layer of CONCAT and a layer of DENSE. First, the features for beat and beat detection are extracted from the original audio signal, that is, the original audio signal is subjected to short-time Fourier transform (STFT), converted from time domain to time-frequency domain, and converted into Mel spectrum features. The specific process has been discussed above and will not be discussed here. Secondly, after obtaining the Mel spectrum features, the Mel spectrum features are used as input features of the beat detection network and the re-beat detection network to predict whether each frame of data is a beat and the probability of a re-beat. During the training process, the beat detection information of each layer of BLSTM can be passed to the corresponding re-beat detection layer, and the target loss function is calculated through the probability output by the model and the beat labeling information and re-beat labeling information. For example, whether each frame feature is a beat frame or a re-beat frame is calculated based on each labeling information, and this information is used as the training target. The loss function is reduced and the model is optimized by training the parameters of the optimization model, and finally the prediction results of the beat detection and re-beat detection of each frame are obtained.

[0091] By training the beat detection network and the beat repetition detection network, we can get the beat prediction and beat repetition prediction results for each frame. Common music time signature types include 2 / 4, 3 / 4 and 4 / 4, such as Figure 5 As shown, the corresponding heavy beat frames, weak beat frames in the beat, and non-beat frames.

[0092] In another implementation, in order to solve the problem of inaccurate detection caused by different styles, style labels and swing labels can be added to the model training for joint training. The model structure is as follows: Figure 6 As shown in the figure, firstly, the features for beat and re-beat detection are extracted from the original audio signal, that is, the original audio signal is subjected to STFT, converted from the time domain to the time-frequency domain, and converted into Mel spectrum features. The specific process has been discussed above and will not be discussed here. Secondly, after obtaining the Mel spectrum features, the Mel spectrum features are used as the input features of the beat detection network and the re-beat detection network to predict whether each frame of data is a beat and the probability of re-beat. At the same time, the Mel spectrum features are used as the input features of the audio style classification network and the audio swingness classification network to predict whether each frame of data is an audio style and audio swingness. During the training process, the output information of each layer of CNN in the audio style classification network can be sequentially input into the corresponding layer in the beat detection network and spliced ​​with the corresponding BLSTM output, and the output information of each layer of CNN in the audio swingness classification network and the beat detection information of each layer of BLSTM can be transmitted to the corresponding re-beat detection layer, and the target loss function is calculated by the probability output by the model and the beat labeling information and the re-beat labeling information, such as calculating whether each frame feature is a beat frame or a re-beat frame according to each labeling information, and using this information as the training target target. By training and optimizing the parameters of the model, the loss function is reduced and the model is optimized, and finally the prediction results of beat detection and rebeat detection for each frame are obtained.

[0093] Figure 7 is a flowchart of a beat-rebeat joint detection method according to an exemplary embodiment. Figure 7 As shown, the beat-rebeat joint detection method uses a pre-trained beat-rebeat joint detection model to perform beat-rebeat joint detection. The beat-rebeat joint detection model includes a beat detection network and a beat detection network. The beat-rebeat joint detection method includes:

[0094] In step S701, an audio signal to be processed is obtained.

[0095] In step S702, the audio features of the audio signal to be processed are input into the beat detection network to obtain the beat point probability of each frame of the audio signal to be processed, wherein the beat point probability represents the probability that each frame of the audio signal to be processed is a beat point. The process of obtaining the audio features has been described in detail above and will not be discussed here.

[0096] According to an exemplary embodiment of the present disclosure, the beat-rebeat joint detection model also includes a pre-trained style classification network, and the style classification network is used to obtain the music type of the audio signal; wherein, the audio features of the audio signal to be processed are input into the beat detection network, including: inputting the audio features of the audio signal to be processed into the style classification network to obtain a first intermediate feature output by at least one intermediate layer in the style classification network; inputting the audio features of the audio signal to be processed and the first intermediate feature into the beat detection network. Through this embodiment, the style classification information is input as a feature into the beat detection network, which improves the learning ability and recognition ability of the network, as well as the detection accuracy of the model. For example, the audio signal is first passed through an audio style classification network. Common music types include: rock, pop, classical, blues, reggae, electric, metal, jazz, etc. Different styles will correspond to different drum beat features, forming a rhythm specific to the style. Sending the style information to the beat detection network will improve the detection accuracy of the beat to a certain extent. The above-mentioned first intermediate feature can be as follows Figure 6 Output features of the Convolutional Neural Networks (CNN) in the style classification network shown.

[0097] According to an exemplary embodiment of the present disclosure, the audio features and the first intermediate features of the audio signal to be processed are input into the beat detection network, including: inputting the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer in the beat detection network, concatenating the third global feature output by the current intermediate layer and the first intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the style classification network to obtain a second concatenated global feature, and inputting the second concatenated global feature into the next intermediate layer of the current intermediate layer. Through this embodiment, the intermediate layer output of the style classification network is used as the input of the intermediate layer of the beat detection network and introduced into the beat detection, so that a beat detection result with a higher accuracy can be obtained. The above-mentioned third global feature can be as follows: Figure 6 The output features of the Bilateral Long Short-Term Memory (BLSTM) network in the beat detection network shown in FIG. The second concatenated global feature can be as follows: Figure 6Output features of the concatenation network CONCAT in the beat detection network shown.

[0098] In step S703, the audio features of the audio signal to be processed are input into the beat detection network to obtain the beat point probability of each frame of data in the audio signal to be processed, wherein the beat point probability represents the probability that each frame of data in the audio signal to be processed is a beat point.

[0099] According to an exemplary embodiment of the present disclosure, the audio features of the audio signal to be processed are input into the beat detection network to obtain the beat point probability of each frame of data in the audio signal to be processed, including: inputting the audio features of the audio signal to be processed into the beat detection network to obtain the global features output by at least one intermediate layer of the beat detection network and the beat point probability of each frame of data in the audio signal to be processed; wherein, the audio features of the audio signal to be processed are input into the rebeat detection network to obtain the predicted probability of the rebeat point of each frame of data in the audio signal to be processed, including: inputting the audio features and global features of the audio signal to be processed into the rebeat detection network to obtain the predicted probability of the rebeat point of each frame of data in the audio signal to be processed. Through this embodiment, the information output by the intermediate layer of the beat detection network is introduced into the rebeat detection network, that is, the beat information is added to the rebeat detection network, which can reduce the error rate of rebeat detection and further improve the accuracy of rebeat detection.

[0100] According to an exemplary embodiment of the present disclosure, the audio features and global features of the audio signal to be processed are input into the beat detection network, including: inputting the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, splicing the first global feature output by the current intermediate layer and the second global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network to obtain a first spliced ​​global feature, and inputting the first spliced ​​global feature into the next intermediate layer of the current intermediate layer. Through this embodiment, the intermediate layer output of the beat detection network is used as the input of the intermediate layer of the beat detection network and introduced into the beat detection, so that a beat detection result with a higher accuracy can be obtained.

[0101] According to an exemplary embodiment of the present disclosure, the beat-re-beat joint detection model also includes a pre-trained swing detection network, and the swing detection network is used to obtain the beat and volume relationship of the audio signal; wherein, the audio features and global features of the audio signal to be processed are input into the beat detection network, including: inputting the audio features of the audio signal to be processed into the swing detection network to obtain a second intermediate feature output by at least one intermediate layer in the swing detection network; and inputting the audio features, global features, and second intermediate features of the audio signal to be processed into the beat detection network. Through this embodiment, the swing classification information is input as a feature into the beat detection network, which improves the learning ability and recognition ability of the network, reduces confusion, and improves the detection accuracy of the model. For example, in addition to the above-mentioned style classification network, the music can also be passed through a swing detection network. Swing music usually gives people a sense of volume that is opposite to the actual strong and weak beats, that is, the volume of the beats is usually weaker, while the volume of non-beats is higher than the beats, so a swinging feeling is formed. This network helps the beat detection network to identify the beats and reduce confusion. The above-mentioned second intermediate feature can be as follows Figure 6 Output features of the Convolutional Neural Networks (CNN) in the sway detection network shown.

[0102] According to an exemplary embodiment of the present disclosure, the audio features, global features and second intermediate features of the audio signal to be processed are input into the beat detection network, including: inputting the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, splicing the fourth global feature output by the current intermediate layer, the second intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the sway detection network and the fifth global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network to obtain a third spliced ​​global feature, and inputting the third spliced ​​global feature into the next intermediate layer of the current intermediate layer. Through this embodiment, the intermediate layer output of the sway detection network is used as the input of the intermediate layer of the beat detection network and introduced into the beat detection, so as to obtain a beat detection result with a higher accuracy. The above-mentioned fourth global feature can be as follows: Figure 6 The output features of BLSTM in the retake detection network shown in FIG. , the fifth global feature can be as follows Figure 6 The output features of BLSTM in the beat detection network shown in FIG. 1 , the third concatenated global feature can be as follows: Figure 6 Output features of the concatenation network CONCAT in the retake detection network shown.

[0103] For example, in order to solve the problem of inaccurate detection caused by different styles, style labels and swing labels can be added to the model training. The model structure is as follows: Figure 6 As shown, first, the features for beat and re-beat detection are extracted from the original audio signal, that is, the original audio signal is subjected to STFT, converted from the time domain to the time-frequency domain, and converted into Mel spectrum features. The specific process has been discussed above and will not be discussed here. Secondly, after obtaining the Mel spectrum features, the Mel spectrum features are used as input features of the beat detection network and the re-beat detection network to predict whether each frame of data is a beat and the probability of re-beat. At the same time, the Mel spectrum features are used as input features of the audio style classification network and the audio swingness classification network to predict whether each frame of data is an audio style and audio swingness. In the actual detection process, the output information of each layer of CNN in the audio style classification network can be sequentially input into the corresponding layer in the beat detection network and spliced ​​with the corresponding BLSTM output, and the output information of each layer of CNN in the audio swingness classification network and the beat detection information of each layer of BLSTM can be transmitted to the corresponding re-beat detection layer. The final output of the model is the prediction result of beat detection and re-beat detection of each frame.

[0104] According to an exemplary embodiment of the present disclosure, the beat-rebeat joint detection model is trained based on the above-mentioned training method of the beat-rebeat joint detection model. According to this embodiment, since the above-mentioned training method of the beat-rebeat joint detection model takes into account the combination of beat information, beat information, style wind and thunder network output information and swing detection network output information, the trained beat-rebeat joint detection model has a higher accuracy, so the model can be used to accurately detect the beat point probability and the beat point probability of the audio signal to be processed.

[0105] In step S704, the beat and re-beat joint detection result of the audio signal to be processed is determined according to the probability that each frame of data in the audio signal to be processed is a beat point and the probability that each frame of data in the audio signal to be processed is a re-beat point.

[0106] Figure 8 is a block diagram of a training device for a beat-repeat joint detection model according to an exemplary embodiment. Figure 8 The beat-rebeat joint detection model includes a beat detection network and a rebeat detection network, and the training device includes: a sample set acquisition unit 80, a beat point probability acquisition unit 82, a rebeat point probability acquisition unit 84, a target loss function determination unit 86 and a training unit 88.

[0107] The sample set acquisition unit 80 is configured to acquire a training sample set, wherein the training sample set includes a plurality of sample audio signals and beat marking information and re-beat marking information corresponding to each sample audio signal; the beat point probability acquisition unit 82 is configured to input the audio features of the sample audio signal into the beat detection network to obtain the beat point prediction probability of each frame of data in the sample audio signal, wherein the beat point prediction probability indicates the prediction probability that each frame of data in the sample audio signal is a beat point; the re-beat point probability acquisition unit 84 is configured to input the audio features of the sample audio signal into the re-beat detection network to obtain the re-beat point prediction probability of each frame of data in the sample audio signal, wherein the re-beat point prediction probability indicates the prediction probability that each frame of data in the sample audio signal is a re-beat point; the target loss function determination unit 86 is configured to calculate the value of the target loss function based on the beat point prediction probability, the re-beat point prediction probability, the corresponding beat marking information and the re-beat marking information; and is configured to jointly train the beat detection network and the re-beat detection network according to the value of the target loss function to obtain a trained beat-re-beat joint detection model.

[0108] According to an exemplary embodiment of the present disclosure, the beat point probability acquisition unit 82 is further configured to input the audio features of the sample audio signal into the beat detection network to obtain the global features output by at least one intermediate layer of the beat detection network and the beat point prediction probability of each frame data in the sample audio signal; the rebeat point probability acquisition unit 84 is further configured to input the audio features and the global features of the sample audio signal into the rebeat detection network to obtain the rebeat point prediction probability of each frame data in the sample audio signal.

[0109] According to an exemplary embodiment of the present disclosure, the beat point probability acquisition unit 84 is also configured to input the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, the first global feature output by the current intermediate layer and the second global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network are spliced ​​to obtain a first spliced ​​global feature, and the first spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

[0110] According to an exemplary embodiment of the present disclosure, the beat-rebeat joint detection model also includes a pre-trained style classification network, which is used to obtain the music type of the sample audio signal; wherein the beat point probability acquisition unit 82 is also configured to input the audio features of the sample audio signal into the style classification network to obtain a first intermediate feature output by at least one intermediate layer in the style classification network; and input the audio features of the sample audio signal and the first intermediate feature into the beat detection network.

[0111] According to an exemplary embodiment of the present disclosure, the beat point probability acquisition unit 82 is also configured to input the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer in the beat detection network, the third global feature output by the current intermediate layer and the first intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the style classification network are spliced ​​to obtain a second spliced ​​global feature, and the second spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

[0112] According to an exemplary embodiment of the present disclosure, the beat-rebeat joint detection model also includes a pre-trained sway detection network; wherein the beat point probability acquisition unit 84 is also configured to input the audio features of the sample audio signal into the sway detection network, the sway detection network is used to obtain the beat and volume relationship of the sample audio signal, and obtain a second intermediate feature output by at least one intermediate layer in the sway detection network; the audio features, global features and second intermediate features of the sample audio signal are input into the beat detection network.

[0113] According to an exemplary embodiment of the present disclosure, the beat point probability acquisition unit 84 is also configured to input the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, the fourth global feature output by the current intermediate layer, the second intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the sway detection network, and the fourth global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network are spliced ​​to obtain a third spliced ​​global feature, and the third spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

[0114] According to an exemplary embodiment of the present disclosure, the target loss function determination unit 86 is also configured to calculate the value of the first target loss function based on the beat point prediction probability and the corresponding beat annotation information; calculate the value of the second target loss function based on the rebeat point prediction probability and the corresponding rebeat annotation information; add the value of the first target loss function and the value of the second target loss function to obtain the value of the target loss function.

[0115] Fig. 9 is a block diagram of a beat-rebeat joint detection device according to an exemplary embodiment. Fig. 9 The beat-rebeat joint detection device uses a pre-trained beat-rebeat joint detection model to perform beat-rebeat joint detection. The beat-rebeat joint detection model includes a beat detection network and a beat detection network. The beat-rebeat joint detection device includes: an audio signal acquisition unit 90, a beat point probability acquisition unit 92, a beat point probability acquisition unit 94 and a determination unit 96.

[0116] The audio signal acquisition unit 90 is configured to acquire the audio signal to be processed; the beat point probability acquisition unit 92 is configured to input the audio features of the audio signal to be processed into the beat detection network to obtain the beat point probability of each frame of data in the audio signal to be processed, wherein the beat point probability represents the probability that each frame of data in the audio signal to be processed is a beat point; the rebeat point probability acquisition unit 94 is configured to input the audio features of the audio signal to be processed into the rebeat detection network to obtain the rebeat point probability of each frame of data in the audio signal to be processed, wherein the rebeat point probability represents the probability that each frame of data in the audio signal to be processed is a rebeat point; the determination unit 96 is configured to determine the beat and rebeat joint detection result of the audio signal to be processed based on the probability that each frame of data in the audio signal to be processed is a beat point and the probability that each frame of data in the audio signal to be processed is a rebeat point.

[0117] According to an exemplary embodiment of the present disclosure, the beat point probability acquisition unit 92 is configured to input the audio features of the audio signal to be processed into the beat detection network, and obtain the global features output by at least one intermediate layer of the beat detection network and the beat point probability of each frame data in the audio signal to be processed; the rebeat point probability acquisition unit 94 is also configured to input the audio features and global features of the audio signal to be processed into the rebeat detection network, and obtain the rebeat point prediction probability of each frame data in the audio signal to be processed.

[0118] According to an exemplary embodiment of the present disclosure, the beat point probability acquisition unit 94 is also configured to input the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, the first global feature output by the current intermediate layer and the second global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network are spliced ​​to obtain a first spliced ​​global feature, and the first spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

[0119] According to an exemplary embodiment of the present disclosure, the beat-rebeat joint detection model also includes a pre-trained style classification network, which is used to obtain the music type of the audio signal; wherein the beat point probability acquisition unit 92 is also configured to input the audio features of the audio signal to be processed into the style classification network to obtain a first intermediate feature output by at least one intermediate layer in the style classification network; and input the audio features of the audio signal to be processed and the first intermediate feature into the beat detection network.

[0120] According to an exemplary embodiment of the present disclosure, the beat point probability acquisition unit 92 is also configured to input the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer in the beat detection network, the third global feature output by the current intermediate layer and the first intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the style classification network are spliced ​​to obtain a second spliced ​​global feature, and the second spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

[0121] According to an exemplary embodiment of the present disclosure, the beat-rebeat joint detection model also includes a pre-trained sway detection network, which is used to obtain the relationship between the beat and volume of the audio signal; wherein the beat point probability acquisition unit 94 is also configured to input the audio features of the audio signal to be processed into the sway detection network to obtain a second intermediate feature output by at least one intermediate layer in the sway detection network; and input the audio features, global features and second intermediate features of the audio signal to be processed into the beat detection network.

[0122] According to an exemplary embodiment of the present disclosure, the beat point probability acquisition unit 94 is also configured to input the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, the fourth global feature output by the current intermediate layer, the second intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the sway detection network, and the fifth global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network are spliced ​​to obtain a third spliced ​​global feature, and the third spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

[0123] According to an exemplary embodiment of the present disclosure, the beat-rebeat joint detection model is trained based on the above-mentioned training method of the beat-rebeat joint detection model.

[0124] According to an embodiment of the present disclosure, an electronic device may be provided. Fig.10 It is a block diagram of an electronic device 1000 according to an embodiment of the present disclosure, and the electronic device includes at least one memory 1001 and at least one processor 1002, wherein the at least one memory stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor, a training method for a beat-repeat joint detection model or a beat-repeat joint detection method according to an embodiment of the present disclosure is executed.

[0125] As an example, the electronic device 1000 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1000 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 1000 may also be part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0126] In the electronic device 1000, the processor 1002 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 1002 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0127] The processor 1002 may execute instructions or codes stored in the memory, wherein the memory 1001 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0128] The memory 1001 may be integrated with the processor 1002, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1001 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1001 and the processor 1002 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1002 can read files stored in the memory 1001.

[0129] In addition, the electronic device 1000 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0130] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the beat-rebeat joint detection model or the beat-rebeat joint detection method of the embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0131] According to an embodiment of the present disclosure, a computer program product is provided, including computer instructions, which, when executed by a processor, implement the training method of the beat-rebeat joint detection model or the beat-rebeat joint detection method of the embodiment of the present disclosure.

[0132] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0133] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for a beat-repeat joint detection model, characterized in that: The beat-rebeat joint detection model includes a beat detection network and a beat-rebeat detection network, and the training method includes: Acquire a training sample set; the training sample set includes a plurality of sample audio signals and beat marking information and re-beat marking information corresponding to each sample audio signal; Inputting the audio features of the sample audio signal into the beat detection network to obtain the beat point prediction probability of each frame of data in the sample audio signal, wherein the beat point prediction probability represents the prediction probability that each frame of data in the sample audio signal is a beat point; Inputting the audio features of the sample audio signal into the rebeat detection network to obtain a rebeat point prediction probability of each frame of data in the sample audio signal, wherein the rebeat point prediction probability represents a prediction probability that each frame of data in the sample audio signal is a rebeat point; Calculating a value of a target loss function based on the beat point prediction probability, the re-beat point prediction probability, the corresponding beat marking information, and the re-beat marking information; According to the value of the target loss function, the beat detection network and the rebeat detection network are jointly trained to obtain a trained beat and rebeat joint detection model; The step of inputting the audio features of the sample audio signal into the beat detection network to obtain the beat point prediction probability of each frame of data in the sample audio signal includes: Inputting the audio features of the sample audio signal into the beat detection network to obtain the global features output by at least one intermediate layer of the beat detection network and the beat point prediction probability of each frame of data in the sample audio signal; The step of inputting the audio features of the sample audio signal into the rebeat detection network to obtain the rebeat point prediction probability of each frame of data in the sample audio signal includes: The audio features of the sample audio signal and the global features are input into the rebeat detection network to obtain the predicted probability of the rebeat point of each frame of data in the sample audio signal.

2. The training method according to claim 1, characterized in that: The step of inputting the audio features and the global features of the sample audio signal into the rebeat detection network comprises: Inputting the audio features of the sample audio signal into the input layer of the rebeat detection network; For each intermediate layer of at least one intermediate layer of the beat detection network, a first global feature output by the current intermediate layer and a second global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network are spliced ​​to obtain a first spliced ​​global feature, and the first spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

3. The training method according to claim 1, characterized in that: The beat-rebeat joint detection model also includes a pre-trained style classification network, and the style classification network is used to obtain the music type of the sample audio signal; The step of inputting the audio features of the sample audio signal into the beat detection network includes: Inputting the audio features of the sample audio signal into the style classification network to obtain a first intermediate feature output by at least one intermediate layer in the style classification network; The audio features of the sample audio signal and the first intermediate features are input into the beat detection network.

4. The training method according to claim 3, characterized in that: The step of inputting the audio feature of the sample audio signal and the first intermediate feature into the beat detection network comprises: Inputting the audio features of the sample audio signal into the input layer of the beat detection network; For each intermediate layer of at least one intermediate layer in the beat detection network, the third global feature output by the current intermediate layer and the first intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the style classification network are spliced ​​to obtain a second spliced ​​global feature, and the second spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

5. The training method according to claim 1, characterized in that: The beat-re-beat joint detection model also includes a pre-trained swing detection network, and the swing detection network is used to obtain the relationship between the beat and volume of the sample audio signal; The step of inputting the audio features and the global features of the sample audio signal into the rebeat detection network comprises: Inputting the audio features of the sample audio signal into the sway detection network to obtain a second intermediate feature output by at least one intermediate layer in the sway detection network; The audio feature of the sample audio signal, the global feature, and the second intermediate feature are input into the rebeat detection network.

6. The training method according to claim 5, characterized in that: The step of inputting the audio feature of the sample audio signal, the global feature, and the second intermediate feature into the rebeat detection network comprises: Inputting the audio features of the sample audio signal into the input layer of the rebeat detection network; For each intermediate layer of at least one intermediate layer of the beat detection network, the fourth global feature output by the current intermediate layer, the second intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the sway detection network, and the fifth global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network are spliced ​​to obtain a third spliced ​​global feature, and the third spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

7. The training method according to claim 1, characterized in that: The calculating the value of the target loss function based on the beat point prediction probability, the re-beat point prediction probability, the corresponding beat marking information and the re-beat marking information includes: Calculating a value of a first objective loss function based on the beat point prediction probability and the corresponding beat labeling information; Calculating a value of a second objective loss function based on the retake point prediction probability and the corresponding retake annotation information; The value of the first objective loss function and the value of the second objective loss function are added to obtain the value of the objective loss function.

8. A beat-repeat joint detection method, characterized in that: The beat-repeat joint detection method uses a pre-trained beat-repeat joint detection model to perform beat-repeat joint detection, the beat-repeat joint detection model includes a beat detection network and a beat detection network, and the beat-repeat joint detection method includes: Obtaining the audio signal to be processed; Inputting the audio features of the audio signal to be processed into the beat detection network to obtain the beat point probability of each frame of data in the audio signal to be processed, wherein the beat point probability represents the probability that each frame of data in the audio signal to be processed is a beat point; Inputting the audio features of the audio signal to be processed into a rebeat detection network to obtain a rebeat point probability of each frame of data in the audio signal to be processed, wherein the rebeat point probability represents the probability that each frame of data in the audio signal to be processed is a rebeat point; Determine a beat-rebeat joint detection result of the audio signal to be processed according to a probability that each frame of data in the audio signal to be processed is a beat point and a probability that each frame of data in the audio signal to be processed is a rebeat point; The step of inputting the audio features of the audio signal to be processed into the beat detection network to obtain the beat point probability of each frame of data in the audio signal to be processed includes: Inputting the audio features of the audio signal to be processed into the beat detection network to obtain the global features output by at least one intermediate layer of the beat detection network and the beat point probability of each frame of data in the audio signal to be processed; The step of inputting the audio features of the audio signal to be processed into the rebeat detection network to obtain the rebeat point probability of each frame of data in the audio signal to be processed includes: The audio features of the audio signal to be processed and the global features are input into the rebeat detection network to obtain the probability of a rebeat point for each frame of data in the audio signal to be processed.

9. The beat-repeat joint detection method according to claim 8, characterized in that: The step of inputting the audio features and the global features of the audio signal to be processed into a rebeat detection network comprises: Inputting the audio features of the audio signal to be processed into the input layer of the rebeat detection network; For each intermediate layer of at least one intermediate layer of the beat detection network, a first global feature output by the current intermediate layer and a second global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network are spliced ​​to obtain a first spliced ​​global feature, and the first spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

10. The beat-repeat joint detection method according to claim 8, characterized in that: The beat-rebeat joint detection model also includes a pre-trained style classification network, and the style classification network is used to obtain the music type of the audio signal; The step of inputting the audio features of the audio signal to be processed into the beat detection network comprises: Inputting the audio features of the audio signal to be processed into the style classification network to obtain a first intermediate feature output by at least one intermediate layer in the style classification network; The audio feature of the audio signal to be processed and the first intermediate feature are input into the beat detection network.

11. The beat-repeat joint detection method according to claim 10, characterized in that: The step of inputting the audio feature of the audio signal to be processed and the first intermediate feature into the beat detection network comprises: Inputting the audio features of the audio signal to be processed into the input layer of the beat detection network; For each intermediate layer of at least one intermediate layer in the beat detection network, the third global feature output by the current intermediate layer and the first intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the style classification network are spliced ​​to obtain a second spliced ​​global feature, and the second spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

12. The beat-repeat joint detection method according to claim 8, characterized in that: The beat-re-beat joint detection model also includes a pre-trained swing detection network, and the swing detection network is used to obtain the relationship between the beat and volume of the audio signal; The step of inputting the audio features and the global features of the audio signal to be processed into a rebeat detection network includes: Inputting the audio feature of the audio signal to be processed into the sway detection network to obtain a second intermediate feature output by at least one intermediate layer in the sway detection network; The audio feature of the audio signal to be processed, the global feature and the second intermediate feature are input into the rebeat detection network.

13. The beat-repeat joint detection method according to claim 12, characterized in that: The step of inputting the audio feature, the global feature and the second intermediate feature of the audio signal to be processed into the rebeat detection network comprises: Inputting the audio features of the audio signal to be processed into the input layer of the rebeat detection network; For each intermediate layer of at least one intermediate layer of the beat detection network, the fourth global feature output by the current intermediate layer, the second intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the sway detection network, and the fifth global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network are spliced ​​to obtain a third spliced ​​global feature, and the third spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

14. The beat-repeat joint detection method according to any one of claims 8 to 13, characterized in that: The beat-repeat joint detection model is trained based on the training method of the beat-repeat joint detection model described in any one of claims 1 to 7.

15. A training device for a beat-repeat joint detection model, characterized in that: The beat-repeat joint detection model includes a beat detection network and a beat detection network, and the training device includes: A sample set acquisition unit is configured to acquire a training sample set, wherein the training sample set includes a plurality of sample audio signals and beat marking information and re-beat marking information corresponding to each sample audio signal; A beat point probability acquisition unit is configured to input the audio features of the sample audio signal into the beat detection network to obtain the beat point prediction probability of each frame of data in the sample audio signal, wherein the beat point prediction probability represents the prediction probability that each frame of data in the sample audio signal is a beat point; a rebeat point probability acquisition unit, configured to input the audio features of the sample audio signal into the rebeat detection network to obtain a rebeat point prediction probability of each frame of data in the sample audio signal, wherein the rebeat point prediction probability represents a prediction probability that each frame of data in the sample audio signal is a rebeat point; a target loss function determination unit, configured to calculate a value of a target loss function based on the beat point prediction probability, the re-beat point prediction probability, the corresponding beat marking information and the re-beat marking information; A training unit is configured to jointly train the beat detection network and the rebeat detection network according to the value of the target loss function to obtain a trained beat-rebeat joint detection model; The beat point probability acquisition unit is further configured to input the audio features of the sample audio signal into the beat detection network to obtain the global features output by at least one intermediate layer of the beat detection network and the beat point prediction probability of each frame of data in the sample audio signal; The rebeat point probability acquisition unit is further configured to input the audio features of the sample audio signal and the global features into the rebeat detection network to obtain the rebeat point prediction probability of each frame of data in the sample audio signal.

16. The training device according to claim 15, characterized in that The beat point probability acquisition unit is also configured to input the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, splice the first global feature output by the current intermediate layer and the second global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network to obtain a first spliced ​​global feature, and input the first spliced ​​global feature to the next intermediate layer of the current intermediate layer.

17. The training device according to claim 15, characterized in that The beat-rebeat joint detection model also includes a pre-trained style classification network, which is used to obtain the music type of the sample audio signal; wherein the beat point probability acquisition unit is also configured to input the audio features of the sample audio signal into the style classification network to obtain a first intermediate feature output by at least one intermediate layer in the style classification network; and input the audio features of the sample audio signal and the first intermediate feature into the beat detection network.

18. The training device according to claim 17, characterized in that The beat point probability acquisition unit is also configured to input the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer in the beat detection network, concatenate the third global feature output by the current intermediate layer and the intermediate feature output by the first intermediate layer corresponding to the current intermediate layer in the style classification network to obtain a second concatenated global feature, and input the second concatenated global feature to the next intermediate layer of the current intermediate layer.

19. The training device according to claim 15, characterized in that The beat-rebeat joint detection model also includes a pre-trained sway detection network, which is used to obtain the beat and volume relationship of the sample audio signal; wherein the beat point probability acquisition unit is also configured to input the audio features of the sample audio signal into the sway detection network to obtain a second intermediate feature output by at least one intermediate layer in the sway detection network; and input the audio features of the sample audio signal, the global features and the second intermediate features into the beat detection network.

20. The training device according to claim 19, characterized in that The beat point probability acquisition unit is also configured to input the audio features of the sample audio signal into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, the fourth global feature output by the current intermediate layer, the second intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the sway detection network, and the fifth global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network are spliced ​​to obtain a third spliced ​​global feature, and the third spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

21. The training device according to claim 15, characterized in that The target loss function determination unit is further configured to calculate a value of a first target loss function based on the beat point prediction probability and the corresponding beat labeling information; Based on the retake point prediction probability and the corresponding retake annotation information, the value of the second objective loss function is calculated; and the value of the first objective loss function and the value of the second objective loss function are added to obtain the value of the objective loss function.

22. A beat and rebeat joint detection device, characterized in that: The beat-repeat joint detection device performs beat-repeat joint detection using a pre-trained beat-repeat joint detection model, wherein the beat-repeat joint detection model includes a beat detection network and a beat detection network, and the beat-repeat joint detection device includes: An audio signal acquisition unit, configured to acquire an audio signal to be processed; A beat point probability acquisition unit is configured to input the audio features of the audio signal to be processed into a beat detection network to obtain the beat point probability of each frame of data in the audio signal to be processed, wherein the beat point probability represents the probability that each frame of data in the audio signal to be processed is a beat point; a rebeat point probability acquisition unit, configured to input the audio features of the audio signal to be processed into the rebeat detection network, and obtain the rebeat point probability of each frame of data in the audio signal to be processed, wherein the rebeat point probability represents the probability that each frame of data in the audio signal to be processed is a rebeat point; A determination unit configured to determine a beat-rebeat joint detection result of the audio signal to be processed according to a probability that each frame of data in the audio signal to be processed is a beat point and a probability that each frame of data in the audio signal to be processed is a rebeat point; The beat point probability acquisition unit is configured to input the audio features of the audio signal to be processed into the beat detection network to obtain the global features output by at least one intermediate layer of the beat detection network and the beat point probability of each frame of data in the audio signal to be processed; The rebeat point probability acquisition unit is further configured to input the audio features of the audio signal to be processed and the global features into the rebeat detection network to obtain the rebeat point probability of each frame of data in the audio signal to be processed.

23. The beat and re-beat joint detection device according to claim 22, characterized in that: The beat point probability acquisition unit is also configured to input the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, concatenate the first global feature output by the current intermediate layer and the second global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network to obtain a first concatenated global feature, and input the first concatenated global feature to the next intermediate layer of the current intermediate layer.

24. The beat and re-beat joint detection device according to claim 22, characterized in that: The beat-rebeat joint detection model also includes a pre-trained style classification network, which is used to obtain the music type of the audio signal; wherein the beat point probability acquisition unit is also configured to input the audio features of the audio signal to be processed into the style classification network to obtain a first intermediate feature output by at least one intermediate layer in the style classification network; and input the audio features of the audio signal to be processed and the first intermediate feature into the beat detection network.

25. The beat and rebeat joint detection device according to claim 24, characterized in that: The beat point probability acquisition unit is also configured to input the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer in the beat detection network, concatenate the third global feature output by the current intermediate layer and the first intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the style classification network to obtain a second concatenated global feature, and input the second concatenated global feature to the next intermediate layer of the current intermediate layer.

26. The beat and re-beat joint detection device according to claim 22, characterized in that: The beat-rebeat joint detection model also includes a pre-trained sway detection network, which is used to obtain the relationship between the beat and volume of the audio signal; wherein the beat point probability acquisition unit is also configured to input the audio features of the audio signal to be processed into the sway detection network to obtain a second intermediate feature output by at least one intermediate layer in the sway detection network; and input the audio features of the audio signal to be processed, the global features and the second intermediate features into the beat detection network.

27. The beat and re-beat joint detection device according to claim 26, characterized in that: The beat point probability acquisition unit is also configured to input the audio features of the audio signal to be processed into the input layer of the beat detection network; for each intermediate layer of at least one intermediate layer of the beat detection network, the fourth global feature output by the current intermediate layer, the second intermediate feature output by the intermediate layer corresponding to the current intermediate layer in the sway detection network, and the fifth global feature output by the intermediate layer corresponding to the current intermediate layer in the beat detection network are spliced ​​to obtain a third spliced ​​global feature, and the third spliced ​​global feature is input to the next intermediate layer of the current intermediate layer.

28. The beat-repeat joint detection device according to any one of claims 22 to 27, characterized in that: The beat-repeat joint detection model is trained based on the training method of the beat-repeat joint detection model described in any one of claims 1 to 7.

29. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the training method of the beat-rebeat joint detection model as described in any one of claims 1 to 7 or the beat-rebeat joint detection method as described in any one of claims 8 to 14.

30. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the training method of the beat-rebeat joint detection model as described in any one of claims 1 to 7 or the beat-rebeat joint detection method as described in any one of claims 8 to 14.

31. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by the processor, the training method of the beat-rebeat joint detection model as described in any one of claims 1 to 7 or the beat-rebeat joint detection method as described in any one of claims 8 to 14 is implemented.