Music rhythm detection method and apparatus, terminal device, and storage medium

Abstract

The application is suitable for the technical field of audio signal rhythm detection, and provides a music rhythm detection method and device, terminal equipment and computer readable storage medium, comprising: obtaining style information of an audio signal; determining model parameters according to the style information; loading the model parameters into a target rhythm recognition model, wherein the target rhythm recognition model is used for recognizing the rhythm of the audio signal input into the target rhythm recognition model, and the target rhythm recognition model is trained by a plurality of first sample audio signals whose style information is known; and obtaining a rhythm detection result of the audio signal according to the audio signal and the target rhythm recognition model loaded with the model parameters. The above method can effectively perform rhythm detection on music signals of different time signatures and style types.

Description

Technical Field

[0001] This application belongs to the field of rhythm detection technology, and in particular relates to a music rhythm detection method, device, terminal equipment and computer-readable storage medium. Background Technology

[0002] The automotive cockpit entertainment system aims to provide users with a more immersive and personalized entertainment experience. Among them, music signal rhythm detection plays a key role. By analyzing the rhythm information of the music signal in real time, lighting effects, seat vibrations, ambient sound effects, etc., can be synchronized with the music signal, thereby enhancing the user's auditory and visual experience.

[0003] Because the car user base is very broad, and the types and characteristics of music signals listened to by users are also very diverse, music signal rhythm detection needs to be adaptable to music signals with different time signatures and styles. Related technologies cannot effectively detect the rhythm of music signals or styles with weak rhythms. Summary of the Invention

[0004] This application provides a detection method, apparatus, terminal device, and computer-readable storage medium, which can effectively detect the rhythm of music signals with different time signatures and genres.

[0005] In a first aspect, embodiments of this application provide a music rhythm detection method, including:

[0006] Obtain the genre information of the audio signal;

[0007] Determine model parameters based on music style information;

[0008] The model parameters are loaded into the target rhythm recognition model, which is used to recognize the rhythm of the audio signal input to the target rhythm recognition model. The target rhythm recognition model is trained from multiple first sample audio signals with known genre information.

[0009] Based on the audio signal and the target rhythm recognition model loaded with the model parameters, the rhythm detection result of the audio signal is obtained.

[0010] In this embodiment, specific model parameters are determined based on the genre information of the audio signal. These parameters are better adapted to the musical characteristics of that genre. The obtained model parameters are loaded into the target rhythm recognition model for rhythm detection. Essentially, a set of adapted recognition parameters is generated for the audio signal based on its genre information. These parameters are applied to the target rhythm recognition model, allowing it to capture the beat of the musical features to be identified. This improves rhythm detection across different genres. In other words, adapted parameters are generated for different genres, and the target rhythm recognition model uses these parameters to improve its ability to capture the beat of different genres, effectively performing rhythm detection and improving the accuracy of rhythm detection across different genres.

[0011] In one possible implementation of the first aspect, obtaining the rhythm detection result of the audio signal based on the audio signal and the target rhythm recognition model loaded with the model parameters includes:

[0012] Convert the audio signal into a frequency domain signal;

[0013] The frequency domain signal is input into the target rhythm recognition model loaded with the model parameters to obtain the beat excitation value sequence corresponding to the audio signal. The beat excitation value sequence includes one or more beat excitation values ​​and is used to represent the probability value of different beat types existing at different times.

[0014] The rhythm detection result of the audio signal is obtained based on the beat excitation value sequence.

[0015] In the embodiments of this application, the excitation value sequence represents the probability value of different beat types existing at different times in the form of probability, which can more accurately identify multiple beat types and improve the flexibility and adaptability of the model.

[0016] In one possible implementation of the first aspect, the rhythm detection result of the audio signal includes a beat type, a beat number, and a beat tempo; obtaining the rhythm detection result of the audio signal based on the beat excitation value sequence includes:

[0017] The beat type is determined by each beat excitation value in the excitation value sequence; the beat type includes strong beats and weak beats;

[0018] The time signature corresponding to the audio signal is determined by the ratio of the number of light beats to the number of heavy beats in each beat type;

[0019] The beat speed is determined by the interval between two adjacent taps in the beat type.

[0020] In this embodiment of the application, the detection result of the audio signal is obtained by decoding the excitation value sequence, which can improve the accuracy of the detection result.

[0021] In one embodiment, determining the model parameters based on the music style information includes:

[0022] Based on the music style information and the target diffusion neural network, the model parameters are determined. The target diffusion neural network is a diffusion neural network obtained when the loss function value of each first sample audio signal converges. When the loss function value of each first sample audio signal converges, the first sample model parameters corresponding to each first sample audio signal output by the target diffusion neural network are optimal. Accordingly, the target rhythm recognition model is also loaded with the first sample model parameters corresponding to each first sample audio signal.

[0023] In the embodiments of this application, the model parameters obtained by using the target diffusion model can better match the style characteristics of the audio signal, which helps to identify the style characteristics using the model parameters in the future and improves the accuracy of rhythm detection.

[0024] In one possible implementation of the first aspect, the method further includes:

[0025] Obtain a set of sample audio signals, the set of sample audio signals including multiple first sample audio signals with known music style information, the music style information of the multiple first sample audio signals being different;

[0026] The genre information corresponding to each of the multiple first sample audio signals is vector-encoded to obtain multiple first sample encoded vectors;

[0027] Generate multiple first sample model parameters based on multiple first sample encoding vectors;

[0028] The target rhythm recognition model is determined based on multiple parameters of the first sample model and the set of sample audio signals.

[0029] In this embodiment, the sample music style information is vector-encoded, and the music style features are integrated into the model training process, so that the model can better understand and distinguish the rhythm features of different music styles. By using diverse training data and music style information, the generated rhythm recognition model can more accurately identify the rhythm of different music styles and improve recognition accuracy.

[0030] In one possible implementation of the first aspect, acquiring the set of sample audio signals includes:

[0031] Acquire a second sample audio signal with known time signature and style information;

[0032] The second sample audio signal is subjected to multiple feature perturbations to obtain a plurality of corresponding first sample audio signals.

[0033] The second sample audio signal and the first sample audio signal are determined as the sample audio signal set.

[0034] In this embodiment of the application, by performing multiple feature perturbations on a small number of audio samples, more training data can be generated, effectively alleviating the problem of insufficient data and improving the sample richness of model training.

[0035] In one possible implementation of the first aspect, determining the target rhythm recognition model based on a plurality of first sample model parameters and the set of sample audio signals includes:

[0036] Each parameter of the first sample model is loaded into the initial rhythm recognition model;

[0037] Each first sample audio signal in the set of sample audio signals is input into the initial rhythm recognition model loaded with the model parameters of each first sample, to obtain a sequence of sample excitation values ​​corresponding to each first sample audio signal;

[0038] The sample excitation value sequence corresponding to any first sample audio signal is compared with the preset excitation value sequence corresponding to any first sample audio signal to obtain the loss function value of any first sample audio signal;

[0039] The target rhythm recognition model is obtained by iteratively training the weights of the initial rhythm recognition model and the initial diffusion neural network until the loss function value of each first sample audio signal converges.

[0040] In this embodiment, by comparing the sample stimulus value sequence with a preset stimulus value sequence, the loss function value is calculated to quantify the difference between the model output and the true value, providing clear guidance for model optimization. Each sample model parameter is loaded into the rhythm recognition model to ensure that the model can fully utilize the feature information of each sample during training, improving the model's training accuracy. Each first sample audio signal is input into the rhythm recognition model to generate a corresponding sample stimulus value sequence, ensuring that the model can accurately capture the rhythmic features of the audio signal. Using the loss function value and sample model parameters, the model parameters are adjusted through optimization algorithms (such as gradient descent) to gradually reduce the loss function value, improving the model's recognition accuracy and robustness.

[0041] Secondly, embodiments of this application provide a rhythm detection device, comprising:

[0042] The information acquisition module is used to acquire the audio signal and the music style information corresponding to the audio signal;

[0043] The parameter determination module is used to determine the model parameters based on the music style information.

[0044] A model loading module is used to load the model parameters into a target rhythm recognition model, wherein the target rhythm recognition model is used to recognize the rhythm of the audio signal input to the target rhythm recognition model, and the target rhythm recognition model is trained from multiple first sample audio signals with known genre information;

[0045] The rhythm detection module is used to obtain the rhythm detection result of the audio signal based on the audio signal and the target rhythm recognition model loaded with the model parameters.

[0046] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the music rhythm detection method as described in any of the first aspects above.

[0047] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the music rhythm detection method as described in any of the first aspects above.

[0048] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the music rhythm detection method of any one of the first aspects described above.

[0049] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the training rhythm recognition model provided in an embodiment of this application;

[0052] Figure 2 This is a schematic diagram of the process of enhancing audio data provided in an embodiment of this application;

[0053] Figure 3 A flowchart illustrating the rhythm recognition model provided in this application embodiment;

[0054] Figure 4 This is a schematic flowchart of the music rhythm detection method provided in the embodiments of this application;

[0055] Figure 5 This is a flowchart illustrating the music rhythm detection method provided in the embodiments of this application. Figure 2 ;

[0056] Figure 6 This is a schematic diagram of the process for obtaining rhythm detection results provided in an embodiment of this application;

[0057] Figure 7 This is a schematic diagram of the structure of the music rhythm detection method provided in the embodiments of this application;

[0058] Figure 8 This is a structural block diagram of the sound detection device provided in the embodiments of this application;

[0059] Figure 9 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0060] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0061] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0062] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0063] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0064] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0065] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0066] The automotive cockpit entertainment system aims to provide users with a more immersive and personalized entertainment experience. Among them, music signal rhythm detection plays a key role. By analyzing the rhythm information of the music signal in real time, lighting effects, seat vibrations, ambient sound effects, etc., can be synchronized with the music signal, thereby enhancing the user's auditory and visual experience.

[0067] Because the automotive user base is very broad, and the types and characteristics of music signals listened to by users are also very diverse, music signal rhythm detection needs to be adaptable to music signals with different time signatures and genres. Related technologies for rhythm detection algorithms mainly fall into two categories: one is based on signal processing methods, which model the rhythm sequence in music according to prior knowledge from expert systems, etc. This type of method has good stability, but due to the diversity of music rhythms themselves, the rules often do not apply, resulting in relatively low accuracy. The other type is based on machine learning methods, which model the rhythm sequence using a certain amount of data and scientific parameters. This type of method can fit the training data using models such as support vector machines and neural networks, exhibiting relatively high robustness. However, achieving high-precision rhythm detection requires a large amount of data and higher model complexity, especially given the limited computing resources of in-vehicle systems. The problem of obtaining high-precision rhythm information with low computational cost urgently needs to be solved.

[0068] To address the problems in the aforementioned related technologies, this application provides a music rhythm detection method. By encoding the genre label information corresponding to the music into the backbone neural network parameters for beat recognition using a diffusion neural network, high-precision rhythm recognition for various genres is achieved. Furthermore, Monte Carlo methods are used to enhance the training data, improving the model's robustness to various music types and covering time signatures with few samples and some classical music and waltzes with less pronounced rhythms. The rhythm recognition model implemented in this application can run in real-time on the vehicle's infotainment system with relatively low computational power, providing real-time music rhythm information, including strong and weak beats, tempo, and time signature, to the in-vehicle entertainment system.

[0069] The music rhythm detection method proposed in this application is generally divided into two stages: rhythm recognition model training and rhythm detection using the trained rhythm recognition model. Before performing rhythm detection on audio signals with few sample time signatures and unclear musical styles, this application first describes the training process of the rhythm recognition model, see [link to relevant documentation]. Figure 1 This is a flowchart illustrating the training rhythm recognition model provided in an embodiment of this application. It is intended as an example and not a limitation. The method may include the following steps:

[0070] S101, Obtain a set of sample audio signals, the set of sample audio signals including multiple first sample audio signals with known genre information.

[0071] As an example, the genre information of the multiple first sample audio signals may differ. Alternatively, the genre information of the multiple first sample audio signals may be the same. It is understood that when the genre information of the multiple first sample audio signals is the same, the target rhythm recognition model obtained through S101-S104 can be used to identify the rhythm of the audio signal with specified genre information. When the genre information of the multiple first sample audio signals differs, the target rhythm recognition model obtained through S101-S104 can be used to identify the rhythm of the audio signal with any genre information.

[0072] In this embodiment, the sample audio signal set may include sample audio signals with different musical styles, particularly sample audio signals with few sample time signatures and some classical music, waltz, and other songs with indistinct rhythms. For music signals with few sample time signatures and musical styles, data augmentation strategies can be used to supplement the data. Data augmentation is an effective method to improve the generalization ability of the model and expand the dataset. For music signals, especially those with few sample time signatures and musical styles with indistinct rhythms, the Monte Carlo method can be used to moderately perturb the audio features to expand the data and obtain the sample audio signal set. The specific methods for supplementing music data are detailed in steps S201-S203.

[0073] In one embodiment, see Figure 2This is a schematic diagram of the process of enhancing audio data provided in the embodiments of this application, such as... Figure 2 As shown, step S101 includes:

[0074] S201, acquire the second sample audio signal with few sample time signatures and style information.

[0075] In this embodiment of the application, music signals are randomly selected from the training set in a uniform distribution as the music signals for data augmentation, i.e., the second sample audio signals. This can be achieved using a random number generator, and then the randomly selected signals are subjected to data augmentation processing.

[0076] S202, the second sample audio signal is subjected to multiple feature perturbations to obtain a plurality of corresponding first sample audio signals.

[0077] In this embodiment of the application, the Monte Carlo method can be used to enhance the selected second sample audio signal. The Monte Carlo method is a method for solving mathematical and physical problems through random sampling. In data enhancement, this method can be used to randomly perturb the audio features.

[0078] Specifically, new samples can be generated by changing the pitch of an audio signal. Pitch changes can be achieved using time-domain or frequency-domain methods; new samples can also be generated by changing the loudness (i.e., volume) of the audio signal. Loudness can be adjusted using linear or logarithmic scales; new samples can also be generated by adding or adjusting reverberation effects. Reverberation effects can be achieved using digital signal processing techniques; new samples can also be generated by changing the spectral components of the audio signal, such as using frequency-domain filters. These perturbations can expand the musical data for few-sample time signatures and musical styles.

[0079] S203, the second sample audio signal and the first sample audio signal are determined as the sample audio signal set.

[0080] In this embodiment of the application, the music data after feature perturbation, namely the first sample audio signal and the second sample audio signal, are used as a set of sample audio signals for data training.

[0081] In the above method, by performing multiple feature perturbations on a small number of audio samples, more training data can be generated, effectively alleviating the problem of insufficient data and improving the sample richness of model training.

[0082] S102, vector encoding is performed on the genre information corresponding to each of the multiple first sample audio signals to obtain multiple first sample encoding vectors.

[0083] In this embodiment of the application, during model training, it is necessary to encode the genre labels corresponding to the genre information of the audio signal in order to correctly process these labels. A common method is to use one-hot encoding. One-hot encoding converts each category into a binary vector, where only one bit is 1 and the rest are 0. This ensures that each category is unique in the vector, and the model can more easily learn the differences between categories.

[0084] For example, suppose the genre labels corresponding to multiple sample audio signals are Pop, Rock, Hip-Hop, etc. The encoding vectors obtained through one-hot encoding can be: Pop-[1,0,0,0,0,0], Rock-[0,1,0,0,0,0], Hip-Hop-[0,0,1,0,0,0], etc. A rhythm recognition model can then be trained based on the encoding vectors corresponding to the genre labels.

[0085] S103, generate multiple first sample model parameters based on multiple first sample encoding vectors.

[0086] In this embodiment, the target rhythm recognition model is based on a neural network architecture. Therefore, during the training process of music from multiple genres, it is necessary to obtain the network parameters of the rhythm recognition model corresponding to each genre, also known as the Temporal Convolutional Network (TCN) parameters (first sample model parameters). These parameters are used to adapt different genres of music to perform information capture in the rhythm recognition model and improve rhythm recognition.

[0087] Specifically, TCN parameters can be obtained by constructing a diffusion neural network, which is a generative model that converts data into random noise by gradually adding noise, and then reverses this process by training a neural network to generate new data samples. The vector codes of different music genres can be used as constraints to generate TCN parameters using a diffusion neural network.

[0088] S104, determine the target rhythm recognition model based on multiple first sample model parameters and the set of sample audio signals.

[0089] In this embodiment of the application, after obtaining the TCN parameters of different genres of music, the parameters are loaded into the initial rhythm recognition model (also known as the rhythm recognition model to be trained), and the initial rhythm recognition model is trained using multiple first sample audio signals to obtain the target rhythm recognition model, as detailed in steps S301-S303.

[0090] In the above method, the sample music style information is vector-encoded, and the music style features are integrated into the model training process, so that the model can better understand and distinguish the rhythm features of different music styles. By using diverse training data and music style information, the generated rhythm recognition model can more accurately identify the rhythm of different music styles and improve the recognition accuracy.

[0091] In one embodiment, see Figure 3 This is a flowchart illustrating the target rhythm recognition model provided in an embodiment of this application, as shown below. Figure 3 As shown, step S104 includes:

[0092] S301, load the parameters of each first sample model into the initial rhythm recognition model respectively.

[0093] In this embodiment of the application, after obtaining the TCN parameters corresponding to different genres of music through the diffusion model, the TCN parameters are loaded into the initial rhythm recognition model with the neural network as the backbone. At this time, the initial rhythm recognition model after loading the TCN parameters is an initial rhythm recognition model to be trained.

[0094] S302, each of the first sample audio signals in the sample audio signal set is input into the initial rhythm recognition model loaded with the model parameters of each first sample, to obtain the sample excitation value sequence corresponding to each first sample audio signal.

[0095] In this embodiment, each audio signal (first sample audio signal) of the genre music corresponding to the TCN parameters is converted into a Mel-spectrum signal, i.e., a frequency domain signal, and input into the initial rhythm recognition model to be trained. The initial rhythm recognition model outputs an excitation value sequence, which is a time series representing the feature values ​​of the audio signal at different time points. Each value in this excitation value sequence represents the probability that the audio signal has a certain beat type at the corresponding time point. For example, if the excitation value sequence is [0.1, 0.8, 0.2, 0.9], it means that the probability of the audio signal having a certain beat type is 0.1 at the first time point, 0.8 at the second time point, and so on.

[0096] S303, compare the sequence of sample excitation values ​​corresponding to any first sample audio signal with the sequence of preset excitation values ​​corresponding to any first sample audio signal to obtain the loss function value of any first sample audio signal.

[0097] In this embodiment, the preset stimulus value sequence is a known, standard sequence of stimulus values ​​corresponding to audio signals, typically annotated by experts or obtained through other reliable methods, used to evaluate the accuracy of the model output. The model's performance can be evaluated by comparing the difference or loss function value between the sample stimulus value sequence corresponding to any first sample audio signal output by the model based on the initial rhythm recognition and the preset sequence values. A smaller loss function value indicates that the model's output is closer to the preset correct value, and the better the model's performance. Common loss functions include Mean Squared Error (MSE) and Cross-Entropy Loss.

[0098] S304, the target rhythm recognition model is obtained by iteratively training the weights of the initial rhythm recognition model and the initial diffusion neural network until the loss function value of each first sample audio signal converges.

[0099] In this embodiment, by jointly optimizing the rhythm recognition model and the diffusion model through a loss function, the fit between the TCN parameter values ​​and the genre music can be improved. By further optimizing the rhythm recognition model using the TCN parameters with high fit, the detection accuracy of the rhythm recognition model can be improved.

[0100] Specifically, it refers to repeatedly executing a certain process. In this context, iteration refers to repeatedly adjusting and optimizing the parameters of the model. Based on the value of the loss function, the gradient of the parameters is calculated through the backpropagation algorithm. Based on the gradient value, the parameters of the weight model corresponding to the initial rhythm model and the initial diffusion model are updated using optimization algorithms (such as gradient descent, Adam, etc.). Through multiple iterations of training, until the loss function value converges, TCN parameters that are more suitable for multiple music genres can be obtained. Then the training of the initial rhythm recognition model ends, and the trained target rhythm recognition model is obtained.

[0101] In the above method, the model parameters of each sample are loaded into the initial rhythm recognition model to ensure that the model can make full use of the feature information of each sample during training, thereby improving the training accuracy of the model. Each first sample audio signal is input into the rhythm recognition model to generate a corresponding sample excitation value sequence, ensuring that the model can accurately capture the rhythm features of the audio signal. Using the loss function value and sample model parameters, the model parameters are adjusted through optimization algorithms (such as gradient descent) to gradually reduce the loss function value and improve the recognition accuracy and robustness of the model.

[0102] The following describes the second stage of rhythm recognition, in which the terminal device uses the target rhythm recognition model obtained from the first stage of training to perform rhythm recognition after the first stage of training is completed.

[0103] It is understood that the terminal device used for training the target rhythm recognition model in the first stage and the terminal device used for music rhythm recognition in the second stage can be the same terminal device or different terminal devices. This application embodiment does not limit this.

[0104] For example, in the first stage, a server can be used as the terminal device for model training. In the second stage, a vehicle can be used as the terminal device for music rhythm recognition.

[0105] See Figure 4 This is a flowchart illustrating the music rhythm detection method provided in the embodiments of this application. Figure 1 ,like Figure 4 As shown, the method includes:

[0106] S401, the terminal device acquires audio signals.

[0107] For example, digital audio interfaces on terminal devices, such as USB, HDMI, and SPDIF (S / PDIF), can directly transmit digital audio signals to the terminal device. This method is commonly used to connect digital audio workstations (DAWs), digital mixing consoles, or other digital audio equipment.

[0108] S402, the terminal device determines the model parameters based on the music style information of the audio signal.

[0109] For example, the musical style information of the audio signal, also known as music genre, musical style, music type, or music style, is not limited in this embodiment. For example, the musical style information of the audio signal can be classical music, pop music, etc.

[0110] As an example, a terminal device can obtain genre information based on the tags of the audio signal, such as tags like Pop or Jazz.

[0111] As an example, S402 can be implemented in the following way: the terminal device obtains the TCN parameters of the audio signal based on the music style information of the audio signal and the target diffusion neural network, and then uses the TCN parameters as model parameters.

[0112] The TCN parameters are obtained from the target diffusion neural network during the training phase, which can better adapt to the stylistic features of the audio signal to be identified.

[0113] S403, the terminal device loads the model parameters into the target rhythm recognition model.

[0114] In this embodiment of the application, the obtained TCN parameters are loaded into the target rhythm recognition model for rhythm detection of the audio signal acquired by the terminal device.

[0115] As an example, the terminal device has a target rhythm recognition model. This target rhythm recognition model can be trained by the terminal device through the above method, or the target rhythm recognition model can be obtained by the terminal device from other devices (such as the cloud). This application embodiment does not limit this.

[0116] As an example, the terminal device can periodically retrieve an updated target rhythm recognition model from the cloud to update the target rhythm recognition model already present in the terminal device.

[0117] As an example, this target rhythm recognition model can be trained using first sample audio signals of various musical styles. Therefore, it can be considered that the target rhythm recognition model can be used to identify the rhythm of audio signals from different musical styles. For instance, the target rhythm recognition model can be trained using first sample audio signals of classical music and first sample audio signals of pop music.

[0118] As another example, the target rhythm recognition model can be trained using multiple first-sample audio signals of a known musical style. This model can then be used to identify the rhythm of audio signals corresponding to that style. For instance, the target rhythm recognition model can be obtained from multiple first-sample audio signals corresponding to classical music. In practical applications, this target rhythm recognition model can then be used to identify the rhythm of audio signals corresponding to classical music.

[0119] Understandably, when the target rhythm recognition model is used to identify the rhythm of an audio signal of a certain genre, the terminal device can obtain the target rhythm recognition models corresponding to different genres. After identifying the genre information of an audio signal, the target rhythm recognition model corresponding to that genre information can be used for rhythm recognition.

[0120] For example, the terminal device includes a target rhythm recognition model 1 and a target rhythm recognition model 2. The target rhythm recognition model 1 is used to recognize the rhythm of popular music, and the target rhythm recognition model 2 is used to recognize the rhythm of classical music. If the terminal device determines that the style information of the audio signal is popular music, the model parameters determined according to the style information of the audio signal can be loaded into the target rhythm recognition model 1 so that the target rhythm recognition model 1 can be used to recognize the rhythm of the audio signal in the future.

[0121] S404, the terminal device obtains the rhythm detection result of the audio signal based on the audio signal and the target rhythm recognition model loaded with the model parameters.

[0122] As an example, the rhythm detection results of an audio signal can include: beat type, beat number, and beat tempo. The rhythm of the audio signal can be inferred through the target rhythm recognition model.

[0123] In the above method, a set of suitable recognition parameters is generated for the audio signal based on the genre information of the audio signal. These parameters are applied to the target rhythm recognition model so that the target rhythm recognition model can use these parameters to capture the beat of the music features to be identified, thereby improving the rhythm detection of different genres of music. In other words, by generating suitable parameters for different genres of music, the target rhythm recognition model can improve its ability to capture the beat of different genres of music and effectively perform rhythm detection, thereby improving the rhythm detection accuracy of different genres of music.

[0124] In one embodiment, see Figure 5 This is a flowchart illustrating the music rhythm detection method provided in the embodiments of this application. Figure 2 ,like Figure 5 As shown, step S404 includes:

[0125] S501, the audio signal is converted into a frequency domain signal.

[0126] In this embodiment, the frequency domain signal can be a Mel spectrum signal. By converting the audio signal into a spectrum signal and remapping the frequency using the Mel scale, the perceptual characteristics of the human ear at different frequencies can be better simulated.

[0127] S502, the frequency domain signal is input into the target rhythm recognition model loaded with the model parameters to obtain the beat excitation value sequence corresponding to the audio signal. The beat excitation value sequence includes one or more beat excitation values ​​and is used to represent the probability value of different beat types existing at different times.

[0128] In this embodiment, the converted Mel spectrum signal is input into the target rhythm recognition model with loaded TCN parameters for inference, and the excitation value sequence of the light and heavy beats of the audio signal is output. By analyzing the excitation value sequence, the rhythm detection result of the audio signal to be recognized can be obtained.

[0129] S503, obtain the rhythm detection result of the audio signal based on the beat excitation value sequence.

[0130] In this embodiment of the application, the excitation value sequence is further decoded to obtain the rhythm detection result of the audio signal to be identified. The decoding steps are as follows: steps S601-S603.

[0131] In the above method, the activation value sequence represents the probability value of different beat types existing at different times in the form of probability, which can more accurately identify multiple beat types and improve the flexibility and adaptability of the model.

[0132] In one embodiment, the rhythm detection result of the audio signal includes the beat type, beat number, and beat tempo. This can be achieved by decoding the excitation value sequence of the light and heavy beats, obtaining the light and heavy beat sequence, tempo, and beat number information, based on the state space, state transition probability model, and observation probability model in a Hidden Markov Model (HMM) used for beat tracking. See also Figure 6 This is a schematic diagram of the process for obtaining rhythm detection results provided in an embodiment of this application, such as... Figure 6 As shown, the decoding steps for the excitation value sequence include:

[0133] S601, the beat type is determined by each beat excitation value in the excitation value sequence; the beat type includes strong beats and weak beats.

[0134] In this embodiment, the decoding process finds the most probable state sequence based on the observation sequence (i.e., the activation sequence) and the HMM model parameters (state transition probabilities and observation probabilities). The most probable state sequence can be found using either the Viterbi Algorithm or the Forward Algorithm.

[0135] S602, the beat number corresponding to the audio signal is determined by the ratio of the number of light beats and heavy beats in each beat type.

[0136] In this embodiment, the time signature is a symbol used in musical notation to represent the rhythmic structure of music, usually written at the beginning of the score. The time signature consists of two numbers, in the form of XY, where X represents the number of beats per measure and Y represents the note value of each beat. The relationship between light and heavy beats can be statistically determined based on the beat type sequence obtained by decoding above. For example, if one heavy beat and three light beats are repeated as a group, the beat type can be obtained as 4 / 4 time, with 4 beats per measure and each beat being a quarter note.

[0137] S603, the beat speed is determined by the interval time between two adjacent taps in the beat type.

[0138] In this embodiment, beat speed refers to the number of beats per minute (BPM) in music. It describes the tempo of the music and is usually used to measure the speed of the music. Beat speed is determined by measuring the time interval between two beats. For example, by using audio analysis tools or manually measuring the time interval between two adjacent beats, denoted as T seconds, the beat speed BPM can be calculated by the following formula: BPM = 60 / T, which gives the speed of the audio signal beat.

[0139] See Figure 7 This is a schematic diagram of the structure of the music rhythm detection method provided in the embodiments of this application, as shown below. Figure 7 As shown, the music rhythm detection method proposed in this application can be divided into two stages: model training and rhythm detection, specifically including:

[0140] Training phase:

[0141] ① Perform multiple feature perturbations on audio signals with few samples or specific musical styles to enhance the music data.

[0142] A set of multiple sample audio signals was obtained;

[0143] ② Encode the genre tag corresponding to each sample audio signal to obtain multiple encoding vectors, and input the multiple encoding vectors into the constructed initial diffusion neural model to obtain the TCN parameters corresponding to each encoding vector, and load the TCN parameters into the initial rhythm recognition model;

[0144] ③ Input each sample audio signal into the initial rhythm recognition model loaded with TCN parameters to obtain the sample beat value sequence corresponding to each sample audio signal, and obtain the loss function value through the preset beat value sequence of each sample audio signal. Iteratively train the weights of the initial rhythm recognition model and the initial diffusion neural network until the loss function value converges to obtain the target rhythm recognition model and the target diffusion neural network.

[0145] Testing phase:

[0146] ① Input the music style information corresponding to the audio signal to be identified into the target diffusion neural network to obtain the TCN parameters corresponding to the audio signal to be identified, and load the TCN parameters into the target rhythm recognition model.

[0147] ② The audio signal to be identified is sent to the target rhythm recognition model loaded with TCN parameters to obtain the beat excitation value corresponding to the audio signal to be identified;

[0148] ③ The beat excitation value is decoded using the HMM model to obtain the music rhythm detection result of the audio signal to be identified, including the beat (strong or weak beat), tempo and time signature.

[0149] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0150] Corresponding to the music rhythm detection method in the above embodiments, Figure 8 This is a structural block diagram of the rhythm detection device provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0151] Reference Figure 8 The rhythm detection device 8 includes:

[0152] Information acquisition module 81 is used to acquire audio signals and the music style information corresponding to the audio signals;

[0153] Parameter determination module 82 is used to determine model parameters based on the music style information;

[0154] The model loading module 83 is used to load the model parameters into the target rhythm recognition model, wherein the target rhythm recognition model is used to recognize the rhythm of the audio signal input to the target rhythm recognition model, and the target rhythm recognition model is trained from multiple first sample audio signals with known genre information;

[0155] The parameter determination module 82 is also used for:

[0156] Based on the music style information and the target diffusion neural network, the model parameters are determined, wherein the target diffusion neural network is the diffusion neural network corresponding to the model parameters loaded into the target rhythm recognition model.

[0157] The rhythm detection device 8 also includes a model training module 84, used for:

[0158] Obtain a set of sample audio signals, the set of sample audio signals including multiple first sample audio signals with known music style information, the music style information of the multiple first sample audio signals being different;

[0159] The genre information corresponding to each of the multiple first sample audio signals is vector-encoded to obtain multiple first sample encoded vectors;

[0160] Generate multiple first sample model parameters based on multiple first sample encoding vectors;

[0161] The target rhythm recognition model is determined based on multiple parameters of the first sample model and the set of sample audio signals.

[0162] Model training module 84 is also used for:

[0163] Acquire second-sample audio signals containing few-sample time signatures and musical style information;

[0164] The second sample audio signal is subjected to multiple feature perturbations to obtain multiple corresponding first sample audio signals, and the second sample audio signal and the first sample audio signal are determined as the sample audio signal set.

[0165] Model training module 84 is also used for:

[0166] Each parameter of the first sample model is loaded into the initial rhythm recognition model;

[0167] Each first sample audio signal in the set of sample audio signals is input into the initial rhythm recognition model loaded with the model parameters of each first sample, to obtain a sequence of sample excitation values ​​corresponding to each first sample audio signal;

[0168] The sample excitation value sequence corresponding to any first sample audio signal is compared with the preset excitation value sequence corresponding to any first sample audio signal to obtain the loss function value of any first sample audio signal;

[0169] The target rhythm recognition model is obtained by iteratively training the weights of the initial rhythm recognition model and the initial diffusion neural network until the loss function value of each first sample audio signal converges.

[0170] The rhythm detection device 8 also includes a rhythm detection module 85, used for:

[0171] This is used to obtain the rhythm detection result of the audio signal based on the audio signal and the target rhythm recognition model loaded with the model parameters.

[0172] The rhythm detection module 85 is also used for:

[0173] Convert the audio signal into a frequency domain signal;

[0174] The frequency domain signal is input into the target rhythm recognition model loaded with the model parameters to obtain the beat excitation value sequence corresponding to the audio signal. The beat excitation value sequence includes one or more beat excitation values ​​and is used to represent the probability value of different beat types existing at different times.

[0175] The rhythm detection result of the audio signal is obtained based on the beat excitation value sequence.

[0176] The rhythm detection module 85 is also used for:

[0177] The beat type is determined by each beat excitation value in the excitation value sequence; the beat type includes strong beats and weak beats;

[0178] The beat number corresponding to the audio signal is determined by the beat type corresponding to each beat excitation value;

[0179] The beat rate is determined by the interval between two adjacent accents in the beat type.

[0180] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0181] in addition, Figure 8 The rhythm detection device shown can be a software unit, a hardware unit, or a combination of software and hardware built into existing terminal devices. It can also be integrated into terminal devices as an independent component or exist as a standalone terminal device.

[0182] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0183] Figure 9 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. For example... Figure 9 As shown, the terminal device 9 of this embodiment includes: at least one processor 90 ( Figure 9 (Only one is shown) a processor, a memory 91, and a computer program 92 stored in the memory 91 and executable on at least one processor 90. When the processor 90 executes the computer program 92, it implements the steps in any of the above-described embodiments of the music rhythm detection method.

[0184] The terminal device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 9 This is merely an example of terminal device 9 and does not constitute a limitation on terminal device 9. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0185] The processor 90 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0186] In some embodiments, memory 91 may be an internal storage unit of terminal device 9, such as a hard disk or memory of terminal device 9. In other embodiments, memory 91 may be an external storage device of terminal device 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on terminal device 9. Furthermore, memory 91 may include both internal storage units and external storage devices of terminal device 9. Memory 91 is used to store operating system, application programs, bootloader, data, and other programs, such as program code of computer programs. Memory 91 can also be used to temporarily store data that has been output or will be output.

[0187] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0188] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0189] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0190] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0191] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0192] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0193] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting music rhythm, characterized in that, The method includes: Obtain the genre information of the audio signal; The model parameters are determined based on the music style information; The model parameters are loaded into the target rhythm recognition model, wherein the target rhythm recognition model is used to recognize the rhythm of the audio signal input to the target rhythm recognition model, and the target rhythm recognition model is trained from multiple first sample audio signals with known genre information; Based on the audio signal and the target rhythm recognition model loaded with the model parameters, the rhythm detection result of the audio signal is obtained; The step of determining model parameters based on the music style information includes: Based on the music style information and the target diffusion neural network, the model parameters are determined. The target diffusion neural network is a diffusion neural network obtained when the loss function value of each first sample audio signal converges. When the loss function value of each first sample audio signal converges, the first sample model parameters corresponding to each first sample audio signal output by the target diffusion neural network are optimal. Accordingly, the target rhythm recognition model is also loaded with the first sample model parameters corresponding to each first sample audio signal. The method further includes: Obtain a set of sample audio signals, the set of sample audio signals including multiple first sample audio signals with known genre information; The genre information corresponding to each of the multiple first sample audio signals is vector-encoded to obtain multiple first sample encoded vectors; Generate multiple first sample model parameters based on multiple first sample encoding vectors; The target rhythm recognition model is determined based on multiple first sample model parameters and the sample audio signal set; Determining the target rhythm recognition model based on multiple first sample model parameters and the sample audio signal set includes: Load the parameters of each of the first sample models into the rhythm recognition model to be trained; Each first sample audio signal in the set of sample audio signals is input into the rhythm recognition model to be trained, which is loaded with the model parameters of each first sample, to obtain a sequence of sample excitation values ​​corresponding to each first sample audio signal; The sample excitation value sequence corresponding to any first sample audio signal is compared with the preset excitation value sequence corresponding to any first sample audio signal to obtain the loss function value of any first sample audio signal; The weights corresponding to the rhythm recognition model to be trained and the initial diffusion neural network are trained through multiple iterations until the loss function value of the first sample audio signal converges. The rhythm recognition model to be trained when the loss function value of each of the first sample audio signals converges is determined as the target rhythm recognition model.

2. The music rhythm detection method as described in claim 1, characterized in that, The step of obtaining the rhythm detection result of the audio signal based on the audio signal and the target rhythm recognition model loaded with the model parameters includes: Convert the audio signal into a frequency domain signal; The frequency domain signal is input into the target rhythm recognition model loaded with the model parameters to obtain the beat excitation value sequence corresponding to the audio signal. The beat excitation value sequence includes one or more beat excitation values ​​and is used to represent the probability value of different beat types existing at different times. The rhythm detection result of the audio signal is obtained based on the beat excitation value sequence.

3. The music rhythm detection method as described in claim 2, characterized in that, The rhythm detection result of the audio signal includes the beat type, beat number, and beat tempo. Obtaining the rhythm detection result of the audio signal based on the beat excitation value sequence includes: The beat type is determined by each beat excitation value in the excitation value sequence; the beat type includes strong beats and weak beats; The time signature corresponding to the audio signal is determined by the ratio of the number of light beats to the number of heavy beats in each beat type; The beat speed is determined by the interval between two adjacent taps in the beat type.

4. The music rhythm detection method as described in claim 3, characterized in that, The acquisition of the sample audio signal set includes: Acquire a second sample audio signal with known time signature and style information; The second sample audio signal is subjected to multiple feature perturbations to obtain multiple corresponding first sample audio signals; The second sample audio signal and the first sample audio signal are determined as the sample audio signal set.

5. A rhythm detection device, characterized in that, include: The information acquisition module is used to acquire the audio signal and the music style information corresponding to the audio signal; The parameter determination module is used to determine the model parameters based on the music style information. A model loading module is used to load the model parameters into a target rhythm recognition model, wherein the target rhythm recognition model is used to recognize the rhythm of the audio signal input to the target rhythm recognition model, and the target rhythm recognition model is trained from multiple first sample audio signals with known genre information; The rhythm detection module is used to obtain the rhythm detection result of the audio signal based on the audio signal and the target rhythm recognition model loaded with the model parameters; The parameter determination module is also used for: Based on the music style information and the target diffusion neural network, the model parameters are determined. The target diffusion neural network is a diffusion neural network obtained when the loss function value of each first sample audio signal converges. When the loss function value of each first sample audio signal converges, the first sample model parameters corresponding to each first sample audio signal output by the target diffusion neural network are optimal. Accordingly, the target rhythm recognition model is also loaded with the first sample model parameters corresponding to each first sample audio signal. The rhythm detection device also includes a rhythm recognition model determination module, used for: Obtain a set of sample audio signals, the set of sample audio signals including multiple first sample audio signals with known genre information; The genre information corresponding to each of the multiple first sample audio signals is vector-encoded to obtain multiple first sample encoded vectors; Generate multiple first sample model parameters based on multiple first sample encoding vectors; The target rhythm recognition model is determined based on multiple first sample model parameters and the sample audio signal set; The rhythm recognition model determination module is also used for: Load the parameters of each of the first sample models into the rhythm recognition model to be trained; Each first sample audio signal in the set of sample audio signals is input into the rhythm recognition model to be trained, which is loaded with the model parameters of each first sample, to obtain a sequence of sample excitation values ​​corresponding to each first sample audio signal; The sample excitation value sequence corresponding to any first sample audio signal is compared with the preset excitation value sequence corresponding to any first sample audio signal to obtain the loss function value of any first sample audio signal; The weights corresponding to the rhythm recognition model to be trained and the initial diffusion neural network are trained through multiple iterations until the loss function value of the first sample audio signal converges. The rhythm recognition model to be trained when the loss function value of each of the first sample audio signals converges is determined as the target rhythm recognition model.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images