Music rhythm detection method and device, terminal equipment and storage medium

Abstract

The invention is suitable for the technical field of audio signal rhythm detection, and provides a music rhythm detection method and device, terminal equipment and a computer readable storage medium, and the method comprises the steps: obtaining music style information of an audio signal; determining model parameters according to the music style information; the model parameters are loaded into a target rhythm recognition model, the target rhythm recognition model is used for recognizing the rhythm of the audio signals input into the target rhythm recognition model, and the target rhythm recognition model is obtained by training a plurality of first sample audio signals with known music style information; and obtaining a rhythm detection result of the audio signal according to the audio signal and the target rhythm recognition model loaded with the model parameters. According to the method, rhythm detection can be effectively carried out on music signals with different beat numbers and music style types.

Description

Technical Field

[0001] The present application belongs to the technical field of rhythm detection, and in particular, relates to a music rhythm detection method, apparatus, terminal device and computer-readable storage medium. Background Art

[0002] The car cockpit entertainment interaction system is designed to provide users with a more immersive and personalized entertainment experience. Among them, music signal rhythm detection plays a key role in this system. By analyzing the rhythm information of the music signal in real time, lighting effects, seat vibrations, ambient sound effects, etc. synchronized with the music signal can be achieved, thereby enhancing the user's auditory and visual experience.

[0003] Since the automobile user group is very broad, the categories and characteristics of music signals listened to by users are also very diverse. Music signal rhythm detection needs to be adaptable to music signals with different time signatures and styles. The relevant technology cannot effectively detect the rhythm of music styles or music signals with weak rhythm. Summary of the invention

[0004] The embodiments of the present application provide a detection method, an apparatus, a terminal device, and a computer-readable storage medium, which can effectively detect the rhythm of music signals of different time signatures and genres.

[0005] In a first aspect, an embodiment of the present application provides a music rhythm detection method, comprising:

[0006] Obtaining music style information of an audio signal;

[0007] Determine model parameters according to music style information;

[0008] Loading the model parameters into a target rhythm recognition model, wherein the target rhythm recognition model is used to recognize the rhythm of an audio signal input to the target rhythm recognition model, and the target rhythm recognition model is trained by a plurality of first sample audio signals whose music style information is known;

[0009] A rhythm detection result of the audio signal is obtained according to the audio signal and the target rhythm recognition model loaded with the model parameters.

[0010] In an embodiment of the present application, specific model parameters are determined according to the music style information of the audio signal, and these parameters can better adapt to the music characteristics of the music style, and the obtained model parameters are loaded into the target rhythm recognition model for rhythm detection. This is equivalent to generating a set of adaptive recognition parameters for the audio signal according to the music style information of the audio signal, and the parameters are applied in the target rhythm recognition model, so that the target rhythm recognition model uses the parameters to capture the beats of the music characteristics to be recognized, thereby improving the rhythm detection of different genres of music. In other words, adaptive parameters are generated for different genres of music, and the target rhythm recognition model uses the parameters to improve the ability to capture the beats of different genres of music, and can effectively perform rhythm detection to improve the rhythm detection accuracy of different genres of music.

[0011] In a possible implementation manner of the first aspect, obtaining a rhythm detection result of the audio signal according to the audio signal and the target rhythm recognition model loaded with the model parameters includes:

[0012] Converting the audio signal into a frequency domain signal;

[0013] Inputting the frequency domain signal into the target rhythm recognition model loaded with the model parameters to obtain a beat excitation value sequence corresponding to the audio signal, wherein the beat excitation value sequence includes one or more beat excitation values, and the beat excitation value sequence is used to represent probability values ​​of different beat types existing at different times;

[0014] A rhythm detection result of the audio signal is obtained according to the beat excitation value sequence.

[0015] In the embodiment of the present application, the excitation value sequence represents the probability values ​​of different beat types existing at different times in the form of probability, which can more accurately identify multiple beat types and improve the flexibility and adaptability of the model.

[0016] In a possible implementation manner of the first aspect, the rhythm detection result of the audio signal includes a beat type, a beat signature, and a beat speed, and obtaining the rhythm detection result of the audio signal according to the beat excitation value sequence includes:

[0017] The beat type is determined by each of the beat excitation values ​​in the excitation value sequence; the beat type includes a heavy beat and a light beat;

[0018] The beat signature corresponding to the audio signal is determined by the ratio of the number of light beats and heavy beats in each beat type;

[0019] The beat speed is determined by the interval time between two adjacent taps in the beat type.

[0020] In the embodiment of the present application, the detection result of the audio signal is obtained by decoding the excitation value sequence, so that the accuracy of the detection result can be improved.

[0021] In one embodiment, determining the model parameters according to the music style information includes:

[0022] The model parameters are determined according to the music style information and the target diffusion neural network. The target diffusion neural network is a diffusion neural network obtained when the loss function value of each of the first sample audio signals converges. When the loss function value of each of the first sample audio signals converges, the first sample model parameters corresponding to each of the first sample audio signals output by the target diffusion neural network are optimal. Correspondingly, the target rhythm recognition model is also loaded with the first sample model parameters corresponding to each of the first sample audio signals.

[0023] In an embodiment of the present application, the model parameters obtained by using the target diffusion model can better match the style characteristics of the audio signal, which is helpful for the subsequent use of the model parameters to identify the style characteristics and improve the accuracy of rhythm detection.

[0024] In a possible implementation manner of the first aspect, the method further includes:

[0025] Acquire a sample audio signal set, where the sample audio signal set includes a plurality of first sample audio signals with known music style information, and the music style information of the plurality of first sample audio signals is different;

[0026] Performing vector encoding on the music style information corresponding to each of the plurality of first sample audio signals to obtain a plurality of first sample encoding vectors;

[0027] Generate a plurality of first sample model parameters according to a plurality of the first sample encoding vectors;

[0028] The target rhythm recognition model is determined according to a plurality of the first sample model parameters and the sample audio signal set.

[0029] In an embodiment of the present application, the sample genre information is vector encoded and the genre features are integrated into the model training process, so that the model can better understand and distinguish the rhythm features of different genres. By utilizing diverse training data and genre information, the generated rhythm recognition model can more accurately identify the rhythm of music of different genres and improve recognition accuracy.

[0030] In a possible implementation manner of the first aspect, obtaining a sample audio signal set includes:

[0031] Obtain a second sample audio signal with known time signature and music style information;

[0032] Performing multiple characteristic perturbations on the second sample audio signal to obtain a corresponding plurality of the first sample audio signals to obtain a corresponding plurality of the first sample audio signals;

[0033] The second sample audio signal and the first sample audio signal are determined as the sample audio signal set.

[0034] In an embodiment of the present application, by performing multiple feature perturbations on a small number of sample audio signals, more training data can be generated, effectively alleviating the problem of insufficient data and improving the sample richness of model training.

[0035] In a possible implementation manner of the first aspect, determining the target rhythm recognition model according to the plurality of first sample model parameters and the sample audio signal set includes:

[0036] Loading each of the first sample model parameters into the initial rhythm recognition model respectively;

[0037] Inputting each of the first sample audio signals in the sample audio signal set into the initial rhythm recognition model loaded with each of the first sample model parameters to obtain a sample excitation value sequence corresponding to each of the first sample audio signals;

[0038] Comparing a sample excitation value sequence corresponding to any of the first sample audio signals with a preset excitation value sequence corresponding to any of the first sample audio signals to obtain a loss function value of any of the first sample audio signals;

[0039] The target rhythm recognition model is obtained by training the corresponding weights of the initial rhythm recognition model and the initial diffusion neural network through multiple iterations until the loss function value of each of the first sample audio signals converges.

[0040] In an embodiment of the present application, by comparing the sample excitation value sequence with the preset excitation value sequence, calculating the loss function value, and quantifying the difference between the model output and the true value, a clear guide is provided for model optimization. Each sample model parameter is loaded into the rhythm recognition model separately to ensure that the model can fully utilize the characteristic information of each sample during the training process, improve the training accuracy of the model, input each first sample audio signal into the rhythm recognition model, generate a corresponding sample excitation value sequence, ensure that the model can accurately capture the rhythm characteristics of the audio signal, use the loss function value and the sample model parameters, adjust the model parameters through an optimization algorithm (such as gradient descent), gradually reduce the loss function value, and improve the recognition accuracy and robustness of the model.

[0041] In a second aspect, an embodiment of the present application provides a rhythm detection device, including:

[0042] An information acquisition module, used to acquire an audio signal and music style information corresponding to the audio signal;

[0043] A parameter determination module, used to determine model parameters according to the music style information;

[0044] A model loading module, used for loading the model parameters into a target rhythm recognition model, wherein the target rhythm recognition model is used to recognize the rhythm of an audio signal input into the target rhythm recognition model, and the target rhythm recognition model is trained by a plurality of first sample audio signals whose music style information is known;

[0045] The rhythm detection module is used to obtain a rhythm detection result of the audio signal according to the audio signal and the target rhythm recognition model loaded with the model parameters.

[0046] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a music rhythm detection method as described in any one of the first aspects above is implemented.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the music rhythm detection method as described in any one of the first aspects above.

[0048] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes any one of the music rhythm detection methods in the first aspect.

[0049] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 It is a flow chart of a training rhythm recognition model provided in an embodiment of the present application;

[0052] Figure 2 is a schematic diagram of a process for enhancing audio data provided by an embodiment of the present application;

[0053] Figure 3 A schematic diagram of a process for determining a rhythm recognition model provided by an embodiment of the present application;

[0054] Figure 4 is a flowchart of a music rhythm detection method provided in an embodiment of the present application;

[0055] Figure 5 The following is a schematic diagram of the process of the music rhythm detection method provided in the embodiment of the present application. Figure 2 ;

[0056] Figure 6 is a schematic diagram of a process for obtaining a rhythm detection result provided in an embodiment of the present application;

[0057] Figure 7 is a structural schematic diagram of a music rhythm detection method provided in an embodiment of the present application;

[0058] Figure 8 is a structural block diagram of a detection device provided in an embodiment of the present application;

[0059] Fig. 9 It is a structural diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0061] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0062] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0063] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0064] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0065] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0066] The car cockpit entertainment interaction system is designed to provide users with a more immersive and personalized entertainment experience. Among them, music signal rhythm detection plays a key role in this system. By analyzing the rhythm information of the music signal in real time, lighting effects, seat vibrations, ambient sound effects, etc. synchronized with the music signal can be achieved, thereby enhancing the user's auditory and visual experience.

[0067] Since the automobile user group is very wide, the types and characteristics of music signals listened by users are also very diverse. Music signal rhythm detection needs to be adaptable to music signals of different time signatures and styles. The beat detection algorithms in related technologies are mainly divided into two categories. One is a method based on signal processing, which models the regularity of the beat sequence in music based on prior knowledge such as expert systems. This type of method has good stability, but due to the diversity of music rhythm itself, the rules are often not applicable, and the accuracy is relatively low; the other is a method based on machine learning, which models the beat sequence through a certain amount of data and scientific parameters. This method can fit the training data through models such as support vector machines and neural networks, and has relatively high robustness. However, achieving a high-precision rhythm detection effect requires a large amount of data and higher model complexity. Especially when the computing resources of the vehicle computer are limited, the problem of obtaining high-precision rhythm information at a low computing cost needs to be solved urgently.

[0068] In order to solve the problems in the above-mentioned related technologies, the present application provides a music rhythm detection method, which realizes high-precision rhythm recognition for each style by encoding the music style label information corresponding to the music into the backbone neural network parameters for beat recognition through a diffusion neural network, and enhances the training data through the Monte Carlo method to improve the robustness of the model for various types of music recognition, which can cover a small number of sample time signatures and some classical music, waltz and other song styles with unclear rhythms. The rhythm recognition model implemented in the present application can run in real time on the vehicle side with less computing power, and provide real-time music rhythm information to the vehicle cockpit entertainment interaction system, including light and heavy beats, speed and time signatures.

[0069] The music rhythm detection method proposed in this application is generally divided into two stages: rhythm recognition model training and application of the trained rhythm recognition model for rhythm detection. Before performing rhythm detection on some audio signals with few samples and unclear time signatures and styles, this application first describes the training process of the rhythm recognition model. Figure 1 , is a flow chart of training a rhythm recognition model provided in an embodiment of the present application. As an example but not a limitation, the method may include the following steps:

[0070] S101, obtaining a sample audio signal set, where the sample audio signal set includes a plurality of first sample audio signals with known music style information.

[0071] As an example, the music style information of multiple first sample audio signals is different. Or the music style information of multiple first sample audio signals is the same. It can be understood that, when the music style information of multiple first sample audio signals is the same, the target rhythm recognition model obtained through S101-S104 can be used to identify the rhythm of the audio signal of the specified music style information. When the music style information of multiple first sample audio signals is different, the target rhythm recognition model obtained through S101-S104 can be used to identify the rhythm of the audio signal of any music style information.

[0072] In an embodiment of the present application, the sample audio signal set may include sample audio signals of different musical styles, especially sample audio signals of songs with few sample time signatures and some classical music, waltz and other styles with unclear rhythms. For music signals with few sample time signatures and styles, data enhancement strategies can be used to supplement their data. Data enhancement is an effective method to improve the generalization ability of the model and expand the data set. For music signals, especially those with few sample time signatures and unclear rhythms, the Monte Carlo method can be used to moderately perturb the audio features to achieve the purpose of expanding the data and obtain a sample audio signal set. The specific method for supplementing the music data is detailed in steps S201-S203.

[0073] In one embodiment, see Figure 2, is a schematic diagram of a process for enhancing audio data provided by an embodiment of the present application, such as Figure 2 As shown, step S101 includes:

[0074] S201, obtaining a second sample audio signal with a small amount of sample time signature and music style information.

[0075] In an embodiment of the present application, a music signal is randomly selected from a training set in a uniform distribution as the music signal for data enhancement, i.e., the second sample audio signal. This can be achieved using a random number generator, and then the randomly selected signal is subjected to data enhancement processing.

[0076] S202: Perform multiple feature perturbations on the second sample audio signal to obtain multiple corresponding first sample audio signals.

[0077] In an embodiment of the present application, the Monte Carlo method can be used to enhance the data of the selected second sample audio signal. The Monte Carlo method is a method for solving mathematical and physical problems through random sampling. In data enhancement, this method can be used to randomly perturb audio features.

[0078] Specifically, new samples can be generated by changing the pitch of an audio signal. Pitch changes can be achieved using time domain or frequency domain methods; new samples can also be generated by changing the loudness (i.e., volume) of an audio signal. Loudness can be adjusted using a linear or logarithmic scale; new samples can also be generated by adding or adjusting reverberation effects. Reverberation effects can be achieved using digital signal processing technology; new samples can also be generated by changing the spectral components of an audio signal, and frequency domain filters can be used to achieve changes in spectral components, etc. The purpose of expanding music data with few sample time signatures and styles can be achieved through the above-mentioned feature perturbations.

[0079] S203: Determine the second sample audio signal and the first sample audio signal as the sample audio signal set.

[0080] In an embodiment of the present application, the music data after feature disturbance, namely the first sample audio signal and the second sample audio signal, is used as a sample audio signal set for data training.

[0081] In the above method, by performing multiple feature perturbations on a few-sample audio signal, more training data can be generated, effectively alleviating the problem of insufficient data and improving the sample richness of model training.

[0082] S102, performing vector encoding on music style information corresponding to each of the plurality of first sample audio signals to obtain a plurality of first sample encoding vectors.

[0083] In an embodiment of the present application, when performing model training, the genre labels corresponding to the genre information of the audio signal need to be encoded so that these labels can be processed correctly. A common method is to use one-hot encoding. One-hot encoding converts each category into a binary vector in which only one position is 1 and the rest are 0. This ensures that each category is unique in the vector and the model can more easily learn the differences between categories.

[0084] For example, assuming that the music genre labels corresponding to the multiple sample audio signals are: Pop, Rock, Hip-Hop and other label information, the encoding vectors obtained by one-hot encoding may be: Pop-[1,0,0,0,0,0], Rock-[0,1,0,0,0,0], Hip-Hop-[0,0,1,0,0,0], etc. The rhythm recognition model may be trained for the encoding vectors corresponding to the music genre labels.

[0085] S103: Generate multiple first sample model parameters according to multiple first sample encoding vectors.

[0086] In the embodiment of the present application, the target rhythm recognition model is based on a neural network as the backbone architecture. Therefore, in the process of training music of multiple genres, it is necessary to obtain the network parameters of the rhythm recognition model corresponding to each genre of music, which are also called temporal convolutional network (TCN) parameters (first sample model parameters). This parameter is used to adapt music of different genres to perform information capture in the rhythm recognition model to improve rhythm recognition.

[0087] Specifically, TCN parameters can be obtained by constructing a diffusion neural network, where the diffusion neural network is a generative model that converts data into random noise by gradually adding noise, and then reverses the process by training a neural network to generate new data samples. The vector encoding of music of different genres can be used as constraints to generate TCN parameters using the diffusion neural network.

[0088] S104: Determine a target rhythm recognition model according to a plurality of first sample model parameters and the sample audio signal set.

[0089] In an embodiment of the present application, after obtaining the TCN parameters of music of different genres, the parameters are loaded into an initial rhythm recognition model (also referred to as a rhythm recognition model to be trained), and the initial rhythm recognition model is trained using multiple first sample audio signals to obtain a target rhythm recognition model, specifically referring to steps S301-S303.

[0090] In the above method, the sample style information is vector encoded and the style features are integrated into the model training process, so that the model can better understand and distinguish the rhythm features of different styles. By utilizing diversified training data and style information, the generated rhythm recognition model can more accurately identify the rhythm of music of different styles and improve the recognition accuracy.

[0091] In one embodiment, see Figure 3 , is a flow chart of determining a target rhythm recognition model provided by an embodiment of the present application, such as Figure 3 As shown, step S104 includes:

[0092] S301, loading each first sample model parameter into an initial rhythm recognition model respectively.

[0093] In an embodiment of the present application, after obtaining the TCN parameters corresponding to different genres of music through the diffusion model, the TCN parameters are loaded into the initial rhythm recognition model with the neural network as the main backbone. At this time, the initial rhythm recognition model after the TCN parameters are loaded is an initial rhythm recognition model to be trained.

[0094] S302: Input each of the first sample audio signals in the sample audio signal set into the initial rhythm recognition model loaded with each of the first sample model parameters to obtain a sample excitation value sequence corresponding to each of the first sample audio signals.

[0095] In an embodiment of the present application, each audio signal (first sample audio signal) of the genre music corresponding to the TCN parameter is converted into a Mel spectrum signal, i.e., a frequency domain signal, and input into the initial rhythm recognition model to be trained. The initial rhythm recognition model will output an excitation value sequence, which is a time series, representing the characteristic values ​​of the audio signal at different time points. Each value in this excitation value sequence represents the probability of the audio signal having a certain beat type at the corresponding time point. For example, if the excitation value sequence is [0.1, 0.8, 0.2, 0.9], it means that at the first time point, the probability of the audio signal having a certain beat type is 0.1, and at the second time point, the probability is 0.8, and so on.

[0096] S303: Compare a sample excitation value sequence corresponding to any first sample audio signal with a preset excitation value sequence corresponding to any first sample audio signal to obtain a loss function value of any first sample audio signal.

[0097] In an embodiment of the present application, the preset excitation value sequence is a known, standard excitation value sequence corresponding to an audio signal, which is usually obtained by expert annotation or other reliable methods, and is used to evaluate the accuracy of the model output. The performance of the model can be evaluated by comparing the difference or loss function value between the sample excitation value sequence corresponding to any first sample audio signal output by the initial rhythm recognition model and the preset sequence value. The smaller the loss function value, the closer the output of the model is to the preset correct value, and the better the performance of the model. Common loss functions include mean square error (MSE), cross entropy loss (Cross-Entropy Loss), etc.

[0098] S304, training the initial rhythm recognition model and the initial diffusion neural network's corresponding weights through multiple iterations until the loss function value of each of the first sample audio signals converges, thereby obtaining the target rhythm recognition model.

[0099] In an embodiment of the present application, the rhythm recognition model and the diffusion model are jointly optimized through a loss function, so that the adaptability of the TCN parameter value to the genre music can be improved. By optimizing the rhythm recognition model with TCN parameters with high adaptability, the detection accuracy of the rhythm recognition model can be improved.

[0100] Specifically, it refers to repeatedly executing a process. In this context, iteration refers to adjusting and optimizing the parameters of the model multiple times. According to the value of the loss function, the gradient of the parameters is calculated through the back propagation algorithm. According to the gradient value, the optimization algorithm (such as gradient descent, Adam, etc.) is used to update the parameters of the weight model corresponding to the initial rhythm model and the initial diffusion model. Through multiple iterative training until the loss function value converges, TCN parameters that are more suitable for multiple genres of music can be obtained. The training of the initial rhythm recognition model is completed, and the trained target rhythm recognition model is obtained.

[0101] In the above method, each sample model parameter is loaded into the initial rhythm recognition model respectively to ensure that the model can fully utilize the characteristic information of each sample during the training process and improve the training accuracy of the model. Each first sample audio signal is input into the rhythm recognition model to generate a corresponding sample excitation value sequence to ensure that the model can accurately capture the rhythm characteristics of the audio signal. The loss function value and the sample model parameters are used to adjust the model parameters through an optimization algorithm (such as gradient descent) to gradually reduce the loss function value and improve the recognition accuracy and robustness of the model.

[0102] The following will describe the second stage of rhythm recognition performed by the terminal device using the target rhythm recognition model obtained by the first stage of training after the first stage of training is completed.

[0103] It is understandable that the terminal device for training the target rhythm recognition model in the first phase and the terminal device for using the target rhythm recognition model for music rhythm recognition in the second phase can be the same terminal device or different terminal devices, and the embodiments of the present application do not limit this.

[0104] For example, in the first stage, the server can be used as a terminal device to perform model training, and in the second stage, the vehicle can be used as a terminal device to perform music rhythm recognition.

[0105] See also Figure 4 , is a schematic diagram of the process of the music rhythm detection method provided in the embodiment of the present application Figure 1 ,like Figure 4 As shown, the method includes:

[0106] S401: The terminal device obtains an audio signal.

[0107] For example, the digital audio interface of the terminal device, such as USB, HDMI, SPDIF (S / PDIF), etc., can transmit the digital audio signal directly to the terminal device. This method is often used to connect digital audio workstations (DAW), digital mixing consoles or other digital audio devices.

[0108] S402: The terminal device determines model parameters according to the music style information of the audio signal.

[0109] For example, the music style information of the audio signal may be called music style, music genre, music type, music school, etc., which is not limited in the embodiments of the present application. For example, the music style information of the audio signal may be classical music, pop music, etc.

[0110] As an example, the terminal device can obtain music genre information according to the label of the audio signal, such as the label is Pop (popular music), Jazz (jazz music), etc.

[0111] As an example, S402 may be implemented in the following manner: the terminal device obtains TCN parameters of the audio signal according to the music style information of the audio signal and the target diffusion neural network, and then uses the TCN parameters as model parameters.

[0112] The TCN parameters are obtained through the target diffusion neural network in the training phase, and can better adapt to the style characteristics of the audio signal to be identified.

[0113] S403: The terminal device loads the model parameters into the target rhythm recognition model.

[0114] In an embodiment of the present application, the obtained TCN parameters are loaded into a target rhythm recognition model for use in performing rhythm detection on an audio signal acquired by a terminal device.

[0115] As an example, the terminal device has a target rhythm recognition model, which can be obtained by training the terminal device through the above method, or the target rhythm recognition model can be obtained by the terminal device from other devices (such as the cloud). The embodiments of the present application are not limited to this.

[0116] As an example, the terminal device may periodically obtain an updated target rhythm recognition model from the cloud to update the target rhythm recognition model in the terminal device.

[0117] As an example, the target rhythm recognition model can be trained by first sample audio signals of known multiple music styles, so it can be considered that the target rhythm recognition model can be used to recognize the rhythm of audio signals of different music styles. For example, the target rhythm recognition model can be trained by first sample audio signals of classical music and first sample audio signals corresponding to pop music.

[0118] As another example, the target rhythm recognition model can be trained by a plurality of first sample audio signals of a certain style of music, and the target rhythm recognition model can be used to recognize the rhythm of an audio signal of a certain style of music. For example, the target rhythm recognition model can be obtained by a plurality of first sample audio signals corresponding to classical music. Then, in actual use, the target rhythm recognition model can be used to recognize the rhythm of an audio signal corresponding to classical music.

[0119] It can be understood that when the target rhythm recognition model is used to identify the rhythm of an audio signal of a certain style, the terminal device can obtain target rhythm recognition models corresponding to different styles, and after identifying the style information of an audio signal, it can use the target rhythm recognition model corresponding to the style information to perform rhythm recognition.

[0120] For example, the terminal device includes a target rhythm recognition model 1 and a target rhythm recognition model 2, wherein the target rhythm recognition model 1 is used to recognize the music rhythm of pop music, and the target rhythm recognition model 2 is used to recognize the music rhythm of classical music. If the terminal device determines that the genre information of the audio signal is pop music, the model parameters determined according to the genre information of the audio signal can be loaded into the target rhythm recognition model 1, so as to facilitate the subsequent use of the target rhythm recognition model 1 to recognize the rhythm of the audio signal.

[0121] S404: The terminal device obtains a rhythm detection result of the audio signal according to the audio signal and the target rhythm recognition model loaded with the model parameters.

[0122] As an example, the rhythm detection result of the audio signal may include: beat type, beat signature and beat speed, and the rhythm of the audio signal may be inferred through the target rhythm recognition model.

[0123] In the above method, a set of adaptive recognition parameters is generated for the audio signal according to the genre information of the audio signal, and the parameters are applied in the target rhythm recognition model, so that the target rhythm recognition model uses the parameters to capture the beat of the music features to be recognized, thereby improving the rhythm detection of music of different genres. In other words, adaptive parameters are generated for music of different genres, and the target rhythm recognition model uses the parameters to improve the ability to capture the beat of music of different genres, and can effectively perform rhythm detection to improve the rhythm detection accuracy of music of different genres.

[0124] In one embodiment, see Figure 5 , is a schematic diagram of the process of the music rhythm detection method provided in the embodiment of the present application Figure 2 ,like Figure 5 As shown, step S404 includes:

[0125] S501: Convert the audio signal into a frequency domain signal.

[0126] In the embodiment of the present application, the frequency domain signal may be a Mel spectrum signal, which is converted into a spectrum signal by converting the audio signal and remapping the frequency using the Mel scale, thereby better simulating the perception characteristics of the human ear to different frequencies.

[0127] S502, input the frequency domain signal into the target rhythm recognition model loaded with the model parameters, and obtain a beat excitation value sequence corresponding to the audio signal, wherein the beat excitation value sequence includes one or more beat excitation values, and the beat excitation value sequence is used to represent the probability values ​​of different beat types existing at different times.

[0128] In an embodiment of the present application, the converted Mel spectrum signal is input into the target rhythm recognition model of the loaded TCN parameters for inference, and an excitation value sequence of the beats and heavy beats of the audio signal is output. By analyzing the excitation value sequence, the rhythm detection result of the audio signal to be identified can be obtained.

[0129] S503: Obtain a rhythm detection result of the audio signal according to the beat excitation value sequence.

[0130] In the embodiment of the present application, the excitation value sequence is further decoded to obtain the rhythm detection result of the audio signal to be identified, and the decoding steps are as follows: steps S601-S603.

[0131] In the above method, the excitation value sequence represents the probability values ​​of different beat types existing at different times in the form of probability, which can more accurately identify multiple beat types and improve the flexibility and adaptability of the model.

[0132] In one embodiment, the rhythm detection result of the audio signal includes the beat type, beat signature and beat speed. Based on the state space, state transition probability model and observation probability model in the Hidden Markov Model (HMM) for beat tracking, the excitation value sequence of the light and heavy beats can be decoded to obtain the light and heavy beat sequence, speed and time signature information. Figure 6 , is a flow chart of obtaining rhythm detection results provided in an embodiment of the present application, such as Figure 6 As shown, the decoding step of the excitation value sequence includes:

[0133] S601, the beat type is determined by each of the beat excitation values ​​in the excitation value sequence; the beat type includes a heavy beat and a light beat.

[0134] In the embodiment of the present application, the decoding process finds the most likely state sequence based on the observation value sequence (i.e., the stimulus value sequence) and the HMM model parameters (state transition probability and observation probability). The most likely state sequence can be found by the Viterbi algorithm or the forward algorithm.

[0135] S602: The beat signature corresponding to the audio signal is determined by the ratio of the number of light beats to the number of heavy beats in each beat type.

[0136] In the embodiment of the present application, the time signature is a symbol used in music notation to indicate the rhythm structure of music, and is usually written at the beginning of the score. The time signature consists of two numbers in the form of XY, where X represents the number of beats per measure and Y represents the note value of each beat. The relationship between the light beat and the heavy beat can be counted according to the beat type sequence obtained by the above decoding. For example, a heavy beat and three light beats are repeated as a group, and the beat type can be obtained as 4 / 4 beat, with 4 beats per measure and each beat is a quarter note.

[0137] S603: The beat speed is determined by the interval time between two adjacent taps in the beat type.

[0138] In the embodiment of the present application, the beat speed refers to the beats per minute (BPM) in the music, which describes the speed of the music and is usually used to measure the speed of the music. The beat speed is determined by measuring the time interval between two taps, such as using an audio analysis tool or manually measuring the time interval between two adjacent taps, recorded as T seconds. The beat speed BPM can be calculated by the following formula: BPM=60 / T, and the speed of the audio signal beat can be obtained.

[0139] See also Figure 7 , is a schematic diagram of the structure of the music rhythm detection method provided in the embodiment of the present application, such as Figure 7 As shown, the music rhythm detection method proposed in this application can be divided into two stages: model training and rhythm detection, specifically including:

[0140] Training phase:

[0141] ① Perform multiple feature perturbations on audio signals with few samples or styles to enhance music data.

[0142] Obtaining a plurality of sample audio signal sets;

[0143] ② Encode the music style label corresponding to each sample audio signal to obtain multiple encoding vectors, and input the multiple encoding vectors into the constructed initial diffusion neural model to obtain the TCN parameters corresponding to each encoding vector, and load the TCN parameters into the initial rhythm recognition model;

[0144] ③ Input each sample audio signal into the initial rhythm recognition model loaded with TCN parameters to obtain the sample beat value sequence corresponding to each sample audio signal, and obtain the loss function value through the preset beat value sequence of each sample audio signal, and iteratively train the weights of the initial rhythm recognition model and the initial diffusion neural network until the loss function value converges to obtain the target rhythm recognition model and target diffusion neural network.

[0145] Detection phase:

[0146] ① Input the music style information corresponding to the audio signal to be recognized into the target diffusion neural network, obtain the TCN parameters corresponding to the audio signal to be recognized, and load the TCN parameters into the target rhythm recognition model.

[0147] ② Send the audio signal to be recognized to the target rhythm recognition model loaded with TCN parameters to obtain the beat excitation value corresponding to the audio signal to be recognized;

[0148] ③ Use the HMM model to decode the beat excitation value to obtain the music rhythm detection result of the audio signal to be identified, including the beat (heavy beat or light beat), speed and time signature.

[0149] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0150] Corresponding to the music rhythm detection method of the above embodiment, Figure 8 Detailed description is given of the structure of the rhythm detection device according to the embodiment of the present application. For the sake of convenience, only the part related to the embodiment of the present application is shown.

[0151] Reference Figure 8 , the rhythm detection device 8 comprises:

[0152] An information acquisition module 81 is used to acquire an audio signal and music style information corresponding to the audio signal;

[0153] A parameter determination module 82, used to determine model parameters according to the music style information;

[0154] A model loading module 83 is used to load the model parameters into a target rhythm recognition model, wherein the target rhythm recognition model is used to recognize the rhythm of an audio signal input to the target rhythm recognition model, and the target rhythm recognition model is trained by a plurality of first sample audio signals whose music style information is known;

[0155] The parameter determination module 82 is further used for:

[0156] The model parameters are determined according to the music style information and the target diffusion neural network, where the target diffusion neural network is a diffusion neural network corresponding to the model parameters loaded into the target rhythm recognition model.

[0157] The rhythm detection device 8 also includes a model training module 84, which is used to:

[0158] Acquire a sample audio signal set, where the sample audio signal set includes a plurality of first sample audio signals with known music style information, and the music style information of the plurality of first sample audio signals is different;

[0159] Performing vector encoding on the music style information corresponding to each of the plurality of first sample audio signals to obtain a plurality of first sample encoding vectors;

[0160] Generate a plurality of first sample model parameters according to a plurality of the first sample encoding vectors;

[0161] The target rhythm recognition model is determined according to a plurality of the first sample model parameters and the sample audio signal set.

[0162] The model training module 84 is also used to:

[0163] Obtain a second sample audio signal with a small amount of sample time signature and music style information;

[0164] Performing multiple feature perturbations on the second sample audio signal to obtain a corresponding plurality of the first sample audio signals, and determining the second sample audio signal and the first sample audio signal as the sample audio signal set.

[0165] The model training module 84 is also used to:

[0166] Loading each of the first sample model parameters into the initial rhythm recognition model respectively;

[0167] Inputting each of the first sample audio signals in the sample audio signal set into the initial rhythm recognition model loaded with each of the first sample model parameters to obtain a sample excitation value sequence corresponding to each of the first sample audio signals;

[0168] Comparing a sample excitation value sequence corresponding to any of the first sample audio signals with a preset excitation value sequence corresponding to any of the first sample audio signals to obtain a loss function value of any of the first sample audio signals;

[0169] The target rhythm recognition model is obtained by training the corresponding weights of the initial rhythm recognition model and the initial diffusion neural network through multiple iterations until the loss function value of each of the first sample audio signals converges.

[0170] The rhythm detection device 8 also includes a rhythm detection module 85, which is used to:

[0171] Used to obtain a rhythm detection result of the audio signal according to the audio signal and the target rhythm recognition model loaded with the model parameters.

[0172] The rhythm detection module 85 is further used for:

[0173] Converting the audio signal into a frequency domain signal;

[0174] Inputting the frequency domain signal into the target rhythm recognition model loaded with the model parameters to obtain a beat excitation value sequence corresponding to the audio signal, wherein the beat excitation value sequence includes one or more beat excitation values, and the beat excitation value sequence is used to represent probability values ​​of different beat types existing at different times;

[0175] A rhythm detection result of the audio signal is obtained according to the beat excitation value sequence.

[0176] The rhythm detection module 85 is further used for:

[0177] The beat type is determined by each of the beat excitation values ​​in the excitation value sequence; the beat type includes a heavy beat and a light beat;

[0178] The beat signature corresponding to the audio signal is determined by the beat type corresponding to each of the beat excitation values;

[0179] The beat speed is determined by the interval time between two adjacent beats in the beat type.

[0180] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0181] in addition, Figure 8 The rhythm detection device shown can be a software unit, a hardware unit, or a combination of software and hardware, which is built into an existing terminal device, or can be integrated into the terminal device as an independent hanger, or can exist as an independent terminal device.

[0182] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0183] Fig. 9 Schematic diagram of the structure of the terminal device provided in the embodiment of the present application. Fig. 9 As shown, the terminal device 9 of this embodiment includes: at least one processor 90 ( Fig. 9 Only one is shown in the figure) a processor, a memory 91, and a computer program 92 stored in the memory 91 and executable on at least one processor 90, and when the processor 90 executes the computer program 92, the steps in any of the above-mentioned music rhythm detection method embodiments are implemented.

[0184] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Fig. 9 It is only an example of the terminal device 9 and does not constitute a limitation on the terminal device 9. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0185] The processor 90 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0186] In some embodiments, the memory 91 may be an internal storage unit of the terminal device 9, such as a hard disk or memory of the terminal device 9. In other embodiments, the memory 91 may also be an external storage device of the terminal device 9, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 9. Further, the memory 91 may also include both an internal storage unit of the terminal device 9 and an external storage device. The memory 91 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of a computer program. The memory 91 may also be used to temporarily store data that has been output or is to be output.

[0187] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0188] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0189] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0190] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0191] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0192] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0193] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A music rhythm detection method, characterized in that: The method comprises: Obtaining music style information of an audio signal; Determine model parameters according to the music style information; Loading the model parameters into a target rhythm recognition model, wherein the target rhythm recognition model is used to recognize the rhythm of an audio signal input to the target rhythm recognition model, and the target rhythm recognition model is trained by a plurality of first sample audio signals whose music style information is known; A rhythm detection result of the audio signal is obtained according to the audio signal and the target rhythm recognition model loaded with the model parameters.

2. The music rhythm detection method as claimed in claim 1, characterized in that: The step of obtaining a rhythm detection result of the audio signal according to the audio signal and the target rhythm recognition model loaded with the model parameters comprises: Converting the audio signal into a frequency domain signal; Inputting the frequency domain signal into the target rhythm recognition model loaded with the model parameters to obtain a beat excitation value sequence corresponding to the audio signal, wherein the beat excitation value sequence includes one or more beat excitation values, and the beat excitation value sequence is used to represent probability values ​​of different beat types existing at different times; A rhythm detection result of the audio signal is obtained according to the beat excitation value sequence.

3. The music rhythm detection method as claimed in claim 2, characterized in that: The rhythm detection result of the audio signal includes a beat type, a beat signature, and a beat speed, and obtaining the rhythm detection result of the audio signal according to the beat excitation value sequence includes: The beat type is determined by each of the beat excitation values ​​in the excitation value sequence; the beat type includes a heavy beat and a light beat; The beat signature corresponding to the audio signal is determined by the ratio of the number of light beats and heavy beats in each beat type; The beat speed is determined by the interval time between two adjacent taps in the beat type.

4. The music rhythm detection method according to claim 1, characterized in that: The determining of the model parameters according to the music style information includes: The model parameters are determined according to the music style information and the target diffusion neural network. The target diffusion neural network is a diffusion neural network obtained when the loss function value of each of the first sample audio signals converges. When the loss function value of each of the first sample audio signals converges, the first sample model parameters corresponding to each of the first sample audio signals output by the target diffusion neural network are optimal. Correspondingly, the target rhythm recognition model is also loaded with the first sample model parameters corresponding to each of the first sample audio signals.

5. The music rhythm detection method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire a sample audio signal set, where the sample audio signal set includes a plurality of first sample audio signals with known music style information; Performing vector encoding on the music style information corresponding to each of the plurality of first sample audio signals to obtain a plurality of first sample encoding vectors; Generate a plurality of first sample model parameters according to a plurality of the first sample encoding vectors; The target rhythm recognition model is determined according to a plurality of the first sample model parameters and the sample audio signal set.

6. The music rhythm detection method as claimed in claim 5, characterized in that: The step of obtaining a sample audio signal set comprises: Obtain a second sample audio signal with known time signature and music style information; Performing multiple feature perturbations on the second sample audio signal to obtain a corresponding plurality of the first sample audio signals; The second sample audio signal and the first sample audio signal are determined as the sample audio signal set.

7. The music rhythm detection method as claimed in claim 5, characterized in that: The step of determining the target rhythm recognition model according to the plurality of the first sample model parameters and the sample audio signal set comprises: Loading each of the first sample model parameters into the rhythm recognition model to be trained; Inputting each of the first sample audio signals in the sample audio signal set into the rhythm recognition model to be trained loaded with each of the first sample model parameters, to obtain a sample excitation value sequence corresponding to each of the first sample audio signals; Comparing a sample excitation value sequence corresponding to any of the first sample audio signals with a preset excitation value sequence corresponding to any of the first sample audio signals to obtain a loss function value of any of the first sample audio signals; Training the weights corresponding to the rhythm recognition model to be trained and the initial diffusion neural network respectively through multiple iterations until the loss function value of the first sample audio signal converges; The rhythm recognition model to be trained when the loss function value of each of the first sample audio signals converges is determined as the target rhythm recognition model.

8. A rhythm detection device, characterized in that: include: An information acquisition module, used to acquire an audio signal and music style information corresponding to the audio signal; A parameter determination module, used to determine model parameters according to the music style information; A model loading module, used for loading the model parameters into a target rhythm recognition model, wherein the target rhythm recognition model is used to recognize the rhythm of an audio signal input into the target rhythm recognition model, and the target rhythm recognition model is trained by a plurality of first sample audio signals whose music style information is known; The rhythm detection module is used to obtain a rhythm detection result of the audio signal according to the audio signal and the target rhythm recognition model loaded with the model parameters.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images