Audio beat analysis method, vehicle, computer equipment and program product

Abstract

The invention provides an audio beat analysis method, a vehicle, computer equipment and a program product, and the method comprises the steps: carrying out the audio signal analysis processing of audio data, and obtaining the amplitude spectrum feature data, pitch feature data and rhythm feature data of the audio data; splicing the amplitude spectrum feature data, the pitch feature data and the rhythm feature data to obtain spliced feature data; inputting the spliced feature data into a deep learning model, and performing feature extraction through feature extractors respectively corresponding to the amplitude spectrum feature data, the pitch feature data and the rhythm feature data in sequence to obtain rhythm features corresponding to the audio data; wherein different feature extractors correspond to different feature extraction scales; and generating beat information of the audio data according to the beat characteristics.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an audio beat analysis method, a vehicle, a computer device, and a program product. Background Art

[0002] Parsing the beats in audio data can help users better understand and learn music. However, traditional audio beat parsing methods not only have the problem of beat parsing misalignment, but also fail to capture the rhythm changes and personalized expressions of music styles because the same parsing parameters are used when parsing audio data of different styles or rhythms. This causes the parsed beats to be seriously inconsistent with the actual beats of the audio data, affecting the accuracy of beat parsing. Summary of the invention

[0003] The embodiments of the present disclosure at least provide an audio beat analysis method, a vehicle, a computer device, and a program product.

[0004] In a first aspect, an embodiment of the present disclosure provides an audio beat parsing method, comprising:

[0005] Performing audio signal analysis processing on the audio data to obtain amplitude spectrum feature data, pitch feature data and rhythm feature data of the audio data;

[0006] splicing the amplitude spectrum feature data, the pitch feature data and the rhythm feature data to obtain spliced ​​feature data;

[0007] The spliced ​​feature data is input into a deep learning model, and feature extraction is performed in sequence by feature extractors corresponding to the amplitude spectrum feature data, the pitch feature data, and the rhythm feature data, respectively, to obtain beat features corresponding to the audio data; wherein different feature extractors have different feature extraction scales;

[0008] The beat information of the audio data is generated according to the beat feature.

[0009] This implementation method can fully mine the feature data of audio data in various aspects by extracting the amplitude spectrum feature data on the audio amplitude dimension, the pitch feature data on the music style dimension, and the rhythm feature data on the rhythm change feature dimension. Then, by splicing the feature data on the three dimensions, the spliced ​​features can fully characterize the music features of the audio data. Since different feature extractors correspond to different feature extraction scales, the spliced ​​features are extracted using feature extractors corresponding to various dimensions, so that the feature data on each dimension can be fully mined and identified at the corresponding scale, ensuring a comprehensive understanding of the prelude and rhythm changes of the audio data, avoiding the influence of blank preludes, music styles and rhythm changes on beat recognition, and adapting to a variety of music styles and rhythm changes, thereby obtaining accurate beat features. Finally, using the beat features, accurate beat information that conforms to the style of the audio data can be generated.

[0010] In a possible implementation, each of the feature extractors includes a convolution layer and an attention extraction layer; and the convolution scales of the convolution layers in the feature extractors corresponding to the amplitude spectrum feature data, the pitch feature data, and the rhythm feature data are different.

[0011] In this implementation, the convolution layer in the feature extractor is used to perform convolution processing on the input features, which can fully mine the fine-grained features in the input features, and then the attention extraction layer is used to extract features, which can extract feature data that can more accurately reflect the beat features. And by extracting features using convolution layers of different convolution scales, it is possible to perform targeted convolution processing on features in each dimension, thereby obtaining more accurate convolution features.

[0012] In a possible implementation, the spliced ​​feature data is input into a deep learning model, and feature extraction is performed in sequence by feature extractors corresponding to the amplitude spectrum feature data, the pitch feature data, and the rhythm feature data, respectively, to obtain a beat feature corresponding to the audio data, including:

[0013] Using a first feature extractor corresponding to the amplitude spectrum feature data, extracting features from the spliced ​​feature data to obtain first intermediate feature data;

[0014] splicing the first intermediate feature and the rhythm feature data and inputting the resultant data into a second feature extractor corresponding to the rhythm feature data to obtain second intermediate feature data;

[0015] splicing the second intermediate feature and the pitch feature data and inputting the resultant data into a third feature extractor corresponding to the pitch feature data to obtain third intermediate feature data;

[0016] The decoder in the deep learning model is used to perform feature decoding processing on the third intermediate feature data to obtain the beat feature.

[0017] In this implementation, the first feature extractor is used to model the first intermediate feature data that can reflect the music features of the audio data at a macro level, and the second feature extractor is used to extract features from the concatenated data of the first intermediate feature data and the rhythm feature data, so that a more microscopic rhythm change feature (i.e., the second intermediate feature data) can be modeled. Finally, the third feature extractor is used to extract features from the concatenated data of the second intermediate feature data and the pitch feature data, and the entire audio data can be represented by features of a higher dimension, thereby achieving accurate characterization of the audio data. Finally, the decoder is used to perform decoding processing, so that accurate beat features can be obtained.

[0018] In a possible implementation, performing audio signal analysis processing on the audio data to obtain amplitude spectrum feature data, pitch feature data, and rhythm feature data of the audio data includes:

[0019] Preprocessing the audio data to obtain each frame of audio signal corresponding to the audio data; the preprocessing includes sampling rate standardization, audio data framing, audio signal normalization and audio signal denoising;

[0020] Performing short-time Fourier transform on each frame of the audio signal to obtain amplitude spectrum characteristics, frequency information, and energy of frequency feature points corresponding to each frame of the audio signal;

[0021] Determine the pitch feature data of the audio data according to the frequency information of the audio signal of each frame;

[0022] The rhythm feature data of the audio data is determined according to the energy of the frequency feature points corresponding to each frame of the audio signal.

[0023] This implementation method can improve the input quality of the model by preprocessing the audio data. By performing a short-time Fourier transform on each frame of audio signal, the audio data can be converted in the time domain and frequency domain to obtain accurate amplitude spectrum features, frequency information, and energy of frequency feature points. By using the frequency information and energy information, the accurate pitch feature data and rhythm feature data corresponding to the audio data can be parsed.

[0024] In a possible implementation manner, determining the rhythm feature data of the audio data according to the energy of the frequency feature points corresponding to each frame of the audio signal includes:

[0025] Determine, from each of the frequency feature points, each beat point corresponding to the audio data according to the energy of each of the frequency feature points and the energy of each of the target feature points located within a preset time interval of the frequency feature point;

[0026] Determine the number of beats per minute of the audio data according to the beat time of each beat point and the time interval between adjacent beat points;

[0027] The rhythm feature data is determined according to the time interval between the adjacent beat points and the number of beats per minute.

[0028] This implementation method uses the energy of the frequency feature point and the energy of the target feature point in the vicinity to accurately determine whether the frequency feature point is an obvious beat point. Using the determined beat point and beat interval, the number of beats per minute that can characterize the rhythm can be determined, and then using the time interval and the number of beats per minute, rhythm feature data that can accurately reflect the rhythm change can be obtained.

[0029] In a possible implementation, determining each beat point corresponding to the audio data from each frequency feature point according to the energy of each frequency feature point and the energy of each target feature point located within a preset time interval of the frequency feature point includes:

[0030] For any of the frequency feature points, if the energy of the frequency feature point exceeds a preset threshold and exceeds the energy of each target feature point within a preset time interval of the frequency feature point, the frequency feature point is determined to be a beat point corresponding to the audio data.

[0031] In this embodiment, the energy peak point is usually a beat point. When the energy of a frequency feature point exceeds a preset threshold and exceeds the energy of a target feature point in the nearby time, it can be said that the frequency feature point is a peak point within a period of time. Using this point as a beat point can achieve accurate identification of the beat point.

[0032] In a possible implementation, the method further includes the step of training the deep learning model:

[0033] Acquire an audio data set; the audio data set includes a plurality of sample audio data, and different audio data have different music styles, different music rhythms, and different music durations;

[0034] Performing audio signal analysis processing on the sample audio data to obtain sample amplitude spectrum feature data, sample pitch feature data and sample rhythm feature data of the sample audio data;

[0035] The sample amplitude spectrum feature data, the sample pitch feature data and the sample rhythm feature data are spliced ​​and input into the deep learning model to be trained to obtain the predicted beat feature of the sample audio data;

[0036] The deep learning model to be trained is iteratively trained by utilizing the feature loss between the predicted beat feature and the label beat feature of the sample audio data until a training cutoff condition is met, thereby obtaining a trained deep learning model.

[0037] This implementation method can improve the prediction accuracy of the model by training the deep learning model, and use the trained deep learning module to perform beat analysis to ensure the accuracy of the beat analysis.

[0038] In a second aspect, an embodiment of the present disclosure further provides a vehicle, including a controller, wherein the controller includes:

[0039] a memory for storing machine-readable instructions; and

[0040] A processor is used to call and execute the machine-readable instructions stored in the memory to implement the above-mentioned audio beat parsing method, or the steps in any possible implementation of the above-mentioned audio beat parsing method.

[0041] In a third aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0042] In a fourth aspect, an optional implementation of the present disclosure further provides a computer program product, which, when executed, executes the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.

[0043] For a description of the effects of the above-mentioned vehicle, computer equipment, and computer program product, please refer to the description of the above-mentioned audio beat analysis method, which will not be repeated here.

[0044] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0046] Figure 1 A flowchart of an audio beat analysis method provided by an embodiment of the present disclosure is shown;

[0047] Figure 2 A schematic diagram of a feature extraction process of a deep learning model provided by an embodiment of the present disclosure is shown;

[0048] Figure 3 A schematic diagram of an audio beat analysis device provided by an embodiment of the present disclosure is shown;

[0049] Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0051] In addition, the terms "first", "second", etc. in the description and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0052] The "multiple or several" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0053] As vehicles develop towards intelligence and technology, the applications configured on vehicles are becoming more and more abundant. Among these applications, there are many applications that rely on music beats, such as music wallpaper applications, car light dance applications, and atmosphere light applications. However, the music beats analyzed by the current beat analysis methods are inaccurate, resulting in poor implementation effects of these applications. Studies have found that conventional music beat analysis software usually has the problem of inaccurate beat analysis, and the reasons for this problem are often: (1) It does not fully consider the prelude at the beginning of the music or cannot accurately identify the music prelude, but directly uses the first second of the music as the beat starting point and then analyzes the subsequent beats. This method often causes beat analysis misalignment when there is a blank or low volume in the prelude, increasing the probability of analysis errors. (2) When analyzing the beat, the characteristics of music of different styles and rhythms are ignored, and the same beat analysis parameters are used for beat analysis of various music, resulting in the inability to capture the real music beat. For example, different musical works have different rhythmic and musical characteristics. For example, the rhythmic styles of jazz and classical music are completely different. If the same analysis parameters are used for beat analysis, it will be impossible to capture the rhythm changes and personalized expressions of a specific musical style. For another example, the rhythm of music is often not static. A piece of music may also have segments with accelerated rhythms and segments with slowed rhythms. If the rhythm changes are ignored for beat analysis and the beat output is performed using a uniform beat interval, the beat information will not match the actual rhythm. Therefore, it can be seen that existing beat analysis software has more or less the problem of inaccurate beat analysis.

[0054] Based on the above research, the present disclosure provides an audio beat analysis method, a vehicle, a computer device and a program product. By extracting the amplitude spectrum feature data on the audio amplitude dimension, the pitch feature data on the music style dimension and the rhythm feature data on the rhythm change feature dimension, it is possible to fully mine the feature data of the audio data in various aspects, and then by splicing the feature data on the three dimensions, the spliced ​​features can fully characterize the music features of the audio data. Since different feature extractors correspond to different feature extraction scales, the spliced ​​features are extracted using feature extractors corresponding to various dimensions, which can fully mine and identify the feature data on each dimension at the corresponding scale, ensure a comprehensive understanding of the prelude and rhythm changes of the audio data, avoid the influence of blank preludes, music styles and rhythm changes on beat recognition, adapt to a variety of music styles and rhythm changes, and thus obtain accurate beat features. Finally, using the beat features, accurate beat information that conforms to the style of the audio data can be generated.

[0055] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0056] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0057] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0058] It should be noted that the specific terms mentioned in the embodiments of the present disclosure include:

[0059] Short-time Fourier transform (STFT) is a mathematical transformation related to Fourier transform, which is used to determine the frequency and phase of the local sinusoidal wave of a time-varying signal.

[0060] WAV format: One of the most common sound file formats, this file can record various mono or stereo sound information and ensure that the sound is not distorted.

[0061] To facilitate understanding of the present embodiment, an audio beat parsing method disclosed in the embodiment of the present disclosure is first introduced in detail. The executor of the audio beat parsing method provided in the embodiment of the present disclosure is generally a terminal device or other processing device with certain computing capabilities, wherein the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a handheld device, a computer device, a server, a vehicle equipment, etc.; in some possible implementation methods, the audio beat parsing method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0062] The audio beat analysis method provided by the embodiment of the present disclosure is described below by taking a computer device as an example of an execution subject.

[0063] like Figure 1 As shown, it is a flowchart of an audio beat parsing method provided by an embodiment of the present disclosure, which may include the following steps:

[0064] S101: Perform audio signal analysis processing on the audio data to obtain amplitude spectrum feature data, pitch feature data and rhythm feature data of the audio data.

[0065] Here, the audio data can be music with any time length, any style, and any rhythm changes. For example, the audio data can be a complete piece of music or a certain music clip. It can also be any of various styles such as jazz, classical music, pop music, rock music, electronic music, hip-hop music, metal music, etc.

[0066] The audio signal analysis process may specifically be to perform signal analysis on the audio data in the time domain and the frequency domain to obtain time domain features and frequency domain features.

[0067] The amplitude spectrum feature data describes the amplitude distribution of the audio signal in the audio data at different frequencies. The pitch feature data is used to describe the pitch change of the audio signal. The pitch feature data can reflect the musical style of the audio data. For example, different musical styles (such as classical, jazz, rock, etc.) have different pitch feature patterns. The rhythm feature data is used to describe the dynamic changes of the rhythm and beat frequency of the audio data.

[0068] As a coarse-grained feature expression, the amplitude spectrum feature data may include pitch feature data and rhythm feature data, but compared with the pitch feature data and rhythm feature data, it also contains coarse-grained information in other dimensions. As a subset of the amplitude spectrum feature, the pitch feature data is a high-level expression of the amplitude spectrum feature, which can be understood as a deep expression in the dimension of musical style. The rhythm feature data is also part of the amplitude spectrum feature, which is a feature expression of a time dimension corresponding to the amplitude spectrum feature, which can be understood as a deep expression in the dimension of rhythm change.

[0069] In specific implementation, for any audio data to be analyzed for beats, the audio data may be first converted into a unified audio format, such as a wav format. Then, digital signal processing technology (such as a processing technology such as Fourier transform) may be used to analyze and extract features of the audio data in this format to obtain amplitude spectrum feature data, pitch feature data, and rhythm feature data of the audio data.

[0070] S102: splicing the amplitude spectrum feature data, the pitch feature data and the rhythm feature data to obtain spliced ​​feature data.

[0071] Exemplarily, the amplitude spectrum feature data, the pitch feature data and the rhythm feature data may be concatenated together in any order to obtain a concatenated feature data.

[0072] S103: Input the concatenated feature data into a deep learning model, and extract features through the feature extractors corresponding to the amplitude spectrum feature data, the pitch feature data, and the rhythm feature data, respectively, to obtain the beat features corresponding to the audio data; wherein different feature extractors have different feature extraction scales.

[0073] Here, the deep learning model is a pre-trained neural network model with powerful feature extraction and pattern recognition capabilities. It can be used to fuse and extract features from audio data, and the analysis of music style and rhythm is added during the extraction process to obtain the beat features of the audio data. The beat feature is used to indicate the position of each music beat in the audio data. The beat feature is a label information similar to 0 / 1, and one label corresponds to a frame of audio signal. One of Tag0 or Tag1 is used to characterize the corresponding frame audio signal as a beat position, and the other is used to characterize the corresponding frame audio signal as a non-beat position. For example, the beat feature can be 0010110001, where the third frame audio signal, the fifth frame audio signal, the sixth frame audio signal, and the tenth frame audio signal are beat positions, and the remaining frame audio signals are non-beat positions.

[0074] Three feature extractors are set in the deep learning model, namely, a feature extractor related to amplitude spectrum feature data, a feature extractor related to pitch feature data, and a feature extractor related to rhythm feature data. Each feature extractor has a different feature extraction scale, which is specifically characterized by having different sizes of convolution kernels in different feature extractors and focusing on extracting features at different levels. The three feature extractors are deployed in the deep learning model in a preset order to extract features of the input features in sequence.

[0075] In a specific implementation, the spliced ​​feature data can be input into a deep learning model, and the feature extractor related to the amplitude spectrum feature data in the deep learning model will perform feature fusion and extraction on the spliced ​​feature data to obtain a first extraction result. The feature extractor related to the rhythm feature data in the deep learning model will perform feature fusion and extraction on the first extraction result to obtain a second extraction result. The feature extractor related to the pitch feature data in the deep learning model will perform feature fusion and extraction on the second extraction result to obtain a third extraction result. Finally, the deep learning model can perform processing such as normalization and decoding on the third extraction result to obtain a beat feature.

[0076] S104: Generate beat information of the audio data according to the beat feature.

[0077] Here, the beat information is used to represent information such as the beat position in the audio data, the time point corresponding to the beat, the beat rhythm change, etc. Compared with the beat features, the beat information is a language that the user can directly understand, and the beat features are machine voice.

[0078] In specific implementation, after obtaining the beat feature, the beat feature can be converted into beat information and output to the user according to the time of the audio signal corresponding to each tag in the beat feature and the meaning corresponding to each tag (for example, whether it is a beat position).

[0079] In this way, by extracting the amplitude spectrum feature data on the audio amplitude dimension, the pitch feature data on the music style dimension, and the rhythm feature data on the rhythm change feature dimension, it is possible to fully mine the feature data of the audio data in various aspects, and then by splicing the feature data on the three dimensions, the spliced ​​features can fully characterize the music features of the audio data. Since different feature extractors correspond to different feature extraction scales, the spliced ​​features are extracted using feature extractors corresponding to various dimensions, which can fully mine and identify the feature data on each dimension at the corresponding scale, ensure a comprehensive understanding of the prelude and rhythm changes of the audio data, avoid the influence of blank prelude, music style and rhythm changes on beat recognition, adapt to a variety of music styles and rhythm changes, and thus obtain accurate beat features. Finally, using the beat features, accurate beat information that conforms to the style of the audio data can be generated.

[0080] In one embodiment, each feature extractor includes a convolution layer and an attention extraction layer; the convolution scales of the convolution layers in the feature extractors corresponding to the amplitude spectrum feature data, the pitch feature data, and the rhythm feature data are different.

[0081] Here, when the convolution scales of the convolution layers in the feature extractor are different, the feature extraction scales corresponding to the feature extractor will also be different. The difference in convolution scales is reflected in the different convolution kernel sizes of the convolution layers in the feature extractor.

[0082] In specific implementation, the deep learning model in the present disclosure can be obtained by designing a traditional transformer model as the backbone model network, and the feature extractor in the present disclosure is obtained by modifying the encoder in the transformer. The specific modification is to add a convolution layer before the attention extraction layer in the encoder to perform feature convolution processing. That is, each feature extractor in the present disclosure includes a convolution layer and an attention extraction layer. The convolution layer first performs convolution fusion processing on the input features, and then the attention extraction layer performs feature extraction.

[0083] Exemplarily, the convolution layer in the feature extractor related to the amplitude spectrum feature data in the deep learning model can be used to first perform feature fusion on the concatenated feature data, and then the attention extraction layer can be used to perform feature extraction on the fused features to obtain a first extraction result. Then, the convolution layer in the feature extractor related to the rhythm feature data can be used to first perform feature fusion on the first extraction result, and then the attention extraction layer can be used to perform feature extraction on the fused features to obtain a second extraction result. Finally, the convolution layer in the feature extractor related to the pitch feature data can be used to first perform feature fusion on the second extraction result, and then the attention extraction layer can be used to perform feature extraction on the fused features to obtain a third extraction result.

[0084] In one embodiment, the three feature extraction layers in the deep learning model may include a first feature extractor corresponding to the amplitude spectrum feature data, a second feature extractor corresponding to the rhythm feature data, and a third feature extractor corresponding to the pitch feature data. The convolution kernels corresponding to the three feature extractors may increase in sequence. For example, the convolution kernel corresponding to the first feature extractor may be 1×1, the convolution kernel corresponding to the second feature extractor may be 1×3, and the convolution kernel corresponding to the third feature extractor may be 1×7. Therefore, the first feature extractor may also be called a low-scale feature extractor, the second feature extractor may also be called a mid-scale feature extractor, and the third feature extractor may also be called a high-scale feature extractor. For the above S103, it can be implemented as follows:

[0085] S103-1: Using a first feature extractor corresponding to the amplitude spectrum feature data, extract features from the concatenated feature data to obtain first intermediate feature data.

[0086] Exemplarily, the concatenated data features can be input into the first feature extractor, and the low-scale convolution layer in the first feature extractor is used to perform feature convolution processing on the concatenated data features, and then the processed features are input into the attention extraction layer in the first feature extractor to extract low-scale features, capture the overall volume change, spectrum characteristics and rhythm change of each frame of audio signal, and obtain the first intermediate feature data. The first intermediate feature data can model the complete music information of the audio data, which can roughly represent the audio data at a macro level.

[0087] S103-2: The first intermediate feature and the rhythm feature data are concatenated and input into a second feature extractor corresponding to the rhythm feature data to obtain second intermediate feature data.

[0088] Exemplarily, after the first feature extractor outputs the first intermediate feature data, the first intermediate feature data and the rhythm feature data can be spliced ​​to obtain a first spliced ​​feature. The first spliced ​​feature is input into the second feature extractor. The first spliced ​​feature is subjected to feature convolution processing using the mesoscale convolution layer in the second feature extractor, and the processed feature is then input into the attention extraction layer in the second feature extractor for mesoscale feature extraction, and the rhythm changes of the music (such as fast and slow changes in rhythm and rapid pitch fluctuations) are captured in fine granularity based on the duration of the audio data to obtain the second intermediate feature data. Among them, the second intermediate feature data can model the complete rhythm information of the audio data.

[0089] S103-3: The second intermediate feature and the pitch feature data are concatenated and input into a third feature extractor corresponding to the pitch feature data to obtain third intermediate feature data.

[0090] Exemplarily, after the second feature extractor outputs the second intermediate feature data, the second intermediate feature data and the pitch feature data can be spliced ​​to obtain a second spliced ​​feature. The second spliced ​​feature is input into the third feature extractor, and the high-scale convolution layer in the third feature extractor is used to perform feature convolution processing on the second spliced ​​feature. The processed feature is then input into the attention extraction layer in the third feature extractor to extract high-scale features, and the pitch change (i.e., style change) of the audio data is learned in a higher-dimensional feature representation to obtain the third intermediate feature data. The third intermediate feature data can model the complete style information of the audio data.

[0091] In the disclosed embodiment, by splicing the amplitude spectrum feature data, the pitch feature data and the rhythm feature data and inputting them into the first feature extractor, splicing the output of the first feature extractor with the rhythm feature data and inputting them into the second feature extractor, and splicing the output of the second feature extractor with the pitch feature data and inputting them into the third feature extractor, it is possible to achieve residual connection of the features for the corresponding feature extractor to learn, thereby enhancing the expressive power and stability of the feature extractor. Since different feature extractors have different convolution scales, using multiple feature extractors can enable the model to fully learn the explicit features and implicit features of audio data at different levels.

[0092] S103-4: Use the decoder in the deep learning model to perform feature decoding processing on the third intermediate feature data to obtain beat features.

[0093] In specific implementation, a decoder is also provided in the deep learning model, and the decoder in the existing transformer model can be directly used. After the third feature extractor outputs the third intermediate feature data, the third intermediate feature data can be input into the decoder in the deep learning model for feature decoding and aggregation processing to obtain the beat feature.

[0094] like Figure 2 As shown, it is a schematic diagram of the feature extraction process of a deep learning model provided by an embodiment of the present disclosure, wherein, after obtaining the amplitude spectrum feature data, the pitch feature data and the rhythm feature data, the three features can be spliced ​​and input into transformer-encoder1 (i.e., the first feature extractor), and then the output of transformer-encoder1 and the rhythm feature data are spliced ​​and input into transformer-encoder2 (i.e., the second feature extractor), and then the output of transformer-encoder2 and the pitch feature data are spliced ​​and input into transformer-encoder3 (i.e., the third feature extractor) to obtain the third intermediate feature data, and the third intermediate feature data is input into transformer-decoder (i.e., decoder) for feature decoding and aggregation processing to obtain the beat feature.

[0095] The disclosed embodiment analyzes the beat of music based on the deep learning method, which can solve the problems of inaccurate beat analysis and lack of personalization in the prior art. Specifically, by constructing a multi-scale feature extraction module and introducing feature extraction under multiple feature scales, the deep learning model can analyze the music signal at different time resolutions, thereby ensuring a comprehensive understanding of the music prelude and rhythm changes, thereby improving the accuracy of beat analysis. In addition, the deep learning model can adapt to audio data under various music styles and music rhythms, realize targeted beat analysis of music under various music styles and rhythms, and output more accurate and personalized beat features. In addition, thanks to the powerful learning ability of the deep learning model, the audio beat analysis method in the present disclosure can be implemented as a customized service for users. For example, by learning the audio data actively provided by the user, the beat information that meets the user's needs can be accurately analyzed for the user, and a personalized beat analysis service can be provided to the user.

[0096] In one embodiment, S101 may be implemented according to the following steps:

[0097] S101 - 1 : Preprocess the audio data to obtain each frame of audio signal corresponding to the audio data; the preprocessing includes sampling rate standardization, audio data framing, audio signal normalization and audio signal denoising.

[0098] Here, the purpose of preprocessing is to improve the input quality of the deep learning model and improve the data quality of the audio data.

[0099] Sampling rate standardization usually refers to unifying the sampling rate of audio signals to a standard value to ensure compatibility and consistency of audio on different devices and platforms. In the wav file format, the sampling rate is an important parameter that defines the number of times the sound signal is sampled per second, usually in Hertz (Hz). For example, CD-quality audio usually uses a sampling rate of 44.1kHz, while higher sampling rates such as 48kHz, 96kHz, or 192kHz may be used in professional audio production. The choice of sampling rate depends on the audio quality requirements and application scenarios. A higher sampling rate can provide a wider frequency range and finer time resolution, but it will also increase the file size.

[0100] The audio data framing may specifically be to divide the audio data into a plurality of frames according to a preset framing interval. The preset framing interval may be set based on experience, and the present disclosure embodiment does not specifically limit it. For example, the preset framing interval may be 16 milliseconds (ms), 32 ms, 64 ms, etc.

[0101] The purpose of audio signal normalization is to make each frame of audio signal within a standard amplitude range, reducing the impact of recording equipment or environment on the audio signal. Audio signal denoising is specifically to remove noise data in the audio signal to improve the quality of the audio signal.

[0102] In a specific implementation, after obtaining the audio data, the audio data can be first converted into a unified audio format to obtain an audio file with a target format. Then, the sampling rate of the audio file can be standardized to obtain audio data with a standard sampling rate. Then, the audio data is framed according to a preset frame interval to obtain audio signals of each frame. Afterwards, audio signal normalization and audio signal denoising can be performed on each frame of audio signal to obtain a preprocessed audio file, which includes each frame of audio signal after preprocessing. Among them, when normalizing the audio signal, methods such as mean normalization and loudness normalization can be used; when denoising the audio signal, methods such as Wiener filtering and spectral subtraction can be used.

[0103] S101 - 2 : Perform short-time Fourier transform on each frame of audio signal to obtain amplitude spectrum characteristics, frequency information, and energy of frequency feature points corresponding to each frame of audio signal.

[0104] Here, the amplitude spectrum feature involves the amplitude distribution of the signal in the frequency domain, which is usually obtained by taking the modulus of the complex vector after performing STFT on the signal. The amplitude spectrum can show the amplitude changes of the audio signal at different frequencies. The frequency information of an audio signal refers to the distribution of different frequency components in the audio signal. The frequency information can characterize and understand the characteristics of the signal, such as pitch, timbre and sound quality. In audio signal processing, energy can be used to measure the strength or loudness of the signal.

[0105] In specific implementation, the preprocessed audio file may be subjected to STFT processing to obtain the amplitude spectrum characteristics of each frame of audio signal, the frequency information of each frame of audio signal, and the energy of the frequency feature points corresponding to each frame of audio signal.

[0106] S101 - 3 : Determine pitch feature data of the audio data according to the frequency information of each frame of the audio signal.

[0107] Here, the pitch feature data is the pitch feature data. The frequency information in the audio signal and the pitch feature data can be converted to each other through a specific conversion method. Pitch refers to the height of the sound, usually expressed in Hertz (Hz), while the frequency information describes the distribution of different frequency components in the audio signal. The fundamental frequency (Pitch) is the basic frequency of the sound, which determines the height of the sound. The fundamental frequency can be calculated by measuring the periodic vibration frequency of the lowest frequency in the sound waveform, and its reciprocal is the period (Period), and the formula is Pitch = 1 / Period.

[0108] In specific implementation, after obtaining the frequency information of each frame of audio, the basic frequency in the frequency information can be used to determine the fundamental frequency information of the audio signal based on the analysis of the frequency information. Then, the pitch feature data is obtained by using the conversion formula in the prior art and the fundamental frequency information.

[0109] S101 - 4 : Determine rhythm feature data of the audio data according to the energy of the frequency feature points corresponding to each frame of the audio signal.

[0110] Here, rhythmic feature data is used to capture the beat frequency and rhythmic dynamics in the music, which is related to the temporal structure of the music.

[0111] In specific implementation, the energy envelope corresponding to the audio data can be constructed according to the energy of each frequency feature point. Then, the peak detection method is applied to the energy envelope to detect the beat point, and the rhythm feature data of the audio data is determined according to the time of the detected beat point. Among them, the peak detection method can identify the beat information in the music and calculate the beat interval, and then detect the acceleration or deceleration of the rhythm in the music through the change of the beat interval, so as to identify the continuous rhythm change pattern.

[0112] In one embodiment, the above S101-4 may be implemented according to the following steps:

[0113] S101-4-1: Determine each beat point corresponding to the audio data from each frequency feature point according to the energy of each frequency feature point and the energy of each target feature point located within a preset time interval of the frequency feature point.

[0114] Here, the preset time interval can be set based on experience and is not specifically limited in the embodiment of the present disclosure.

[0115] In specific implementation, after constructing an energy envelope using the energy of each frequency feature point, the peak detection method can be used to detect beat points. Specifically, each frequency feature point in the energy envelope can be traversed in turn, and for the currently traversed frequency feature point, each target feature point that is separated from the frequency feature point by a time interval less than or equal to a preset time interval can be screened out from each frequency feature point based on the time corresponding to the frequency feature point and the preset time interval. Optionally, the target feature point can also be screened out using a set distance. For example, for each frequency feature point, each target feature point that is less than a set distance can be screened out from each other frequency feature point based on the distance between each other frequency feature point and the frequency feature point.

[0116] Furthermore, it is possible to determine whether the energy of the currently traversed frequency feature point exceeds the energy of each target feature point based on the energy of the currently traversed frequency feature point and the energy of each target feature point. If so, the currently traversed frequency feature point can be used as a beat point and the next frequency feature point can be traversed until all frequency feature points are traversed and the beat points corresponding to the audio data are obtained. If not, it can be determined that the currently traversed frequency feature point is not a beat point, and the next frequency feature point can be traversed until all frequency feature points are traversed and the beat points corresponding to the audio data are obtained.

[0117] In one embodiment, the above S101-4-1 may be implemented according to the following steps:

[0118] For any frequency feature point, if the energy of the frequency feature point exceeds a preset threshold and exceeds the energy of each target feature point within a preset time interval of the frequency feature point, the frequency feature point is determined to be a beat point corresponding to the audio data.

[0119] Here, the preset threshold is a pre-set energy threshold, which can be set based on experience and is not specifically limited in the embodiments of the present disclosure.

[0120] For example, for any frequency feature point, it can be determined whether the energy of the frequency feature point exceeds the preset threshold and the energy of each target feature point at the same time. If so, it can be determined that the frequency feature point is a peak point and the point is used as a beat point of the audio data. If not, it can be determined that the frequency feature point is not a beat point.

[0121] S101-4-2: Determine the number of beats per minute of the audio data based on the beat time of each beat point and the time interval between adjacent beat points.

[0122] Here, the beats per minute (BPM) refers to the number of sound beats emitted during a one-minute time period. The beat time of the beat point can specifically be the time point corresponding to the audio signal corresponding to the beat point in the audio data.

[0123] In specific implementation, the time interval between adjacent beat points can be determined according to the beat time of the beat points. Then, the BPM corresponding to the audio data can be determined according to the beat time and the time interval. Here, since the BPM in different 1 minutes may be different, the embodiment of the present disclosure can obtain the BPM corresponding to each 1 minute in the audio data.

[0124] S101-4-3: Determine rhythm feature data based on the time interval between adjacent beat points and the number of beats per minute.

[0125] In specific implementation, the time interval between adjacent beat points and the number of beats per minute can be used as rhythm feature data of the audio data. If the obtained number of beats per minute includes multiple beats per minute, each beat per minute can be used as rhythm feature data; the mean, maximum value, variance, etc. of multiple beats per minute can also be used as rhythm feature data.

[0126] Optionally, after obtaining the time intervals between adjacent beat points, each time interval may be further analyzed to obtain beat interval analysis features, such as the changing trend of the beat interval, the change in the duration of the beat interval, the potential regularity of the beat interval, etc. Then, the beat interval analysis features and the number of beats per minute may be used as rhythm feature data of the audio data.

[0127] In one embodiment, since the audio beat analysis method provided by the embodiment of the present disclosure needs to use a deep learning module to extract beat features, the embodiment of the present disclosure also provides the following steps of training a deep learning model as shown in step 1 to step 4:

[0128] Step 1: Obtain an audio data set; the audio data set includes multiple sample audio data, and different audio data have different music styles, different music rhythms, and different music durations.

[0129] Here, the audio data set includes a large amount of sample audio data, and each sample audio data can be preprocessed audio data. Among the large amount of sample audio data, audio data of various music styles, various music rhythms and various music durations can be included. There are differences between different audio data at least in one or more of music style, music rhythm and music duration. Among them, among the large amount of sample audio data, some sample audio data may have no prelude, some may have a complete prelude, and some may have a prelude including a low volume and a blank. That is, in the audio data set, various sample audio data can be collected as much as possible for deep learning model learning.

[0130] In specific implementation, a large amount of audio data of various music styles and music rhythms can be collected, and these audio data can be pre-processed to obtain sample audio data.

[0131] Step 2: Perform audio signal analysis on the sample audio data to obtain sample amplitude spectrum feature data, sample pitch feature data and sample rhythm feature data of the sample audio data.

[0132] Here, the sample amplitude spectrum feature data is the amplitude spectrum feature data of the sample audio data, the sample pitch feature data is the pitch feature data of the sample audio data, and the sample rhythm feature data is the sample rhythm feature data of the sample audio data.

[0133] Regarding the step of performing audio signal analysis processing on the sample audio data, reference may be made to the above-mentioned step S101, which will not be described in detail here.

[0134] Step 3: Concatenate the sample amplitude spectrum feature data, the sample pitch feature data, and the sample rhythm feature data and input them into the deep learning model to be trained to obtain the predicted beat features of the sample audio data.

[0135] Here, the predicted beat feature is the beat feature predicted by the deep learning model to be trained for the sample audio data.

[0136] Exemplarily, for any sample audio data, the sample amplitude spectrum feature data, sample pitch feature data and sample rhythm feature data corresponding to the sample audio data can be spliced ​​to obtain sample spliced ​​data features, and then the sample spliced ​​data features are input into the deep learning model to be trained to obtain the predicted beat features of the sample audio data. Among them, the feature extraction and fusion process of the sample spliced ​​data features by the deep learning model to be trained can refer to the feature extraction and fusion process of the spliced ​​data features by the deep learning model mentioned above, which will not be repeated here.

[0137] Step 4: Use the feature loss between the predicted beat features and the labeled beat features of the sample audio data to iteratively train the deep learning model to be trained until the training cutoff condition is met to obtain a trained deep learning model.

[0138] Here, the label beat feature is a manually labeled feature of the sample audio data, and is used as the feature truth value for loss determination. The training cutoff condition may be that the prediction accuracy of the deep learning model obtained through training reaches a preset accuracy, and / or the number of iterative training rounds reaches a preset number of rounds.

[0139] In specific implementation, the predicted beat features of each sample audio data and the labeled beat features of each sample audio data can be used to determine the feature loss of the deep learning model to be trained when predicting the beat features, and then the deep learning model to be trained with the feature loss is iteratively trained until the training cutoff condition is met, and the trained deep learning model is used as the trained deep learning model. After the model training is completed, the trained deep learning model can be used to parse the music beats of the audio data.

[0140] In one example, the audio beat parsing method provided by the embodiment of the present disclosure can be applied to a vehicle. When a user performs audio recognition in a vehicle, the audio beat parsing method provided by the embodiment of the present disclosure can be used to parse the audio provided by the user, thereby outputting corresponding beat information, and using the beat information to control related vehicle applications.

[0141] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0142] In addition, an embodiment of the present disclosure further provides a vehicle, wherein the vehicle includes a controller, wherein the controller includes:

[0143] a memory for storing machine-readable instructions; and

[0144] The processor is used to call and execute the machine-readable instructions stored in the memory to implement the steps of the audio beat parsing method as described in the above embodiments.

[0145] Based on the same inventive concept, an audio beat analysis device corresponding to the audio beat analysis method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned audio beat analysis method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0146] like Figure 3FIG. 1 is a schematic diagram of an audio beat analysis device provided by an embodiment of the present disclosure, comprising:

[0147] The parsing module 301 is used to perform audio signal parsing processing on the audio data to obtain amplitude spectrum feature data, pitch feature data and rhythm feature data of the audio data;

[0148] A splicing module 302 is used to splice the amplitude spectrum feature data, the pitch feature data and the rhythm feature data to obtain spliced ​​feature data;

[0149] The extraction module 303 is used to input the spliced ​​feature data into a deep learning model, and extract the features through the feature extractors corresponding to the amplitude spectrum feature data, the pitch feature data and the rhythm feature data, respectively, to obtain the beat features corresponding to the audio data; wherein different feature extractors have different feature extraction scales;

[0150] The generating module 304 is used to generate the beat information of the audio data according to the beat feature.

[0151] In a possible implementation, each of the feature extractors includes a convolution layer and an attention extraction layer; and the convolution scales of the convolution layers in the feature extractors corresponding to the amplitude spectrum feature data, the pitch feature data, and the rhythm feature data are different.

[0152] In a possible implementation, the extraction module 303, after inputting the spliced ​​feature data into a deep learning model and extracting features through feature extractors corresponding to the amplitude spectrum feature data, the pitch feature data, and the rhythm feature data, respectively, to obtain the beat features corresponding to the audio data, is used to:

[0153] Using a first feature extractor corresponding to the amplitude spectrum feature data, extracting features from the spliced ​​feature data to obtain first intermediate feature data;

[0154] splicing the first intermediate feature and the rhythm feature data and inputting the resultant data into a second feature extractor corresponding to the rhythm feature data to obtain second intermediate feature data;

[0155] splicing the second intermediate feature and the pitch feature data and inputting the resultant data into a third feature extractor corresponding to the pitch feature data to obtain third intermediate feature data;

[0156] The decoder in the deep learning model is used to perform feature decoding processing on the third intermediate feature data to obtain the beat feature.

[0157] In a possible implementation, the parsing module 301, when performing audio signal parsing processing on the audio data to obtain the amplitude spectrum feature data, pitch feature data and rhythm feature data of the audio data, is used to:

[0158] Preprocessing the audio data to obtain each frame of audio signal corresponding to the audio data; the preprocessing includes sampling rate standardization, audio data framing, audio signal normalization and audio signal denoising;

[0159] Performing short-time Fourier transform on each frame of the audio signal to obtain amplitude spectrum characteristics, frequency information, and energy of frequency feature points corresponding to each frame of the audio signal;

[0160] Determine the pitch feature data of the audio data according to the frequency information of the audio signal of each frame;

[0161] The rhythm feature data of the audio data is determined according to the energy of the frequency feature points corresponding to each frame of the audio signal.

[0162] In a possible implementation manner, the analysis module 301, when determining the rhythm feature data of the audio data according to the energy of the frequency feature points corresponding to each frame of the audio signal, is used to:

[0163] Determine, from each of the frequency feature points, each beat point corresponding to the audio data according to the energy of each of the frequency feature points and the energy of each of the target feature points located within a preset time interval of the frequency feature point;

[0164] Determine the number of beats per minute of the audio data according to the beat time of each beat point and the time interval between adjacent beat points;

[0165] The rhythm feature data is determined according to the time interval between the adjacent beat points and the number of beats per minute.

[0166] In a possible implementation, the parsing module 301, when determining each beat point corresponding to the audio data from each frequency feature point according to the energy of each frequency feature point and the energy of each target feature point located within a preset time interval of the frequency feature point, is used to:

[0167] For any of the frequency feature points, if the energy of the frequency feature point exceeds a preset threshold and exceeds the energy of each target feature point within a preset time interval of the frequency feature point, the frequency feature point is determined to be a beat point corresponding to the audio data.

[0168] In a possible implementation, the device further includes:

[0169] The training module 305 is used to train the deep learning model according to the following steps:

[0170] Acquire an audio data set; the audio data set includes a plurality of sample audio data, and different audio data have different music styles, different music rhythms, and different music durations;

[0171] Performing audio signal analysis processing on the sample audio data to obtain sample amplitude spectrum feature data, sample pitch feature data and sample rhythm feature data of the sample audio data;

[0172] The sample amplitude spectrum feature data, the sample pitch feature data and the sample rhythm feature data are spliced ​​and input into the deep learning model to be trained to obtain the predicted beat feature of the sample audio data;

[0173] The deep learning model to be trained is iteratively trained by utilizing the feature loss between the predicted beat feature and the label beat feature of the sample audio data until a training cutoff condition is met, thereby obtaining a trained deep learning model.

[0174] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0175] Based on the same technical concept, the embodiment of the present application also provides a computer device. Figure 4 FIG. 1 is a schematic diagram of a computer device provided in an embodiment of the present application, including:

[0176] Processor 401, memory 402 and bus 403. The memory 402 stores machine-readable instructions executable by the processor 401, and the processor 401 is used to execute the machine-readable instructions stored in the memory 402. When the machine-readable instructions are executed by the processor 401, the processor 401 executes the following steps: S101: Perform audio signal analysis processing on the audio data to obtain amplitude spectrum feature data, pitch feature data and rhythm feature data of the audio data; S102: Splice the amplitude spectrum feature data, pitch feature data and rhythm feature data to obtain spliced ​​feature data; S103: Input the spliced ​​feature data into the deep learning model, and extract features through feature extractors corresponding to the amplitude spectrum feature data, pitch feature data and rhythm feature data respectively, to obtain the beat features corresponding to the audio data; wherein different feature extractors correspond to different feature extraction scales; and S104: Generate beat information of the audio data according to the beat features.

[0177] The above-mentioned memory 402 includes internal memory 4021 and external memory 4022; the memory 4021 here is also called internal memory, which is used to temporarily store the calculation data in the processor 401, as well as the data exchanged with the external memory 4022 such as the hard disk. The processor 401 exchanges data with the external memory 4022 through the internal memory 4021. When the computer device is running, the processor 401 communicates with the memory 402 through the bus 403, so that the processor 401 executes the execution instructions mentioned in the above-mentioned method embodiment.

[0178] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the audio beat parsing method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0179] The computer program product of the audio beat analysis method provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the audio beat analysis method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0180] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0181] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0182] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0184] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0185] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0186] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. An audio beat analysis method, characterized in that: include: Performing audio signal analysis processing on the audio data to obtain amplitude spectrum feature data, pitch feature data and rhythm feature data of the audio data; splicing the amplitude spectrum feature data, the pitch feature data and the rhythm feature data to obtain spliced ​​feature data; The spliced ​​feature data is input into a deep learning model, and feature extraction is performed in sequence by feature extractors corresponding to the amplitude spectrum feature data, the pitch feature data, and the rhythm feature data, respectively, to obtain beat features corresponding to the audio data; wherein different feature extractors have different feature extraction scales; The beat information of the audio data is generated according to the beat feature.

2. The method according to claim 1, characterized in that Each of the feature extractors includes a convolution layer and an attention extraction layer; the convolution scales of the convolution layers in the feature extractors corresponding to the amplitude spectrum feature data, the pitch feature data, and the rhythm feature data are different.

3. The method according to claim 1 or 2, characterized in that: The spliced ​​feature data is input into a deep learning model, and feature extraction is performed in sequence by feature extractors corresponding to the amplitude spectrum feature data, the pitch feature data, and the rhythm feature data, respectively, to obtain the beat feature corresponding to the audio data, including: Using a first feature extractor corresponding to the amplitude spectrum feature data, extracting features from the spliced ​​feature data to obtain first intermediate feature data; splicing the first intermediate feature and the rhythm feature data and inputting the resultant data into a second feature extractor corresponding to the rhythm feature data to obtain second intermediate feature data; splicing the second intermediate feature and the pitch feature data and inputting the resultant data into a third feature extractor corresponding to the pitch feature data to obtain third intermediate feature data; The decoder in the deep learning model is used to perform feature decoding processing on the third intermediate feature data to obtain the beat feature.

4. The method according to claim 1, characterized in that The audio signal analysis processing is performed on the audio data to obtain amplitude spectrum feature data, pitch feature data and rhythm feature data of the audio data, including: Preprocessing the audio data to obtain each frame of audio signal corresponding to the audio data; the preprocessing includes sampling rate standardization, audio data framing, audio signal normalization and audio signal denoising; Performing short-time Fourier transform on each frame of the audio signal to obtain amplitude spectrum characteristics, frequency information, and energy of frequency feature points corresponding to each frame of the audio signal; Determine the pitch feature data of the audio data according to the frequency information of the audio signal of each frame; The rhythm feature data of the audio data is determined according to the energy of the frequency feature points corresponding to each frame of the audio signal.

5. The method according to claim 4, characterized in that The step of determining the rhythm feature data of the audio data according to the energy of the frequency feature points corresponding to each frame of the audio signal comprises: Determine, from each of the frequency feature points, each beat point corresponding to the audio data according to the energy of each of the frequency feature points and the energy of each of the target feature points located within a preset time interval of the frequency feature point; Determine the number of beats per minute of the audio data according to the beat time of each beat point and the time interval between adjacent beat points; The rhythm feature data is determined according to the time interval between the adjacent beat points and the number of beats per minute.

6. The method according to claim 5, characterized in that The determining, from each of the frequency feature points, each beat point corresponding to the audio data according to the energy of each of the frequency feature points and the energy of each of the target feature points located within a preset time interval of the frequency feature point, comprises: For any of the frequency feature points, if the energy of the frequency feature point exceeds a preset threshold and exceeds the energy of each target feature point within a preset time interval of the frequency feature point, the frequency feature point is determined to be a beat point corresponding to the audio data.

7. The method according to claim 1, characterized in that The method also includes the step of training the deep learning model: Acquire an audio data set; the audio data set includes a plurality of sample audio data, and different audio data have different music styles, different music rhythms, and different music durations; Performing audio signal analysis processing on the sample audio data to obtain sample amplitude spectrum feature data, sample pitch feature data and sample rhythm feature data of the sample audio data; The sample amplitude spectrum feature data, the sample pitch feature data and the sample rhythm feature data are spliced ​​and input into the deep learning model to be trained to obtain the predicted beat feature of the sample audio data; The deep learning model to be trained is iteratively trained by utilizing the feature loss between the predicted beat feature and the label beat feature of the sample audio data until a training cutoff condition is met, thereby obtaining a trained deep learning model.

8. A vehicle, characterized in that: A controller is included, the controller comprising: a memory for storing machine-readable instructions; and A processor, configured to call and execute the machine-readable instructions stored in the memory to implement the steps of the audio beat parsing method as described in any one of claims 1 to 7.

9. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the steps of the audio beat parsing method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is executed by a computer device, the computer device executes the steps of the audio beat analysis method according to any one of claims 1 to 7.

Images