Audio data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses an audio data processing method and device, a storage medium and electronic equipment. The method comprises the following steps: determining audio data required in a target model training process; determining a target spectrum based on the audio data; window division and splicing are carried out on the target spectrum to obtain a first feature sequence, the first feature sequence comprises a plurality of first features recorded according to a specified time sequence, and first overlapping data exist between two first features which are continuous in time; and determining audio features based on the first feature sequence. According to the method, window division and splicing are carried out on a target frequency spectrum to realize data enhancement of audio features, after data enhancement processing, long-sequence audio data are split into a plurality of data fragments with overlapped parts, multiple times of reasoning can be carried out, and the data fragments have relevance, so that the accuracy of reasoning is improved. Misrecognition caused by single reasoning of the target model can be effectively prevented, and the recognition accuracy of the target model is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to an audio data processing method, device, storage medium and electronic device. Background Art

[0002] The music style classification algorithm is an important application algorithm in the field of music information retrieval and music analysis. It aims to input a piece of audio data and, after processing, automatically identify and classify the style or type of music tracks, such as rock, pop, classical, etc.

[0003] The implementation of the music style classification algorithm is mainly achieved through two parts. The first part is audio feature extraction, and the second part is to classify the music style using a model based on the extracted audio features.

[0004] When extracting audio features, the Mel frequency cepstral coefficient is more in line with the human hearing mechanism, but in actual engineering applications, the amount of calculation required for feature extraction is greater than that of Fourier transform, which makes music classification more time-consuming and occupies too much hardware resources. In the selection of models, for existing machine learning algorithms, it is difficult to achieve effective classification in the face of complex semantic information of continuous and nonlinear music, and the generalization ability is poor. At the same time, it takes a lot of time to adjust parameters and filters. Therefore, deep learning methods are more used in the process of music classification. In deep learning research, the RNN (Recurrent Neural Network) model has memory ability and can capture the timing information of audio data. However, it is slow and difficult to parallelize when calculating, and the RNN architecture is prone to gradient disappearance or gradient explosion problems. The convolution operation of the CNN (Convolutional Neural Networks) model can be calculated in parallel, making it faster to train, and its shared parameter mechanism reduces the risk of overfitting. However, the lack of memory ability in the CNN structure will ignore the timing information of audio data. In the data processing part, usually after feature extraction, it is directly passed into the model for inference. The use of long sequence data can easily lead to inaccurate inference results, and this processing method will lead to insufficient use of data during training. Summary of the invention

[0005] The present application provides an audio data processing method, device, storage medium and electronic device, the purpose of which is to achieve data enhancement of music features, ensure the completeness and relevance of the data, and the full use of the data also improves the accuracy of the algorithm. In addition, a time domain convolution model is used to take into account the timing information of the audio data, improve the algorithm accuracy, retain the model parallel characteristics, and improve the model inference rate.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] An audio data processing method, comprising:

[0008] Determine the audio data required for the target model training process; the target model is used to: determine the music style of the audio data based on the audio features with time series attributes extracted from the audio data;

[0009] Based on the audio data, a target spectrum is determined; the target spectrum includes a plurality of frequencies, and the plurality of frequencies are recorded in a specified time sequence;

[0010] The target spectrum is windowed and spliced ​​to obtain a first feature sequence; the first feature sequence includes a plurality of first features recorded in a specified time sequence, and there is first overlapping data between two first features that are consecutive in time; the first feature is a combination of a plurality of frequencies that are consecutive in time; the first overlapping data represents the frequency corresponding to the same timestamp;

[0011] The audio feature is determined based on the first feature sequence.

[0012] Optionally, determining a target frequency spectrum based on the audio data includes:

[0013] Based on the audio data, target time series data is determined; the target time series data comprises a plurality of sound wave features, and the plurality of sound wave features are recorded in a specified time sequence;

[0014] Based on the target time series data, a target frequency spectrum is determined.

[0015] Optionally, determining a target spectrum based on the target time series data includes:

[0016] The target time series data is framed to obtain a second feature sequence; the second feature sequence includes a plurality of second features recorded in the specified time sequence; the second feature is a combination of a plurality of acoustic wave features that are continuous in time; wherein there is second overlapping data between two second features that are continuous in time, and the second overlapping data includes acoustic wave features corresponding to the same timestamp;

[0017] Windowing the second feature sequence to obtain a third feature sequence; the third feature sequence includes a plurality of third features recorded in a specified time sequence; the third feature is a combination of a plurality of second features that are continuous in time;

[0018] Perform Fourier transform on the third characteristic sequence to obtain a target spectrum.

[0019] Optionally, the target model includes a time-domain convolutional network.

[0020] Optionally, determining the audio feature based on the first feature sequence includes:

[0021] Based on the first feature sequence, determine a two-dimensional tensor; wherein the two-dimensional tensor is composed of R×L scalar elements, R represents the total number of first features included in the first feature sequence, L represents the total number of frequencies included in the first feature, and the scalar element represents the frequency;

[0022] The two-dimensional tensor is expanded and stacked to obtain the audio feature; the audio feature includes T target elements recorded in a specified time sequence, T and R have the same value, the data dimension of the target element is k×m, k and L have the same value, and m represents the number of stacking times.

[0023] Optionally, the target model includes an activation function, and the activation function is used to determine the style probability distribution of the audio feature; the style probability distribution is used to characterize the probability values ​​of various music styles.

[0024] Optionally, the target model is specifically used to: determine the style probability distribution of the T target elements based on the T target elements as input; for each target element, determine the music style of the target element based on the music style with the highest probability value in the style probability distribution; determine the target music style as the output from the music styles of the T target elements, wherein the number proportion of the target music style is the highest among the music styles of the T target elements.

[0025] An audio data processing device, comprising:

[0026] An audio determination unit, used to determine the audio data required for the target model training process; the target model is used to: determine the music style of the audio data based on the audio features with time series attributes extracted from the audio data;

[0027] A spectrum determination unit, configured to determine a target spectrum based on the audio data; the target spectrum includes a plurality of frequencies, and the plurality of frequencies are recorded in a specified time sequence;

[0028] A data enhancement unit is used to perform window division and splicing on the target spectrum to obtain a first feature sequence; the first feature sequence includes a plurality of first features recorded in the specified time sequence, and there is first overlapping data between two first features that are continuous in time; the first feature is a combination of a plurality of frequencies that are continuous in time; the first overlapping data represents the frequency corresponding to the same timestamp;

[0029] A feature determination unit is used to determine the audio feature based on the first feature sequence.

[0030] Optionally, the spectrum determination unit is specifically used to:

[0031] Based on the audio data, target time series data is determined; the target time series data includes a plurality of sound wave features, and the plurality of sound wave features are recorded in a specified time sequence;

[0032] Based on the target time series data, a target frequency spectrum is determined.

[0033] Optionally, the spectrum determination unit is specifically used to:

[0034] The target time series data is framed to obtain a second feature sequence; the second feature sequence includes a plurality of second features recorded in the specified time sequence; the second feature is a combination of a plurality of acoustic wave features that are continuous in time; wherein there is second overlapping data between two second features that are continuous in time, and the second overlapping data includes acoustic wave features corresponding to the same timestamp;

[0035] Windowing the second feature sequence to obtain a third feature sequence; the third feature sequence includes a plurality of third features recorded in the specified time sequence; the third feature is a combination of a plurality of second features that are continuous in time;

[0036] Perform Fourier transform on the third characteristic sequence to obtain a target spectrum.

[0037] Optionally, the target model includes a time-domain convolutional network.

[0038] Optionally, the feature determination unit is specifically used to:

[0039] Based on the first feature sequence, determine a two-dimensional tensor; wherein the two-dimensional tensor is composed of R×L scalar elements, R represents the total number of first features included in the first feature sequence, L represents the total number of frequencies included in the first feature, and the scalar element represents the frequency;

[0040] The two-dimensional tensor is expanded and stacked to obtain the audio feature; the audio feature includes T target elements recorded in a specified time sequence, T and R have the same value, the data dimension of the target element is k×m, k and L have the same value, and m represents the number of stacking times.

[0041] Optionally, the target model includes an activation function, and the activation function is used to determine the style probability distribution of the audio feature; the style probability distribution is used to characterize the probability values ​​of various music styles.

[0042] Optionally, the target model is specifically used to: determine the style probability distribution of T target elements based on multiple of the first features as input; for each target element, determine the music style of the target element based on the music style with the highest probability value in the style probability distribution; determine the target music style as output from the music styles of the T target elements, wherein the number proportion of the target music style is the highest among the music styles of the T target elements.

[0043] A storage medium includes a stored program, wherein the program executes the audio data processing method when executed by a processor.

[0044] An electronic device, comprising: a processor, a memory and a bus; the processor and the memory are connected via the bus;

[0045] The memory is used to store programs, and the processor is used to run programs, wherein the program executes the audio data processing method when the processor runs it.

[0046] The technical solution provided by the present application determines the audio data required for the target model training process; based on the audio data, the target spectrum is determined. The target spectrum is divided into windows and spliced ​​to obtain a first feature sequence, and the first feature sequence includes multiple first features recorded in a specified time sequence, and there is a first overlapping data between two first features that are consecutive in time. Based on the first feature sequence, the audio features are determined. The present application achieves data enhancement of audio features by dividing and splicing the target spectrum into windows. After data enhancement processing, the long sequence of audio data will be split into multiple data segments with overlapping parts, which can be inferred multiple times, and each data segment is correlated, which can effectively prevent the misrecognition caused by a single inference of the target model and improve the recognition accuracy of the target model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flowchart of an audio data processing method provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of another audio data processing method provided in an embodiment of the present application;

[0050] Figure 3 A flowchart of another audio data processing method provided in an embodiment of the present application;

[0051] Figure 4 A flowchart of another audio data processing method provided in an embodiment of the present application;

[0052] Figure 5 A flowchart of another audio data processing method provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the architecture of an audio data processing device provided in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of a network structure provided for an embodiment of the present application;

[0055] Figure 8 A schematic diagram of the window division and splicing principle provided in an embodiment of the present application;

[0056] Fig. 9 A flowchart of another audio data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0059] In related technologies, audio feature data is usually directly passed into the model for inference. This application uses audio features to perform data enhancement, fully utilizes limited data fragments and retains the correlation between data, effectively avoids misidentification caused by single inference, and improves recognition accuracy. In addition, compared to the common method of using CNN networks to classify music styles, a time series prediction mechanism is added, which takes into account the time series information of audio data, improves the accuracy of the algorithm, and retains the parallel advantage to increase the inference rate.

[0060] Embodiment 1

[0061] like Figure 1 As shown, it is a flowchart of an audio data processing method provided in an embodiment of the present application, including the following steps.

[0062] S101: Determine the audio data required for the target model training process.

[0063] The target model is used to determine the musical style of the audio data based on the audio features with time series attributes extracted from the audio data.

[0064] Optionally, the target model includes TCN (Temporal Convolutional Networks, temporal convolutional network).

[0065] It can be understood that the process of training the target model based on the audio features of the audio data as training samples is a conventional technical means familiar to those skilled in the art. For example, a training set is pre-based on multiple audio features, and a music style label is set for each sample (i.e., audio feature) in the training set. Each sample is used as the input of a preset TCN model, and the music style label of each sample is used as the training target to train the target model.

[0066] Optionally, the target model includes an activation function, which is used to determine the style probability distribution of the audio feature, and the style probability distribution is used to characterize the probability values ​​of various music styles.

[0067] In some examples, the target model is a TCN model, and the last layer in the TCN model can be set to a softmax layer, and the softmax layer can be regarded as the performance of the activation function in the TCN model.

[0068] In a possible implementation, the various music styles include rock, pop, classical, and hip-hop, and the style probability distribution of the audio features determined by the softmax layer may be: rock 50%, pop 20%, classical 5%, and hip-hop 25%. Rock 50% represents the probability value of rock, that is, the probability that the music style of the audio data is rock. Pop 20% represents the probability value of pop, that is, the probability that the music style of the audio data is pop. Classical 5% represents the probability value of classical, that is, the probability that the music style of the audio data is classical. Hip-hop 25% represents the probability value of hip-hop, that is, the probability that the music style of the audio data is hip-hop.

[0069] In some examples, the target model is Figure 7 The CNN model shown in Figure 2 can be used to establish a mapping relationship between music features and music styles. Figure 7 In the model structure shown, the CNN model is composed of convolutional layers, pooling layers, random inactivation layers, fully connected layers and softmax layers.

[0070] Since audio data is time series data, in order to better identify the time series characteristics of audio data, the target model can be designed as a TCN model. The model structure of the TCN model can be found in Figure 7 shown.

[0071] It should be noted that compared with the method of using CNN model to classify music styles, the TCN model adds a timing prediction mechanism, takes into account the timing characteristics of audio data, and improves the accuracy of the algorithm. Compared with existing CNN models, RNN models and other network models, the use of TCN mechanism for timing prediction can improve the inference rate.

[0072] exist Figure 7 In the model structure shown, the TCN model includes multiple network layers, and the multiple network layers include at least network layer 1, network layer i, TCN layer, network layer n and network layer n+1, i is a positive integer greater than 1, and n is a positive integer greater than i+1. All network layers from network layer 1 to network layer i, and all network layers from network layer i+2 to network layer n, are based on two-dimensional convolutional layers, two-dimensional pooling layers, and random inactivation layers. The TCN layer uses a convolutional network with a TCN mechanism, which can extract temporal features from audio data through convolution operations. Network layer n+1 is based on a fully connected layer, a random inactivation layer, and a softmax layer.

[0073] It is understandable that the TCN model adopts the TCN structure and can process audio data through the idea of ​​time domain convolution, which makes up for the shortcoming of CNN structure that ignores the timing information of audio data, while retaining the advantages of parallel computing, and can maintain a faster reasoning speed while improving the accuracy of the algorithm.

[0074] In a possible implementation, the audio features are first input into a two-dimensional convolutional layer, and the audio features are convolved through the two-dimensional convolutional layer to obtain more feature information of the audio features, further improving the recognition ability of the TCN model. Then, the convolved audio features are input into a two-dimensional pooling layer for a maximum pooling operation to reduce the number of parameters of the audio features, expand the receptive field of the audio features, and make the audio features nonlinear. Finally, the audio features after maximum pooling are input into a random dropout layer to eliminate overfitting of the audio features.

[0075] In some examples, the duration of the audio shown in the audio data can be set by a technician according to actual conditions. For example, the audio duration can be limited to N seconds, where N is a positive integer.

[0076] S102: Determine a target frequency spectrum based on the audio data.

[0077] The target spectrum includes multiple frequencies, and the multiple frequencies are recorded in a specified time sequence.

[0078] In some examples, the audio duration of the audio data is 100 seconds, and the target spectrum may include frequencies corresponding to the 100 seconds recorded in order from early to late in terms of playback time, with a sampling time of 2 seconds per interval, a total of 50 frequencies, and the 50 frequencies are sorted in order from early to late in terms of playback timestamps.

[0079] Optionally, the implementation process of determining the target spectrum based on audio data can be found in Figure 2 The steps are shown and their explanations.

[0080] S103: Perform window division and splicing on the target spectrum to obtain a first characteristic sequence.

[0081] Among them, the first feature sequence includes multiple first features recorded in a specified time order, and there is first overlapping data between two temporally consecutive first features. The first feature is a combination of multiple temporally consecutive frequencies, and the first overlapping data represents the frequency corresponding to the same timestamp.

[0082] In the process of window division and splicing, in some examples, the target spectrum is recorded as f_matrix, each f (which can be understood as frequency) in f_matrix is ​​recorded as a frame, and the target spectrum is windowed and spliced ​​to obtain t time windows, and the data length of each time window is set to L frames (which can be understood as a combination of L frequencies that are continuous in time). In order to achieve data enhancement, it is required that the data in each time window has K frames of data overlap, so window splicing is required. Specifically, starting from the second time window, part of the data in each subsequent time window takes the tail of the previous time window, and finally T first features are obtained, and there is first overlapping data between two first features that are continuous in time, and the first overlapping data includes K frequencies.

[0083] In a possible implementation, the number T of first features included in the first feature sequence may be determined based on formula (1).

[0084] T=(t*LL) / (LK)+1 (1)

[0085] In a possible implementation, the target spectrum with an audio duration of N seconds is windowed and spliced. The processing method of each time window in the window division and splicing process can be found in Fig. 9 shown.

[0086] In a possible implementation, the frequencies shown in the target spectrum include R1, R2, R3, R4, R5, R6, R7, and R8. The target spectrum is divided into windows, and the sampling data length of the window function is 2. Then the obtained characteristic sequences are Q1, Q2, Q3, and Q4, wherein Q1 is a combination of R1 and R2, Q2 is a combination of R3 and R4, Q3 is a combination of R5 and R6, and Q4 is a combination of R7 and R8. Secondly, Q1, Q2, Q3, and Q4 are windowed and spliced ​​to obtain the first characteristic sequence of S1, S2, S3, S4, S5, S6, and S7, wherein S1 is a combination of R1 and R2, S2 is a combination of R2 and R3, S3 is a combination of R3 and R4, S4 is a combination of R4 and R5, S5 is a combination of R5 and R6, S6 is a combination of R6 and R7, and S7 is a combination of R7 and R8.

[0087] It can be understood that the presence of first overlapping data between two first features that are consecutive in time can ensure that the transition between two first features that are adjacent in time sequence is smoother, so as to improve the reliability of the audio feature.

[0088] In some examples, S2 and S3 are two first features that are continuous in time, and the first overlapping data is R3.

[0089] It should be noted that the data between each time window (i.e., each first feature) has an overlapping part, so as to retain the correlation of the data between the time windows, realize the reuse of data segments, avoid the misrecognition caused by a single inference of the target model, and improve the recognition accuracy of the target model.

[0090] S104: Determine audio features based on the first feature sequence.

[0091] Wherein, based on the first feature sequence, an audio feature is determined as an input of the target model to obtain an output result of the target model, and the output result is recorded as a music style.

[0092] Optionally, for the audio features as the input of the target model, the process of inferring the music style of the target model can be seen in Figure 4 Steps shown and explanations of the steps.

[0093] Optionally, the implementation process of determining the audio feature based on the first feature sequence can be found in Figure 5 Steps shown and explanations of the steps.

[0094] In some examples, combined Figure 2 , Figure 3 , Figure 4 as well as Figure 5 The method shown in the figure can be used to implement the music style classification of audio data. Fig. 9 As shown in the figure, it can be summarized as follows: the audio data is decoded in advance, and then the decoded audio data is resampled and framed to obtain an audio signal (i.e., target time series data). Next, the feature of the audio signal is extracted to obtain feature data (i.e., target spectrum). Then, the feature data is enhanced by window division and splicing to obtain audio features. The audio features are used as the input of the target model to predict the music style and obtain the music classification results.

[0095] The process shown in S101-S104 above realizes data enhancement of audio features by windowing and splicing the target spectrum. After data enhancement processing, the long sequence of audio data will be split into multiple data segments with overlapping parts, which can be inferred multiple times, and each data segment is related, which can effectively prevent the misidentification caused by a single inference of the target model and improve the recognition accuracy of the target model. In addition, compared with the common method of using CNN network to classify music styles, a time series prediction mechanism is added, the time series information of audio data is considered, and the algorithm accuracy is improved. Compared with time series prediction mechanisms such as LSTM and RNN, the use of TCN mechanism for time series prediction can improve the inference rate.

[0096] Embodiment 2

[0097] like Figure 2 As shown, it is a flow chart of another audio data processing method provided in an embodiment of the present application, including the following steps.

[0098] S201: Determine target time series data based on audio data.

[0099] The target time series data includes a plurality of acoustic wave features, and the plurality of acoustic wave features are recorded in a specified time sequence.

[0100] In some examples, after the audio data is obtained, the audio data needs to be decoded to obtain an audio signal (which can be understood as a digital signal), and the audio signal is regarded as target time series data.

[0101] It should be noted that audio signals are information carriers of frequency and amplitude changes of regular sound waves with voice, music and marketing. According to the characteristics of sound waves, audio signals can be classified into regular audio and irregular sound. Among them, regular audio can be divided into voice, music and sound effects. Regular audio can be regarded as a continuously changing analog signal, which can be represented by a continuous curve and can be called a sound wave. Sound waves have three important parameters, namely frequency, amplitude and phase, which determine the sound wave characteristics of audio signals.

[0102] In some examples, after obtaining the audio signal obtained after decoding the audio data, the audio signal can be resampled, and the resampled audio signal is used as the target time series data. Resampling is an image processing technology, which is mainly used to interpolate information of another type of pixel based on information of one type of pixel. In the field of signal processing, resampling can change the spatial resolution of the audio signal and set rules for the aggregation value or interpolation of all new data points. The size of the data point can be changed, but the range of the audio signal will remain unchanged, so that each data point in the resampled audio signal will have a unique attribute value.

[0103] It is understandable that the audio signal obtained after decoding the audio data can obtain an audio signal that meets the specified sampling rate and data input size by resampling the audio signal, ensuring that the target timing data finally obtained meets the subsequent frame requirements.

[0104] S202: Determine a target frequency spectrum based on the target time series data.

[0105] Among them, determining the target spectrum based on the target time series data is essentially to convert the target time series data into corresponding target frequency domain data, thereby realizing the conversion of audio data from the time domain to the frequency domain.

[0106] Optionally, the target spectrum can be determined based on the target time series data. Figure 3 Steps shown and explanations of the steps.

[0107] It is understandable that the target time series data is a one-dimensional time domain signal, and it is difficult to intuitively see the frequency change pattern. For this reason, the time domain signal is converted into a frequency domain signal to obtain the change pattern of the frequency distribution of the audio data over time, and then based on the change pattern of the frequency distribution over time, the audio features of the audio data are extracted to ensure that the audio features are objective and effective.

[0108] In some examples, the target spectrum can be represented based on a spectrogram, which is a graph showing the change of speech spectrum over time. The vertical axis of the spectrogram represents frequency, and the horizontal axis represents time. The strength of any specified frequency component at a specified time can be represented by the grayscale or hue of the corresponding point. Generally speaking, a spectrogram displays a large amount of information related to the sentence characteristics of speech, combines the points of the spectrogram and the time domain waveform, and clearly shows the change of the speech spectrum over time, or can also be understood as a dynamic spectrum.

[0109] The process shown in S201-S202 above can convert audio data from time domain signals to frequency domain signals, so as to intuitively obtain the change pattern of frequency distribution over time, so as to extract objective and effective audio features from the change pattern of frequency distribution over time.

[0110] Embodiment 3

[0111] like Figure 3 As shown, it is a flow chart of another audio data processing method provided in an embodiment of the present application, including the following steps.

[0112] S301: Frame the target time series data to obtain a second feature sequence.

[0113] The second feature sequence includes a plurality of second features recorded in a specified time sequence, and the second feature is a combination of a plurality of acoustic wave features that are continuous in time.

[0114] The so-called framing means that several audio sampling points are divided into one frame. Within this frame, the characteristics of the audio signal can be considered stable. In the related art, a complete audio signal is usually non-steady-state, that is, the characteristics of the audio signal basically change with time. However, due to the inherent characteristics of sound waves, within a short time range (such as 10-30ms), the characteristics of the audio signal remain basically unchanged, that is, the audio signal is short-term stable. For this reason, when performing frequency domain analysis on the audio signal, it is necessary to process it frame by frame, and the frame length can be set according to the actual situation.

[0115] In some examples, for audio data with an audio duration of N seconds, the frame length used to divide the target time series data into frames can be set to l (the value of l is a power of 2), and l temporally continuous sound wave features are selected as a frame. Each frame is represented as signal_flow and can be regarded as a second feature.

[0116] In a possible implementation, the target time series data includes multiple feature data, namely X1, X2, X3, X4, X5, X6, X7, X8, X9, X10, X11, and X12, and the frame length can be set to 4. The target time series data is framed, and the second feature sequence obtained is: S1, S2, and S3. Among them, S1 is a combination of X1, X2, X3, and X4, S2 is a combination of X5, X6, X7, and X8, and S3 is a combination of X9, X10, X11, and X12.

[0117] Optionally, there is second overlapping data between two temporally consecutive second features, and the second overlapping data includes sound wave features corresponding to the same timestamp.

[0118] In order to make a smooth transition between frames, there can generally be a certain overlap between frames (acoustic wave features corresponding to the same timestamp). The time difference between the starting positions of two adjacent frames can be regarded as frame shift (i.e., the number of acoustic wave features contained in the second overlapping data). The frame shift can be regarded as 1 / 2 or 3 / 4 of the frame length.

[0119] In a possible implementation, the target time series data includes multiple feature data, namely X1, X2, X3, X4, X5, X6, X7, X8, X9, X10, X11, and X12. The frame length can be set to 4. The target time series data is framed, and the second feature sequence obtained is: S1, S2, S3, S4, and S5. Among them, S1 is a combination of X1, X2, X3, and X4, S2 is a combination of X3, X4, X5, and X6, S3 is a combination of X5, X6, X7, and X8, S4 is a combination of X7, X8, X9, and X10, and S5 is a combination of X9, X10, X11, and X12.

[0120] S302: Windowing the second feature sequence to obtain a third feature sequence.

[0121] The third feature sequence includes a plurality of third features recorded in a specified time sequence, and the third feature is a combination of a plurality of second features that are continuous in time.

[0122] The so-called windowing means: dividing the signal into different time windows so as to apply Fourier transform or other spectrum analysis methods to each time window. In the related art, windowing is to reduce spectrum leakage, and a series of processing is performed on each time window to make it smoother. Spectrum leakage refers to the leakage of the spectrum energy of the signal into other frequencies during the spectrum estimation process, causing blurring or distortion of the spectrum peak. Windowing can reduce spectrum leakage, thereby improving the accuracy of spectrum estimation. Generally speaking, a window function is usually used to window the second characteristic sequence to obtain a third characteristic sequence. The window function used in this embodiment includes but is not limited to a rectangular window, a Hanning window, a Hamming window, and a Blackman window. Different window functions differ in amplitude attenuation or frequency domain resolution in different frequency bands, so the selected window function may depend on the characteristics and requirements of the signal.

[0123] In some examples, the implementation process of windowing the second feature sequence can be shown in formula (2).

[0124] Audio_signal=window_function*signal_flow (2)

[0125] In formula (2), Audio_signal represents the third feature, window_function represents the window function, and signal_flow represents the second feature.

[0126] It can be understood that windowing the second feature sequence is essentially filtering the second feature sequence.

[0127] S303: Perform Fourier transform on the third characteristic sequence to obtain a target spectrum.

[0128] Among them, the implementation principle and purpose of Fourier transform are conventional technical means in the field of mathematics, which will not be repeated here. It should be noted that since the second characteristic sequence is obtained by framing the target time series data, framing reduces the computing resource consumption of each Fourier transform. Therefore, the computing power consumed by Fourier transforming the third characteristic sequence is relatively small.

[0129] In some examples, by performing Fourier transform on the third characteristic sequence, a transform result x[i] can be obtained, and then the real part x[i] in the transform result x[i] is calculated. real and the imaginary part x[i] imag By performing square and root operations, the calculation result f[i] can be obtained. Finally, the target spectrum is generated based on the calculation result f[i].

[0130] In one possible implementation, for the real part x[i] real and the imaginary part x[i] imagThe implementation process of taking the square root can be seen in formula (3).

[0131] f[i] = sqrt(x[i] real 2 +x[i] imag 2 ) (3)

[0132] In formula (3), i∈[0, l / 2), l represents the frame length (ie, the number of acoustic wave features contained in the second feature).

[0133] The process shown in the above S301-S303 can convert the target time series data into a target spectrum, and realize the conversion of audio data from the time domain to the frequency domain, so as to obtain more reliable and intuitive audio features.

[0134] Embodiment 4

[0135] like Figure 4 As shown, it is a flow chart of another audio data processing method provided in an embodiment of the present application, including the following steps.

[0136] S401: Based on T target elements as input, determine the style probability distribution of the T target elements.

[0137] Among them, the audio features include T target elements, T represents the total number of first features contained in the first feature sequence, and each target element is used to characterize the first feature under a specified dimension (which can be regarded as a window feature). Based on the activation function in the target model, after determining T target elements as input, the style probability distribution of the T target elements can be obtained.

[0138] In some examples, the style probability distributions of the T target elements may be different.

[0139] In some examples, the style probability distribution can be expressed based on a 1×n matrix, where n represents the total number of music style categories. Therefore, the style probability distribution of T target elements can be expressed as shown in formula (4).

[0140] windows_probability[i]=[a 0 ,a 1 ,......,a n-1 ] 1×n (4)

[0141] In formula (4), i represents the index of each target element, [a 0 ,a 1 ,......,a n-1 ] 1×n Represents a matrix, and each value in the matrix represents the probability value of each type of music style.

[0142] S402: For each target element, based on the music style with the highest probability value in the style probability distribution, determine the music style of the target element.

[0143] The implementation process of determining the music style of the target element based on the music style with the highest probability value in the style probability distribution can be shown in formula (5).

[0144] music_gen[i]=locate(max{windows_probability[i]}) (5)

[0145] In formula (5), music_gen[i] represents the music style of each target element.

[0146] S403: Determine a target music style as output from the music styles of the T target elements.

[0147] Among them, the proportion of the target music style is the highest among the music styles of the T target elements.

[0148] In some examples, the implementation process of determining the target music style as the output from the music styles of T target elements can be shown in formula (6).

[0149] result=max_num(music_gen[i]) (6)

[0150] In a possible implementation, the T target elements include element A, element B, element C, and element D, wherein the music style of element A is hip-hop, the music style of element B is hip-hop, the music style of element C is pop, and the music style of element D is hip-hop. Obviously, the proportion of hip-hop is the highest among all music styles, so the target music style is determined to be hip-hop.

[0151] The process shown in S401-S403 above uses the style probability distribution of multiple first features to infer the target music style, so that the recognition ability of the target model is effectively improved.

[0152] Embodiment 5

[0153] like Figure 5 As shown, it is a flow chart of another audio data processing method provided in an embodiment of the present application, including the following steps.

[0154] S501: Determine a two-dimensional tensor based on the first feature sequence.

[0155] The two-dimensional tensor is composed of R×L scalar elements, R represents the total number of first features contained in the first feature sequence, L represents the total number of frequencies contained in the first feature, and the scalar element represents the frequency.

[0156] S502: Expand and stack the two-dimensional tensor to obtain audio features.

[0157] The audio feature includes T target elements recorded in a specified time sequence, T and R have the same value, that is, the number of target elements can be determined based on the first feature included in the first feature sequence, and the data dimension of the target element is k×m, k and L have the same value, and m represents the number of stacking times. By increasing the dimension of the target elements in the audio feature through dimension expansion and stacking, the data enhancement of the audio feature can be achieved.

[0158] Generally speaking, the implementation principles of dimension expansion and stacking are conventional technical means in the field of mathematics and will not be elaborated here.

[0159] The process shown in S501-S503 above can perform data enhancement on the first feature sequence again to achieve data enhancement on the audio features.

[0160] Embodiment 6

[0161] Corresponding to the audio data processing method provided in the above-mentioned embodiment of the present application, the embodiment of the present application also provides an audio data processing device.

[0162] like Figure 6 , which is a schematic diagram of the architecture of an audio data processing device provided in an embodiment of the present application, including the units shown below.

[0163] The audio determination unit 100 is used to determine the audio data required for the target model training process; the target model is used to determine the musical style of the audio data based on the audio features with time series attributes extracted from the audio data.

[0164] The spectrum determination unit 200 is used to determine a target spectrum based on the audio data; the target spectrum includes a plurality of frequencies, and the plurality of frequencies are recorded in a specified time sequence.

[0165] The windowing processing unit 300 is used to divide and splice the target spectrum into windows to obtain a first feature sequence; the first feature sequence includes multiple first features recorded in a specified time sequence, and there is first overlapping data between two first features that are consecutive in time; the first feature is a combination of multiple frequencies that are consecutive in time; the first overlapping data represents the frequency corresponding to the same timestamp.

[0166] The feature determination unit 400 is used to determine the audio feature based on the first feature sequence.

[0167] Optionally, the spectrum determination unit 200 is specifically used to: determine target time series data based on audio data; the target time series data includes multiple sound wave features, and the multiple sound wave features are recorded in a specified time sequence; determine the target spectrum based on the target time series data.

[0168] Optionally, the spectrum determination unit 200 is specifically used to: frame the target time series data to obtain a second feature sequence; the second feature sequence includes multiple second features recorded in a specified time sequence; the second feature is a combination of multiple acoustic wave features that are continuous in time, and there is second overlapping data between two second features that are continuous in time, and the second overlapping data includes acoustic wave features corresponding to the same timestamp; window the second feature sequence to obtain a third feature sequence; the third feature sequence includes multiple third features recorded in a specified time sequence; the third feature is a combination of multiple second features that are continuous in time; and perform Fourier transform on the third feature sequence to obtain the target spectrum.

[0169] Optionally, the target model includes a temporal convolutional network.

[0170] Optionally, the feature determination unit 400 is specifically used to: determine a two-dimensional tensor based on a first feature sequence; wherein the two-dimensional tensor is composed of R×L scalar elements, R represents the total number of first features contained in the first feature sequence, L represents the total number of frequencies contained in the first feature, and the scalar element represents the frequency; expand and stack the two-dimensional tensor to obtain audio features; the audio features include T target elements recorded in a specified time sequence, T and R have the same value, the data dimension of the target element is k×m, k and L have the same value, and m represents the number of stacking times.

[0171] Optionally, the target model includes an activation function, which is used to determine the style probability distribution of the audio feature; the style probability distribution is used to characterize the probability values ​​of various music styles.

[0172] Optionally, the target model is specifically used to: determine the style probability distribution of T target elements based on T target elements as input; for each target element, determine the music style of the target element based on the music style with the highest probability value in the style probability distribution; determine the target music style as the output from the music styles of the T target elements, wherein the proportion of the target music style is the highest among the music styles of the T target elements.

[0173] Each unit shown above realizes data enhancement of audio features by windowing and splicing the target spectrum. After data enhancement processing, the long sequence of audio data will be split into multiple data segments with overlapping parts, which can be inferred multiple times, and each data segment is related, which can effectively prevent the misidentification caused by a single inference of the target model and improve the recognition accuracy of the target model. In addition, compared with the common method of using CNN network to classify music styles, a time series prediction mechanism is added, the time series information of audio data is considered, and the algorithm accuracy is improved. Compared with time series prediction mechanisms such as LSTM and RNN, the use of TCN mechanism for time series prediction can improve the inference rate.

[0174] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the audio data processing method provided by the present application.

[0175] The present application also provides an electronic device, including: a processor, a memory and a bus. The processor and the memory are connected via the bus, the memory is used to store programs, and the processor is used to run the programs, wherein the audio data processing method provided by the present application is executed when the programs are run.

[0176] In addition, the functions described above in the embodiments of the present application may be at least partially performed by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0177] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0178] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. An audio data processing method, characterized in that: include: Determine the audio data required for the target model training process; The target model is used to: determine the music style of the audio data based on the audio features with time series attributes extracted from the audio data; Based on the audio data, a target spectrum is determined; the target spectrum includes a plurality of frequencies, and the plurality of frequencies are recorded in a specified time sequence; The target spectrum is windowed and spliced ​​to obtain a first feature sequence; the first feature sequence includes a plurality of first features recorded in a specified time sequence, and there is first overlapping data between two first features that are consecutive in time; the first feature is a combination of a plurality of frequencies that are consecutive in time; the first overlapping data represents the frequency corresponding to the same timestamp; The audio feature is determined based on the first feature sequence.

2. The method according to claim 1, characterized in that Based on the audio data, determining a target frequency spectrum includes: Based on the audio data, target time series data is determined; the target time series data includes a plurality of sound wave features, and the plurality of sound wave features are recorded in a specified time sequence; Based on the target time series data, a target frequency spectrum is determined.

3. The method according to claim 2, characterized in that Determining a target frequency spectrum based on the target time series data includes: The target time series data is framed to obtain a second feature sequence; the second feature sequence includes a plurality of second features recorded in a specified time sequence; the second feature is a combination of a plurality of acoustic wave features that are continuous in time; wherein there is second overlapping data between two second features that are continuous in time, and the second overlapping data includes acoustic wave features corresponding to the same timestamp; Windowing the second feature sequence to obtain a third feature sequence; the third feature sequence includes a plurality of third features recorded in a specified time sequence; the third feature is a combination of a plurality of second features that are continuous in time; Perform Fourier transform on the third characteristic sequence to obtain a target spectrum.

4. The method according to claim 1, characterized in that The target model includes a time-domain convolutional network.

5. The method according to claim 1, characterized in that Determining the audio feature based on the first feature sequence includes: Based on the first feature sequence, determine a two-dimensional tensor; wherein the two-dimensional tensor is composed of R×L scalar elements, R represents the total number of first features included in the first feature sequence, L represents the total number of frequencies included in the first feature, and the scalar element represents the frequency; The two-dimensional tensor is expanded and stacked to obtain the audio feature; the audio feature includes T target elements recorded in a specified time sequence, T and R have the same value, the data dimension of the target element is k×m, k and L have the same value, and m represents the number of stacking times.

6. The method according to claim 5, characterized in that The target model includes an activation function, which is used to determine the style probability distribution of the audio feature; the style probability distribution is used to characterize the probability values ​​of various types of music styles.

7. The method according to claim 6, characterized in that The target model is specifically used to: determine the style probability distribution of the T target elements based on the T target elements as input; for each target element, determine the music style of the target element based on the music style with the highest probability value in the style probability distribution; and determine the target music style as output from the music styles of the T target elements, wherein the number proportion of the target music style is the highest among the music styles of the T target elements.

8. An audio data processing device, characterized in that: include: An audio determination unit, used to determine audio data required for a target model training process; The target model is used to: determine the music style of the audio data based on the audio features with time series attributes extracted from the audio data; A spectrum determination unit, configured to determine a target spectrum based on the audio data; the target spectrum includes a plurality of frequencies, and the plurality of frequencies are recorded in a specified time sequence; A data enhancement unit is used to perform window division and splicing on the target spectrum to obtain a first feature sequence; the first feature sequence includes a plurality of first features recorded in a specified time sequence, and there is first overlapping data between two first features that are continuous in time; the first feature is a combination of a plurality of frequencies that are continuous in time; the first overlapping data represents the frequency corresponding to the same timestamp; A feature determination unit is used to determine the audio feature based on the first feature sequence.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program, when executed by a processor, executes the audio data processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: processor, memory, and bus; The processor is connected to the memory via the bus; The memory is used to store programs, and the processor is used to run programs, wherein the program, when run by the processor, executes the audio data processing method according to any one of claims 1 to 7.

Images