An audio segmentation and classification method based on multi-granularity slicing

Abstract

This invention discloses an audio segmentation and classification method based on multi-granularity slicing, comprising: preprocessing the audio to obtain an audio file with a uniform sampling rate; slicing the audio file at different time granularities; extracting MFCC features from each slice at different time granularities and then performing image processing; establishing an image classification convolutional neural network model and training and validating it; inputting the processed audio into the image classification convolutional neural network model to obtain the classification result of each slice; and performing aggregation analysis based on the classification results to obtain the segmentation points and segment types of the audio file. This invention, by segmenting long audio files at different time granularities, utilizing an image classification convolutional neural network model for type judgment and classification, and finally performing aggregation analysis, can quickly and accurately find the segmentation points between different types of audio and determine the audio types of the audio segments before and after the segmentation points.

Description

Technical Field

[0001] This invention belongs to the field of computer auditory technology, specifically relating to an audio segmentation and classification method based on multi-granularity slices. Background Technology

[0002] Audio segmentation and classification technology refers to the use of signal processing and pattern recognition methods to segment continuous content in long audio files into segments and identify the type of each segment. Currently, commonly used audio segmentation methods include finding abrupt changes in audio data or features for segmentation, and performing local similarity analysis on the audio signal before segmentation. Common classification methods primarily employ neural network classification models to categorize and manage the segmented audio. However, these segmentation and classification methods cannot automatically segment audio files of varying lengths or with different precision requirements. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing an audio segmentation and classification method based on multi-granularity slicing. This method segments and slices audio at different time granularities, classifies the slices, and then aggregates them by type to obtain the audio cutting points and segment types. It is suitable for automatic segmentation of audio with different lengths and different precision requirements.

[0004] The technical solution adopted in this invention is: an audio segmentation and classification method based on multi-granularity slices, comprising the following steps:

[0005] S1: Preprocess the audio to obtain an audio file with a uniform sampling rate;

[0006] S2: For audio files with a uniform sampling rate, slice them sequentially from beginning to end according to the corresponding time granularity at different time granularities;

[0007] S3: Extract MFCC features from each slice at different time granularities, and visualize the extracted feature data to obtain MFCC feature images;

[0008] S4: Establish an image classification convolutional neural network model and construct a sample set of audio. Process 80% of the audio in the sample set according to the above steps S1-S3 and associate it with the corresponding classification type label. Then input it into the image classification convolutional neural network model to complete the training of the image classification convolutional neural network model. Use 20% of the audio in the sample set as a test set to verify the trained image classification convolutional neural network model. After verification, the final image classification convolutional neural network model is obtained.

[0009] S5: Process the audio to be cut according to the above steps S1-S3 to obtain the MFCC feature image of each slice at different time granularities, and input the obtained MFCC feature image into the final image classification convolutional neural network model to obtain the classification result of each slice;

[0010] S6: By aggregating and analyzing the classification results of each slice at different time granularities, the segmentation points and segment types of the audio based on the minimum granularity accuracy are obtained.

[0011] Preferably, step S1, the preprocessing of the audio, includes the following steps:

[0012] S11: Select the appropriate decoder based on the audio file format to decode and obtain audio sampling data;

[0013] S12: Save the audio sampling data as an uncompressed WAV audio file;

[0014] S13: Analyze the sampling rate of the audio file to ensure that the sampling rate of the audio file is uniform, and resample the audio that does not meet the requirements.

[0015] Preferably, the time granularity in step S2 is determined by defining the audio frame length and number of frames of the audio file.

[0016] Preferably, the audio frame lengths defined in different time granularities in step S2 are the same, the minimum time granularity is determined according to the number of frames, and the remaining time granularities are multiples of the minimum time granularity.

[0017] Preferably, the MFCC feature extraction in step S3 includes sequentially performing audio data pre-emphasis, audio signal windowing, audio signal discrete Fourier transform, Mel filtering, calculating Fbank features, inverse discrete cosine transform calculation, difference and energy, and composing MFCC features.

[0018] Preferably, the aggregation analysis of the classification results in step S6 includes parallel aggregation analysis and serial aggregation analysis.

[0019] Preferably, the parallel aggregation analysis includes: performing classification aggregation analysis on classification slices at different time granularities simultaneously.

[0020] Preferably, the serial aggregation analysis includes: performing classification aggregation analysis on the classification slices at different time granularities in sequence from the largest time granularity to the smallest time granularity.

[0021] The beneficial effects of this invention are as follows: This invention segments long audio files using different time granularities, classifies and aggregates the audio slices generated by the segments at different time granularities after type determination, and finally performs aggregate analysis on the classified audio slices, thereby quickly and accurately finding the cutting points between different types of audio and determining the audio types of the audio segments before and after the cutting points. Attached Figure Description

[0022] Figure 1 This is a flowchart of the processing flow of the present invention;

[0023] Figure 2 This is a schematic diagram of the polymerization analysis in Embodiment 1 of the present invention;

[0024] Figure 3 This is a schematic diagram of the aggregation analysis in Embodiment 2 of the present invention. Detailed Implementation

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0026] Example

[0027] like Figure 1 As shown, the audio segmentation and classification method based on multi-granularity slices provided in this embodiment includes the following steps:

[0028] S1: Audio preprocessing: Select the appropriate decoder according to the audio file format to decode and obtain audio sampling data, and then save the audio sampling data as an uncompressed WAV audio file; perform audio sampling rate analysis on the saved uncompressed WAV audio file, and resample audio files that do not meet the standard to ensure that the sampling rate of the preprocessed audio files is uniform; in this embodiment, a sampling rate of 44.1KHz is used as the uniform sampling rate for audio files;

[0029] S2: For audio files with a uniform sampling rate, slice them sequentially from beginning to end according to the corresponding time granularity at different time granularities. In this embodiment, different time granularities are determined by defining the same audio frame length and different frame numbers. Simultaneously, using other frame numbers as multiples of the minimum frame number to obtain different time granularities effectively improves the efficiency of subsequent aggregation analysis. In this embodiment, the audio frame length is defined as 26ms, and the frame number is selected as 10, 20, 40, and 80 frames, i.e., the four different time granularities are 260ms, 520ms, 1040ms, and 2080ms, respectively. This means slicing is performed according to different time granularities, i.e., the same audio file is sliced ​​in 260ms... At a time granularity of s, each audio segment is 260ms; at 520ms, each audio segment is 520ms; at 1040ms, each audio segment is 1040ms; and at 2080ms, each audio segment is 2080ms. Since the audio is divided into several segments from beginning to end under different time granularity definitions, the number of segments is the total duration of the audio divided by the duration of each segment. The calculated number of segments is rounded down. Finally, there will be a special segment whose duration is less than the duration of each segment. If the total duration of the audio divided by the duration of each segment is an integer, then the special segment at the end does not exist.

[0030] S3: Perform the following processing on each slice at different time granularities in sequence:

[0031] Audio data pre-emphasis processing: High-frequency energy in the speech spectrum experiences significant attenuation during propagation; the higher the sound frequency, the greater the attenuation. Therefore, pre-emphasis is needed to compensate for the attenuation. The pre-emphasis calculation formula for the speech sample value k[n] at time n is as follows:

[0032] H[n]=k[n]-α*k[n-1]α∈[0.9, 1]

[0033] The pre-weighting coefficient α typically ranges from 0.9 to 1.

[0034] Audio signal windowing: Since the Discrete Fourier Transform operation in the subsequent processing is performed under the assumption that the audio signal is a periodic signal, direct processing may result in spectral leakage. Therefore, windowing the audio signal can make the short audio data segments periodic, which can be better processed by Discrete Fourier Transform in the future.

[0035] The audio is segmented into M segments of length N by windowing. The Hamming window function is then used to process each segment, converting it into signal data with periodic characteristics.

[0036]

[0037] When h0 = 0.53836, L = N-1, the Hamming window function can be obtained;

[0038] Suppose that at time n of a certain audio segment, the value of the signal is:

[0039] y[n]=x[n]*ω[n]

[0040] After the above operations, each segment of the audio signal will resemble a periodic signal;

[0041] The Discrete Fourier Transform (DFT) of an audio signal is used to map a windowed audio signal segment from the time domain to the frequency domain. Assuming the number of sampling points after windowing is N, the DFT is performed using the following formula:

[0042]

[0043] Mel filtering: Fourier transform yields the amplitude of the signal at different frequency components, and squaring these components gives the energy corresponding to each frequency component. Since the human ear is more sensitive to low frequencies and less sensitive to high frequencies above 1000Hz, mapping frequencies to Mel frequencies simulates the human ear's response to audio signals. The mapping relationship between Mel frequencies and frequencies is expressed by the following formula:

[0044]

[0045] Calculate the Fbank feature: Taking the logarithm of the Mel feature Y[m] from the previous step yields the Fbank feature, i.e., the logarithmic power spectrum, which contains both low-frequency and high-frequency information.

[0046] Fbank = log Y[m]

[0047] Inverse Discrete Cosine Transform Calculation: Cepstral Analysis of Logarithmic Power Spectrum Using Inverse Discrete Cosine Transform

[0048]

[0049] Difference and Energy: For each frame, the first 12 cepstral coefficients calculated using inverse discrete cosine transform are selected. Then, the energy is used as the 13th feature. The energy feature can be obtained from the windowed time-domain signal. Assuming the window length is [t1, t2], the energy is:

[0050]

[0051] Features can characterize acoustic properties, and the changes of features over time can also characterize acoustic properties. We add the changes of features over time to the original 13-dimensional features, that is, we calculate δ and doubleδ.

[0052] The δ feature represents the change in cepstral coefficients and energy between frames, while doubleδ represents the change in the δ feature between frames. The original MFCC feature of cepstral coefficients and energy is represented as c[n], and the δ feature and doubleδ feature can be represented as:

[0053]

[0054]

[0055] MFCC feature combination: The 12 cepstral coefficients, 12 delta cepstral coefficients, 12 doubleδ cepstral coefficients, 1 energy, 1δ energy, and 1 doubleδ energy obtained above can form a 39-dimensional MFCC feature.

[0056] The MFCC features obtained from each audio segment are visualized to obtain the MFCC feature image of each audio segment.

[0057] S4: Establish an image classification convolutional neural network model and construct a sample set of audio.

[0058] Select a sufficient number of known audio files and manually segment and label them according to time range. Label each audio file with a type, such as music, speech, song, noise; or more detailed classifications, such as soft music, high-pitched music, male songs, female songs, male speech, female speech, noise, etc.

[0059] The selected known audio is sliced ​​according to the steps S1 and S2 above. Each slice is mapped to the marked segment according to its time period (slices at the marked points are discarded to improve the accuracy of training data). This allows all slices to be labeled with type in batches.

[0060] By performing the above S3 step on the slices labeled with type, the MFCC feature images can be processed and each MFCC feature image can be assigned a type label to obtain the sample set audio data.

[0061] Therefore, the training set of the image classification convolutional neural network model at this time granularity is obtained by using MFCC feature images with type labels from 80% of the sample set data. After training, the MFCC feature images with type labels from the remaining 20% ​​of the sample set data are input into the trained image classification convolutional neural network model for verification. After the verification achieves the required classification effect, the final image classification convolutional neural network model is obtained.

[0062] In this embodiment, four time granularities are defined: 260ms, 520ms, 1040ms, and 2080ms. Therefore, in the image classification convolutional neural network model, a training set corresponding to each time granularity should be established. Thus, in the image classification convolutional neural network model, the training set corresponding to each time granularity corresponds to the classification result type array corresponding to the corresponding time granularity.

[0063] S5: Process the audio to be cut according to the above steps S1-S3 to obtain the MFCC feature image of each slice at four time granularities, that is, to obtain four sets of MFCC feature images. Input the four sets of MFCC feature images into the final image classification convolutional neural network model for parallel processing to obtain the classification results of the four audio slices.

[0064] S6: As Figure 2 As shown, parallel aggregation analysis is performed on the classification results of four sets of audio slices. That is, all classified slices at the same time granularity are aggregated and connected according to the audio timeline, and synchronous analysis begins. Specifically, the cutting point is determined from the aggregation of the largest granularity classified slices, and the audio slices before and after this cutting point are aggregated and analyzed at the next time granularity, and so on, until the smallest time granularity, thus obtaining the cutting point and segment type of the audio file at the smallest time granularity. Parallel processing requires sufficient computing resources, which can realize layer-by-layer parallel aggregation analysis of each granularity at different time granularities, effectively improving the efficiency of audio segmentation.

[0065] Example 2

[0066] This embodiment is basically the same as Embodiment 1, except that, under conditions of insufficient computing power, a serial aggregation analysis is performed on the classification results of the four sets of audio slices, such as... Figure 3 As shown, the aggregation analysis starts from the largest granularity classification slice to obtain the next layer of time granularity slices covered by the slices before and after the cutting point. The image classification convolutional neural network model is calculated iteratively to finally obtain the audio segmentation point and segment type of the smallest time granularity. The serial processing effectively reduces the computational load of the image classification convolutional neural network model, which is suitable for use when computing resources are insufficient. However, because it performs layer-by-layer serial slice classification aggregation analysis according to different time granularities, its efficiency is low.

[0067] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications and substitutions based on the technical solutions and inventive concepts provided by the present invention should be covered within the scope of protection of the present invention.

Claims

1. An audio segmentation and classification method based on multi-granularity slices, characterized in that: Includes the following steps: S1: Preprocess the audio to obtain an audio file with a uniform sampling rate; S2: For audio files with a uniform sampling rate, slice them sequentially from beginning to end according to the corresponding time granularity at different time granularities; S3: Extract MFCC features from each slice at different time granularities, and visualize the extracted feature data to obtain MFCC feature images; S4: Establish an image classification convolutional neural network model and construct a sample set of audio. Process 80% of the audio in the sample set according to the above steps S1-S3 and associate it with the corresponding classification type label. Then input it into the image classification convolutional neural network model to complete the training of the image classification convolutional neural network model. Use 20% of the audio in the sample set as a test set to verify the trained image classification convolutional neural network model. After verification, the final image classification convolutional neural network model is obtained. S5: Process the audio to be cut according to the above steps S1-S3 to obtain the MFCC feature image of each slice at different time granularities, and input the obtained MFCC feature image into the final image classification convolutional neural network model to obtain the classification result of each slice; S6: By aggregating and analyzing the classification results of each slice at different time granularities, the segmentation points and segment types of the audio based on the minimum granularity accuracy are obtained.

2. The audio segmentation and classification method based on multi-granularity slices according to claim 1, characterized in that: Step S1, the preprocessing of the audio, includes the following steps: S11: Select the appropriate decoder based on the audio file format to decode and obtain audio sampling data; S12: Save the audio sampling data as an uncompressed WAV audio file; S13: Analyze the sampling rate of the audio file to ensure that the sampling rate of the audio file is uniform, and resample the audio that does not meet the requirements.

3. The audio segmentation and classification method based on multi-granularity slices according to claim 1 or 2, characterized in that: The time granularity in step S2 is determined by defining the audio frame length and number of frames of the audio file.

4. The audio segmentation and classification method based on multi-granularity slices according to claim 3, characterized in that: In step S2, the audio frame lengths defined in different time granularities are the same. The minimum time granularity is determined based on the number of frames, and the remaining time granularities are multiples of the minimum time granularity.

5. An audio segmentation and classification method based on multi-granularity slices according to claim 1 or 4, characterized in that: Step S3, MFCC feature extraction, includes sequentially performing audio data pre-emphasis, audio signal windowing, audio signal discrete Fourier transform, Mel filtering, Fbank feature calculation, inverse discrete cosine transform calculation, difference and energy, and composition of MFCC features.

6. The audio segmentation and classification method based on multi-granularity slices according to claim 1, characterized in that: The aggregation analysis of classification results in step S6 includes parallel aggregation analysis and serial aggregation analysis.

7. The audio segmentation and classification method based on multi-granularity slices according to claim 6, characterized in that: The parallel aggregation analysis includes performing classification aggregation analysis on classification slices at different time granularities simultaneously.

8. The audio segmentation and classification method based on multi-granularity slices according to claim 6, characterized in that: The serial aggregation analysis includes performing classification aggregation analysis on the classification slices at different time granularities in sequence from the largest time granularity to the smallest time granularity.

Images