A hierarchical fusion audio data enhancement method and system

Abstract

The present invention proposes a hierarchical fusion audio data enhancement method and system, the method comprising: collecting an original audio signal X; using a WSOLA algorithm to perform time domain signal compression and expansion on the audio signal X to obtain a compressed and expanded audio signal X o ; The processed audio X o Mixed with the original audio X to form a new training set S x ; Extract the frequency of each audio in the training set; Use the SWIPE algorithm to estimate the fundamental frequency and construct the fundamental frequency set S f ; Use the perturbed fundamental frequency to normalize the frequency and construct the frequency set S F ; Use fast Fourier transform to extract acoustic features. The audio data enhancement method proposed in the present invention improves the recognition performance of the model for noisy audio and can be applied to a variety of speech tasks, including but not limited to: audio classification, voiceprint recognition and speech recognition.

Description

Technical Field

[0001] The invention relates to an audio data enhancement method applicable to a variety of audio tasks, and belongs to the field of audio data processing. Background Art

[0002] Currently, most audio tasks rely on the amount of labeled data. Generally, the larger the amount of data, the better the model effect. For audio tasks under low-resource conditions, data augmentation is a simple and effective way to construct new samples. Using data augmentation technology, the model can extract stable speech representations under small sample conditions, and the recognition effect is greatly improved compared to the original training method.

[0003] Existing research focuses on data enhancement in both front-end and feature aspects to improve recognition performance in unknown environments. Adding parameterized reverberation, offset, and speed perturbations to the original audio can simulate noise in real environments. Using this method for data enhancement can greatly improve the robustness of the model. In addition, amplifying data using vocal tract length perturbation technology has also been proven to be effective. In addition, data enhancement methods based on signal compression have been successfully used in the field of sound attack detection. This method achieves data enhancement by compressing and expanding the original signal based on the a-law and μ-law signal compression methods.

[0004] In addition to data enhancement at the front end, audio features can also be enhanced. Existing research has achieved data enhancement of audio features based on fundamental frequency normalization, which constructs multiple similar frequencies by adding different degrees of perturbation to the fundamental frequency. This method can achieve good results in speech recognition tasks. Summary of the invention

[0005] The technical problem to be solved by the present invention is: for audio classification tasks: on the one hand, the cost of obtaining a large amount of external noise data to achieve data enhancement is high; on the other hand, the samples constructed by the existing methods are relatively limited.

[0006] In order to solve the above technical problems, a technical solution of the present invention is to provide a hierarchical fusion audio data enhancement method, which is characterized by comprising the following steps:

[0007] a) Collect the original signal and save it in the form of digital signal as audio X;

[0008] b) Perform time domain signal compression and expansion on audio X to obtain the compressed and expanded audio X o :

[0009] When compressing and expanding the audio signal X in the time domain, the waveform similarity overlap superposition algorithm is used to introduce a series of distortions while retaining the harmonic signal, thereby obtaining the compressed and expanded audio signal X. o ;

[0010] c) The compressed and expanded audio X o Mixed with the original audio X to form a new training set S x ;

[0011] d) For the training set S x Perform frequency extraction on each audio in to obtain frequency f;

[0012] e) Extract the base frequency of each audio and obtain the base frequency f o，def ;

[0013] d) For the fundamental frequency f o，def Perturbations are added to form the base frequency set S f :

[0014] For the fundamental frequency f o，def Add disturbances, add frequency offsets of ±20, ±40 and ±60 respectively, and the perturbed fundamental frequency f o With the original fundamental frequency f o，def Together they form the baseband set S f ;

[0015] e) Use base frequency set S f Normalize the frequency and construct the frequency set S F :

[0016] Use the base frequency set S f Normalize the frequency features. For the spectrum graph corresponding to the current audio, use step d) to extract the fundamental frequency of each frame and calculate the median of the fundamental frequency, denoted as f o，audio , and then use the base frequency set S f Each value in normalizes the frequency value on the Mel scale:

[0017] f norm =f orig -(f o，audio -f o，def )

[0018] In the formula, f orig Indicates the frequency value in Mel scale, f o，audio Indicates the median of the fundamental frequency of all frames in the current audio, f o，def Indicates the default fundamental frequency;

[0019] The normalized f norm The frequency set S F ;

[0020] f) Use frequency set S F Perform acoustic feature extraction

[0021] Use frequency set S FThe elements in the audio signal are converted into energy distribution in the frequency domain by fast Fourier transform. Different energy distributions can represent the characteristics of different speech.

[0022] Preferably, any audio frame in the original audio X is defined as the first audio frame, and step b) specifically includes the following steps:

[0023] Select a second audio frame within the left and right range of the first audio frame, and align the phase parameter of the second audio frame with the phase of the first audio frame;

[0024] Search for the third audio frame in the range [-Δmax, Δmax], where Δmax is set to half an audio period, and then calculate the correlation coefficients between frames in the range, and select the audio frame with the highest similarity to the second audio frame as the third audio frame;

[0025] The first audio frame, the second audio frame, and the third audio frame are concatenated with the same step length, and the overlapping parts are added.

[0026] Preferably, in step c), the compressed and expanded audio X o Add the same annotations as the original audio X and add them to the original dataset to form a new training set S x .

[0027] Preferably, in step d), the training set S x The audio in is framed, windowed and Mel-scaled to extract the audio frequency features and obtain the frequency f.

[0028] Preferably, in step e), the SWIPE algorithm is used to estimate the fundamental frequency, and the significance of the peak value of the amplitude spectrum at each integer multiple of the frequency f relative to the two valley values ​​adjacent to it is measured by the peak-to-valley distance, and the function is defined as:

[0029]

[0030] Where: d k (f) represents the peak-to-valley distance; k=1, 2, ..., n, represents the multiple, and |X(kf)| represents the peak value of the amplitude spectrum at k times the frequency f;

[0031] The significance is expressed by the average peak-to-valley distance of each harmonic, as shown in the following formula:

[0032]

[0033] The final estimated fundamental frequency based on the significance is expressed as f o，def , the fundamental frequency f o，def Used for subsequent feature normalization.

[0034] Another technical solution of the present invention is to provide a hierarchical fusion audio data enhancement system, characterized in that it includes:

[0035] Signal compression and expansion unit: used to compress and expand the time domain signal of audio X to obtain the compressed and expanded audio X o , when compressing and expanding the audio X in the time domain, a waveform similarity overlap algorithm is used to introduce a series of distortions while retaining the harmonic signal, thereby obtaining the compressed and expanded audio X o ;

[0036] Training set construction unit: used to compress and expand the audio X o Mixed with the original audio X to form a new training set S x ;

[0037] Frequency extraction unit: used to extract training set S x The frequency of each audio in is obtained, and the frequency f is obtained;

[0038] Base frequency extraction unit: used to extract the base frequency of each audio and obtain the base frequency f o，def ;

[0039] Base frequency disturbance adding unit: used to add the base frequency f o，def Perturbations are added to form the base frequency set S f , where for the fundamental frequency f o，def When adding disturbance, frequency offsets of ±20, ±40, and ±60 are added respectively, and the resulting fundamental frequency f after disturbance is o With the original fundamental frequency f o，def Together they form the baseband set S f ;

[0040] Frequency normalization unit: used to use the base frequency set S f Normalize the frequency and construct the frequency set S F , where the base frequency set S is used f When normalizing the frequency features, for the spectrum corresponding to the current audio, use step d) to extract the fundamental frequency of each frame and calculate the median of the fundamental frequency, denoted as f o，audio , and then use the base frequency set S f Each value in normalizes the frequency value on the Mel scale:

[0041] f norm =f orig -(f o，audio -f o，def )

[0042] In the formula, f orig Indicates the frequency value in Mel scale, f o，audio Indicates the median of the fundamental frequency of all frames in the current audio, fo，def Represents the default fundamental frequency; the normalized f norm The frequency set S F

[0043] Acoustic feature extraction unit: used to use the frequency set S F To extract acoustic features, the frequency set S F The signal is transformed into energy distribution in the frequency domain by fast Fourier transform based on the elements in . Different energy distributions can represent the characteristics of different speech.

[0044] The present invention proposes a hierarchical fusion data enhancement method, which first performs compression and expansion operations on the original audio signal. Time-scale modification (TSM) is a technology that can change the speed of audio without changing its pitch, and is very important in current audio signal processing. The TSM algorithm divides a segment of audio signal into different frames, and then performs a series of processing on each frame, such as stretching, compression, etc., and finally re-superimposes these frames into a synthetic signal. The present invention implements compression and expansion of audio signals based on a waveform similarity overlapping superposition algorithm, and uses this method for data enhancement, applies the processed signal to the construction of new audio, and realizes data enhancement without using additional audio. These audio signals will be used for subsequent feature extraction. During the extraction process, feature-level data enhancement will be performed, and it is considered to add small disturbances to the sound frequency to improve the robustness of the model against noise. The method based on fundamental frequency normalization can be used to construct new samples similar to the original audio, reducing the impact of audio differences caused by environmental noise on recognition results.

[0045] The present invention proposes a hierarchical fusion audio data enhancement method, which integrates the existing audio data enhancement method and divides the enhancement into different stages. The method can be used for a variety of audio tasks and has a certain degree of universality. The present invention can achieve audio data enhancement without using additional noise data, reducing the dependence of existing data enhancement methods on external noise data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of an audio data enhancement method of the present invention;

[0047] Figure 2 It is a calculation flow chart of audio feature data enhancement of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with specific examples. It should be understood that these examples are only to help those of ordinary skill in the art understand the principles and knowledge of the present invention, and are not intended to limit the scope of the present invention, and cannot be considered to limit the application scenarios of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art may make various changes or modifications to the present invention, but the deformation, changes and conversions made to the embodiments based on the principles and purposes of the present invention also fall within the scope defined by the claims attached to this application. And it is obvious that this specification is only exemplified by preferred embodiments, and there is no need to exhaust all embodiments.

[0049] like Figure 1 As shown, the present invention proposes a set of audio data enhancement method processes, including: audio signal data enhancement and audio feature data enhancement. The audio signal data enhancement includes signal compression and expansion and training set construction; the audio feature data enhancement includes frequency extraction, baseband extraction, baseband perturbation addition, frequency normalization and acoustic feature extraction. The specific implementation process is as follows:

[0050] Step 1: Use the sensor to collect the original signal and save it as a digital signal as audio X.

[0051] Step 2: Perform time domain signal compression and expansion on audio X to obtain compressed and expanded audio X o

[0052] The waveform similarity overlap superposition algorithm is used to introduce a series of distortions while retaining the harmonic signal, thereby obtaining the compressed and expanded audio X o Define any audio frame in the original audio as the first audio frame, and the specific operations of the second step are as follows:

[0053] (1) selecting a second audio frame within the left and right range of the first audio frame, and aligning a phase parameter of the second audio frame with a phase parameter of the first audio frame;

[0054] (2) Searching for the third audio frame in the range [-Δmax, Δmax], where Δmax is set to half an audio period, and then calculating the correlation coefficients between the frames in the range, and selecting the audio frame with the highest similarity to the second audio frame as the third audio frame;

[0055] (3) The first audio frame, the second audio frame, and the third audio frame are concatenated with the same step length, and the overlapping parts are added.

[0056] Step 3: X the compressed audio o Mixed with the original audio X to form a new training set S x

[0057] The compressed audio Xo Add the same annotations as the original audio X and add them to the original dataset to form a new training set S x .

[0058] Step 4: Training set S x Perform frequency extraction on each audio in to obtain frequency f;

[0059] For the training set S x The audio in the video is framed, windowed, and Mel-scaled to extract the audio frequency features. The basic process is as follows: Figure 2 As shown, the specific operations are as follows:

[0060] (1) Audio framing

[0061] Since the audio signal is unstable as a whole, it is impossible to directly perform frequency domain conversion on the entire speech. Framing divides a segment of audio into multiple shorter segments, and each short audio segment is considered stable. After framing, the speech can be considered as a stable signal. In this embodiment, 25ms is used as the frame length, but in order to avoid signal mutations between frames, a frame is taken every 10ms, that is, the frame shift is 10ms.

[0062] (2) Windowing

[0063] Use the window function to slide on the split frames so that the amplitude of a frame signal gradually changes to zero at both ends, thereby highlighting the middle part of a frame. This can improve the resolution of the subsequent Fourier transform results. Since the window function will attenuate the amplitude at both ends of each frame, and the framing will cause overlapping parts between adjacent frames, so that each sampling point can be highlighted by the window function.

[0064] In this embodiment, the window function w(n) uses a Hamming window, as shown in the following formula:

[0065]

[0066] Wherein: n represents the position of the n+1th sampling point on a frame signal; N represents the Hamming window length, that is, the total number of sampling points, which is taken as 256 in this embodiment.

[0067] (3) Mel-scale conversion

[0068] For the frequencies contained in the current frame, the Mel scale is used for transformation, as shown in the following formula:

[0069]

[0070] Where: f is the frequency after transformation; f' is the frequency before transformation.

[0071] Step 5: Extract the baseband of each audio

[0072] This embodiment uses the SWIPE algorithm to estimate the fundamental frequency. The peak-to-valley distance is used to measure the significance of the peak value of the amplitude spectrum at each integer multiple of the frequency f relative to the two valley values ​​adjacent to it. The function is defined as:

[0073]

[0074] Where: d k (f) represents the peak-to-valley distance; k=1, 2, ..., n, represents the multiple, and |X(kf)| represents the peak value of the amplitude spectrum at k times the frequency f;

[0075] The significance is expressed by the average peak-to-valley distance of each harmonic, as shown in the following formula:

[0076]

[0077] The final estimated fundamental frequency based on the significance is expressed as f o，def , the fundamental frequency f o，def Used for subsequent feature normalization.

[0078] Step 6: Add baseband perturbations to form baseband set S f

[0079] For the fundamental frequency f o，def Add perturbations, add frequency offsets of ±20, ±40, and ±60, i.e., Mel scale offsets, and obtain the perturbed fundamental frequency f o With the original fundamental frequency f o，def Together they form the baseband set S f .

[0080] Step 7: Use the base frequency set to normalize the frequency and construct the frequency set S F

[0081] Use the base frequency set S f Normalize the frequency features. For the spectrum graph corresponding to the current audio, use the method in step 6 to extract the fundamental frequency of each frame and calculate the median of the fundamental frequency, denoted as f o，audio , and then use the base frequency set S f Each value in normalizes the frequency value on the Mel scale:

[0082] f norm =f orig -(f o，audio -f o，def )

[0083] In the formula, f orig Indicates the frequency value in Mel scale, f o，audio Indicates the median of the fundamental frequency of all frames in the current audio, f o，def Indicates the default fundamental frequency;

[0084] The normalized f norm The frequency set S F .

[0085] Step 8: Use frequency set S F Perform acoustic feature extraction

[0086] Taking the elements in the frequency concentration as the reference frequency, the signal is converted into energy distribution in the frequency domain by Fast Fourier Transform (FFT). Different energy distributions can represent the characteristics of different speech.

[0087] Perform N-point FFT on each frame signal to calculate the spectrum, where N is 256. The fast Fourier transform S(t, f) of audio X is expressed as:

[0088]

[0089] Where X(n) represents the audio signal at the nth sampling position in the tth frame, and w(nt) represents the window function.

[0090] Afterwards, various acoustic features can be extracted according to the actual task to obtain acoustic features such as Fbank or MFCC as the input of the corresponding model.

[0091] Since the method provided by the present invention does not require additional noise data and only needs to operate on the input audio signal without changing the label, it has wider applicability than existing task-related data enhancement methods. The method provided by the present invention can be used in general audio classification, voiceprint recognition or speech recognition tasks.

Claims

1. A hierarchical fusion audio data enhancement method, characterized in that: The following steps are involved: a) Collect the original signal and save it in the form of digital signal as audio X; b) Perform time domain signal compression and expansion on audio X to obtain the compressed and expanded audio X o : When compressing and expanding the audio signal X in the time domain, the waveform similarity overlap superposition algorithm is used to introduce a series of distortions while retaining the harmonic signal, thereby obtaining the compressed and expanded audio signal X. o ; c) The compressed and expanded audio X o Mixed with the original audio X to form a new training set S x ; d) For the training set S x Perform frequency extraction on each audio in to obtain frequency f; e) Extract the base frequency of each audio and obtain the base frequency f o，def ; d) For the fundamental frequency f o，def Perturbations are added to form the base frequency set S f : For the fundamental frequency f o，def Add disturbances, add frequency offsets of ±20, ±40 and ±60 respectively, and the perturbed fundamental frequency f o With the original fundamental frequency f o，def Together they form the baseband set S f ; e) Use base frequency set S f Normalize the frequency and construct the frequency set S F : Use the base frequency set S f Normalize the frequency features. For the spectrum graph corresponding to the current audio, use step d) to extract the fundamental frequency of each frame and calculate the median of the fundamental frequency, denoted as f o，audio , and then use the base frequency set S f Each value in normalizes the frequency value on the Mel scale: f norm =f orig -(f o，audio -f o，def ) In the formula, f orig Indicates the frequency value in Mel scale, f o，audio Indicates the median of the fundamental frequency of all frames in the current audio, f o，def Indicates the default fundamental frequency; The normalized f norm The frequency set S F ; f) Use frequency set S F Perform acoustic feature extraction With frequency set S F Based on the elements in , the signal is transformed into energy distribution in the frequency domain by fast Fourier transform. Different energy distributions can represent the characteristics of different speech.

2. The hierarchical fusion audio data enhancement method according to claim 1, characterized in that: Define any audio frame in the original audio X as the first audio frame, then step b) specifically includes the following steps: A second audio frame is selected within the left and right range of the first audio frame, and a phase parameter of the second audio frame is aligned with the phase of the first audio frame; Search for the third audio frame in the range [-Δmax, Δmax], where Δmax is set to half an audio period, and then calculate the correlation coefficients between frames in the range, and select the audio frame with the highest similarity to the second audio frame as the third audio frame; The first audio frame, the second audio frame, and the third audio frame are concatenated with the same step length, and the overlapping parts are added.

3. The hierarchical fusion audio data enhancement method according to claim 1, characterized in that: In step c), the compressed and expanded audio X o Add the same annotations as the original audio X and add them to the original dataset to form a new training set S x .

4. The hierarchical fusion audio data enhancement method according to claim 1, characterized in that: In step d), the training set S x The audio in is framed, windowed and Mel-scaled to extract the audio frequency features and obtain the frequency f.

5. The hierarchical fusion audio data enhancement method according to claim 1, characterized in that: In step e), the fundamental frequency is estimated using the SWIPE algorithm, and the significance of the peak value of the amplitude spectrum at each integer multiple of the frequency f relative to the two valley values ​​adjacent to it is measured by the peak-to-valley distance. The function is defined as: Where: d k (f) represents the peak-to-valley distance; k=1, 2, ..., n, represents the multiple, and |X(kf)| represents the peak value of the amplitude spectrum at k times the frequency f; The significance is expressed by the average peak-to-valley distance of each harmonic, as shown in the following formula: The final estimated fundamental frequency based on the significance is expressed as f o，def , the fundamental frequency f o，def Used for subsequent feature normalization.

6. A hierarchical fusion audio data enhancement system, characterized in that: include: Signal compression and expansion unit: used to compress and expand the time domain signal of audio X to obtain the compressed and expanded audio X o , when compressing and expanding the audio X in the time domain, a waveform similarity overlap algorithm is used to introduce a series of distortions while retaining the harmonic signal, thereby obtaining the compressed and expanded audio X o ; Training set construction unit: used to compress and expand the audio X o Mixed with the original audio X to form a new training set S x ; Frequency extraction unit: used to extract training set S x The frequency of each audio in is obtained, and the frequency f is obtained; Base frequency extraction unit: used to extract the base frequency of each audio and obtain the base frequency f o，def ; Base frequency disturbance adding unit: used to add the base frequency f o，def Perturbations are added to form the base frequency set S f , where for the fundamental frequency f o，def When adding disturbance, frequency offsets of ±20, ±40, and ±60 are added respectively, and the resulting fundamental frequency f after disturbance is o With the original fundamental frequency f o，def Together they form the baseband set S f ; Frequency normalization unit: used to use the base frequency set S f Normalize the frequency and construct the frequency set S F , where the base frequency set S is used f When normalizing the frequency features, for the spectrum corresponding to the current audio, use step d) to extract the fundamental frequency of each frame and calculate the median of the fundamental frequency, denoted as f o，audio , and then use the base frequency set S f Each value in normalizes the frequency value on the Mel scale: f norm =f orig -(f o，audio -f o，def ) In the formula, f orig Indicates the frequency value in Mel scale, f o，audio Indicates the median of the fundamental frequency of all frames in the current audio, f o，def Represents the default fundamental frequency; the normalized f norm The frequency set S F Acoustic feature extraction unit: used to use the frequency set S F To extract acoustic features, the frequency set S F The signal is transformed into energy distribution in the frequency domain by fast Fourier transform based on the elements in . Different energy distributions can represent the characteristics of different speech.

Images