A speech recognition data enhancement method based on feature replacement and masking of spectrograms

Abstract

The application discloses a speech recognition data enhancement method based on feature replacement and masking of a spectrum diagram, and comprises the following steps: acquiring acoustic features of original speech; performing time replacement and masking on the acoustic features in a time dimension, randomly selecting a time block size that needs to be exchanged and masked, randomly selecting and exchanging two time blocks from a spectrum diagram, and masking speech signal features of one of the time blocks; performing frequency replacement and masking on the acoustic features in a frequency dimension, randomly selecting a frequency block size that needs to be exchanged and masked, randomly selecting and exchanging two frequency blocks from the spectrum diagram, and masking speech signal features of one of the frequency blocks; obtaining newly generated acoustic features, and generating a new spectrum diagram according to the newly generated acoustic features. The method directly acts on speech features instead of original speech, does not need to generate additional speech data, saves resource space and computing time, and does not need to generate an audio signal again.

Description

Technical Field

[0001] This invention relates to the field of speech recognition processing technology, and in particular to a speech recognition data enhancement method based on spectrogram feature replacement and masking. Background Technology

[0002] Speech recognition technology enables machines to understand human language and is figuratively described as a "computer's auditory system." The purpose of speech recognition is to convert sound into corresponding text using computers, involving knowledge from multiple fields such as signal processing, computer science and technology, mathematics, and acoustics. Currently, it has very wide applications in industries, healthcare, and smart homes, such as map navigation apps, voice input methods, smart speakers, and mobile phone voice assistants—all applications of speech recognition technology in daily life.

[0003] Since the beginning of the 21st century, speech recognition technology has developed rapidly, mainly in two aspects. One is the traditional speech recognition architecture, which achieves speech recognition through acoustic models, language models, dictionaries, and decoders. The other is end-to-end speech recognition, which has become a research hotspot. It does not require pre-alignment of speech frames with Hidden Markov Models (HMMs) and does not rely on pronunciation dictionaries, making speech recognition simpler. To improve speech recognition performance, related research mainly focuses on designing better network structures, such as Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), and end-to-end architectures. However, these models are prone to overfitting and require large amounts of training data. Data augmentation improves the model's generalization ability by adding additional training data.

[0004] Most existing speech enhancement methods focus on the audio signal, such as Vocal Tract Length Normalization (VTLN), adding noise, and sound velocity perturbation. These methods not only require space to store the newly generated audio signal but also necessitate feature extraction from it. Recently, Daniel S. Park et al. from Google proposed an online speech data enhancement method called Spectrum Augmentation. This method does not require generating additional audio; it directly applies to the spectrogram, masking it in both the time and frequency dimensions to achieve data enhancement. However, this method cannot maximize the enhancement of the data. Summary of the Invention

[0005] This invention provides a speech recognition data enhancement method based on spectrogram feature replacement and masking to overcome the above-mentioned technical problems.

[0006] A speech recognition data augmentation method based on spectrogram feature replacement and masking includes,

[0007] Step 1: Obtain the original speech and extract its acoustic features Z. X,Y X represents the time dimension, and Y represents the frequency dimension;

[0008] Step 2: In the time dimension, perform time replacement and masking on the acoustic features. Specifically, this includes randomly selecting t from a uniform distribution from 0 to X, where t represents the length of the time block in the time dimension; randomly selecting the first time sampling starting point t0 from [0, Xt); and randomly selecting the second time sampling starting point t1 from [t0+t, Xt).

[0009] Get the data from column t0 to column t0+t of the acoustic feature and represent it as Z1. Get the data from column t1 to column t1+t of the acoustic feature and represent it as Z2. Store the data of Z1 in columns t1 to t1+t of the acoustic feature. Fill all the data in columns t0 to t0+t of the acoustic feature with 0. Save the modified acoustic feature as the first acoustic feature.

[0010] Step 3: In the frequency dimension, perform frequency replacement and masking on the acoustic features. Specifically, this includes randomly selecting f from a uniform distribution from 0 to Y, where f represents the length of the frequency block in the frequency dimension.

[0011] Randomly select a first frequency sampling starting point f0 from [0, Yf), and randomly select a second frequency sampling starting point f1 from [f0+f, Yf).

[0012] Get the data from row f0 to row f0+f of the acoustic feature and represent it as Z3. Get the data from row f1 to row f1+f of the acoustic feature and represent it as Z4. Store the data of Z3 in row f1 to row f1+f of the acoustic feature. Fill all the data from row f0 to row f0+f of the acoustic feature with 0. Save the modified acoustic feature as the second acoustic feature.

[0013] Step 4: Fuse the first acoustic feature and the second acoustic feature to obtain the newly generated acoustic feature. Specifically, this includes saving the data from column t0 to column t0+t of the first acoustic feature to column t0 to column t0+t of the acoustic feature, saving the data from column t1 to column t1+t of the first acoustic feature to column t1 to column t1+t of the acoustic feature, saving the data from row f0 to row f0+f of the second acoustic feature to row f0 to row f0+f of the acoustic feature, saving the data from row f1 to row f1+f of the second acoustic feature to row f1 to row f1+f, saving the modified acoustic feature as the newly generated acoustic feature, and generating a new spectrogram based on the newly generated acoustic feature.

[0014] Preferably, the extraction of acoustic features from the original speech includes extracting FBank acoustic features or MFCC acoustic features from the original speech.

[0015] Preferably, the extraction of FBank acoustic features from the original speech includes,

[0016] Step 1a: Preprocess the original speech;

[0017] Step 1b: Use the Fast Fourier Transform algorithm to convert the preprocessed original speech into a frequency domain signal;

[0018] Step 1c: Define N sets of filters, use the filter banks to smooth the frequency domain signal, and obtain the smoothed spectrum signal;

[0019] Step 1d: Calculate the logarithmic energy of the filter bank based on the smoothed spectral signal, and use the filter bank characteristics as FBank acoustic features.

[0020] Preferably, the extraction of MFCC acoustic features from the original speech includes,

[0021] Step 2a: Preprocess the original speech;

[0022] Step 2b: Use the Fast Fourier Transform algorithm to convert the preprocessed original speech into a frequency domain signal;

[0023] Step 2c: Define M groups of filters, use the filter groups to smooth the frequency domain signal, and obtain the smoothed spectrum signal;

[0024] Step 2d: Calculate the logarithmic energy of the filter bank based on the smoothed spectral signal;

[0025] Step 2f: Obtain the acoustic features of MFCC by performing discrete cosine transform according to formula (1).

[0026]

[0027] Where M represents the number of filters in the filter bank, 1≤m≤M, s(m) is the logarithmic energy of the filter bank, L is the dimension of the MFCC eigenvector, and C(n) is used as the acoustic feature of the MFCC.

[0028] A speech recognition data augmentation system based on spectrogram feature replacement and masking includes an acoustic feature acquisition module, a time-dimensional replacement and masking module, a frequency-dimensional replacement and masking module, and a new acoustic feature generation module.

[0029] The acoustic feature acquisition module is used to acquire the original speech and extract the acoustic features Z of the original speech. X,Y X represents the time dimension, and Y represents the frequency dimension;

[0030] The temporal dimension replacement and masking module is used to perform temporal replacement and masking of acoustic features in the temporal dimension. Specifically, it includes randomly selecting t from a uniform distribution from 0 to X, where t represents the length of the time block in the temporal dimension; randomly selecting a first time sampling starting point t0 from [0, Xt); and randomly selecting a second time sampling starting point t1 from [t0+t, Xt).

[0031] Get the data from column t0 to column t0+t of the acoustic feature and represent it as Z1. Get the data from column t1 to column t1+t of the acoustic feature and represent it as Z2. Store the data of Z1 in columns t1 to t1+t of the acoustic feature. Fill all the data in columns t0 to t0+t of the acoustic feature with 0. Save the modified acoustic feature as the first acoustic feature.

[0032] The frequency dimension replacement and masking module is used to perform frequency replacement and masking on acoustic features in the frequency dimension. Specifically, it involves randomly selecting f from a uniform distribution from 0 to Y, where f represents the length of the frequency block in the frequency dimension.

[0033] Randomly select a first frequency sampling starting point f0 from [0, Yf), and randomly select a second frequency sampling starting point f1 from [f0+f, Yf).

[0034] Get the data from row f0 to row f0+f of the acoustic feature and represent it as Z3. Get the data from row f1 to row f1+f of the acoustic feature and represent it as Z4. Store the data of Z3 in row f1 to row f1+f of the acoustic feature. Fill all the data from row f0 to row f0+f of the acoustic feature with 0. Save the modified acoustic feature as the second acoustic feature.

[0035] The new acoustic feature generation module is used to fuse the first acoustic feature and the second acoustic feature to obtain a newly generated acoustic feature. Specifically, it includes saving the data from column t0 to column t0+t of the first acoustic feature to column t0 to column t0+t of the acoustic feature, saving the data from column t1 to column t1+t of the first acoustic feature to column t1 to column t1+t of the acoustic feature, saving the data from row f0 to row f0+f of the second acoustic feature to row f0 to row f0+f of the acoustic feature, saving the data from row f1 to row f1+f of the second acoustic feature to row f1 to row f1+f, saving the modified acoustic feature as the newly generated acoustic feature, and generating a new spectrogram based on the newly generated acoustic feature.

[0036] This invention provides a speech recognition data augmentation method based on spectrogram feature replacement and masking. By performing feature replacement and masking in the time and frequency dimensions to generate a new spectrogram, it can solve the problems of insufficient existing speech data, lack of diversity, and excessively large datasets caused by traditional data augmentation methods. This invention can directly apply to speech features instead of the original speech, without generating additional speech data, saving resources, space and computing time. It does not require regenerating audio signals and directly applies to the spectrogram, effectively achieving data expansion. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the method of the present invention;

[0039] Figure 2 This is a schematic diagram of the replacement and masking in the time dimension of the present invention;

[0040] Figure 3 This is a schematic diagram of the replacement and masking in the frequency dimension of the present invention;

[0041] Figure 4 This is a schematic diagram of the replacement and masking of the frequency dimension in this invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Figure 1 This is a flowchart of the method of the present invention, as shown below. Figure 1 As shown, the method in this embodiment may include:

[0044] Step 1: Obtain the original speech and extract its acoustic features Z. X,Y X represents the time dimension, Y represents the frequency dimension, and the extraction of acoustic features from the original speech includes extracting FBank acoustic features or MFCC acoustic features from the original speech.

[0045] The extracted FBank acoustic features of the original speech include,

[0046] Step 1a: Preprocess the original speech, including pre-emphasis, framing, and windowing.

[0047] Pre-emphasis is used to boost the high-frequency signals of the original speech. Because low-frequency sounds travel a greater distance across the basilar membrane of the inner ear than higher-frequency sounds, low frequencies tend to mask high frequencies. Furthermore, during human vocalization, factors such as the vocal cords and lips cause high-frequency signals to attenuate easily. Loss of high-frequency signals can lead to poor phoneme modeling capabilities in acoustic models. Therefore, it is necessary to boost the high-frequency signals of speech to highlight high-frequency information; this process is also known as pre-emphasis.

[0048] Framing is used to segment the original speech signal. Since the overall speech signal is unstable and time-varying, to reduce the impact of this characteristic, the speech can generally be considered a relatively stable signal for a short period (10ms to 30ms). The segmented speech is called a speech frame, and the duration of the speech frame is called the frame length, which is set to 25ms. To avoid excessive changes between adjacent frames, a certain overlap is set between adjacent speech frames during framing. The difference in the start time between two adjacent frames is called the frame shift.

[0049] Windowing is used to reduce the loss of speech energy in the original speech, and it can reduce the leakage of speech energy loss. Hamming windows are generally used. A Hamming window is a function shaped like a window. Each frame is multiplied by a Hamming window to increase the continuity of the left and right ends of the speech frame. After windowing, Fourier transform is required on the data in each window. It is assumed that the signal in each window represents a periodic signal. Usually, a short segment of audio data does not have obvious periodicity. After adding a Hamming window, the speech signal has periodicity, as shown in equation (1):

[0050]

[0051] Where N is the window length, i.e. the number of sampling points, N = 512. After adding the Hamming window, only the speech signal in the middle is reflected, and the speech information on both sides will be lost. Therefore, when framing, only 1 / 3 or 1 / 2 of the window will be moved, and the frame shift is set to 10ms.

[0052] Step 1b: Use the Fast Fourier Transform (FFT) algorithm to convert the original speech into a frequency domain signal. Since it is difficult to see the characteristics of a signal by transforming it in the time domain, it is usually converted into the energy distribution in the frequency domain. The time domain signal is converted into a frequency domain signal by using FFT.

[0053] Step 1c: Smooth the spectrum of the frequency domain signal using a filter bank to eliminate harmonics and highlight the original sound's formants. The human ear has different perceptual abilities for different frequency domain signals; the Mel filter bank can mimic the human auditory system's frequency perception to filter frequency signals. The masking critical bandwidth for low-frequency sounds is smaller than that for high-frequency sounds; therefore, a bandpass filter is arranged from low to high frequencies, arranged in descending order of critical bandwidth.

[0054] Using filters can smooth the spectrum, eliminate harmonics, and highlight the original speech resonances. The conversion between the Mel frequency and the actual frequency f is shown in equation (2):

[0055]

[0056] Within the speech spectrum, several bandpass filters H are set. m (k), 1≤m≤M, where M is the number of filters. The frequency response is shown in equation (3):

[0057]

[0058] Where k represents the sampling point number, and f(m) is the number of FFT points corresponding to the center frequency of the m-th filter.

[0059] Step 1d: Calculate the logarithmic energy of the filter bank. Calculate the logarithmic energy according to formula (4), where M is the number of triangular bandpass filters.

[0060]

[0061] When the number of triangular filters is 40, the corresponding acoustic features are 40-dimensional Fourier-transform-based log filter-bank (FBank) features, and the filter-bank features are used as FBank acoustic features.

[0062] The extracted MFCC acoustic features of the original speech include,

[0063] Step 2a: Preprocess the original speech, including pre-emphasis, framing, and windowing. Pre-emphasis is used to enhance the high-frequency signal of the original speech, framing is used to segment the original speech, and windowing is used to reduce the loss of speech energy in the original speech.

[0064] Step 2b: Use the Fast Fourier Transform algorithm to convert the preprocessed original speech into a frequency domain signal;

[0065] Step 2c: Define M groups of filters, use the filter groups to smooth the spectrum of the frequency domain signal, and obtain the smoothed spectrum signal;

[0066] Step 2d: Calculate the logarithmic energy of the filter bank based on the smoothed spectral signal;

[0067] Step 2f: Obtain the acoustic features of MFCC by performing discrete cosine transform according to formula (5).

[0068]

[0069] Where M represents the number of filters in the filter bank, s(m) is the logarithmic energy of the filter bank, and L is the dimension of the MFCC feature vector, 1≤m≤M. C(n) is used as the acoustic feature of the MFCC, typically L is set to 13. First-order and second-order differences are then performed to obtain a 39-dimensional MFCC feature, which can represent the dynamic characteristics of speech.

[0070] Step 2: In the time dimension, perform temporal replacement and masking on the acoustic features. The result of temporal replacement and masking is as follows: Figure 2 As shown, firstly, from a uniform distribution of time replacement and masking parameters (upper limits for replacement and masking time lengths), the size of the time block to be swapped and masked is randomly selected. Then, two time blocks are randomly selected from the spectrogram. Next, these two time blocks are swapped, and then the speech signal features of one of the time blocks are masked. Based on the aforementioned design concept, the corresponding steps include:

[0071] From a uniform distribution ranging from 0 to X, t is randomly selected, where t represents the length of the time block in the time dimension.

[0072] A first time sampling starting point t0 is randomly selected from [0, Xt), and a second time sampling starting point t1 is randomly selected from [t0+t, Xt).

[0073] Get the data from column t0 to column t0+t of the acoustic feature and represent it as Z1. Get the data from column t1 to column t1+t of the acoustic feature and represent it as Z2. Store the data of Z1 in columns t1 to t1+t of the acoustic feature. Fill all the data in columns t0 to t0+t of the acoustic feature with 0. Save the modified acoustic feature as the first acoustic feature.

[0074] Step 3: In the frequency dimension, perform frequency substitution and masking on the acoustic features. The results of frequency substitution and masking are as follows: Figure 3 As shown, firstly, from a uniform distribution of frequency replacement and masking parameters (the upper limit of the frequency size for replacement and masking), the size of the frequency block to be swapped and masked is randomly selected. Then, two frequency blocks are randomly selected from the spectrogram. Next, these two frequency blocks are swapped, and then the speech signal features of one of the frequency blocks are masked. Based on the aforementioned design concept, the corresponding steps include:

[0075] From a uniform distribution from 0 to Y, f is randomly selected, where f represents the length of the frequency block in the frequency dimension.

[0076] Randomly select a first frequency sampling starting point f0 from [0, Yf), and randomly select a second frequency sampling starting point f1 from [f0+f, Yf).

[0077] Get the data from row f0 to row f0+f of the acoustic feature and represent it as Z3. Get the data from row f1 to row f1+f of the acoustic feature and represent it as Z4. Store the data of Z3 in row f1 to row f1+f of the acoustic feature. Fill all the data from row f0 to row f0+f of the acoustic feature with 0. Save the modified acoustic feature as the second acoustic feature.

[0078] Step 4: Fuse the first acoustic feature and the second acoustic feature to obtain a newly generated acoustic feature. This includes saving the data from column t0 to column t0+t of the first acoustic feature to column t0 to column t0+t of the second acoustic feature, saving the data from column t1 to column t1+t of the first acoustic feature to column t1 to column t1+t of the second acoustic feature, saving the data from row f0 to row f0+f of the second acoustic feature to row f0 to row f0+f of the second acoustic feature, and saving the data from row f1 to row f1+f of the second acoustic feature to row f1 to row f1+f. Save the modified acoustic feature as the newly generated acoustic feature. Generate a new spectrogram based on the newly generated acoustic feature. The frequency dimension and frequency dimension replacement and masking are illustrated as follows. Figure 4 As shown, the spectrogram is used to train the speech recognition model, including training using an end-to-end architecture based on a convolution-augmented Transformer (conformer).

[0079] A speech recognition data augmentation system based on spectrogram feature replacement and masking includes an acoustic feature acquisition module, a time-dimensional replacement and masking module, a frequency-dimensional replacement and masking module, and a new acoustic feature generation module.

[0080] The acoustic feature acquisition module is used to acquire the original speech and extract the acoustic features Z of the original speech. X,Y X represents the time dimension, and Y represents the frequency dimension;

[0081] The temporal dimension replacement and masking module is used to perform temporal replacement and masking of acoustic features in the temporal dimension. Specifically, it includes randomly selecting t from a uniform distribution from 0 to X, where t represents the length of the time block in the temporal dimension; randomly selecting a first time sampling starting point t0 from [0, Xt); and randomly selecting a second time sampling starting point t1 from [t0+t, Xt).

[0082] Get the data from column t0 to column t0+t of the acoustic feature and represent it as Z1. Get the data from column t1 to column t1+t of the acoustic feature and represent it as Z2. Store the data of Z1 in columns t1 to t1+t of the acoustic feature. Fill all the data in columns t0 to t0+t of the acoustic feature with 0. Save the modified acoustic feature as the first acoustic feature.

[0083] The frequency dimension replacement and masking module is used to perform frequency replacement and masking on acoustic features in the frequency dimension. Specifically, it involves randomly selecting f from a uniform distribution from 0 to Y, where f represents the length of the frequency block in the frequency dimension.

[0084] Randomly select a first frequency sampling starting point f0 from [0, Yf), and randomly select a second frequency sampling starting point f1 from [f0+f, Yf).

[0085] Get the data from row f0 to row f0+f of the acoustic feature and represent it as Z3. Get the data from row f1 to row f1+f of the acoustic feature and represent it as Z4. Store the data of Z3 in row f1 to row f1+f of the acoustic feature. Fill all the data from row f0 to row f0+f of the acoustic feature with 0. Save the modified acoustic feature as the second acoustic feature.

[0086] The new acoustic feature generation module is used to fuse the first acoustic feature and the second acoustic feature to obtain a newly generated acoustic feature. Specifically, it includes saving the data from column t0 to column t0+t of the first acoustic feature to column t0 to column t0+t of the acoustic feature, saving the data from column t1 to column t1+t of the first acoustic feature to column t1 to column t1+t of the acoustic feature, saving the data from row f0 to row f0+f of the second acoustic feature to row f0 to row f0+f of the acoustic feature, saving the data from row f1 to row f1+f of the second acoustic feature to row f1 to row f1+f, saving the modified acoustic feature as the newly generated acoustic feature, and generating a new spectrogram based on the newly generated acoustic feature.

[0087] Overall beneficial effects:

[0088] This invention provides a speech recognition data augmentation method based on spectrogram feature replacement and masking. By performing feature replacement and masking in the time and frequency dimensions to generate a new spectrogram, it can solve the problems of insufficient existing speech data, lack of diversity, and excessively large datasets caused by traditional data augmentation methods. This invention can directly apply to speech features instead of the original speech, without generating additional speech data, saving resources, space and computing time. It does not require regenerating audio signals and directly applies to the spectrogram, effectively achieving data expansion.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition data augmentation method based on feature replacement and masking of spectrograms, characterized in that, comprising, Step one, obtaining original speech, extracting acoustic features of the original speech X represents the time dimension, and Y represents the frequency dimension. Step two, time replacement and masking of the acoustic features in the time dimension, specifically including, randomly selecting t from a uniform distribution from 0 to X, where t represents the length of the time block in the time dimension, randomly selecting the first time sampling start point from [0, , , , , , acoustic feature column data in the acoustic feature column data in the acoustic feature column data in the acoustic feature column data in the acoustic feature column data in the acoustic feature column data in the acoustic feature column data in the acoustic feature column data in the acoustic feature column data in the acoustic feature column data in the acoustic feature column data in the acoustic feature column Step three, frequency replacement and masking of the acoustic feature in the frequency dimension, specifically including, randomly selecting wherein, denotes the length of the frequency block in the frequency dimension, from [0, randomly selects a first frequency sampling start point from from [ , randomly selects a second frequency sampling start point from , acoustic feature of the first row to the nth row is obtained and represented as data of the first row to the nth row and represented as data of the first row to the nth row and represented as data of the first row to the nth row is obtained and represented as data of the first row to the nth row and represented as data of the first row to the nth row is obtained and represented as data of the first row to the nth row is obtained and represented as data of the first row to the nth row is obtained and represented as data of the first row to the nth row is obtained and represented as data of the first row to the nth row is obtained and represented as data of the first row to the nth row is obtained and represented as data of the first row to the nth row is obtained and represented as Step four, fusing the first acoustic feature and the second acoustic feature to obtain a newly generated acoustic feature, specifically including saving the data of the first acoustic feature from the first column to the second column to the first acoustic feature from the third column to the fourth column, saving the data of the first acoustic feature from the fifth column to the sixth column to the acoustic feature from the seventh column to the eighth column, saving the data of the first acoustic feature from the ninth column to the tenth column to the acoustic feature from the eleventh column to the twelfth column, saving the data of the second acoustic feature from the first row to the second row to the acoustic feature from the third row to the fourth row, saving the data of the second acoustic feature from the fifth row to the sixth row to the acoustic feature from the seventh row to the eighth row, saving the data of the second acoustic feature from the ninth row to the tenth row to the acoustic feature from the eleventh row to the twelfth row, saving the modified acoustic feature as a newly generated acoustic feature, and generating a new spectrogram according to the newly generated acoustic feature. ​​​​​​​​​​​​​​​​ the extracting of the acoustic features of the original speech comprises extracting FBank acoustic features or MFCC acoustic features of the original speech; the extracting of the FBank acoustic features of the original speech comprises, Step 1a, preprocessing the original speech; Step 1b, converting the preprocessed original speech into a frequency domain signal using a fast Fourier transform algorithm; Step 1c, defining N groups of filters, and performing smoothing processing on the frequency domain signal using a filter bank to obtain a smoothed frequency spectrum signal; Step 1d, calculating the log energy of the filter bank according to the smoothed frequency spectrum signal, and taking the filter bank features as the FBank acoustic features.

2. The speech recognition data augmentation method based on feature replacement and masking of spectrograms according to claim 1, characterized in that, the extracting of the MFCC acoustic features of the original speech comprises, Step 2a, preprocessing the original speech; Step 2b, converting the preprocessed original speech into a frequency domain signal using a fast Fourier transform algorithm; Step 2c, defining M groups of filters, and performing smoothing processing on the frequency domain signal using a filter bank to obtain a smoothed frequency spectrum signal; Step 2d, calculating the log energy of the filter bank according to the smoothed frequency spectrum signal; Step 2f, performing discrete cosine transform according to formula (1) to obtain MFCC acoustic features, （1） wherein represents the number of filters in the filter bank, , is the log energy of the filter bank, is the dimension of the MFCC feature vector, and is the MFCC acoustic feature.

3. A speech recognition data augmentation system based on feature replacement and masking of spectrograms, characterized in that, comprising an acoustic feature acquisition module, a time dimension replacement and masking module, a frequency dimension replacement and masking module, and a new acoustic feature generation module, The acoustic feature acquisition module is configured to acquire the original speech and extract acoustic features of the original speech X represents a time dimension, and Y represents a frequency dimension. The temporal replacement and masking module is used to perform temporal replacement and masking of acoustic features in the temporal dimension. Specifically, it includes randomly selecting t from a uniform distribution from 0 to X, where t represents the length of the time block in the temporal dimension, from [0, ..., X]. The first sampling starting point is randomly selected from the data. ,from[ , The second time sampling starting point is randomly selected from the data. , acoustic feature column to the data in the column and represented as , acoustic feature column to the data in the column and represented as , the data of is saved in acoustic feature column to the data in the column, the data of acoustic feature column to the data in the column is filled with 0, and the modified acoustic feature is saved as the first acoustic feature; The frequency dimension replacement and masking module is configured to perform frequency replacement and masking on the acoustic feature in the frequency dimension, and specifically includes randomly selecting wherein, represents the length of the frequency block in the frequency dimension, from [0, randomly selects a first frequency sampling start point from from [ , randomly selects a second frequency sampling start point from , acoustic feature of the first row to the row and represented as , acoustic feature of the first row to the row and represented as , acoustic feature of the first row to the row and represented as , acoustic feature of the first row to the row and represented as , acoustic feature of the first row to the row and represented as , acoustic feature of the first row to the row and represented as saving the modified acoustic features as second acoustic features; The new acoustic feature generation module is used to fuse the first acoustic feature and the second acoustic feature to obtain the newly generated acoustic feature. Specifically, it includes fusing the first acoustic feature into the second acoustic feature. Listed to number The data of the column is saved to the first acoustic feature. Listed to number The first acoustic feature of the column Listed to number The data of the column is saved to the first acoustic feature. Listed to number The column will be the second acoustic feature of the first Arrive at the The data of the row is saved to the first acoustic feature. Arrive at the Okay, let's talk about the second acoustic feature. Arrive at the Save the data of the row to the first row. Arrive at the Okay, save the modified acoustic features as the newly generated acoustic features, and generate a new spectrogram based on the newly generated acoustic features; the extracting of the acoustic features of the original speech comprises extracting FBank acoustic features or MFCC acoustic features of the original speech; the extracting of the FBank acoustic features of the original speech comprises, Step 1a, preprocessing the original speech; Step 1b, converting the preprocessed original speech into a frequency domain signal using a fast Fourier transform algorithm; Step 1c, defining N groups of filters, and performing smoothing processing on the frequency domain signal using a filter bank to obtain a smoothed frequency spectrum signal; Step 1d, calculating the log energy of the filter bank according to the smoothed frequency spectrum signal, and taking the filter bank features as the FBank acoustic features.

Images