A speech recognition data enhancement method based on feature clipping and translation

Abstract

The present invention relates to a speech recognition data enhancement method based on feature clipping and translation, and belongs to the field of speech recognition processing technology. The method clips and translates the audio signal features in the time dimension and the frequency dimension respectively. Among them, in the time dimension, the feature value of a certain time period is randomly selected, the feature of the time period is clipped, and then the unclipped feature value is translated along the time dimension direction; in the frequency dimension direction, the feature value of a certain frequency segment is randomly selected, and then the feature of the frequency segment is clipped, and the unclipped feature value is translated along the frequency direction. The method does not need to regenerate the audio signal or extract features, which avoids the waste of storage space and running time. It has better data enhancement effect than the existing data enhancement technology and effectively improves the speech recognition effect.

Description

Technical Field

[0001] The invention relates to a speech recognition data enhancement method based on feature clipping and translation, and belongs to the technical field of speech recognition processing. Technical Background

[0002] With the rapid development of computer technology, computer technology has been applied to various fields of society. At the same time, it has caused difficulties in processing massive amounts of voice data and human-computer interaction. The goal of speech recognition is to automatically convert human speech into text through computers. With the further development of artificial intelligence and deep learning technology, speech recognition technology has made significant progress.

[0003] The main difficulties faced by speech recognition are insufficient training corpus and lack of diversity, which leads to overfitting of the recognition model and poor generalization ability. This greatly limits the development of speech recognition technology. Currently, the commonly used solutions include Dropout, L1, L2 regularization, Batch Normalization, transfer learning, pre-training, etc.

[0004] However, these methods all start from identifying models, trying to find a model that best fits the training data from a family of models with stronger generalization capabilities. However, data augmentation starts from data and generates pseudo data based on prior knowledge from existing data to increase data diversity, thereby achieving the purpose of enhancing the generalization ability of the model and preventing the model from overfitting.

[0005] Most of the existing speech data enhancement methods focus on the audio signal end. For example, the principle of noise addition is to add environmental noise to the original audio signal to generate a new audio signal. However, this type of speech data enhancement method not only requires space to store the newly generated audio, but also requires more time to extract features from the newly generated audio. In addition, there are some online speech data enhancement methods, such as SpecAugment proposed by Daniel S. Park et al., which achieves the purpose of data enhancement by adding random masks to the feature end. However, the degree of disturbance of SpecAugment on the data is not ideal and cannot enhance the data to the greatest extent. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and to solve the technical problems in speech recognition technology such as insufficient data volume and lack of diversity of speech data sets, and traditional data enhancement easily causing the data set to be too large. A speech recognition data enhancement method based on feature clipping and translation is creatively proposed.

[0007] The innovation of the present invention is that the audio signal features are clipped and translated in the time dimension and frequency dimension respectively. In the time dimension, the feature value of a certain time period is randomly selected, the feature of the time period is clipped, and then the unclipped feature value is translated along the time dimension; in the frequency dimension, the feature value of a certain frequency segment is randomly selected, and then the feature of the frequency segment is clipped, and the unclipped feature value is translated along the frequency direction.

[0008] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0009] A speech recognition data enhancement method based on feature clipping and translation operates on the feature values ​​of audio signals.

[0010] First, for the input speech segment, the waveform file of its audio signal is input into the system model for feature extraction to obtain the feature matrix F of the audio signal (such as Mel spectrum feature matrix), and this two-dimensional matrix is ​​the eigenvalue.

[0011] The feature extraction method may adopt the MFCC (Mel-Frequency Cepstral Coefficients) method or other audio feature extraction methods.

[0012] Then, according to the upper limit of the length of the randomly selected cropping time period, the starting point and length of the randomly selected cropping time period are sampled; according to the upper limit of the length of the randomly selected cropping frequency segment, the starting point and length of the randomly selected cropping frequency segment are sampled.

[0013] Afterwards, a time clipping translation matrix L is generated according to the start point and length of the randomly selected clipping time period; and a frequency clipping translation matrix R is generated according to the start point and length of the randomly selected clipping frequency segment.

[0014] Then, the time clipping translation matrix L and the frequency clipping translation matrix R are used to perform matrix transformation on the feature matrix F to obtain the new audio signal feature

[0015]

[0016] Exploiting new audio signal characteristics Train the speech recognition model and complete speech recognition data enhancement.

[0017] Beneficial Effects

[0018] Compared with the prior art, the method of the present invention does not need to regenerate the audio signal or extract features, thus avoiding the waste of storage space and running time. It has better data enhancement effect than the existing data enhancement technology and can effectively improve the speech recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the process of the present invention.

[0020] Figure 2 Schematic diagram of cropping and translation of the spectrum graph in the time dimension using this method.

[0021] Figure 3 Schematic diagram of cropping and translation of the spectrum graph in the feature dimension based on this method. DETAILED DESCRIPTION

[0022] The present invention is further described in detail below with reference to the accompanying drawings.

[0023] like Figure 1 As shown, a speech recognition data enhancement method based on feature clipping and translation includes the following steps:

[0024] Step 1: Input the waveform file of the audio signal into the system model to obtain the feature matrix F of the audio signal.

[0025] The audio signal feature F is a two-dimensional matrix. Specifically, the system model needs to extract features from the waveform file of the audio signal. The feature extraction method can use MFCC (Mel-Frequency Cepstral Coefficients) or other audio feature extraction methods. After the audio signal is feature extracted, the feature value is a two-dimensional array, where the first dimension represents time and the second dimension represents frequency. The two-dimensional matrix is ​​defined as F, and its matrix size is M×N. The constant M represents the length of the audio signal feature used in the time dimension, and the constant N represents the length of the audio signal feature used in the frequency dimension.

[0026] Step 2: Uniformly sample a value t from 0 to T as the length of the clipping time period.

[0027] The variable t represents the length of the randomly selected clipping time period, and the constant T represents the upper limit of the length t of the randomly selected clipping time period.

[0028] Step 3: Uniformly sample a value t0 from 0 to Mt as the starting point of the clipping time end.

[0029] The variable t0 represents the starting point of the randomly selected clipping time period.

[0030] Step 4: Generate the time clipping translation matrix L based on t and t0 obtained in steps 2 and 3:

[0031]

[0032] Among them, E1 represents the unit matrix of shape t0×t0, and E2 represents the unit matrix of shape (Mt-t0)×(Mt-t0). Figure 2 shown.

[0033] Step 5: Uniformly sample a value k from 0 to K as the length of the cropped frequency segment.

[0034] The constant K represents the upper limit of the length of the randomly selected cropped frequency segment, and the variable k represents the length of the randomly selected cropped frequency segment.

[0035] Step 6: Uniformly sample a value k0 from 0 to Nk as the starting point of the clipping frequency segment.

[0036] The variable k0 represents the starting point of the randomly selected clipping frequency segment.

[0037] Step 7: Generate the clipping translation matrix R based on k and k0 obtained in steps 5 and 6:

[0038]

[0039] Among them, E3 represents the identity matrix of shape k0×k0, and E4 represents the identity matrix of shape (Nk-k0)×(Nk-k0). Figure 3 shown.

[0040] Step 8: Perform matrix transformation on the audio signal feature F and the cropping and translation matrices L and R obtained in steps 4 and 7 to obtain a new audio signal feature

[0041]

[0042] Step 9: Get new audio signal features Then, model training is performed.

[0043] Here, an end-to-end model with a Transformer architecture can be used for training.

[0044] Comparative Verification

[0045] The method of the present invention is used in a speech recognition system. The data set is AISHELL-1, and Transformer is used to recognize Chinese speech into pinyin. The word error rate (WER) is used to evaluate the recognition effect, and the results are shown in Table 1.

[0046] Table 1: Speech recognition results of various data augmentation methods

[0047]

[0048] It can be seen that compared with the SpecAugment method, which has the best data enhancement effect in current speech data enhancement technology, the data enhancement method based on feature cropping and translation proposed in the present invention has a lower word error rate than this method, indicating that the data enhancement method proposed in the present invention has significant technical effects.

Claims

1. A speech recognition data enhancement method based on feature clipping and translation, characterized in that: The following steps are involved: First, for the input speech segment, the waveform file of its audio signal is input into the system model for feature extraction to obtain the feature matrix F of the audio signal. The feature matrix F is a two-dimensional matrix, and the two-dimensional matrix is ​​the eigenvalue; In the feature matrix F, the first dimension represents time, the second dimension represents frequency, and the matrix size is M×N, where the constant M represents the length of the audio signal feature used in the time dimension, and the constant N represents the length of the audio signal feature used in the frequency dimension; Then, according to the upper limit of the length of the randomly selected cropping time period, the starting point and length of the randomly selected cropping time period are sampled; according to the upper limit of the length of the randomly selected cropping frequency segment, the starting point and length of the randomly selected cropping frequency segment are sampled; Afterwards, a time clipping translation matrix L is generated according to the start point and length of the randomly selected clipping time segment; a frequency clipping translation matrix R is generated according to the start point and length of the randomly selected clipping frequency segment; Let variable t represent the length of the randomly selected cropping time period, let constant T represent the upper limit of the length t of the randomly selected cropping time period, let variable t0 represent the starting point of the randomly selected cropping time period, let constant K represent the upper limit of the length of the randomly selected cropping frequency segment, let variable k represent the length of the randomly selected cropping frequency segment; Uniformly sample a value t from 0 to T as the length of the clipping time period, and uniformly sample a value t0 from 0 to Mt as the starting point of the clipping time end; based on t and t0, generate the time clipping translation matrix L: Among them, E1 represents the identity matrix of shape t0×t0, and E2 represents the identity matrix of shape (Mt-t0)×(Mt-t0); From 0 to K, a value k is uniformly sampled as the length of the cropped frequency segment, and from 0 to Nk, a value k0 is uniformly sampled as the starting point of the cropped frequency segment; based on k and k0, the cropping translation matrix R is generated: Among them, E3 represents the identity matrix of shape k0×k0, and E4 represents the identity matrix of shape (Nk-k0)×(Nk-k0); Then, the time clipping translation matrix L and the frequency clipping translation matrix R are used to perform matrix transformation on the feature matrix F to obtain the new audio signal feature Finally, using the new audio signal features Train the speech recognition model and complete speech recognition data enhancement.

2. A method for speech recognition data enhancement based on feature clipping and translation as claimed in claim 1, characterized in that: Using audio signal characteristics The model is trained using an end-to-end Transformer architecture.

3. A method for speech recognition data enhancement based on feature clipping and translation as claimed in claim 1, characterized in that: The MFCC method is used to extract features from the waveform file of the audio signal.

Images