Acoustic language classification method based on Transform and improved Ecapa-Tdnn
By combining Transformer and improving Ecapa-Tdnn methods, the model architecture and feature extraction capabilities are optimized, and the limitations of speech recognition technology in language classification are solved, and the classification accuracy and computing efficiency are achieved. It is suitable for multilingual environments .
Patent Information
- Application Number
- CN202411295026.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-15
- Publication Date
- 2024-12-06
AI Technical Summary
The existing speech recognition technology has limitations in language classification. Traditional methods have limited performance in complex speech signal processing. The Transformer model performs weakly in short-term speech feature extraction, while Ecapa-Tdnn is not suitable for language classification tasks. , Global information capture capability is insufficient, making it difficult to effectively process long-term sequence data.
Combining the global modeling ability of the Transformer model and improving the local feature extraction ability of Ecapa-Tdnn, we optimize the model architecture, enhance the front-end convolution module, introduce multi-scale feature aggregation strategy, and improve the adaptability and accuracy of the model in language classification.
It significantly improves the accuracy and robustness of language classification, reduces the demand for computing resources, and is suitable for speech recognition tasks in multilingual environments, providing a more reliable solution.
Smart Images

Figure CN119091856A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of speech recognition, and in particular relates to an acoustic language classification method based on Transformer and improved Ecapa-Tdnn. Background Art
[0002] With the deepening development of globalization, the use of different languages and dialects is becoming more and more common. How to accurately and quickly identify and classify languages has become an important research topic in the field of speech processing. Traditional language recognition methods usually rely on specific speech features and statistical models, such as Gaussian mixture models (GMM) and hidden Markov models (HMM). However, these methods often have limited performance when faced with complex and changing speech signals.
[0003] In recent years, deep learning technology has made significant progress in speech recognition and natural language processing. Among them, convolutional neural networks (CNN) and recurrent neural networks (RNN) are widely used in language recognition tasks, but they have certain limitations in processing long time series and capturing global dependencies. In order to overcome these limitations, the Transformer model was introduced into this field. The Transformer model achieves efficient modeling of long-distance dependencies through the self-attention mechanism, and due to its highly parallelized architecture, it significantly improves the speed of training and reasoning.
[0004] However, there are still some challenges in using the Transformer model alone for language classification. Although the Transformer model is good at capturing global information, it is relatively weak in extracting short-term speech features, especially when processing local features in the frequency and time dimensions, which may not be as good as specially designed acoustic models. In addition, the Transformer model has a large number of parameters and high computing resource requirements, which may bring certain burdens in practical applications.
[0005] Ecapa-Tdnn is a deep neural network architecture optimized for speech processing. The model performs well in speaker recognition tasks by emphasizing channel attention mechanisms, propagation, and aggregation of multi-scale features. However, Ecapa-Tdnn also has some shortcomings when dealing with language classification tasks: (1) Lack of targeting. Ecapa-Tdnn is mainly designed for speaker recognition, and its structure may not be completely suitable for language classification tasks. (2) Insufficient ability to capture global information. When processing long time series data, Ecapa-Tdnn will suffer from performance degradation due to the lack of global information capture capability. (3) Feature extraction is limited. Although Ecapa-Tdnn performs well in extracting local features of speech signals, it still has limitations in capturing long-distance dependencies across time and frequency.
[0006] In order to improve the performance of language classification, the present invention proposes an acoustic language classification method based on Transformer and improved Ecapa-Tdnn. This method combines the advantages of the Transformer model and Ecapa-Tdnn and overcomes their respective shortcomings.
[0007] Specifically, the method of the present invention includes the following improvements: (1) Combining global and local features. By combining the global modeling capability of the Transformer model and improving the local feature extraction capability of Ecapa-Tdnn, the model can simultaneously capture the global dependencies and local details of the speech signal, thereby improving the classification accuracy. (2) Optimizing the model architecture, improving on the basis of ECAPA-TDNN, and enhancing its adaptability in language classification tasks. For example, by adjusting the structure of the front-end convolution module, it is more suitable for the extraction of language features. (3) Multi-scale feature aggregation: Through the multi-scale feature aggregation strategy, the model can effectively utilize speech features of different scales to enhance the robustness and accuracy of classification.
[0008] In summary, the acoustic language classification method based on Transformer and improved Ecapa-Tdnn proposed in the present invention fully utilizes the advantages of the two models and improves the performance and efficiency of language classification. This method not only achieves significant improvement in classification accuracy, but also shows high practicality in computing resource utilization and practical applications, providing a more reliable solution for related research and applications in the field of speech processing. Summary of the invention
[0009] The present invention proposes an acoustic language classification method based on Transformer and improved Ecapa-Tdnn, the purpose of which is to improve the performance and efficiency of language classification by improving the model structure and parameters, combining the global modeling capability of the Transformer model with the advantages of Ecapa-Tdnn in speech feature extraction.
[0010] The present invention is achieved through the following technical solutions:
[0011] An acoustic language classification method based on Transformer and improved Ecapa-Tdnn specifically includes the following steps:
[0012] S1, preprocess the speech data and extract the Fbank features of the speech data;
[0013] S2, perform data enhancement to obtain a processed spectrum graph;
[0014] S3. The processed input is subjected to acoustic language classification through the improved Ecapa-Tdnn model.
[0015] Furthermore, in the step S1, the preprocessing of the speech data includes: initializing the audio segment, converting the audio data samples into floating point data of type float32; setting the sampling rate of the audio data to 16kHz, and normalizing the volume of the audio data; and using an audio enhancement method of speech rate enhancement.
[0016] Furthermore, in step S1, the Fbank feature of the speech data is extracted by the following steps: converting the audio waveform into a digital signal by sampling and quantizing; performing a discrete Fourier transform on the audio data after framing to obtain spectrum information, using 80 Mel filters to collect energy in different frequency bands, mapping the spectrum energy to the Mel frequency scale, each filter focusing on a specific frequency range, simulating the human ear's perception of different frequencies, and thus obtaining the Fbank feature.
[0017] Furthermore, in step S2, the means of data enhancement include: realizing data enhancement by masking random time and frequency regions on the spectrogram of the audio data. Taking the frequency mask operation as an example, firstly, the batch size, time dimension and frequency dimension of the spectrogram are obtained, and then the frequency mask length and position of each sample are randomly generated. After the mask is generated, the eigenvalue at the mask position is set to zero, and then the processed spectrogram is returned. The time mask operation is the same.
[0018] Furthermore, in step S3, the overall structure of the improved Ecapa-Tdnn model includes: a feature extraction part, a feature fusion part, and a classification part.
[0019] Furthermore, in step S3, the feature extraction part is mainly composed of the following steps: the input data is subjected to feature extraction through a front-end convolution module consisting of a convolution layer and two basic residual blocks, and then through three consecutive enhanced residual blocks. Each enhanced residual block performs feature processing and adaptive adjustment of channel relationships through the Res2Net structure and SE mechanism. The output of each block is added to the previous input to form a residual connection. Finally, the outputs of the three blocks are spliced in the channel dimension to form a large feature map.
[0020] Furthermore, in the step S3, the front-end convolution module of Ecapa-Tdnn only extracts features through one-dimensional convolution Conv1d and simple activation functions, and batch normalization, and has limited ability to extract complex data and high-level features. Therefore, in the present invention, the front-end convolution module including the basic residual block is upgraded, including three parts: preliminary convolution layer, residual layer, and post-processing convolution layer. The preliminary convolution layer includes a Conv2d layer for performing convolution operations on the input to increase the number of channels; a BatchNorm2d layer for batch normalization of the convolution results. The residual blocks stacked after the convolution layer are used to further extract features, and the post-processing convolution layer includes a Conv2d layer to adjust the number of channels and spatial dimensions. This front-end convolution module can extract more complex and high-order features, and the processing of input data is more comprehensive and detailed. Due to the addition of residual connections, it helps to alleviate the gradient vanishing problem and improve the training stability and performance of the model.
[0021] Furthermore, in step S3, the basic residual block includes two convolutional layers, each of which is followed by a batch normalization layer, and a shortcut connection to ensure that the dimensions of the input and output are consistent. The enhanced residual block first implements feature decomposition and combination through grouped convolution to improve the granularity of feature extraction, and then follows the convolutional layer with ReLU activation and batch normalization for feature conversion, and then uses the channel attention mechanism to enhance the weights of important features through global average pooling and fully connected layers.
[0022] Furthermore, in step S3, the feature fusion part is composed of the following steps: multiple Transformer encoder layers, each layer is composed of a multi-head self-attention mechanism and a feedforward neural network, which is used to capture the global dependency of the input features. The concatenated feature map is input into the Transformer encoder for global feature modeling, and then the features are further processed through a 1D convolution layer, ReLU activation and batch normalization.
[0023] Furthermore, in step S3, the classification part is composed of the following steps: feature aggregation pooling layer, fully connected layer and output layer. In step S3, the feature aggregation pooling layer uses the attention statistical pooling layer to perform feature aggregation through weighted average and standard deviation. After feature aggregation, the pooled features are mapped to the target category space through two layers of fully connected layers and batch normalization. A batch normalization layer is added after the fully connected layer to stabilize the training process and accelerate convergence. Finally, the output layer calculates the probability value of each category through the softmax function to achieve acoustic language classification.
[0024] Beneficial effects: This invention innovatively combines the global modeling capability of the Transformer model with the local feature extraction advantage of Ecapa-Tdnn. By optimizing the front-end convolution module, introducing the Transformer encoder and the multi-scale feature aggregation strategy, the model's feature extraction capability for speech signals in the frequency and time dimensions is enhanced, and the accuracy, robustness and computational efficiency of language classification are improved, making it suitable for speech recognition tasks in multilingual environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 The present invention provides a flowchart of an acoustic language classification method based on Transformer and improved Ecapa-Tdnn.
[0026] Figure 2 Schematic diagram of the improved Ecapa-Tdnn model proposed in the present invention.
[0027] Figure 3 Schematic diagram of the front-end convolution module. DETAILED DESCRIPTION
[0028] The following is a detailed description of an embodiment of the present invention in conjunction with the accompanying drawings: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiments.
[0029] The following describes in detail the specific implementation process of an acoustic language classification method based on Transformer and improved Ecapa-Tdnn.
[0030] First, we prepare the multilingual speech dataset CSS10, which includes five languages: Chinese, Japanese, French, German, and Russian. The original audio clips are basically no longer than 10 seconds, with a total of about 15,000 audio clips in WAV format. The training set and validation set are divided into a ratio of nine to one.
[0031] Then read the audio data, split the audio path and label into three paths: training data, verification data, and classification label. Preprocess the speech data, initialize the audio segment, and convert the audio data sample into float32 type floating point data; set the sampling rate of the audio data to 16kHz, normalize the audio data volume, and set the normalized volume decibel value to -20; use the speech rate perturbation enhanced audio enhancement method, randomly select a speed ratio from 0.9, 1.0, 1.1 and apply it to the audio segment, so that the playback speed of the audio segment is different, enhancing the robustness of the model.
[0032] The audio preprocessing steps are as follows: (1) pre-emphasize the input audio signal; (2) frame division, that is, divide the audio into small segments, each frame lasting about 20 to 40 milliseconds, and there will be a certain overlap between each frame; (3) add a window function to each frame signal.
[0033] Then, a fast Fourier transform (FFT) is performed to obtain the spectrum information, and 80 Mel filters are used to collect the energy of different frequency bands, and the spectrum energy is mapped to the Mel frequency scale. Each filter focuses on a specific frequency range to simulate the human ear's perception of different frequencies. In this way, the Fbank feature is obtained.
[0034] Data augmentation is achieved by masking random time and frequency regions on the spectrogram of audio data. The width range of the frequency mask is (0,8), and the width range of the time mask is (0,10). Taking the frequency mask operation as an example, first obtain the shape of the input tensor, that is, the batch size, time dimension and frequency dimension of the spectrogram, and then randomly generate the frequency mask length and position of each sample. For each sample, a frequency mask length is randomly generated within the given frequency mask width range, that is, (0,8). The specific steps are as follows: Generate a tensor with a tensor size of (batch_size,1), and then expand the generated mask length tensor mask_len to a shape of (batch_size,1,1) through unsqueeze to facilitate subsequent broadcast operations with the frequency dimension. At the same time, on the feature dimension of each sample, a starting position is randomly generated as the starting point of the mask. The specific steps are as follows: a tensor is generated, the size of which is (batch_size, 1), and the generated mask position tensor mask_pos is also expanded to the shape of (batch_size, 1, 1) by unsqueeze. After the mask is generated, the eigenvalues at the mask position are set to zero, and then the processed spectrum is returned. The time mask operation is similar, and the mask operation is performed on the time dimension.
[0035] The processed audio information is used as input data through the improved ECAPA-TDNN model, whose overall structure includes: feature extraction part, feature fusion part, and classification part.
[0036] First is the feature extraction part. The input data is extracted through a front-end convolution module consisting of a convolution layer and two basic residual blocks. The feature dimension is converted into a form that is not suitable for subsequent processing through Conv2d convolution and subsequent reshape operations. Then, three consecutive enhanced residual blocks are used. Each enhanced residual block performs feature processing and adaptive adjustment of channel relationships through the Res2Net structure and SE mechanism. The output of each block is added to the previous input to form a residual connection. Finally, the outputs of the three blocks are spliced in the channel dimension to form a large feature map.
[0037] The basic residual block contains two convolutional layers, each followed by a batch normalization layer and a shortcut connection to ensure that the dimensions of the input and output are consistent. The enhanced residual block first implements feature decomposition and combination through grouped convolution to improve the granularity of feature extraction, and then uses ReLU activation and batch normalization after the convolution layer for feature conversion, and then uses the channel attention mechanism to enhance the weights of important features through global average pooling and fully connected layers.
[0038] The feature fusion part consists of multiple Transformer encoder layers to capture the global dependencies of the input features. The concatenated feature maps are input into the Transformer encoder for global feature modeling, and then further processed through a 1D convolution layer, ReLU activation, and batch normalization.
[0039] The entire Transformer encoder module consists of multiple Transformer layers, each of which contains two sub-layers, a multi-head self-attention mechanism, and a feed-forward neural network.
[0040] The multi-head self-attention mechanism captures information in different subspaces through multiple attention heads:
[0041]
[0042] Among them, Q (Query), K (Key), and V (Value) are all obtained by linear transformation of the input. The multi-head mechanism calculates multiple attentions in parallel, then concatenates them and obtains the final output through linear transformation.
[0043] The feedforward neural network consists of two linear transformations and an activation function ReLU:
[0044] FFN(x)=max(0,xW1+b1)W2+b2
[0045] After being processed by the previous convolutional layer, the shape of the data is (N, 3*C, T / 8, F). After being transposed and stacked through the Transformer layer, the long-range dependencies between time steps are captured, and the dimension remains unchanged. The data is transposed again to keep the shape unchanged from the original input.
[0046] The classification part consists of a feature aggregation pooling layer and a fully connected layer. The feature aggregation pooling layer uses the attention statistics pooling layer to perform feature aggregation through weighted average and standard deviation, where the weighted sum calculation formula is:
[0047]
[0048] where αt is the attention weight, e t is the calculated unnormalized score.
[0049] The input features are weighted summed using the attention weights to obtain a global representation:
[0050]
[0051] After feature aggregation, the pooled features are mapped to the target category space through two fully connected layers and batch normalization. A batch normalization layer is added after the fully connected layer to stabilize the training process and accelerate convergence. Finally, the output layer calculates the probability value of each category through the softmax function to achieve acoustic language classification.
[0052] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. An acoustic language classification method based on Transformer and improved Ecapa-Tdnn, characterized in that: The specific steps include: S1, preprocess the speech data and extract the Fbank features of the speech data; S2, perform data enhancement to obtain a processed spectrum graph; S3. The processed input is subjected to acoustic language classification through the improved Ecapa-Tdnn model.
2. The acoustic language classification method based on Transformer and improved Ecapa-Tdnn according to claim 1, characterized in that: In the step S1, the preprocessing of the speech data includes: initializing the audio segment, converting the audio data samples into floating point data of type float32; setting the sampling rate of the audio data to 16kHz, normalizing the volume of the audio data; and using an audio enhancement method of speech rate enhancement.
3. The acoustic language classification method based on Transformer and improved Ecapa-Tdnn according to claim 1, characterized in that: In step S1, the Fbank feature of the speech data is extracted by the following steps: converting the audio waveform into a digital signal by sampling and quantizing; performing discrete Fourier transform on the audio data after framing to obtain spectrum information, using 80 Mel filters to collect energy in different frequency bands, mapping the spectrum energy to the Mel frequency scale, each filter is concentrated in a specific frequency range, simulating the human ear's perception of different frequencies, and thus obtaining the Fbank feature.
4. The acoustic language classification method based on Transformer and improved Ecapa-Tdnn according to claim 1, characterized in that: In step S2, the means of data enhancement include: realizing data enhancement by masking random time and frequency regions on the spectrogram of the audio data. Taking the frequency mask operation as an example, firstly, the batch size, time dimension and frequency dimension of the spectrogram are obtained, and then the frequency mask length and position of each sample are randomly generated. After the mask is generated, the feature value at the mask position is set to zero, and then the processed spectrogram is returned. The time mask operation is the same.
5. The acoustic language classification method based on Transformer and improved Ecapa-Tdnn according to claim 1, characterized in that: In step S3, the overall structure of the improved Ecapa-Tdnn model includes: a feature extraction part, a feature fusion part, and a classification part.
6. The acoustic language classification method based on Transformer and improved Ecapa-Tdnn according to claim 1, characterized in that: In step S3, the feature extraction part is mainly composed of the following steps: the input data is subjected to feature extraction by a front-end convolution module consisting of a convolution layer and two basic residual blocks, and then passes through three consecutive enhanced residual blocks. Each enhanced residual block performs feature processing and adaptive adjustment of channel relationships through the Res2Net structure and SE mechanism. The output of each block is added to the previous input to form a residual connection. Finally, the outputs of the three blocks are spliced in the channel dimension to form a large feature map.
7. The acoustic language classification method based on Transformer and improved Ecapa-Tdnn according to claim 1, characterized in that: In the step S3, the front-end convolution module of Ecapa-Tdnn only extracts features through one-dimensional convolution Conv1d and simple activation functions, and batch normalization, and has limited ability to extract complex data and high-level features. Therefore, in the present invention, the front-end convolution module including the basic residual block is upgraded, including three parts: a preliminary convolution layer, a residual layer, and a post-processing convolution layer. The preliminary convolution layer includes a Conv2d layer for performing convolution operations on the input to increase the number of channels; a BatchNorm2d layer for batch normalization of the convolution results, and the residual blocks stacked after the convolution layer are used to further extract features. The post-processing convolution layer includes a Conv2d layer to adjust the number of channels and spatial dimensions. This front-end convolution module can extract more complex and high-order features, and the processing of input data is more comprehensive and detailed. Due to the addition of residual connections, it helps to alleviate the gradient vanishing problem and improve the training stability and performance of the model.
8. The acoustic language classification method based on Transformer and improved Ecapa-Tdnn according to claim 1, characterized in that: In step S3, the basic residual block includes two convolutional layers, each of which is followed by a batch normalization layer and a shortcut connection to ensure that the dimensions of the input and output are consistent. The enhanced residual block first implements feature decomposition and combination through grouped convolution to improve the granularity of feature extraction, and then connects ReLU activation and batch normalization after the convolution layer for feature conversion. Then, the channel attention mechanism is used to enhance the weights of important features through global average pooling and fully connected layers.
9. The acoustic language classification method based on Transformer and improved Ecapa-Tdnn according to claim 1, characterized in that: In step S3, the feature fusion part is composed of the following steps: multiple Transformer encoder layers, each layer is composed of a multi-head self-attention mechanism and a feedforward neural network, which is used to capture the global dependency of the input features, and the concatenated feature map is input into the Transformer encoder for global feature modeling, and then the features are further processed through a 1D convolution layer, ReLU activation and batch normalization.
10. The acoustic language classification method based on Transformer and improved Ecapa-Tdnn according to claim 1, characterized in that: In step S3, the classification part is composed of the following steps: feature aggregation pooling layer, fully connected layer and output layer. In step S3, the feature aggregation pooling layer uses the attention statistical pooling layer to perform feature aggregation through weighted average and standard deviation. After feature aggregation, the pooled features are mapped to the target category space through two layers of fully connected layers and batch normalization. A batch normalization layer is added after the fully connected layer to stabilize the training process and accelerate convergence. Finally, the output layer calculates the probability value of each category through the softmax function to realize acoustic language classification.
