An end-to-end speech recognition method based on a multi-path convolutional network

Abstract

The application discloses an end-to-end speech recognition method based on a multi-channel convolutional network, which is specially used for improving the recognition of the coarse-granularity pronunciation unit in Chinese pronunciation in the Chinese speech recognition process, adopts a multi-channel convolutional network (MCNN) to extract features of speech data from the time frame of pronunciation and the frequency spectrum of pronunciation; a multi-channel convolutional network of the MCNN is added to the input front end of the transformer to obtain the local features of the speech in advance, the transformer network is input into the training of the speech recognition, and a CTC structure is further added after the Encoder layer, finally, a joint training model is used to assist the training of the CTC, so that the convergence speed of the overall model is accelerated. A large number of experiments prove that the application has stronger robustness and higher recognition rate than general speech recognition models, and because the model can be modularized, stronger feasibility is provided for the later improvement.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, specifically relating to an end-to-end speech recognition method. Background Technology

[0002] Automatic speech recognition (ASR), as an important means of human-computer interaction, aims to enable computers to "understand" human speech and extract the textual information contained within it. With continuous development, speech recognition has evolved into a comprehensive technology involving multiple disciplines such as acoustics, linguistics, digital signal processing, and statistical pattern recognition.

[0003] Speech recognition has seen new developments in the era of deep learning. Traditional speech recognition models typically consist of three parts: an acoustic model (AM), a lexicon, and a language model (LM). Each part requires separate learning and training. Establishing an end-to-end learning mechanism that allows model training to bypass the lexicon and language model, truly achieving direct speech-to-text transcription, has become a current research hotspot. Currently, there are two main approaches to achieving end-to-end learning: Connectionist Temporal Classification (CTC) and Sequence-to-Sequence Model based on Attention.

[0004] For Chinese speech recognition, the pronunciation granularity of each character is large to a large extent. Using fine-grained speech recognition methods does not work very well for Chinese speech recognition. In addition, a large amount of speech data in real speech environments contains noise. The Transformer model does not perform well in processing noisy speech data, while CTC performs better in processing real noisy data because it is very effective in forcing alignment of sequence data.

[0005] MCNN (Multi-channel Convolutional Neural Network) primarily expands the width of traditional CNNs by employing a three-channel design. It uses three convolutional layers to extract speech information, fully extracting local features from the high-dimensional information of speech. It can extract additional detailed features from speech in terms of width. However, MCNN alone can only extract speech features from each frame, but it cannot capture the relationships between different frames. Therefore, a transformer is needed to compensate for this deficiency.

[0006] Proposed by Google's machine translation team in 2017, Transformer is an end-to-end framework system based on a self-attention mechanism. It abandons traditional CNNs and RNNs, and the entire network structure is composed entirely of self-attention mechanisms. A trainable neural network based on Transformer can be built by stacking Transformers, which consists of two structures: Encoder and Decoder. A standard Transformer architecture consists of 12 layers of Encoder-Decoder, providing a more efficient solution for the seq2seq problem.

[0007] The Temporal Classification (CTC) model, proposed by Graves et al., is a temporal classification method that avoids manual alignment of input and output, making it well-suited for applications such as speech recognition or OCR. The input sequence X = [x1, x2, ..., xT] and the label data Y = [y1, y2, ..., yU], such as speech files and text files in speech recognition, aim to find a mapping from X to Y. This algorithm for classifying temporal data is called Temporal Classification. However, CTC as a loss function cannot establish the relationships between each frame of speech, so it needs to collaborate with other neural networks. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention provides an end-to-end speech recognition method based on multi-channel convolutional networks (MCNNs). Specifically designed to improve the coarse-grained recognition of articulatory units in Chinese speech recognition, this method employs MCNNs to extract features from speech data based on the time frame and frequency spectrum of the pronunciation. An additional MCNN is added to the input front of the transformer to pre-acquire local speech features, which are then input into the transformer network for speech recognition training. A CTC structure is added after the Encoder layer. Finally, a joint training model is used with CTC as an auxiliary training method to accelerate the convergence speed of the overall model. Extensive experiments demonstrate that this invention has stronger robustness and a recognition rate exceeding that of general speech recognition models. Furthermore, because the model can be modularly assembled, it provides greater feasibility for future improvements.

[0009] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0010] Step 1: Convert the sampling rate of all speech data to aHz by resampling;

[0011] wav = resampel(wav, a)

[0012] Step 2: Perform speech preprocessing on the resampled speech data (wav);

[0013] The Fbank output mode is adopted. The first step is pre-emphasis processing, which filters the low-frequency band of the speech signal through a high-pass filter. The second step is to perform frame processing on the speech signal after passing through the high-pass filter, dividing the variable-length audio into small segments of fixed length to obtain the framed speech signal. Then, a sliding Hamming window function is set to perform Fourier transform on these framed speech signals to transform the speech data from a time domain signal to a frequency domain signal. Finally, the obtained frequency domain signal is processed by log-Mel spectrum filtering to complete the Fbank calculation.

[0014] wav_frame=hamming(wav, 10ms)

[0015] wav_data = fft(wav_frame)

[0016] wav Fbank = log_mel(wav_data)

[0017] Step 3: After performing Fbank processing on the speech data in Step 2, a speech data wav with 80-dimensional spectral features is obtained. Fbank[batch_size, wav_input, 80], where batch_size represents the size of each batch of data and wav_input represents the length of the input speech;

[0018] All voice data is padded with length so that the dimension of wav_input reaches 200:

[0019] wav_data′[batch_size, 200, 80]

[0020] =padding(wav_data[batch_size,wav_input,80])

[0021] Step 4: Input the speech data obtained in Step 3 into the multi-channel convolutional network MCNN;

[0022] The speech data is input into three channels, all with identical structures. First, it enters a first-level 2D convolutional CNN layer, where each convolutional layer includes a batch normalization (BatchNorm2d) and ReLU activation function. Each CNN has 16 filter banks, each with a 3x3 kernel size and a stride of 1. The output of this first-level 2D convolutional CNN layer is then input into a second, identical 2D convolutional layer, followed by a 2D max-pooling layer. Finally, it is input into a Dropout layer with a deactivation rate of 0.3. The speech data parameters then become [1, 100, 40, 16], where the first parameter is bitch. The first parameter is the size of the audio file, the second is the audio length, the third is the feature dimension of the audio file, and the fourth is the filter size of the convolutional layer. The data is then fed into the third Conv2D 2D convolutional layer, where the number of filter banks is changed to 32 compared to the first layer, resulting in output speech data parameters of [1, 50, 20, 32]. After two more convolutional layers, the data enters the sixth Conv2D 2D convolutional layer, where the number of filter banks is changed to 64 compared to the first layer, resulting in output speech data parameters of [1, 25, 10, 64]. Finally, the data is reshaped in the reshape layer to re-merge the speech data wav_data [1, 25, 640].

[0023] wav_data′ i =Maxpool(BatchNorm2d(Relu(Conv2D(wav_data′)), epsilon=0.0002), pool_size=[2, 2])

[0024] wav_data i =reshape(1,25,640)(wav_data′i i = 1, 2, 3 represent 3 channels;

[0025] Step 5: Merge the speech data output from the three channels while keeping the length dimension of the input speech unchanged. Merge the spectral feature dimensions and output the data. Then, the data enters the fully connected layer FC. The first fully connected layer Dense1 has an input of 1920 and an output of 1024. The second fully connected layer Dense2 has an input of 1024 and an output of 512. The third fully connected layer Dense3 has an input of 512 and an output of 320. Each pair of fully connected layers is connected by a batch normalization BatchNorm2d and a ReLU activation function. The final output is wav_data″[1, 25, 320].

[0026] wav_data″=FC(Merge(wav_data1, wav_data2, wav_data3))

[0027] Step 6: Construct a speech recognition network based on the Transformer model, including an encoder and a decoder, where both the encoder and decoder are implemented by a multi-head attention mechanism (MHA); both the encoder and decoder consist of a multi-head attention mechanism and a position feedforward network (FFN) module, and residual connections and layer normalization are used after each sub-layer.

[0028] The data output from the MCNN layer of the multi-channel convolutional network is fed into the Transformer model, where the encoder will output the input speech data wav″. data = (X1, X2, ..., X T The multi-head attention mechanism maps the data to latent space states (h1, h2, ..., h...). N The decoder then decodes the text labels provided: label(Y1, Y2, ..., Y...). L The latent space states (h1, h2, ..., h) output by the joint encoder layer N Decoding yields the final predicted target sequence pre_label. T (Y1, Y2, ..., Y) L );

[0029] wav_data″′(h1, h2, ..., h N = Encoder(X1, X2, ..., X...) r )

[0030] pre_label T (Y1, Y2, ..., Y) L)=Decoder((Y1,Y2,...,Y L-1 ), (h1, h2, ..., h N ))

[0031] Step 7: Convert the hidden state transition matrix (h1, h2, ..., h...) obtained by the encoder into a single matrix. N The input is fed into the CTC structure, and the forward and backward algorithms of CTC are used to force monotonic alignment between the speech and label sequences. CTC uses a greedy search method called greedy_search.

[0032] pre_label c (Y1, Y2, ..., Y) L )

[0033] =CTC_greedy_search((Y1,Y2,...,Y L-1 ), (h1, h2, ..., h N ))

[0034] L MTL =λL CTC +(1-λ)L attention

[0035] Where λ is a hyperparameter, L CTC L represents the CTC loss function. attention Let L represent the attention loss function. MTL This represents the multi-task loss function that combines the two.

[0036] Preferably, a = 16000.

[0037] Preferably, in step 2, the variable-length audio is divided into segments of fixed length, each segment being 10-30ms in length.

[0038] The beneficial effects of this invention are as follows:

[0039] 1. A method using MCNN (Multi-path Convolutional Neural Network) to process high-dimensional speech data was proposed. By using three pooling layers, the number of model parameters and the redundancy of speech features were reduced, making the model more lightweight.

[0040] 2. An MCNN method was designed to obtain local features of speech in advance in the front acoustic unit, thereby solving the problem that self-attention in transformer only focuses on global features, making the focus of the acoustic model in speech recognition model more comprehensive.

[0041] 3. A framework based on multi-path convolutional networks for speech recognition was designed. Extensive experiments demonstrated that the model has stronger robustness and a recognition rate exceeding that of general speech recognition models. Furthermore, because the model can be modularly assembled, it provides greater feasibility for future improvements. Attached Figure Description

[0042] Figure 1 This is a network diagram of the Multi-Path Convolutional Neural Network (MCNN) of this invention.

[0043] Figure 2 This invention is based on a self-attention mechanism architecture using dot products.

[0044] Figure 3 This invention relates to the multi-head attention mechanism architecture.

[0045] Figure 4 This invention is an end-to-end speech recognition model based on multi-path convolutional networks. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] The purpose of this invention is to provide an end-to-end speech recognition framework based on multi-path convolutional networks to improve the performance of speech recognition.

[0048] The purpose of this invention is to improve the coarse-grained recognition of articulatory units in Chinese speech recognition. It employs a Multi-Channel Convolutional Neural Network (MCNN) to extract features from speech data based on the time frame and frequency spectrum of the pronunciation. Furthermore, it addresses the issue in Transformer-based speech recognition frameworks where self-attention only focuses on global features while neglecting local features. The MCNN convolutional neural network can extract local features from the speech data before inputting them into the Transformer network, thereby improving the performance of Chinese speech recognition.

[0049] The relevant definitions in this invention are as follows:

[0050] Definition 1: Speech resampling is the process of uniformly changing the sampling rate of the input speech data to 16000Hz, with the main purpose of reducing data dimensionality.

[0051] We define 2Fbank as a speech feature parameter extraction method, which is mainly based on cepstral extraction. Its advantage is that it is more in line with the human hearing principle.

[0052] The 3relu activation function is defined to provide a nonlinear element to neurons, allowing neural networks to approximate any nonlinear function.

[0053] The 4Maxpooling layer is defined as having the effects of dimensionality reduction, reducing the number of parameters, removing redundant information, compressing features, simplifying network complexity, reducing computation, and reducing memory consumption.

[0054] Definition 5: The function of a high-pass filter is to allow frequencies above a certain cutoff frequency to pass through, removing unnecessary low-frequency components or low-frequency interference from the signal.

[0055] We define self-attention as a self-attention mechanism, where Q, K, and V matrices represent the query, keyword, and value vectors, respectively. The global attention probability is calculated by multiplying the dot products of these matrix vectors. The formula for calculating attention is as follows:

[0056]

[0057] The 7CER (Character Error Rate) is an important metric for evaluating ASR (Automatic Sentence Recognition) performance. It assesses the error rate between the predicted text and the target text. Therefore, a lower 7CER is better. Furthermore, since the smallest unit for Chinese sentence recognition is the Chinese character, the 7CER effectively measures the quality of Chinese ASR. Here, S represents the number of substitutions that occur when converting the predicted text to the target text, D represents the number of errors that occur, I represents the number of insertions that occur, and N represents the total number of characters in the target text.

[0058]

[0059] This invention is based on the following principle: In Chinese speech recognition, it improves the coarse-grained recognition of articulatory units in Chinese pronunciation. It employs a Multi-CNN (Multi-path Convolutional Network) to extract features from the speech data based on the time frame and frequency spectrum of the pronunciation. While transformer-based speech recognition utilizes a self-attention mechanism, obtaining the correlation between each output frame through the dot product of the Q-query vector, K-keyword vector, and V-value vector, these correlations only pertain to global features and cannot effectively capture local features of the speech data. Therefore, an MCNN multi-path convolutional network is added at the input front of the transformer to pre-acquire local speech features. Speech recognition training is performed within the input transformer network, and a CTC (Choice-Track Convolutional) structure is added after the Encoder layer. CTC_greedy_search (a greedy search algorithm) is used for prediction training. Finally, a joint training model is used with CTC as an auxiliary training method, thereby accelerating the convergence speed of the overall model.

[0060] The specific steps are as follows:

[0061] 1. Because the sampling rates of voice data are different (such as 48000Hz, 16000Hz, etc.), the sampling rate of all voice data is converted to 16000Hz through resampling (that is, 16000 points will be sampled for one second of voice).

[0062] wav = resampel(wav, 16000)

[0063] 2. In the preprocessing stage of the resampled speech data wav, we adopt the output mode of Fbank. The first step of pre-emphasis processing is to filter the low-frequency bands of the speech signal through a high-pass filter. The second step is to perform frame segmentation processing on the speech signal after passing through the high-pass filter, that is, to divide the variable-length audio into small segments of fixed length, generally 10-30ms per frame, and then obtain the framed speech signal wav[X1, X2, X3, ..., Xn]. Then, a sliding Hamming window function is set to perform Fourier transform on these framed sentences (discrete Fourier transform is used because it is digital audio), transforming the speech data from a time domain signal to a frequency domain signal. Finally, the acquired frequency domain signal is processed by log-Mel spectrum filtering to complete the Fbank calculation.

[0064] wav_frame=hamming(wav, 10ms)

[0065] wav_data = fft(wav_frame)

[0066] wav Fbank= log_mel(wav_data)

[0067] 3. After F-bank processing of the speech data in step 2, we obtain an 80-dimensional spectral feature: wav_data(batch_size, wav_input, 80) (where batch_size represents the size of each batch of data, and wav_input represents the length of the input speech). For the subsequent convolutional network, we will perform concatenation and padding operations on the speech data. Depending on the size of the data in the dataset, we pad all data to increase the length of wav_input to 200. The labels corresponding to the lengths of the padded speech are all set to... <block>.

[0068] wav_data[batch_size, 200, 80]

[0069] =padding(wav_data[batch_size,wav_input,80])

[0070] 4. The processed speech data is input into MCNN (Multi-Way Convolutional Network). MCNN extracts more useful features from the time and frequency dimensions. The speech data is input into three first-layer two-dimensional convolutional (CNN) layers. Each convolutional layer is accompanied by a batch normalization (BatchNorm2d) and ReLU activation function. Each CNN has 16 filter banks, each filter kernel size is 3*3, and the stride is 1. After the output of the first layer, it is input into the second layer, which is exactly the same as the previous two-dimensional convolutional layer. Then it is connected to a 2D max pooling layer. Finally, it is input into the Dropout layer with a deactivation rate of 0.3. The speech data becomes wav_data[1, 100, 40, 16], and then continues into the third (Conv2D) 2D convolutional layer. The number of filter banks in the convolutional layer is changed to 32, and the output speech data becomes wav_data[1, 50, 20, 32]. Finally, it enters the sixth (Conv2D) 2D convolutional layer, where the number of filter banks is changed to 64, and the output speech data becomes wav_data[1, 25, 10, 64]. Then, in the reshape layer (which represents changing the data dimensions), the speech data wav_data[1, 25, 640] is re-merged.

[0071] wav_data = Conv2D(wav_data)

[0072] wav_data = Relu(wav_data)

[0073] wav_data=BatchNorm2d(wav_data, epsilon=0.0002)

[0074] wav_data=Maxpool(wav_data, pool_size=[2, 2])

[0075] wav_data=reshape(1,25,640)(wav_data)

[0076] 5. Merge the speech data output from the three channels, keeping the dimensions of `wav_input` (representing the length of the input speech) unchanged. Merge the dimensions of `wav_input_feature` (representing the 80-dimensional spectral features after Fbank processing) to output `wav_data[1, 25, 1920]`, which then enters a fully connected layer (FC). The first fully connected layer is Dense1 (input = 1920, output = 1024), the second is Dense2 (input = 1024, output = 512), and the third is Dense3 (input = 512, output = 320). Each step is accompanied by a batch normalization (BatchNorm2d) and ReLU activation function, finally outputting `wav_data[1, 25, 320]`. (Wav_data1, wav_data2, and wav_data3 represent the processed speech data of each channel, respectively.)

[0077] wav_data=Merge(wav_data1, wav_data2, wav_data3)

[0078] wav_data = FC(wav_data)

[0079] 6. Begin constructing the Transformer model's speech recognition network, including an encoder and a decoder, both of which utilize a multi-head attention (MHA) mechanism. The encoder and decoder consist of a MHA mechanism and a position feedforward network (FFN) module, using skip connections and layer normalization after each sub-layer. The difference between the encoder and decoder lies in the decoder's use of a MHA mechanism that masks future information, preventing the decoder from seeing future label information. It feeds the data output from the multi-path convolutional network (MCNN) layers into the Transformer model, essentially pre-collecting local features, allowing the Transformer model to obtain more comprehensive features. Specifically, the encoder takes the input speech data wav_data = (x1, x2, ..., x...). T The data is mapped to the latent space state through a multi-head attention mechanism: wav_data = (h1, h2, ..., h...). N Then the decoder layer uses the provided text label label = (y1, y2, ..., y3) L-1 The latent space states (h1, h2, ..., h) are output by the joint encoder layer. N The decoding generates the final predicted target sequence pre_label = (y1, y2, ..., y3). L ).

[0080] wav_data(h1, h2, ..., h N = Encoder(X1, X2, ..., X...) T )

[0081] pre_label r (Y1, Y2, ..., Y) L )=Decoder((Y1,Y2,...,Y L-1 ), (h1, h2, ..., h N ))

[0082] 7. Then, the hidden state transition matrix (h1, h2, ..., h) obtained from the Encoder layer is... N The input is fed into the CTC structure. The forward and backward algorithms of CTC can force monotonic alignment between the speech and label sequences. CTC employs a greedy search method. One advantage of using CTC as an auxiliary task is that it accelerates the training speed of the neural network. Joint learning using CTC and Attention allows training that should converge at epoch=9 to converge at epoch=5. Here, λ is a hyperparameter.

[0083] pre-label c (Y1, Y2, ..., Y) L )

[0084] =CTC_greedy_search((Y1,Y2,...,Y L-1 ), (h1, h2, ..., h N ))

[0085] L MTL =λL cTC +(1-λ)L attention Specific implementation examples:

[0087] 1. Experimental Data

[0088] The data used consists of two datasets: THCHS30, an open-source Chinese speech database published by the Center for Speech and Language Technology (CSLT) at Tsinghua University, which can be used for the development of Chinese speech recognition systems. The speech data in this dataset was recorded in a quiet office environment, with a total duration exceeding 30 hours. The sampling frequency is 16kHz, and the sample size is 16 bits.

[0089] 2. Experimental Environment

[0090] Experimental hardware environment: Processor: 11th Gen Intel(R) Core(TM) i7-11700; Memory: 64GB; GPU: NVIDIA 3090; Experimental software environment: PyTorch deep learning environment under Windows.

[0091] 3. Model parameters

[0092] Multi-channel Convolutional Network (MCNN): This is a three-channel, six-layer Conv2D network (composed of a two-dimensional convolutional network). The first two convolutional layers have a filter size of 16, a 3x3 kernel, and a ReLU activation function. The middle two convolutional layers have only a filter size of 32. The last two convolutional layers have only a filter size of 64. Each convolutional layer is followed by a batch normalization layer with eps set to 0.0002. Then, every two layers are followed by a max pooling layer with a pooling size of [2, 2], followed by a dropout layer.

[0093] The deactivation ratio of the layer is set to 0.3.

[0094] The Transformer model consists of 6 encoder layers and 6 decoder layers, both of which are constructed using multi-head attention and feed-forward mechanisms. There are 8 attention heads, and each layer has an output dimension of 512.

[0095] 4. Experimental Results and Evaluation

[0096] The designed model was evaluated on the THCHS30 dataset. The baseline acoustic model we compared it to was the baseline model of this invention without MCNN. The results are shown in the table below:

[0097]

[0098] Comparing the results with the baseline model reveals that the multi-channel convolutional network (MCNN) model reduced the word error rate by 2.12% and 3.13% on the validation and test sets, respectively. MCNN demonstrates a significant improvement in speech recognition by enhancing its ability to extract coarse-grained data and also aids in local feature extraction. Furthermore, the addition of CTC loss allows training to converge at epoch 5, offering substantial advantages in training performance.< / block>

Claims

1. An end-to-end speech recognition method based on multi-path convolutional networks, characterized in that, Includes the following steps: Step 1: Convert the sampling rate of all speech data to aHz by resampling; wav = resampel(wav, a) Step 2: Perform speech preprocessing on the resampled speech data (wav); The Fbank output mode is adopted. The first step is pre-emphasis processing, which filters the low-frequency band of the speech signal through a high-pass filter. The second step is to perform frame processing on the speech signal after passing through the high-pass filter, dividing the variable-length audio into small segments of fixed length to obtain the framed speech signal. Then, a sliding Hamming window function is set to perform Fourier transform on these framed speech signals to transform the speech data from a time domain signal to a frequency domain signal. Finally, the obtained frequency domain signal is processed by log-Mel spectrum filtering to complete the Fbank calculation. wav_frame = hamming(wav, 10ms) wav_data = fft(wav_frame) wav Fbank =log_mel(wav_data) Step 3: After performing Fbank processing on the speech data in Step 2, a speech data wav with 80-dimensional spectral features is obtained. Fbank [batch_size, wav_input, 80], where batch_size represents the size of each batch of data and wav_input represents the length of the input speech; All voice data is padded with length so that the dimension of wav_input reaches 200: wav_data′[batch_size,200,80] =padding(wav_data[batch_size,wav_input,80]) Step 4: Input the speech data obtained in Step 3 into the multi-channel convolutional network MCNN; The speech data is input into three channels, all with identical structures. First, it enters a first-level 2D convolutional CNN layer, where each convolutional layer includes a batch normalization (BatchNorm2d) and ReLU activation function. Each CNN has 16 filter banks, each with a 3x3 kernel size and a stride of 1. The output of this first-level 2D convolutional CNN layer is then input into a second, identical 2D convolutional layer, followed by a 2D max-pooling layer. Finally, it is input into a Dropout layer with a deactivation rate of 0.

3. The speech data parameters then become [1, 100, 40, 16], where the first parameter is bitch. The first parameter is the size of the audio file, the second is the audio length, the third is the feature dimension of the audio file, and the fourth is the filter size of the convolutional layer. The data is then fed into the third Conv2D 2D convolutional layer, where the number of filter banks is changed to 32 compared to the first layer, resulting in output speech data parameters of [1, 50, 20, 32]. After two more convolutional layers, the data enters the sixth Conv2D 2D convolutional layer, where the number of filter banks is changed to 64 compared to the first layer, resulting in output speech data parameters of [1, 25, 10, 64]. Finally, the data is reshaped in the reshape layer to re-merge the speech data wav_data [1, 25, 640]. wav_data′ i =Maxpool(BatchNorm2d(Relu(Conv2D(wav_data′)),epsolon=0.0002),pool_size=[2,2]) wav_data i =reshape(1,25,640)(wav_data′ i i = 1, 2, 3 represents 3 channels; Step 5: Merge the speech data output from the three channels while keeping the length dimension of the input speech unchanged. Merge the spectral feature dimensions and output the data. Then, the data enters the fully connected layer FC. The first fully connected layer Dense1 has an input of 1920 and an output of 1024. The second fully connected layer Dense2 has an input of 1024 and an output of 512. The third fully connected layer Dense3 has an input of 512 and an output of 320. Each pair of fully connected layers is connected by a batch normalization BatchNorm2d and a ReLU activation function. The final output is wav_data″[1,25,320]. wav_data″=FC(Merge(wav_data1, wav_data2, wav_data3)) Step 6: Construct a speech recognition network based on the Transformer model, including an encoder and a decoder, where both the encoder and decoder are implemented by a multi-head attention mechanism (MHA); both the encoder and decoder consist of a multi-head attention mechanism and a position feedforward network (FFN) module, and residual connections and layer normalization are used after each sub-layer. The data output from the MCNN layer of the multi-channel convolutional network is fed into the Transformer model, where the encoder will output the input speech data wav″. data =(X1,X2,…,X) T The multi-head attention mechanism maps the data to latent space states (h1, h2, ..., h). N The decoder then decodes the text labels provided: label(Y1,Y2,…,Y). L The latent space states (h1, h2, ..., h) output by the joint encoder layer N Decoding yields the final predicted target sequence pre_label. T (Y1,Y2,…,Y L ); wav_data″′(h1,h2,…,h N )=Encoder(X1,X2,…,X T ) pre_label T (Y1,Y2,…,Y L )=Decoder((Y1,Y2,…,Y L-1 ),(h1,h2,…,h N )) Step 7: Convert the hidden state transition matrix (h1, h2, ..., h) obtained by the encoder into a single matrix. N The input is fed into the CTC structure, and the forward and backward algorithms of CTC are used to force monotonic alignment between the speech and label sequences. CTC uses a greedy search method called greedy_search. pre_label C (Y1,Y2,…,Y L ) =CTC_greedy_search((Y1,Y2,…,Y L-1 ),(h1,h2,…,h N )) THE MTL =λL CTC +(1-λ)L attention Where λ is a hyperparameter, L CTC L represents the CTC loss function. attention Let L represent the attention loss function. MTL This represents the multi-task loss function that combines the two.

2. The end-to-end speech recognition method based on a multi-path convolutional network according to claim 1, characterized in that, The value of a is 16000.

3. The end-to-end speech recognition method based on a multi-path convolutional network according to claim 1, characterized in that, In step 2, the variable-length audio is divided into segments of fixed length, each segment being 10-30ms in length.

Images