PCVCs end-to-end speech recognition method based on ResNet-GAU model
Through the ResNet-GAU model combined with CTC and transfer learning and data enhancement technology, the problem of high data labeling cost and noise sensitivity in civil aviation speech recognition is solved, the recognition accuracy and robustness are improved, and the speech recognition performance in civil aviation field is achieved promote.
Patent Information
- Application Number
- CN202210616528.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-01
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-06-01
AI Technical Summary
In the civil aviation field, existing speech recognition technology faces problems such as high data labeling costs, sensitive noise interference and insufficient training data, resulting in insufficient recognition accuracy and robustness.
Using the end-to-end speech recognition method based on the ResNet-GAU model, the time-frequency domain features are extracted through ResNet, the GAU captures the long-distance dependence of sequences, and combines CTC to perform hard alignment of speech signals and text labels, using transfer learning and data augmentation technology Optimize model performance.
It improves the accuracy and robustness of speech recognition in civil aviation, reduces the impact of noise interference on performance, enhances the generalization ability of the model, and achieves the improvement of recognition performance on target tasks.
Smart Images

Figure CN115019801B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech recognition technology, and in particular to a PCVCs end-to-end speech recognition method based on a ResNet-GAU model. Background Art
[0002] The main task of air traffic control (ATC) is to prevent collisions and maintain and accelerate the orderly flow of air traffic. Control instructions are mainly issued through pilot-controller voice calls (PCVCs), and the accuracy of instructions is extremely important for ensuring flight safety. In the continuous development of air transportation, the contradiction between traffic growth and flight safety has become more prominent, and artificial intelligence technology has gradually been integrated into air traffic control, such as consistency monitoring between controller instructions and pilot repetitions, and post-event voice analysis. The research and application of new technologies can effectively reduce the workload of controllers and improve operational efficiency and safety levels.
[0003] Automatic speech recognition (ASR) technology was introduced into the ATC field as early as the early 1990s. Wang J et al. used the open source English dataset librispeech to pre-train the model, and then used the ATC dataset to fine-tune the model to solve the problem of scarce civil aviation data; Ajay et al. proposed a deep learning-based algorithm to transcribe aviation instructions. In order to expand the training set, non-aviation speech was incorporated to improve generalization. Y. Wang et al. applied DNN to speech separation. Considering the temporal dynamic characteristics of speech, structured perceptrons and conditional random fields were introduced to optimize the model, which significantly improved the robustness of noisy ASR; D. Wang et al. explored time-frequency masking, spectrum mapping, and end-to-end time domain mapping methods, and achieved better enhanced performance; Zhou K et al. used convolution invariance to overcome the diversity of speech signals themselves, solve the noise problem of data, and used the hybrid CTC-attention model to achieve good recognition results on ATC data. Lin Y et al. used a multi-scale convolutional network to extract local features of the spectrum graph and designed a single framework based on three cascade modules to solve the ASR problem of mixed Chinese and English. Helmke et al. used ASR technology and radar technology to build an assistant-based speech recognition system, which can significantly reduce the task pressure of air traffic controllers. Based on the ASR toolkit Kaldi, Helmke et al. transcribed the content of ground-to-air conversations. After testing, they found that using this technology to assist air traffic controllers can effectively reduce the unplanned flight time of aircraft and improve control efficiency and economic benefits.
[0004] Despite a lot of research, speech recognition in air traffic control still needs to be improved in terms of recognition rate, overall performance, and technology integration, mainly including:
[0005] (1) Most speech recognition systems with high accuracy require a large amount of speech data with text annotations to train the model. The data scale often needs to reach hundreds or even thousands of hours, while in the industry, the training data can even reach tens of thousands or even hundreds of thousands of hours. However, such a large amount of data is difficult to obtain in a specific field such as civil aviation, because the annotation of civil aviation speech data requires professionals, and it is too costly to obtain a large-scale and high-quality data set.
[0006] (2) The most advanced speech recognition systems can achieve extremely high recognition accuracy under experimental conditions. However, in actual applications, it is often difficult to ensure that they are always in an ideal environment. Therefore, many systems often have significantly reduced performance in some complex environments. One of the key reasons for this phenomenon is the existence of noise interference. For recognition algorithms that are not optimized for noise robustness, even weak noise may cause a significant drop in performance.
[0007] (3) The end-to-end speech recognition system simultaneously trains the acoustic and language models, which simplifies the training process. However, since the decoder replaces the language model in the traditional architecture, only a limited number of paired audio-text data are used during training, which is insufficient to fully cover the full range of scenarios with professional terms and instructions in the civil aviation field. Summary of the invention
[0008] To solve the above problems, the present invention provides a PCVCs end-to-end speech recognition method based on the ResNet-GAU model, in which ResNet uses the translation invariance and local correlation of CNN to extract the time-frequency domain information of the speech signal; GAU uses the gated single-head attention mechanism to not only better capture the long-distance dependency of the sequence to obtain a larger receptive field and context information, but also has a faster training convergence speed; CTC uses the introduction of blank and the generation of repeated tokens to solve the problem of hard alignment of speech signals and text labels. The technical solution is as follows:
[0009] The PCVCs end-to-end speech recognition method based on the ResNet-GAU model includes the following steps:
[0010] ResNet is used to extract the time-frequency domain features of the speech signal to be recognized;
[0011] Through multiple serially connected GAU modules, the word order information of time-frequency domain features is captured;
[0012] The word order information is passed into the Dense layer, and after a linear transformation in the feature space, the final classification prediction probability is obtained through the softmax layer.
[0013] Preferably, the method of extracting the time-frequency domain features of the speech signal to be recognized by using ResNet comprises the following steps:
[0014] The input speech to be recognized passes through the 7×7 Conv Layer. When the convolution kernel is working, it will regularly scan the input features, perform matrix element multiplication and sum of the input features in the receptive field, and add the deviation;
[0015] The output feature map is passed to the Max Pooling Layer for feature selection and information filtering; the PoolingLayer contains a pre-set pooling function that replaces the result of a single point in the feature map with the feature map statistics of its adjacent area;
[0016] The feature vector after maximum pooling is sequentially passed through a stack composed of different numbers of Residual blocks to obtain highly purified features. Shortcut connections are introduced between the stacks inside the blocks to pass the input across layers and add the convolution results, alleviating the gradient vanishing problem caused by increasing the depth of CNN.
[0017] The features of the final output of ResNet are reduced in dimension through Flattenlayer and used as the input of multiple series-connected GAUmodules.
[0018] Preferably, it also includes:
[0019] Absolute position encoding is added to the input to ensure consistency in the time dimension. Shortcut connections are introduced between GAUmodules to pass the input of the GAU module across layers and add it to the convolution result.
[0020] Preferably, the capturing of word order information of time-frequency domain features comprises the following steps:
[0021] The input time-frequency domain features pass through Layernorm, and then pass through Dense Layer and nonlinear transformation to obtain the gating matrix U and the Value, Query and Key for calculating attention;
[0022] U=φ u (XW u ) V=φ v (XW v )∈R T×e
[0023] Q&K=φ q&k (XW q&k )∈R T×s
[0024] Where X∈R T×d , T represents the sequence length, d represents the model size; W U , W V ∈R d×e , e represents expandedintermediate size; W q&k ∈R d×s , s represents headsize; represents the activation function;
[0025] Calculate the attention score using relu 2 Activate the function and add relativepositionbias:
[0026] A=Relu 2 (QK T +B)∈R T×T
[0027] O=(U⊙AV)W o ∈R T×d
[0028] Among them, A is the attention matrix, which is responsible for fusing the information between tokens; ⊙ represents element-wise multiplication;
[0029] Dropoutlayer is used to prevent overfitting during model training.
[0030] Preferably, the method further includes training a speech recognition model composed of ResNet and a plurality of serially connected GAU modules, comprising the following steps:
[0031] Using feature enhancement technology on speech data in a source domain speech database to obtain an enhanced source domain speech database;
[0032] Transfer learning is introduced to fine-tune the speech recognition model trained in the enhanced source domain speech database in the target domain air traffic control speech database to obtain a trained speech recognition model.
[0033] Preferably, the speech recognition model is trained on a large-scale source domain Aishell corpus, and then the trained model is further fine-tuned on a small-scale target domain ATC corpus.
[0034] Preferably, the method of using feature enhancement technology on the speech data in the source domain speech database comprises the following steps:
[0035] The time-frequency masking technology in SpecAugment is used. The maximum range of continuous masking in the time and frequency domain axes is set to 250ms and 100Hz. A t(f) is uniformly sampled in the range of [0,250] and [0,100] of the source domain Aishell corpus. A point t0(f0) is randomly determined in the range of [0,τ-t] and [0,τ-f]. Starting from the position t0(f0), masking is performed continuously t(f) times along the time and frequency domain axes, where τ represents the time dimension and v represents the frequency domain dimension.
[0036] Add Gaussian white noise with a certain signal-to-noise ratio to the source domain Aishell corpus;
[0037] Perform speech rate enhancement operations in the source domain Aishell corpus.
[0038] Preferably, it also includes automatically aligning input and output sequences of unequal lengths according to a time series classification algorithm, and introducing a special character blank so that each frame of the speech sequence can predict its conditional probability distribution on the output label sequence one by one.
[0039] Preferably, it also includes:
[0040] The speech enhancement technology is used to extract useful speech signals from the electromagnetic interference noise background to be recognized as the speech signals to be recognized.
[0041] Beneficial effects of the present invention:
[0042] The present invention designs a new end-to-end speech recognition model ResNet-GAU-CTC, in which ResNet extracts the time-frequency domain features of the speech signal, the GAU module captures the global interaction information between sequences, and the CTC automatically aligns the input and output sequences of unequal lengths. In response to the challenges faced by ATC Chinese speech recognition, two methods are proposed here, transfer learning and data enhancement technology. The former solves the problems of scarcity of calibration data in the ATC field and special pronunciation of some terms; the latter expands the source field data, enriches the diversity of data features, reduces the differences in data distribution between different fields, and improves the generalization ability of the model after transfer learning. The CER of this model on the source task and test set is 11.1%; on the target task, ResNet50_GAU@48 achieved the best recognition performance, and the CER dropped to 8.0%. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 This is the overall architecture diagram of the recognition model of the method of the present invention;
[0044] Figure 2 A schematic diagram of the process of transfer learning according to an embodiment of the present invention;
[0045] Figure 3 This is a time distribution diagram of an embodiment of the present invention;
[0046] Figure 4 A schematic diagram of visualizing a speech signal according to an embodiment of the present invention;
[0047] Figure 5 A schematic diagram of a GAU model according to an embodiment of the present invention;
[0048] Figure 6 Residual blocks in ResNet with different numbers of layers in the embodiment of the present invention;
[0049] Figure 7 It is the beam search (beam width=2) of the embodiment of the present invention;
[0050] Figure 8 is the pre-training loss of different models in the embodiments of the present invention;
[0051] Fig. 9 Pre-trained CERs for different models of the embodiments of the present invention;
[0052] Fig.10 is the training loss of the GAU module with different layers in the ATC corpus according to the embodiment of the present invention;
[0053] Fig.11 Graph 1 shows the CER of GAU modules with different numbers of layers in the ATC corpus according to the embodiment of the present invention. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0055] Example 1
[0056] The present invention discloses a PCVCs end-to-end speech recognition method based on a ResNet-GAU model.
[0057] The following steps are involved:
[0058] S1: ResNet is used to extract the time-frequency domain features of the speech signal to be recognized.
[0059] The input speech to be recognized passes through the 7×7 Conv Layer. When the convolution kernel is working, it will regularly scan the input features, perform matrix element multiplication and sum of the input features in the receptive field, and add the deviation;
[0060] The output feature map is passed to the Max Pooling Layer for feature selection and information filtering; the PoolingLayer contains a pre-set pooling function that replaces the result of a single point in the feature map with the feature map statistics of its adjacent area;
[0061] The feature vector after maximum pooling is sequentially passed through a stack composed of different numbers of Residual blocks to obtain highly purified features. Shortcut connections are introduced between the stacks inside the blocks to pass the input across layers and add the convolution results, alleviating the gradient vanishing problem caused by increasing the depth of CNN.
[0062] The features of the final output of ResNet are reduced in dimension through the Flatten layer and used as the input of multiple series-connected GAU modules.
[0063] S2: Capture the word order information of time-frequency domain features through multiple serially connected GAU modules.
[0064] Absolute position encoding is added to the input to ensure consistency in the time dimension. Shortcut connections are introduced between GAU modules to pass the input of the GAU module across layers and add it to the convolution result.
[0065] The processing process of each GAUmodule includes the following steps:
[0066] The input time-frequency domain features pass through Layernorm, and then pass through Dense Layer and nonlinear transformation to obtain the gating matrix U and the Value, Query and Key for calculating attention;
[0067] U=φ u (XW u ) V=φ v (XW v )∈R T×e
[0068] Q&K=φ q&k (XW q&k )∈R T×s
[0069] Where X∈R T×d , T represents the sequence length, d represents the model size; W U , W V ∈R d×e , e represents expandedintermediate size; Wq&k ∈R d×s , s represents head size; represents the activation function;
[0070] Calculate the attention score using relu 2 Activate the function and add relative position bias:
[0071] A=Relu 2 (QK T +B)∈R T×T
[0072] O=(U⊙AV)W o ∈R T×d
[0073] Among them, A is the attention matrix, which is responsible for fusing the information between tokens; ⊙ represents element-wise multiplication;
[0074] The Dropout layer is used to prevent overfitting during model training.
[0075] S3: The word order information is passed into the Dense layer, and after a linear transformation in the feature space, the final classification prediction probability is obtained through the softmax layer.
[0076] Furthermore, the training of a speech recognition model composed of ResNet and multiple serially connected GAU modules includes the following steps:
[0077] The feature enhancement technology is used on the speech data in the source domain speech database to obtain the enhanced source domain speech database; transfer learning is introduced to fine-tune the speech recognition model trained in the enhanced source domain speech database in the target domain air traffic control speech database to obtain the trained speech recognition model. That is, the model is trained on the large-scale source domain Aishell corpus, and then the trained model is further fine-tuned on the small-scale target domain ATC corpus.
[0078] Among them, the feature enhancement technology is used for the speech data in the source domain speech database, including the following steps:
[0079] The time-frequency masking technology in SpecAugment is used. The maximum range of continuous masking in the time and frequency domain axes is set to 250ms and 100Hz. A t(f) is uniformly sampled in the range of [0,250] and [0,100] of the source domain Aishell corpus. A point t0(f0) is randomly determined in the range of [0,τ-t] and [0,τ-f]. Starting from the position t0(f0), masking is performed continuously t(f) times along the time and frequency domain axes, where τ represents the time dimension and v represents the frequency domain dimension.
[0080] Add Gaussian white noise with a certain signal-to-noise ratio to the source domain Aishell corpus;
[0081] Perform speech rate enhancement operations in the source domain Aishell corpus.
[0082] Furthermore, the input and output sequences of unequal lengths are automatically aligned according to the time series classification algorithm, and the special character blank is introduced to enable each frame of the speech sequence to predict its conditional probability distribution on the output label sequence one by one.
[0083] Preferably, a speech enhancement technique is used to extract a useful speech signal from the electromagnetic interference noise background to be identified as the speech signal to be identified.
[0084] In this embodiment,
[0085] 1. Data analysis and processing:
[0086] Two methods are proposed to improve the generalization ability of the model on the target task.
[0087] 1.1 Transfer Learning
[0088] Transfer learning relaxes the constraint in traditional machine learning that training data and test data must be independent and identically distributed. It can thus mine the domain-invariant essential features and structures between two different but interrelated domains, allowing supervised information such as labeled data to be transferred and reused between domains.
[0089] Two concepts are proposed here, domain and task. The domain consists of two parts: the feature space X and the marginal distribution P(x) of the feature space, where {x1,x2,…,xn}∈X. Given a domain D={X,P(x)}, the task consists of the label space Y and the target prediction function f(.). Given a source domain Ds and source domain learning task Ts, a target domain Dt and target domain task Tt (where Ds is not equal to Dt or Ts is not equal to Tt), transfer learning uses the knowledge in the source domain Ds and Ts to improve or optimize the target prediction function f in the target domain Dt. t (.) learning effect.
[0090] The introduction of transfer learning solves the problems of scarcity of labeled data and special pronunciation in the civil aviation field, such as Figure 2 As shown in the figure, the model is first trained on the large-scale source domain Aishell corpus to give the model better transcription ability, and then the model is further fine-tuned on the small-scale target domain ATC corpus. Finally, the model not only performs well on the source task, but also retains the generalization ability on the target task.
[0091] 1.2 Data Augmentation
[0092] Data augmentation is to make limited data generate more value equivalent to valid data without substantially increasing the extended data. This method of indirectly increasing the number of samples and enriching the diversity of samples can reduce the model's dependence on certain attributes, thereby improving the model's generalization ability. However, a large number of data augmentation times cannot continuously improve the performance of the model. Only at an appropriate level of data augmentation can it have a positive effect on the transfer learning model.
[0093] The selected source domain Aishell corpus was recorded in a quiet indoor environment, and the recording texts involved 11 fields such as smart home, unmanned driving, industrial production, etc. We sorted and compared the basic information of the source domain dataset and the target domain dataset, as shown in Table 1.
[0094] Table 1 Basic information of the dataset
[0095]
[0096] The number of samples in the Aishell corpus is about three times that of the ATC corpus, and the speech rates of the two datasets are 3.16 words / second and 4.75 words / second respectively. At the same time, considering that the ATC corpus uses concise language and one sample represents one control instruction, the distribution of single sample duration in the two datasets is also statistically analyzed. Figure 3 As shown, the duration of the two different types of samples is mostly distributed between 2 and 8 seconds, and it can be seen that the distribution of sample durations remains almost consistent.
[0097] In order to reduce the differences in the nature of samples from different fields, we used three methods to expand the Aishell corpus of the source field, referring to the basic characteristics of the two datasets. Figure 4 As shown in (A) and (B), the Mel spectrogram of the speech signal of a sample in the ATCcorpus and Aishellcorpus is visualized respectively. Figure 4As shown in (C), the time-frequency masking technology in SpecAugment is used. The maximum range of continuous masking in the time (frequency domain) axis direction is set to 250ms (100Hz), a t(f) is uniformly sampled in the range [0,250] ([0,100]), a point t0(f0) is randomly determined in the range [0,τ-t] ([0,τ-f]), and masking is performed continuously t(f) times along the time (frequency domain) axis direction starting from t0(f0). τ represents the time dimension and v represents the frequency domain dimension.
[0098] In the ATC field, there is a complex of noise in the control speech formed by interference from signals from different sources. Using synthetic Gaussian noise to blend into the source field data is a simple and good approximate simulation when dealing with this complexity and not knowing the distribution of the noise. Figure 4 As shown in (D), the Melspectrogram generated after adding Gaussian white noise with a certain signal-to-noise ratio is shown. Figure 4 As shown in (E), the speed enhancement operation is performed. Referring to the average speaking speed of the two data, we use 1.5x for transformation. The speech signal after the above three changes is called improved Aishell corpus.
[0099] 2 Methods
[0100] 2.1 GAU Model
[0101] Transformer has made many breakthroughs in the fields of NLP and CV by leveraging the advantages of MHSA's ability to fully integrate global information and powerful parallel computing capabilities. However, most Transformers are still limited to short context sizes due to the quadratic complexity of the input length-due to limited memory capacity, earlier information has to be discarded. As a new Transformer variant, GAU still has quadratic complexity, but compared to the standard Transformer, it has faster training speed, lower video memory occupancy and better results. The core idea is to use self-attention and GLU as a unified layer and share their calculations as much as possible. Doing so not only achieves higher computational efficiency, but also naturally enables a powerful attention gating mechanism. At present, GAU has achieved remarkable success in NLP tasks. Therefore, we also tried to explore the applicability of GAU in the field of ASR. The model structure is as follows Figure 5 shown.
[0102] The input time-frequency domain features are processed through Layer norm, Dense Layer and nonlinear transformation to obtain the gating matrix U and the Value, Query and Key for calculating attention.
[0103]
[0104] in, T represents the sequence length, d represents the model size; e represents expanded intermediate size; s represents head size; represents the activation function;
[0105] Calculate the attention score using relu 2 Activate the function and add relative position bias:
[0106]
[0107] Among them, A is the attention matrix, which is responsible for fusing the information between tokens; ⊙ represents element-wise multiplication;
[0108] The Dropout layer is used to prevent overfitting during model training.
[0109] 2.2 Overall Model Architecture
[0110] Mel spectrogram is one of the visualization methods of speech signals. It contains both time domain and frequency domain information. We use it as the input of the model. The overall structure of the model consists of two parts: Figure 1 shown.
[0111] Considering that CNN can provide translation-invariant convolution in time and space, applying its idea to the acoustic modeling of ASR can use the invariance of convolution to overcome the diversity of the speech signal itself. Therefore, the first part is composed of ResNet.
[0112] First, the input passes through the 7×7 Conv Layer. When the convolution kernel is working, it will regularly scan the input features, perform matrix element multiplication and sum the input features within the receptive field, and superimpose the deviation. Secondly, the output feature map is passed to the MaxPooling Layer for feature selection and information filtering. The Pooling Layer contains a preset pooling function that replaces the result of a single point in the feature map with the feature map statistics of its adjacent area. Finally, the feature vector after maximum pooling passes through different numbers of Residual blocks (such as Figure 6The stack is composed of layers (as shown in Figure 1) to obtain highly purified features. Within the block, shortcut connections are introduced between the stacks to pass the input across layers and add the convolution results, alleviating the gradient vanishing problem caused by increasing the depth in CNN. The Flatten layer plays a bridging role in the overall architecture of the model, reducing the dimension of the features finally output by ResNet as the input of the second part.
[0113] The second part is mainly composed of multiple GAU modules connected in series. Although this module based on the attention mechanism solves the problem that RNN cannot calculate in parallel, it cannot capture word order information. We add absolute position encoding to the input to ensure consistency in the time dimension. Shortcut connections are introduced between GAU modules, and the input of the GAU module is passed across layers and added to the result of convolution, making the forward and backward propagation of information smoother. The output is passed to the Dense layer, linearly transformed in the feature space, and then passed through the softmax layer to calculate the final classification prediction probability.
[0114] 2.3 Training and Decoding
[0115] Training process: The introduction of CTC solves the problem of different lengths of input and output sequences in the ASR field that are difficult to align. By introducing the special character blank, each frame of the speech sequence can predict its conditional probability distribution on the output label sequence one by one. Given an input sequence X (x1, x2, ..., xT) of length T, V represents the dictionary, The vector y output at time t t Through the softmax layer, we get the probability of token k:
[0116]
[0117] V ~T Indicates that V ~ Defined as the set of all paths of length T. For a given input sequence X, V ~T The conditional probability calculation formula for any path a in is:
[0118]
[0119] In the calculation of path probability, the length of the input speech sequence is equal to the length of the output path, which is mainly ensured by inserting blanks between output labels and generating heavy tokens. The actual speech sequence length is much longer than the label sequence length, thus forming a many-to-one mapping from input to output. To obtain the predicted sequence, path aggregation is required (merging the same continuous labels and deleting blanks in the path) and summing all path probabilities.
[0120]
[0121] Among them, β -1 ( y ) represents the set V ~T All paths corresponding to the label sequence in . The final objective function of CTC is the negative logarithmic probability sum of all labels, and the loss can be minimized by training the network through multiple iterations of back propagation.
[0122] CTC(x)=-logP r (y|x)
[0123] Decoding process: Abandon the traditional greedy algorithm and use beam search. Figure 7 As shown in the figure, a search tree is established using a breadth-first strategy. At each level of the tree, the nodes are sorted according to the heuristic cost, and then only a predetermined number of nodes are left. These nodes continue to expand at the next level, and the other nodes are pruned. The beam width can be predetermined or variable. You can first search according to a minimum beam width. If no suitable solution is found, expand the beam width and search again. Finally, the difference between the search solution and the standard solution is evaluated by the CER based on the Levenshtein distance.
[0124] In this embodiment:
[0125] The input of the present invention is unified as Mel spectrogram of shape = (-1, 3, 64, 512), and the four dimensions represent batch size, channel, height and width respectively. The batch size is 64. In the training phase, the forward and backward algorithm is used to calculate the CTC objective function loss. The initial learning rate is set to 0.0001, and the hyperparameters β1 = 0.9, β2 = 0.98, ε = 10 -9The Adam optimizer updates the model weights. In the inference process, a beam search method with a width of 5 is used to obtain the final predicted text. In the extended Aishell corpus, the experimental results of ResNet34 combined with BiLSTM, BiGRU, MHSA+GLU and GAU are compared; in the ATC corpus, the experimental results are analyzed by changing the number of layers of the GAU module and using different convolutional structures.
[0126] 1. Experimental results of different models in extended Aishell corpus pre-training
[0127] Our model architecture takes ResNet34_GAU@24 as an example, and the parameter settings are shown in Table 2. In the comparison model, the number of stacked layers of BiLSTM and BiGRU models is 4, and the number of hidden units in each layer is 512. The number of stacked layers of MHSA+GLU is 24, and each layer uses 8 self-attention heads, and the dimension of each attention head is 64.
[0128] Table 2. details of architecture
[0129]
[0130]
[0131] The specific training process loss curve is as follows Figure 8 As shown in the figure, the number of iterations is 20 epochs, a total of 74980 steps, where each step contains 64 samples. In the first 9000 steps, the convergence speed of BiLSTM is obviously too slow, BiGRU and MHSA+GLU are similar, while the loss curve of GAU has a larger slope and drops faster after 1000 steps, and the convergence speed is obviously better than the other three models. In the last 9000 steps, the convergence effect of BiLSTM and BiGRU is similar, and the fluctuation range of training loss is large; while the amplitude of the loss curve of MHSA+GLU and GAU is relatively smaller.
[0132] After each round of training, each model is verified and tested. Fig. 9 (A) and (B) show that the lowest CERs are 15.5%, 17.6% and 15.6%, 17.4% respectively; the lowest CERs achieved by MHSA+GLU and GAU are 10.9%, 12.5% and 10.3%, 11.1% respectively. On the validation set, our model reduces the CER by 33.5% compared to the model based on recurrent neural network and 5.5% compared to the model based on multi-head attention; on the test set, the relative reduction is 36.4% and 11.2%.
[0133] The comparison of the four types of models in terms of CER, total number of parameters, running time and training time is shown in Table 3. The total number of model parameters is 63.3M, which is 1.35, 1.54 and 0.68 times that of BiLSTM, BiGRU and MHSA+GLU respectively. At the same time, with less time per step and total training time, it is more competitive in recognition effect than other models.
[0134] Table 3. Evaluation metrics of different models
[0135]
[0136] 2. Experimental results of GAU modules with different layers in ATC corpus
[0137] Our model structure has achieved good results on the source task. In order to explore the impact of changing the number of layers on the final recognition results of the target task, we saved the optimal weights of GAU modules with different numbers of layers during the extended Aishell corpus test as the initialization parameters for training on the ATC corpus dataset. Fig.10 As shown in the figure, the number of iterations is 10 epochs, a total of 15760 steps, and each step contains 64 samples. The 12-, 24-, 36-, and 48-layer GAU modules all have faster convergence speeds. From 15000 steps to the end of the iteration, the convergence effect of different numbers of layers is shown. It can be found that as the number of layers increases, the convergence effect becomes more stable.
[0138] like Fig.11 As shown in (A) and (B), the CER comparison of GAU modules with different numbers of layers on the ATC corpus validation and test sets. It can be found that after one round of iteration, the CER of the 12-layer GAU module has reached about 19%, while the 24, 36 and 48-layer GAU modules are about 15%. As the number of iterations increases, the lowest CERs of the 12-layer, 24-layer, 36-layer and 48-layer GAU modules are 9.0%, 8.5%, 7.7%, 7.4% and 9.7%, 9.2%, 8.6%, 8.2%, respectively. The CER of the 48-layer GAU module is relatively lower than that of the 12-layer GAU module by 17.8% and 15.5%.
[0139] As shown in Table 4, as the number of layers accumulates, the total number of model parameters and training time increase linearly. The increase in the number of GAU modules connected by residuals significantly improves the recognition accuracy.
[0140] Table 4. Evaluation indicators of GAU modules with different layers in ATC corpus
[0141]
[0142] 3. Experimental results of different convolutional structures in ATC corpus
[0143] We also tried to explore the impact of different convolutional structures on the experimental results of the target task, as shown in Table 5. In order to ensure the approximate total number of parameters, the GAU module uses 48 layers and selects VGG16, VGG19, ResNet34 and ResNet50 as experimental controls. The training time of each network architecture is not much different, each step is 1.88s, 2.31s, 1.76s and 2.01s respectively, and the total training time is kept within 5h. The CER of each model on the validation and test sets is 8.0%, 7.8%, 7.9%, 7.7% and 8.3%, 8.2%, 8.2%, 8.0% respectively. Although the model based on the ResNet structure is slightly better than the VGG structure, the different convolutional structures do not improve the final recognition rate much.
[0144] Table 5. Evaluation indicators of different convolutional structures in ATC corpus
[0145]
[0146] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. The PCVCs end-to-end speech recognition method based on the ResNet-GAU model is characterized by: The following steps are involved: ResNet is used to extract the time-frequency domain features of the speech signal to be recognized; Through multiple serially connected GAU modules, the word order information of time-frequency domain features is captured; The word order information is passed into the Dense layer, and after a linear transformation in the feature space, the final speech classification prediction probability is obtained through the softmax layer; The method of extracting the time-frequency domain features of the speech signal to be recognized by using ResNet includes the following steps: The input speech to be recognized passes through the 7×7 Conv Layer. When the convolution kernel is working, it will regularly scan the input features, perform matrix element multiplication and sum of the input features in the receptive field, and add the deviation; The output feature map is passed to the Max Pooling Layer for feature selection and information filtering; the Pooling Layer contains a pre-set pooling function that replaces the result of a single point in the feature map with the feature map statistics of its adjacent area; The feature vector after maximum pooling is sequentially passed through a stack composed of different numbers of Residual blocks to obtain highly purified features. Shortcut connections are introduced between the stacks inside the blocks to transfer the input across layers and add the convolution result to alleviate the gradient vanishing problem caused by increasing the depth in CNN. The features of the final output of ResNet are reduced in dimension through the Flatten layer and used as the input of multiple GAU modules connected in series; Also includes: Absolute position encoding is added to the input to ensure consistency in the time dimension. Shortcut connections are introduced between GAU modules to pass the input of the GAU module across layers and add it to the convolution result.
2. The PCVCs end-to-end speech recognition method based on the ResNet-GAU model according to claim 1, characterized in that: The method of capturing word order information of time-frequency domain features comprises the following steps: The input time-frequency domain features pass through Layernorm, and then pass through DenseLayer and nonlinear transformation to obtain the gating matrix U and the Value, Query and Key for calculating attention; U=φ u (XW u )V=φ v (XW v )∈R T×e Q&K=φ q&k (XW q&k )∈R T×s Where X∈R T×d , T represents the sequence length, d represents the model size; W U , W V ∈R d×e , e represents expandedintermediate size; W q&k ∈R d×s , s represents head size; represents the activation function; Calculate the attention score using relu 2 Activate the function and add relative position bias: A=Relu 2 (QK T +B)∈R T×T O=(U⊙AV)W o ∈R T×d Among them, A is the attention matrix, which is responsible for fusing the information between tokens; ⊙ represents element-wise multiplication; The Dropout layer is used to prevent overfitting during model training.
3. The PCVCs end-to-end speech recognition method based on the ResNet-GAU model according to claim 1, characterized in that: It also includes training a speech recognition model consisting of ResNet and multiple serially connected GAU modules, including the following steps: Using feature enhancement technology on speech data in a source domain speech database to obtain an enhanced source domain speech database; Transfer learning is introduced to fine-tune the speech recognition model trained in the enhanced source domain speech database in the target domain air traffic control speech database to obtain a trained speech recognition model.
4. The PCVCs end-to-end speech recognition method based on the ResNet-GAU model according to claim 3 is characterized in that: The speech recognition model is trained on a large-scale source domain Aishell corpus, and then fine-tuned on a small-scale target domain ATC corpus.
5. The PCVCs end-to-end speech recognition method based on the ResNet-GAU model according to claim 4, characterized in that: The method of using feature enhancement technology on the speech data in the source domain speech database comprises the following steps: The time-frequency masking technology in SpecAugment is used. The maximum range of continuous masking in the time and frequency domain axes is set to 250ms and 100Hz. A t(f) is uniformly sampled in the range of [0,250] and [0,100] of the source domain Aishell corpus. A point t0(f0) is randomly determined in the range of [0,τ-t] and [0,τ-f]. Starting from the position t0(f0), masking is performed continuously t(f) times along the time and frequency domain axes, where τ represents the time dimension and v represents the frequency domain dimension. Add Gaussian white noise with a certain signal-to-noise ratio to the source domain Aishell corpus; Perform speech rate enhancement operations in the source domain Aishell corpus.
6. The PCVCs end-to-end speech recognition method based on the ResNet-GAU model according to claim 1, characterized in that: It also includes automatically aligning input and output sequences of unequal lengths according to the time series classification algorithm, and introducing the special character blank so that each frame of the speech sequence can predict its conditional probability distribution on the output label sequence one by one.
7. The PCVCs end-to-end speech recognition method based on the ResNet-GAU model according to claim 1, characterized in that: Also includes: The speech enhancement technology is used to extract useful speech signals from the electromagnetic interference noise background to be recognized as the speech signals to be recognized.
