Air traffic control instruction end-to-end speech recognition method under high noise condition

Abstract

The invention relates to the technical field of air traffic control and voice recognition, in particular to an air traffic control instruction end-to-end voice recognition method under a high-noise condition, which comprises the following steps: preprocessing to-be-recognized voice, and extracting to obtain original voice frequency characteristics; carrying out noise reduction intensity control, frequency band gain control and fundamental tone tracking processing on the original voice frequency characteristics by adopting a self-adaptive attention noise reduction module to obtain noise-reduced voice; preprocessing the noise-reduced voice, and extracting to obtain noise-reduced voice frequency characteristics; a voice recognition module is adopted to carry out shared encoder coding, connection time sequence classification decoder decoding and attention-based encoder-decoder decoding on the noise reduction voice frequency characteristics, and an output voice recognition transcription text is obtained; according to the invention, the voice recognition precision of the air traffic control instruction under the high-noise condition can be improved.

Description

Technical Field

[0001] The invention relates to the technical field of air traffic control and speech recognition, and in particular to an end-to-end speech recognition method for air traffic control instructions under high noise conditions. Background Art

[0002] Air traffic control instructions are specific operational instructions issued by air traffic controllers to pilots. Through air traffic control instructions, air traffic controllers provide pilots with real-time information about the airport scene and airspace, and instruct pilots to complete specific operations. Pilots complete corresponding operations according to the instructions, such as takeoff, landing, changing flight altitude, speed or direction, etc., so as to maintain air traffic order, prevent collisions between aircraft, and ensure that aircraft fly according to the prescribed route and speed.

[0003] Air traffic management mainly transmits instructions and information through voice communication between pilots and air traffic controllers, but this process relies on manual recognition, which is prone to "mistakes, forgetting and omissions", leading to frequent unsafe incidents. Therefore, it is necessary to introduce a voice recognition system to recognize the instructions and reply voices of controllers and pilots in real time, thereby reducing misunderstandings and forgetfulness.

[0004] Since the pilot's communication environment is the cabin, there is a serious problem of noise interference, and the air traffic control voice is transmitted through ground-to-air very high frequency communication, which leads to high-frequency signal loss and clipping in the audio. These complex communication environment characteristics increase the difficulty of recognition, especially when facing problems such as multi-source noise superposition and frequency band instability. It is difficult for existing technical solutions to obtain accurate speech recognition results under these conditions. At present, the air traffic control command voice recognition method commonly uses a hybrid speech recognition model that mainly trains each module separately. It has the defects of insufficient model collaborative optimization, limited information sharing between modules, and the easy occurrence of local optimal solutions during the training process, making it difficult to achieve the overall optimal effect.

[0005] In summary, the existing technology lacks noise reduction optimization design for complex multi-source noise and communication environment characteristics, especially lacks targeted processing of factors such as cabin noise and very high frequency communication distortion. At the same time, the various parts of the hybrid recognition model cannot be jointly optimized, resulting in insufficient accuracy and robustness of speech recognition in complex environments, making it difficult to meet actual needs. Summary of the invention

[0006] In view of the above problems, the present invention provides an end-to-end speech recognition method for air traffic control commands under high noise conditions, which solves the problem of inaccurate recognition caused by multi-source high noise conditions and difficulty in joint optimization of models in the prior art.

[0007] The present invention provides an end-to-end speech recognition method for air traffic control instructions under high noise conditions, which uses a trained end-to-end speech recognition model to perform speech recognition, and includes the following steps:

[0008] Step S1, preprocessing the speech to be recognized to extract the frequency characteristics of the original speech;

[0009] Step S2, using an adaptive attention noise reduction module to perform noise reduction intensity control, frequency band gain control, and pitch tracking processing on the original speech frequency characteristics to obtain noise-reduced speech;

[0010] Step S3, preprocessing the noise reduction speech to extract the noise reduction speech frequency features;

[0011] Step S4: using a speech recognition module to encode the noise reduction speech frequency features using a shared encoder, decode them using a connected temporal classification decoder, and decode them using an attention-based encoder-decoder to obtain an output speech recognition transcription text.

[0012] Preferably, the step S1 specifically includes:

[0013] Step S1-1, performing pre-emphasis, frame windowing, fast Fourier transform and frequency band division processing on the speech to be recognized to obtain filter bank features;

[0014] Step S1-2, obtaining the pitch period and spectrum non-stationarity measurement of the speech to be recognized;

[0015] Step S1-3, taking the first-order and second-order derivatives of the filter bank characteristics, the pitch period, and the spectrum non-stationarity measure together as the original speech frequency features.

[0016] Preferably, in step S1:

[0017] The expression of the processing method of the pre-emphasis is:

[0018] x′(n′)=x(n′)-0.97·x(n′-1)

[0019] Among them, x(n′) represents the value of the n′th sampling point of the speech signal, x(n′-1) represents the sampling value of the n′-1th point of the speech signal, 0.97 is the coefficient of the high-pass filter, which determines the proportion of high-frequency compensation, and x′(n′) represents the value of the n′th sampling point of the speech signal after pre-emphasis processing;

[0020] The expression of the window function of the frame-by-frame windowing is:

[0021]

[0022] Where w(n) represents the window function value of the nth sampling point in the frame, and N is the total number of sampling points in each frame;

[0023] The calculation expression of the filter bank characteristics is:

[0024] F b =log(E(b)+∈)

[0025]

[0026] Among them, F b is the filter bank characteristic of the bth frequency band, E(b) is the energy of the bth frequency band of the speech signal, ∈ is a small constant, and w b (k) is the amplitude of frequency band b at frequency k, and |X(k)| represents the amplitude of the spectrum value corresponding to frequency k.

[0027] Preferably, the step S2 specifically includes:

[0028] Step S2-1, using an adaptive controller to perform convolutional neural network and gated recurrent unit processing on the original speech frequency characteristics to obtain noise reduction strength;

[0029] Step S2-2, using the RNN-GRU network to process the original speech frequency features, and obtaining the frequency band gain of each frequency band of the audio frequency domain signal through smooth interpolation;

[0030] Step S2-3, detecting a pitch period in the speech signal, and determining a gain signal through a comb filter based on the pitch period;

[0031] Step S2-4, the gain signal is controlled by the noise reduction strength and the frequency band gain to obtain a noise-reduced frequency domain signal, and the noise-reduced speech is obtained by performing an IFFT inverse fast Fourier transform.

[0032] Preferably, in step S2-1, the expression of the convolutional neural network and gated recurrent unit processing process is:

[0033]

[0034] in, is the FBank feature vector of the input original speech, Conv represents the convolution operation, BN represents batch normalization, MaxPool represents the maximum pooling, GRU represents the gated recurrent unit, FC represents the fully connected layer, is the intermediate eigenvector of the network of the adaptive controller;

[0035] In step S2-2, the expression of the processing process of the RNN-GRU network is:

[0036]

[0037] Among them, E s (b) represents the energy of the speech signal in the bth frequency band, E x (b) represents the energy of the noisy speech signal in the bth frequency band, Indicates the dynamically adjusted noise reduction strength;

[0038] Step S2-3 specifically includes: obtaining a frequency band-dependent filter coefficient α based on a comb filter of the pitch period b , where the filter coefficient α b The expression is:

[0039]

[0040] Among them, min(·,·) means taking the minimum value, is the complex value of the signal in frequency band k, is the complex conjugate value of the fundamental tone signal in frequency band k, |X k | 2 is the power of the spectrum value in frequency band k, |P k | 2 is the power of the fundamental tone signal in frequency band k, k is the index of the frequency band, and K is the total number of frequency bands.

[0041] Preferably, the step S4 specifically includes:

[0042] Step S4-1, inputting the noise reduction speech frequency feature into the shared encoder to obtain a high-dimensional representation;

[0043] Step S4-2, inputting the high-dimensional representation into a time series classification decoder to obtain an activation value;

[0044] Step S4-3: The attention-based encoder-decoder re-scores the activation value to generate a final speech recognition transcription text.

[0045] Preferably, in step S4-1, the shared encoder is composed of multiple Conformer layers, and the structural expression of the Conformer layer is:

[0046]

[0047] in, is the FBank feature vector of the input denoised speech, FFN is a fully connected feedforward network, MHSA is a multi-head self-attention mechanism, Conv is a convolution operation, and Layernorm is a layer normalization operation. is the intermediate feature vector of the Conformer layer, y c A high-dimensional representation of the output;

[0048] In step S4-2, the expression of the connection timing classification decoder calculation process is:

[0049]

[0050] Among them, p(l|x) is the probability of generating the output sequence l given the input feature sequence x, represents the sum of all possible alignment paths s, |l ′ | is the output sequence l ′ The length of α t (s) is the forward variable, which represents the cumulative probability of aligning path s at time step t, β t (s) is the backward variable, which represents the cumulative probability of the alignment path s at time step t, Indicates that at time step t, the output sequence l ′ s The activation value of

[0051] In step S4-3, the calculation process of the attention-based encoder-decoder is expressed as:

[0052]

[0053] Where K is the hidden state of the encoder, V is the output of the encoder, and Q is the input of the decoder. N is the number of output vectors of the encoder, and the weight α is i represents the correlation between the current time step of the decoder and the i-th hidden state of the encoder, thereby helping the model generate a more accurate output sequence, v i is the i-th context vector.

[0054] Preferably, the end-to-end speech recognition model is trained, and the trained end-to-end speech recognition model is used for air traffic control command speech recognition, and the training steps specifically include:

[0055] Step S5, constructing an air traffic control environment noise-clean voice simulation data set and an air traffic control instruction voice-text data set;

[0056] Step S6: use the simulated noise-clean speech data set to pre-train the adaptive attention noise reduction module; use the air traffic control command speech-text data set to separately train the speech recognition module; use the air traffic control command speech-text data set to jointly train the end-to-end speech recognition model.

[0057] Preferably, the step S5 specifically includes:

[0058] Step S5-1, using an open source noise dataset and an open source clean speech Chinese dataset to respectively form a noise dataset and a clean speech dataset in the air traffic control environment noise-clean speech simulation dataset;

[0059] Step S5-2, using the air traffic control system operation production data as raw data, processing the raw data to form an air traffic control instruction voice-text data set;

[0060] Step S5-2 specifically includes:

[0061] Step S5-2-1, dividing the original data into speech segments, marking the range of the segments containing human voice according to the change of the audio spectrum, and using the existing automatic speech recognition model to recognize the segments containing human voice to obtain the transcription text of each segment containing human voice;

[0062] Step S5-2-2, manually annotating the vocal segment and the corresponding transcribed text;

[0063] Step S5-2-3, perform text screening on the manual annotation results, remove unclear parts, divide the speech and corresponding text that constitute a single sentence instruction, and form the air traffic control instruction speech-text dataset.

[0064] Preferably, the step S6 specifically includes:

[0065] Determine the loss function of the adaptive attention denoising module:

[0066]

[0067] in, is the loss function of the adaptive attention denoising module, is the gain estimate of the b-th frequency band, and the exponent ζ is a perceptual parameter that controls the degree of noise suppression;

[0068] Determine the loss function of the speech recognition module:

[0069] L c (x,y)=γL CTC (x,y)+(1-γ)L AED (x,y)

[0070] Among them, L c (x, y) is the loss function of the speech recognition module, x, y are the input speech features and the target text sequence, L CTC (x,y),L AED (x, y) are the losses of CTC encoder and AED encoder respectively, γ is the first balancing weight coefficient;

[0071] Determine the loss function of the end-to-end model:

[0072] L total (x,y)=L c (x,y)+μWER

[0073] Among them, L total (x, y) is the loss function of the end-to-end model, WER is the word error rate (WordError Rate) of the recognition result, and μ is the second balance weight coefficient;

[0074] With the goal of minimizing the loss functions of the adaptive attention denoising module, the speech recognition module and the end-to-end model, pre-training of the adaptive attention denoising module, separate training of the speech recognition module and joint training of the end-to-end speech recognition model are completed.

[0075] Compared with the prior art, the present invention has at least the following beneficial effects:

[0076] (1) The present invention uses an adaptive attention noise reduction module (ABNS) to control noise reduction intensity, frequency band gain control, and pitch tracking. Through a noise reduction module specifically for air traffic control speech, it can effectively suppress noise interference and reduce audio clipping distortion, thereby improving the accuracy of speech recognition in complex environments. It can still stably recognize and extract clear speech information under high noise conditions.

[0077] (2) The present invention establishes an air traffic control environment noise-clean speech simulation data set and an air traffic control command speech-text data set, and conducts special training on air traffic control commands, thereby enhancing the accuracy and robustness of the speech recognition module.

[0078] (3) The present invention combines the adaptive attention noise reduction module with the speech recognition module to construct an end-to-end recognition model, realizing the joint optimization of the entire model. This not only improves the recognition effect, but also greatly improves the recognition speed and robustness, and can better cope with complex noise conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The drawings are only for the purpose of illustrating particular embodiments and are not to be construed as limiting the invention.

[0080] Figure 1 The present invention provides a flow chart of an end-to-end speech recognition method for air traffic control instructions under high noise conditions.

[0081] Figure 2 A flowchart of the joint pre-training and model use of the end-to-end model provided by the present invention.

[0082] Figure 3A flow chart of noise suppression by the adaptive attention noise reduction module provided by the present invention.

[0083] Figure 4 A flow chart of end-to-end speech recognition model training steps is provided for the present invention. DETAILED DESCRIPTION

[0084] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0085] In order to illustrate the effectiveness of the method proposed by the present invention, the above technical solution of the present invention is described in detail below through a specific embodiment. Figure 1 As shown, a method for end-to-end speech recognition of air traffic control instructions under high noise conditions is disclosed, and the specific implementation steps include: using an end-to-end speech recognition model to perform speech recognition, including the following steps:

[0086] Step S1: pre-process the speech to be recognized and extract the original speech frequency features.

[0087] like Figure 2 As shown, the speech signal is first pre-emphasized, that is, the signal passes through a high-pass filter to compensate for the loss of high-frequency components, so that the speech signal retains sufficient high-frequency information during subsequent processing. The expression of the processing method is:

[0088] x′(n′)=x(n′)-0.97·x(n′-1)

[0089] Among them, x(n′) represents the value of the n′th sampling point of the speech signal, x(n′-1) represents the sampling value of the n′-1th point of the speech signal, 0.97 is the coefficient of the high-pass filter, which determines the proportion of high-frequency compensation, and x′(n′) represents the value of the n′th sampling point of the speech signal after pre-emphasis processing.

[0090] The speech is framed and windowed, a 20 millisecond window is selected, and a 50% overlap is used to divide the speech signal into multiple segments, and each segment is windowed. The expression of the window function for window division is:

[0091]

[0092] Wherein, w(n) represents the window function value of the nth sampling point in the frame, and N is the total number of sampling points in each frame.

[0093] Perform fast Fourier transform on each windowed segment to convert the time domain signal into a frequency domain signal. The conversion method is as follows:

[0094]

[0095] Wherein, X(k) represents the spectrum value corresponding to frequency k, x′(n) is the time domain sampling signal, and the number of fast Fourier transform points is set to N.

[0096] Finally, the spectrum X(k) is divided into frequency bands through a set of Mel-scaled triangular filters, and the logarithm of the energy in each frequency range is taken to obtain the filter bank feature (FilterBank, Fbank), which is called Fbank feature.

[0097] The specific steps are as follows: define the band energy as E(b), E(b) is the energy of the bth frequency band, and set w b (k) is the amplitude of band b at frequency k, let ∑ b w b (k) = 1, completing the normalization of the amplitude of all frequency bands. For the transformed signal X(k), the energy in the frequency band is as follows:

[0098]

[0099] The energy of the frequency band is logarithmically transformed to obtain FBank:

[0100] F b =log(E(b)+∈)

[0101] Among them, F b is the FBank feature of the b-th frequency band, and ∈ is a small constant.

[0102] Get the pitch period and spectral non-stationarity measure of the speech segment.

[0103] The first-order and second-order derivatives of the FBank feature, the fundamental frequency period, and the spectrum non-stationarity measure are used together as the extracted original speech frequency features for subsequent noise reduction processing.

[0104] Step S2: Adopting an adaptive attention noise reduction module, performing noise reduction intensity control, frequency band gain control and pitch tracking processing on the original speech frequency characteristics to obtain noise-reduced speech.

[0105] Specifically, if Figure 3As shown, the present invention adopts an adaptive attention-based noise suppression (ABNS) module to achieve speech noise reduction. The adaptive attention-based noise suppression module includes an adaptive controller, a recurrent neural network-gated recurrent unit network and a pitch tracking process.

[0106] The present invention adopts an adaptive controller to control the noise reduction strength based on the frequency characteristics, adopts a recursive neural network-gated recurrent unit network to control the frequency band gain based on the frequency characteristics, adopts a comb filter to perform pitch tracking based on the frequency characteristics, and finally obtains noise-reduced speech. The following is a detailed introduction

[0107] (1) Adopt an adaptive controller to obtain the noise reduction strength.

[0108] The adaptive controller uses the input FBank features to dynamically adjust the noise reduction strength parameter α of each frame to control the noise reduction strength of the noise reduction module in all frequency bands. The adaptive controller is used to mine the relationship between FBank features and speech recognition word error rate (WER), so as to generate appropriate noise reduction strength in different noise environments.

[0109] The adaptive controller combines a convolutional neural network (CNN) and a gated recurrent unit (GRU), and the input is the FBank feature vector of the original speech. Set the convolution layer, batch normalization, maximum pooling, GRU layer, fully connected layer, and finally output the denoising strength α.

[0110] The network structure expression of the adaptive controller is:

[0111]

[0112]

[0113] in, is the FBank feature vector of the input original speech, Conv represents the convolution operation, BN represents batch normalization, MaxPool represents the maximum pooling, GRU represents the gated recurrent unit, FC represents the fully connected layer, is the intermediate feature vector of the network of the adaptive controller. Through the calculation of the neural network in the adaptive controller, the parameters for dynamically adjusting the noise reduction intensity of each frame can be obtained.

[0114] (2) Use RNN-GRU network to obtain frequency band gain.

[0115] In this step, the present invention inputs the FBank feature obtained in step S1 into the RNN-GRU network, and obtains the gain g of each frequency band after feature extraction. b , gb represents the gain of the bth frequency band, indicating the noise suppression strength of this frequency band, g b The calculation formula is:

[0116]

[0117] Among them, E s (b) represents the energy of the speech signal in the bth frequency band, E x (b) represents the energy of the noisy speech signal in the bth frequency band, Indicates the dynamically adjusted noise reduction strength.

[0118] Through the neural network calculation of the RNN-GRU network, the frequency band gain parameters of each frequency band of the audio frequency domain signal can be obtained.

[0119] (3) The filter outputs the noise-reduced speech.

[0120] The band gain parameters of each frequency band output by the neural network are interpolated to obtain the optimal gain on each frequency band (bin). Considering the optimal band gain, the interpolation gain expression applied to each frequency band (bin) is:

[0121]

[0122] where r(k) is the gain of frequency band k.

[0123] The present invention detects the pitch period in the speech signal and processes the signal of the pitch period through a comb filter. The comb filter based on the pitch period uses a frequency band-dependent filter coefficient α b , and processed in the frequency domain. The pitch correlation P of band b b Defined as:

[0124]

[0125] in, is the complex value of the signal in frequency band k, is the complex conjugate value of the fundamental tone signal in frequency band k, |X k | 2 is the power of the spectrum value in frequency band k, |P k | 2 is the power of the fundamental tone signal in frequency band k, k is the index of the frequency band, and K is the total number of frequency bands.

[0126] α b The value depends on the frequency band P b and g b , the expression is:

[0127]

[0128] Among them, min(·,·) means taking the minimum value.

[0129] The gain signal output by the filter is superimposed on the original signal after the noise reduction strength parameter α controls the noise reduction strength of the noise reduction module in all frequency bands to form a noise-reduced frequency domain signal. The noise-reduced speech is obtained through IFFT inverse fast Fourier transform as the input for the subsequent speech recognition step.

[0130] Step S3: preprocess the noise reduction speech and extract the noise reduction speech frequency features.

[0131] The present invention performs preprocessing similar to that in step S1 on the noise-reduced speech obtained in step S2, and obtains the FBank feature of the noise-reduced speech through operations such as pre-emphasis, frame windowing, fast Fourier transform, and frequency band division.

[0132] Step S4: using a speech recognition module to encode the noise reduction speech frequency features using a shared encoder, decode them using a connected temporal classification decoder, and decode them using an attention-based encoder-decoder to obtain an output speech recognition transcription text.

[0133] The speech recognition module of the present invention comprises a shared encoder, a connectionist temporal classification (CTC) decoder and an attention-based encoder (AED).

[0134] In this step, the FBank features of the denoised speech are input into the shared encoder, which converts the input sequence into a set of high-dimensional representations, consisting of multiple Conformer layers. The structure of the Conformer layer can be expressed by the equation:

[0135]

[0136]

[0137] in, is the FBank feature vector of the input denoised speech, FFN is a fully connected feedforward network, MHSA is a multi-head self-attention mechanism, Conv is a convolution operation, and Layernorm is a layer normalization operation. is the intermediate feature vector of the Conformer layer, y c Represents a high-dimensional representation of the output.

[0138] The CTC decoder of the present invention processes the high-dimensional representation of the shared encoder output. The CTC decoder receives the feature sequence from the shared encoder and maps it to the CTC activation, expressed as:

[0139]

[0140] Among them, p(l|x) is the probability of generating the output sequence l given the input feature sequence x, π is the alignment path, which represents the sequence output by the network, including repeated characters and empty characters, and P(π|x) is the probability of the alignment path π, which is usually expressed as the product of the predicted probabilities of each time step on the path. Where p(π t |x) is the predicted character π at time step t t The probability of B(π) is the mapping from the alignment path π to the final label sequence l, B -1 (l) represents the set of all possible paths that can generate the label sequence l.

[0141] After completing CTC decoding, the AED decoder re-scores the CTC activations and generates the final recognition result. The AED decoder processes the input denoised speech signal into hidden representations that capture the contextual information of the input sequence. The attention mechanism then calculates the correlation between the decoder's current time step and the encoder's hidden state, and assigns weights based on these correlations to generate a weighted context vector. The decoder uses this context vector and the previously generated output to generate the next output label. This process is repeated until the complete output sequence is generated. The expression of this process is:

[0142]

[0143] Where K is the hidden state of the encoder, x is the output of the encoder, and N is the input of the decoder. N is the number of output vectors of the encoder, and the weight α is i represents the correlation between the current time step of the decoder and the i-th hidden state of the encoder, thereby helping the model generate a more accurate output sequence, v i is the i-th context vector.

[0144] Through the above steps, the shared encoder converts the input sequence into a set of high-dimensional representations. The CTC decoder receives the high-dimensional representations from the shared encoder and maps them to CTC activations. After completing CTC decoding, the AED decoder re-scores the CTC activations and generates the final speech recognition transcription text.

[0145] The present invention also includes the step of training the adaptive attention noise reduction module in the end-to-end speech recognition model and the neural network in the speech recognition module, including constructing a data set and separate training and joint training, such as Figure 4 As shown, specifically including:

[0146] Step S5: construct an air traffic control environment noise-clean speech simulation dataset and an air traffic control instruction speech-text dataset.

[0147] In this embodiment, the air traffic control environment noise-clean speech simulation data set is divided into a noise data set and a clean speech data set. The noise data set adopts the open source data set Noise-92, and the clean speech data adopts the open source data set Aishell-1 Chinese data set.

[0148] For the air traffic control command voice-text dataset, the air traffic control system operation production data is used. The original data is collected from the control call voice data recorded by the air traffic management system of a certain airport tower control unit, mainly Chinese control call voice data, including some English control call voice data.

[0149] The raw data is processed to form an air traffic control command voice-text data set. The processing process of the raw data in the air traffic control command voice-text data set is as follows:

[0150] 1) Use the existing automatic speech recognition model to divide and machine-recognize human voice segments.

[0151] The original data is divided into segments of less than 5 minutes, and the range of the human voice segment is marked according to the changes in the audio spectrum. The existing pre-trained automatic speech recognition model is used for recognition, and the transcribed text obtained by recognition is used as a reference for the next step.

[0152] 2) Build a labeling platform and a proofreading platform to manually perform segmentation, proofreading, and text annotation. Formulate labeling rules, including audio labeling difficulty classification, segmentation labeling standards, and text labeling standards; visualize the voice segments, labels, and transcribed texts obtained in the previous step, and have professionals further process the segmentation and labeling according to the labeling rules, accurately divide the scope, and correct incorrect labels. After the labeling is completed, the results of the professional division and labeling are randomly assigned to quality inspectors for review to improve the accuracy of the labeling.

[0153] 3) Filter the text content, remove unclear parts, and divide the original data to form the voice set and text set of the single-sentence instruction. Export the manually reviewed annotation content, clean the annotation data, and remove the data with invalid annotations, incomplete start and end times, and unknown characters in the transcribed text. Divide the text set into the "corresponding voice id transcribed text content" format according to the annotation situation.

[0154] Step S6, using the air traffic control environment noise-clean voice simulation data set and the air traffic control instruction voice-text data set to train the end-to-end speech recognition model, including:

[0155] Pre-training the adaptive attention noise reduction module using the simulated noise-clean speech data set;

[0156] The recognition module is trained separately using an air traffic control command speech-text dataset;

[0157] The end-to-end speech recognition model is jointly trained using an air traffic control command speech-text dataset.

[0158] The specific steps include: first pre-training the adaptive attention denoising module using the simulated noise-clean speech dataset, and using the mean square error (MSE) of the exponential power of the frequency band gain as the loss function for the RNN network in the denoising module, expressed as:

[0159]

[0160] in, is the loss function of the adaptive attention denoising module, is the gain estimate for the bth frequency band, and the exponent ζ is a perceptual parameter that controls the degree of noise suppression.

[0161] After that, the recognition module is trained separately, and some air traffic control command speech-text data sets are input to train the CTC encoder and AED encoder. That is, the loss function consists of two parts, and the expression is

[0162] L c (x,y)=γL CTC (x,y)+(1-γ)L AED (x,y)

[0163] Among them, L c (x, y) is the loss function of the speech recognition module, x, y are the input speech features and the target text sequence, L CTC (x,y),L AED (x, y) are the losses of CTC encoder and AED encoder respectively, and γ is the first balancing weight coefficient.

[0164] Finally, the end-to-end model is jointly trained by inputting the air traffic control command speech-text dataset and jointly training through back propagation. The training loss is as follows:

[0165] L total (x,y)=L c (x,y)+μWER

[0166] Among them, L total (x, y) is the loss function of the end-to-end model, WER is the word error rate (WordError Rate) of the recognition result, and μ is the second balance weight coefficient.

[0167] In the above steps, the adaptive attention denoising module is back-propagation trained with the goal of minimizing the loss function of the adaptive attention denoising module, the speech recognition module and the end-to-end model.

[0168] Compared with the prior art, the present invention provides an end-to-end speech recognition method for control instructions under high noise conditions. Taking into account the problem of inaccurate recognition caused by the lack of noise reduction design targeting the characteristics of air traffic control speech and the inability to jointly optimize the various parts of the hybrid recognition model, the present invention designs a noise reduction module targeting the characteristics of air traffic control calls to improve the speech quality and recognition accuracy of air traffic control instruction speech; a comprehensive noise reduction module design is used to construct an end-to-end model for speech recognition, and an adaptive attention module is designed to optimize the noise reduction module, so as to achieve overall optimization of the model and improve the recognition ability of the model.

[0169] Although the specific embodiments of the present invention depict various actions or steps in a specific order, this should be understood as requiring such actions or steps to be performed in the specific order shown or in a sequential order, or requiring all illustrated actions or steps to be performed to obtain the desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination. The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0170] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for end-to-end speech recognition of air traffic control instructions under high noise conditions, characterized in that: Using the trained end-to-end speech recognition model for speech recognition includes the following steps: Step S1, preprocessing the speech to be recognized to extract the frequency characteristics of the original speech; Step S2, using an adaptive attention noise reduction module to perform noise reduction intensity control, frequency band gain control, and pitch tracking processing on the original speech frequency characteristics to obtain noise-reduced speech; Step S3, preprocessing the noise reduction speech to extract the noise reduction speech frequency features; Step S4: using a speech recognition module to encode the noise reduction speech frequency features using a shared encoder, decode them using a connected temporal classification decoder, and decode them using an attention-based encoder-decoder to obtain an output speech recognition transcription text.

2. The end-to-end speech recognition method for air traffic control instructions under high noise conditions according to claim 1 is characterized in that: The step S1 specifically includes: Step S1-1, performing pre-emphasis, frame windowing, fast Fourier transform and frequency band division processing on the speech to be recognized to obtain filter bank features; Step S1-2, obtaining the pitch period and spectrum non-stationarity measurement of the speech to be recognized; Step S1-3, taking the first-order and second-order derivatives of the filter bank characteristics, the pitch period, and the spectrum non-stationarity measure together as the original speech frequency features.

3. The end-to-end speech recognition method for air traffic control instructions under high noise conditions according to claim 2 is characterized in that: In step S1: The expression of the processing method of the pre-emphasis is: x′(n′)=x(n′)-0.97·x(n′-1) Among them, x(n′) represents the value of the n′th sampling point of the speech signal, x(n′-1) represents the sampling value of the n′-1th point of the speech signal, 0.97 is the coefficient of the high-pass filter, which determines the proportion of high-frequency compensation, and x′(n′) represents the value of the n′th sampling point of the speech signal after pre-emphasis processing; The expression of the window function of the frame-by-frame windowing is: Where w(n) represents the window function value of the nth sampling point in the frame, and N is the total number of sampling points in each frame; The calculation expression of the filter bank characteristics is: F b =log(E(b)+∈) Among them, F b is the filter bank characteristic of the bth frequency band, E(b) is the energy of the bth frequency band of the speech signal, ∈ is a small constant, and w b (k) is the amplitude of frequency band b at frequency k, and |X(k)| represents the amplitude of the spectrum value corresponding to frequency k.

4. The end-to-end speech recognition method for air traffic control instructions under high noise conditions according to claim 3 is characterized in that: The step S2 specifically includes: Step S2-1, using an adaptive controller to perform convolutional neural network and gated recurrent unit processing on the original speech frequency characteristics to obtain noise reduction strength; Step S2-2, using the RNN-GRU network to process the original speech frequency features, and obtaining the frequency band gain of each frequency band of the audio frequency domain signal through smooth interpolation; Step S2-3, detecting a pitch period in the speech signal, and determining a gain signal through a comb filter based on the pitch period; Step S2-4, the gain signal is controlled by the noise reduction strength and the frequency band gain to obtain a noise-reduced frequency domain signal, and the noise-reduced speech is obtained by performing an IFFT inverse fast Fourier transform.

5. The end-to-end speech recognition method for air traffic control instructions under high noise conditions according to claim 4 is characterized in that: In step S2-1, the expression of the convolutional neural network and gated recurrent unit processing process is: in, is the FBank feature vector of the input original speech, Conv represents the convolution operation, BN represents batch normalization, MaxPool represents the maximum pooling, GRU represents the gated recurrent unit, FC represents the fully connected layer, x′ a ,x" a is the intermediate eigenvector of the network of the adaptive controller; In step S2-2, the expression of the processing process of the RNN-GRU network is: Among them, E s (b) represents the energy of the speech signal in the bth frequency band, E x (b) represents the energy of the noisy speech signal in the bth frequency band, Indicates the dynamically adjusted noise reduction strength; Step S2-3 specifically includes: obtaining a frequency band-dependent filter coefficient α based on a comb filter of the pitch period b , where the filter coefficient α b The expression is: Among them, min(·,·) means taking the minimum value, is the complex value of the signal in frequency band k, is the complex conjugate value of the fundamental tone signal in frequency band k, |X k | 2 is the power of the spectrum value in frequency band k, |P k | 2 is the power of the fundamental tone signal in frequency band j, k is the index of the frequency band, and K is the total number of frequency bands.

6. The end-to-end speech recognition method for air traffic control instructions under high noise conditions according to claim 5 is characterized in that: The step S4 specifically includes: Step S4-1, inputting the noise reduction speech frequency feature into the shared encoder to obtain a high-dimensional representation; Step S4-2, inputting the high-dimensional representation into a time series classification decoder to obtain an activation value; Step S4-3: The attention-based encoder-decoder re-scores the activation value to generate a final speech recognition transcription text.

7. The end-to-end speech recognition method for air traffic control instructions under high noise conditions according to claim 6 is characterized in that: In step S4-1, the shared encoder is composed of multiple Conformer layers, and the structural expression of the Conformer layer is: in, is the FBank feature vector of the input denoised speech, FFN is a fully connected feedforward network, MHSA is a multi-head self-attention mechanism, Conv is a convolution operation, and Layernorm is a layer normalization operation. x′ c ,x" c is the intermediate feature vector of the Conformer layer, y c A high-dimensional representation of the output; In step S4-2, the expression of the connection timing classification decoder calculation process is: Among them, p(l|x) is the probability of generating the output sequence l given the input feature sequence x, π is the alignment path, B -1 (l) represents the set of all possible paths that can generate the label sequence l, and P(π|x) represents the probability of aligning the path π; In step S4-3, the calculation process of the attention-based encoder-decoder is expressed as: Where K is the hidden state of the encoder, V is the output of the encoder, and Q is the input of the decoder. N is the number of output vectors of the encoder, and the weight α is i represents the correlation between the current time step of the decoder and the i-th hidden state of the encoder, thereby helping the model generate a more accurate output sequence, v i is the i-th context vector.

8. The end-to-end speech recognition method for air traffic control instructions under high noise conditions according to any one of claims 1 to 7, characterized in that: The end-to-end speech recognition model is trained and used for air traffic control command speech recognition. The training steps specifically include: Step S5, constructing an air traffic control environment noise-clean voice simulation data set and an air traffic control instruction voice-text data set; Step S6: use the simulated noise-clean speech data set to pre-train the adaptive attention noise reduction module; use the air traffic control command speech-text data set to separately train the speech recognition module; use the air traffic control command speech-text data set to jointly train the end-to-end speech recognition model.

9. The end-to-end speech recognition method for air traffic control instructions under high noise conditions according to claim 8, characterized in that: The step S5 specifically includes: Step S5-1, using an open source noise dataset and an open source clean speech Chinese dataset to respectively form a noise dataset and a clean speech dataset in the air traffic control environment noise-clean speech simulation dataset; Step S5-2, using the air traffic control system operation production data as raw data, processing the raw data to form an air traffic control instruction voice-text data set; Step S5-2 specifically includes: Step S5-2-1, dividing the original data into speech segments, marking the range of the segments containing human voice according to the change of the audio spectrum, and using the existing automatic speech recognition model to recognize the segments containing human voice to obtain the transcription text of each segment containing human voice; Step S5-2-2, manually annotating the vocal segment and the corresponding transcribed text; Step S5-2-3, perform text screening on the manual annotation results, remove unclear parts, divide the speech and corresponding text that constitute a single sentence instruction, and form the air traffic control instruction speech-text dataset.

10. The end-to-end speech recognition method for air traffic control instructions under high noise conditions according to claim 9, characterized in that: The step S6 specifically includes: Determine the loss function of the adaptive attention denoising module: in, is the loss function of the adaptive attention denoising module, is the gain estimate of the b-th frequency band, and the exponent ζ is a perceptual parameter that controls the degree of noise suppression; Determine the loss function of the speech recognition module: L c (x,y)=γL CTC (x,y)+(1-γ)L AED (x,y) Among them, L c (x, y) is the loss function of the speech recognition module, x, y are the input speech features and the target text sequence, L CTC (x,y),L AED (x, y) are the losses of CTC encoder and AED encoder respectively, γ is the first balancing weight coefficient; Determine the loss function of the end-to-end model: L total (x,y)=L c (x,y)+μWER Among them, L total (x, y) is the loss function of the end-to-end model, WER is the word error rate (Word ErrorRate) of the recognition result, and μ is the second balance weight coefficient; With the goal of minimizing the loss functions of the adaptive attention denoising module, the speech recognition module and the end-to-end model, pre-training of the adaptive attention denoising module, separate training of the speech recognition module and joint training of the end-to-end speech recognition model are completed.

Images