Air traffic control command end-to-end speech recognition method under high noise condition

Abstract

The application relates to the field of air traffic control and speech recognition technology, and particularly relates to an air traffic control instruction end-to-end speech recognition method under high noise conditions, which comprises the following steps: pre-processing to-be-recognized speech to extract original speech frequency features; adopting an adaptive attention noise reduction module to perform noise reduction intensity control, frequency band gain control and pitch tracking processing on the original speech frequency features to obtain noise reduction speech; pre-processing the noise reduction speech to extract noise reduction speech frequency features; adopting a speech recognition module to perform shared encoder coding, connection time sequence classification decoder decoding and attention-based encoder-decoder decoding on the noise reduction speech frequency features to obtain output speech recognition transcription text; and the application can improve the speech recognition accuracy of air traffic control instructions under high noise conditions.

Description

Technical Field

[0001] This invention relates to the fields of air traffic control and speech recognition technology, specifically to an end-to-end speech recognition method for air traffic control commands under high noise conditions. Background Technology

[0002] Air traffic control instructions are specific operational commands issued by air traffic controllers to pilots. Through these instructions, controllers provide pilots with real-time situational information about the airport and airspace, directing them to perform specific maneuvers. Pilots then perform corresponding maneuvers, such as takeoff, landing, or changes in altitude, speed, or direction, thereby maintaining air traffic order, preventing collisions between aircraft, and ensuring that aircraft fly along prescribed routes and speeds.

[0003] Air traffic management primarily relies on voice communication between pilots and air traffic controllers to transmit instructions and information. However, this process depends on manual recognition, which is prone to errors, omissions, and oversights, leading to frequent safety incidents. Therefore, it is necessary to introduce a voice recognition system to recognize the instructions and responses of controllers and pilots in real time, thereby reducing misunderstandings and forgetting.

[0004] Because pilots communicate in the cockpit, which suffers from severe noise interference, and because air traffic control voice communication is transmitted via VHF ground-to-air communication, high-frequency signal loss and audio clipping occur. These complex communication environment characteristics exacerbate the difficulty of recognition, especially when faced with multi-source noise superposition and frequency band instability. Existing technologies struggle to achieve accurate speech recognition results under these conditions. Currently, air traffic control command speech recognition methods commonly use hybrid speech recognition models that primarily train each module individually. These models suffer from insufficient collaborative optimization, limited information sharing between modules, and a tendency to encounter local optima during training, making it difficult to achieve overall optimal performance.

[0005] In summary, existing technologies lack noise reduction optimization designs for complex multi-source noise and communication environment characteristics, especially lacking targeted processing for factors such as cabin noise and VHF communication distortion. Furthermore, the various components of the hybrid recognition model cannot achieve joint optimization, resulting in insufficient accuracy and robustness of speech recognition in complex environments, making it difficult to meet practical needs. Summary of the Invention

[0006] In view of the above problems, the present invention provides an end-to-end speech recognition method for air traffic control commands under high noise conditions, which solves the problem of inaccurate recognition caused by multi-source high noise conditions and the difficulty of joint optimization of models in the prior art.

[0007] This invention provides an end-to-end speech recognition method for air traffic control commands under high noise conditions. The method employs a trained end-to-end speech recognition model and includes the following steps:

[0008] Step S1: Preprocess the speech to be recognized to extract the original speech frequency features;

[0009] Step S2: The adaptive attention noise reduction module is used to perform noise reduction intensity control, frequency band gain control and pitch tracking processing on the original speech frequency features to obtain the noise-reduced speech;

[0010] Step S3: Preprocess the denoised speech to extract the frequency features of the denoised speech;

[0011] Step S4: The speech recognition module is used to encode the noise-reduced speech frequency features using a shared encoder, decode them using a connected temporal classification decoder, and decode them using an attention-based encoder-decoder to obtain the output speech recognition transcribed text.

[0012] Preferably, step S1 specifically includes:

[0013] Step S1-1: Perform pre-emphasis, frame windowing, fast Fourier transform, and frequency band division on the speech to be recognized to obtain filter bank features.

[0014] Step S1-2: Obtain the pitch period and spectral nonstationarity measure of the speech to be recognized;

[0015] Steps S1-3: The first and second derivatives of the filter bank features, the pitch period, and the spectral nonstationarity measure are used together as the original speech frequency features.

[0016] Preferably, in step S1:

[0017] The expression for the pre-emphasis processing method is:

[0018] x′(n′)=x(n′)-0.97·x(n′-1)

[0019] Where x(n′) represents the value of the n′th sampling point of the speech signal, x(n′-1) represents the sampling value of the n′-1th point of the speech signal, 0.97 is the coefficient of the high-pass filter, which determines the proportion of high-frequency compensation, and x′(n′) represents the value of the n′th sampling point of the speech signal after pre-emphasis processing;

[0020] The expression for the window function of the frame-by-frame windowing is:

[0021]

[0022] Where w(n) represents the window function value of the nth sampling point in the frame, and N is the total number of sampling points in each frame;

[0023] The calculation expression for the filter bank characteristics is as follows:

[0024] F b =log(E(b)+∈)

[0025]

[0026] Among them, F b Here, E(b) represents the filter bank characteristics of the b-th frequency band, E(b) is the energy of the b-th frequency band of the speech signal, ∈ is a small constant, and w b (k) represents the amplitude of band b at frequency k, and |X(k)| represents the amplitude of the spectral value corresponding to frequency k.

[0027] Preferably, step S2 specifically includes:

[0028] Step S2-1: Use an adaptive controller to process the original speech frequency features using a convolutional neural network and a gated recurrent unit to obtain the noise reduction effect;

[0029] Step S2-2: The original speech frequency features are processed using an RNN-GRU network, and the frequency band gain of each frequency band of the audio frequency domain signal is obtained through smooth interpolation.

[0030] Step S2-3: Detect the pitch period in the speech signal, and determine the gain signal based on the pitch period using a comb filter;

[0031] Step S2-4: The gain signal is controlled by the noise reduction intensity and the frequency band gain to obtain the noise-reduced frequency domain signal, and then the noise-reduced speech is obtained by IFFT inverse fast Fourier transform.

[0032] Preferably, in step S2-1, the expression for the processing procedure of the convolutional neural network and the gated recurrent unit is:

[0033]

[0034] in, Here, is the FBank feature vector of the input original speech, Conv represents convolution operation, BN represents batch normalization, MaxPool represents max pooling, GRU is gated recurrent unit, and FC is fully connected layer. It is the intermediate feature vector of the adaptive controller network;

[0035] In step S2-2, the expression for the processing procedure of the RNN-GRU network is as follows:

[0036]

[0037] Among them, E s (b) represents the energy of the speech signal in the b-th frequency band, E x (b) represents the energy of the noisy speech signal in the b-th frequency band. Indicates the dynamically adjusted noise reduction intensity;

[0038] Steps S2-3 specifically include: obtaining the frequency band-dependent filter coefficients α based on the fundamental period of the comb filter. b Where the filter coefficient α b The expression is:

[0039]

[0040] Where min(·,·) represents taking the minimum value. Let k be the complex value of the signal in frequency band k. Let |X| be the complex conjugate value of the fundamental signal in frequency band k. k | 2 It is the power of the spectral value of frequency band k, |P k | 2 It is the power of the pitch signal in frequency band k, where k is the index of the frequency band and K is the total number of frequency bands.

[0041] Preferably, step S4 specifically includes:

[0042] Step S4-1: Input the noise-reduced speech frequency features into the shared encoder to obtain a high-dimensional representation;

[0043] Step S4-2: Input the high-dimensional representation into the temporal classification decoder to obtain the activation value;

[0044] Step S4-3: The attention-based encoder-decoder re-scores the activation values ​​to generate the final speech recognition transcribed text.

[0045] Preferably, in step S4-1, the shared encoder consists of multiple Conformer layers, and the structural expression of the Conformer layer is as follows:

[0046]

[0047] in, This is the FBank feature vector of the input denoised speech, FFN is a fully connected feedforward network, MHSA is a multi-head self-attention mechanism, Conv is a convolution operation, and Layernorm is a layer normalization operation. It is the intermediate feature vector of the Conformer layer, y c This represents the high-dimensional representation of the output.

[0048] In step S4-2, the expression for the calculation process of the connection time-series classification decoder is:

[0049]

[0050] Where p(l|x) is the probability of generating an output sequence l given an input feature sequence x. This represents summing over all possible alignment paths s, |l ′ | is the output sequence l ′ The length, α t (s) is a forward variable, representing the cumulative probability of aligning path s at time step t, β t (s) is a backward variable, representing the cumulative probability of aligning path s at time step t. This indicates the output sequence l at time step t. ′ s The activation value;

[0051] In step S4-3, the expression for the calculation process of the attention-based encoder-decoder is:

[0052]

[0053] Where K is the encoder's hidden state, V is the encoder's output, Q is the decoder's input, N is the number of encoder output vectors, and the weights α are... i This represents the correlation between the decoder's current time step and the encoder's i-th hidden state, thus helping the model generate a more accurate output sequence, v. i It is the i-th context vector.

[0054] Preferably, the end-to-end speech recognition model is trained, and the trained end-to-end speech recognition model is used for air traffic control command speech recognition. The training steps specifically include:

[0055] Step S5: Construct an air traffic control environment noise-clean speech simulation dataset and an air traffic control instruction speech-text dataset;

[0056] Step S6: Pre-train the adaptive attention noise reduction module using the simulated noise-clean speech dataset; train the speech recognition module separately using the air traffic control instruction speech-text dataset; and jointly train the end-to-end speech recognition model using the air traffic control instruction speech-text dataset.

[0057] Preferably, step S5 specifically includes:

[0058] Step S5-1: Use open-source noise dataset and open-source clean speech Chinese dataset to construct the noise dataset and clean speech dataset in the air traffic control environment noise-clean speech simulation dataset, respectively.

[0059] Step S5-2: Using the air traffic control system operation production data as raw data, process the raw data to form an air traffic control command voice-text dataset;

[0060] Step S5-2 specifically includes:

[0061] Step S5-2-1: Divide the original data into speech segments, mark the range of human voice segments according to the changes in the audio spectrum, and use the existing automatic speech recognition model to identify the human voice segments to obtain the transcribed text of each human voice segment.

[0062] Step S5-2-2: Manually annotate the voice-containing segments and the corresponding transcribed text;

[0063] Step S5-2-3: Filter the manually labeled results to remove unclear parts, and divide them into speech and text of single-sentence instructions to form the air traffic control instruction speech-text dataset.

[0064] Preferably, step S6 specifically includes:

[0065] Determine the loss function of the adaptive attention denoising module:

[0066]

[0067] in, The loss function for the adaptive attention denoising module. For the gain estimation of the b-th frequency band, the exponent ζ is a sensing parameter that controls the degree of noise suppression;

[0068] Determine the loss function of the speech recognition module:

[0069] L c (x,y)=γL CTC (x,y)+(1-γ)L AED (x,y)

[0070] Among them, L c (x, y) is the loss function of the speech recognition module, where x and y are the input speech features and the target text sequence, respectively. CTC (x,y),L AED (x,y) represent the losses of the CTC encoder and the AED encoder, respectively, and γ is the first balancing weight coefficient;

[0071] Determine the loss function for the end-to-end model:

[0072] L total (x,y)=L c (x,y)+μWER

[0073] Among them, L total (x,y) is the loss function of the end-to-end model, WER is the word error rate of the recognition result, and μ is the second balancing weight coefficient;

[0074] With the goal of minimizing the loss function of the adaptive attention noise reduction module, the speech recognition module, and the end-to-end model, the pre-training of the adaptive attention noise reduction module, the individual training of the speech recognition module, and the joint training of the end-to-end speech recognition model are completed.

[0075] Compared with the prior art, the present invention has at least the following beneficial effects:

[0076] (1) This invention employs an Adaptive Attention Noise Reduction (ABNS) module for noise reduction intensity control, frequency band gain control, and pitch tracking. Through a noise reduction module specifically designed for air traffic control speech, it can effectively suppress noise interference and reduce audio clipping distortion, thereby improving the speech recognition accuracy in complex environments. It can still stably recognize and extract clear speech information under high noise conditions.

[0077] (2) This invention enhances the accuracy and robustness of the speech recognition module by establishing an air traffic control environment noise-clean speech simulation dataset and an air traffic control instruction speech-text dataset, and by specifically training the speech recognition module for air traffic control instructions.

[0078] (3) This invention combines the adaptive attention noise reduction module with the speech recognition module to build an end-to-end recognition model, realizing the joint optimization of the model as a whole. This not only improves the recognition effect, but also greatly improves the recognition speed and robustness, and can better cope with complex noise conditions. Attached Figure Description

[0079] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0080] Figure 1 The flowchart of the end-to-end speech recognition method for air traffic control commands under high noise conditions provided by the present invention is shown.

[0081] Figure 2 This is a flowchart of the joint pre-training and model usage of the end-to-end model provided by the present invention.

[0082] Figure 3This is a flowchart illustrating the noise suppression process performed by the adaptive attention noise reduction module provided in this invention.

[0083] Figure 4 This invention provides a flowchart of the end-to-end speech recognition model training steps. Detailed Implementation

[0084] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0085] To illustrate the effectiveness of the method proposed in this invention, the following detailed description of the above technical solution is provided through a specific embodiment, such as... Figure 1 As shown, an end-to-end speech recognition method for air traffic control commands under high noise conditions is disclosed. The specific implementation steps include: performing speech recognition using an end-to-end speech recognition model, including the following steps:

[0086] Step S1: Preprocess the speech to be recognized and extract the original speech frequency features.

[0087] like Figure 2 As shown, the speech signal is first pre-emphasized, meaning it passes through a high-pass filter to compensate for high-frequency component loss, ensuring sufficient high-frequency information is retained during subsequent processing. The expression for this processing method is:

[0088] x′(n′)=x(n′)-0.97·x(n′-1)

[0089] Where x(n′) represents the value of the n′th sampling point of the speech signal, x(n′-1) represents the sampling value of the n′-1th point of the speech signal, 0.97 is the coefficient of the high-pass filter, which determines the proportion of high-frequency compensation, and x′(n′) represents the value of the n′th sampling point of the speech signal after pre-emphasis processing.

[0090] The speech signal is divided into multiple segments using frame segmentation and windowing. A 20-millisecond window is selected, and 50% overlap is applied. Each segment is then windowed. The expression for the window function is:

[0091]

[0092] Where w(n) represents the window function value of the nth sampling point in the frame, and N is the total number of sampling points in each frame.

[0093] Perform a Fast Fourier Transform on each windowed segment to convert the time-domain signal into a frequency-domain signal, as shown in the following equation:

[0094]

[0095] Where X(k) represents the spectral value corresponding to frequency k, x′(n) is the time-domain sampled signal, and the number of points of the fast Fourier transform is set to N.

[0096] Finally, the spectrum X(k) is divided into frequency bands by a set of Mel-scale triangular filters. The logarithm of the energy in each frequency range is taken to obtain the filter bank characteristics (FilterBank, Fbank), which are called Fbank characteristics.

[0097] The specific steps are as follows: define the frequency band energy as E(b), where E(b) is the energy of the b-th frequency band, and let w b (k) represents the amplitude of band b at frequency k. Let ∑ b w b (k) = 1, completing the normalization of amplitudes across all frequency bands. For the transformed signal X(k), the energy in the frequency band is as follows:

[0098]

[0099] The FBank is obtained by performing a logarithmic transformation on the energy of the frequency band:

[0100] F b =log(E(b)+∈)

[0101] Among them, F b is the FBank feature of the b-th frequency band, and ∈ is a small constant.

[0102] Obtain the pitch period and spectral nonstationarity measure of a speech segment.

[0103] The first and second derivatives of the FBank features, the pitch period, and the spectral nonstationarity measure are used together as the extracted original speech frequency features for subsequent noise reduction processing.

[0104] Step S2: Using an adaptive attention noise reduction module, noise reduction intensity control, frequency band gain control, and pitch tracking processing are applied to the original speech frequency features to obtain noise-reduced speech.

[0105] Specifically, such as Figure 3As shown, this invention employs an Adaptive Attention-Based Noise Suppression (ABNS) module to achieve speech noise reduction. The ABNS module includes an adaptive controller, a recurrent neural network-gated recurrent unit network, and a pitch tracking processing procedure.

[0106] This invention employs an adaptive controller to control the noise reduction intensity based on the frequency characteristics, a recurrent neural network-gated recurrent unit network to control the bandwidth gain based on the frequency characteristics, and a comb filter to perform pitch tracking based on the frequency characteristics, ultimately obtaining the denoised speech. A detailed description follows.

[0107] (1) An adaptive controller is used to obtain the noise reduction effect.

[0108] The adaptive controller utilizes the input FBank features to dynamically adjust the noise reduction intensity parameter α for each frame, controlling the noise reduction intensity of the noise reduction module across all frequency bands. The adaptive controller is used to mine the relationship between FBank features and the word error rate (WER) of speech recognition, thereby generating appropriate noise reduction intensities under different noise environments.

[0109] The adaptive controller combines a convolutional neural network (CNN) and a gated recurrent unit (GRU), with the input being the FBank feature vector of the original speech. The system consists of convolutional layers, batch normalization, max pooling, GRU layers, and fully connected layers, ultimately outputting the noise reduction intensity α.

[0110] The network structure expression for the adaptive controller is:

[0111]

[0112]

[0113] in, Here, is the FBank feature vector of the input original speech, Conv represents convolution operation, BN represents batch normalization, MaxPool represents max pooling, GRU is gated recurrent unit, and FC is fully connected layer. This is the intermediate feature vector of the adaptive controller's network. Through neural network calculations within the adaptive controller, parameters for dynamically adjusting the noise reduction intensity for each frame can be obtained.

[0114] (2) Use RNN-GRU network to obtain bandwidth gain.

[0115] In this step, the FBank features obtained in step S1 are input into the RNN-GRU network, and after feature extraction, the gain g of each frequency band is obtained. b gb G represents the gain of the b-th frequency band, and g represents the noise suppression capability of that frequency band. b The calculation formula is:

[0116]

[0117] Among them, E s (b) represents the energy of the speech signal in the b-th frequency band, E x (b) represents the energy of the noisy speech signal in the b-th frequency band. This indicates the dynamically adjusted noise reduction intensity.

[0118] By using the neural network calculation of the RNN-GRU network, the frequency band gain parameters of each frequency band of the audio frequency domain signal can be obtained.

[0119] (3) The filtered output is noise-reduced speech.

[0120] Interpolate the band gain parameters of each frequency band in the neural network output to obtain the optimal gain for each band (bin). Considering the optimal band gain, the interpolation gain expression applied to each band (bin) is as follows:

[0121]

[0122] Where r(k) is the gain of frequency band k.

[0123] This invention detects the pitch period in a speech signal and processes the pitch period signal using a comb filter. The comb filter, based on the pitch period, uses a frequency band-dependent filtering coefficient α. b And processed in the frequency domain. The pitch correlation P of frequency band b b Defined as:

[0124]

[0125] in, Let k be the complex value of the signal in frequency band k. Let |X| be the complex conjugate value of the fundamental signal in frequency band k. k | 2 It is the power of the spectral value of frequency band k, |P k | 2 It is the power of the pitch signal in frequency band k, where k is the index of the frequency band and K is the total number of frequency bands.

[0126] α b The value of P depends on the frequency band. b and g b The expression is:

[0127]

[0128] Here, min(·,·) represents taking the minimum value.

[0129] The gain signal output by the filter is superimposed on the original signal after the noise reduction intensity parameter α controls the noise reduction intensity of the noise reduction module across all frequency bands, forming a noise-reduced frequency domain signal. This signal is then subjected to IFFT (Inverse Fast Fourier Transform) to obtain the noise-reduced speech, which serves as the input for subsequent speech recognition steps.

[0130] Step S3: Preprocess the denoised speech and extract the frequency features of the denoised speech.

[0131] The present invention performs preprocessing on the denoised speech obtained in step S2 in a similar manner to that in step S1, including pre-emphasis, frame windowing, fast Fourier transform, and frequency band division, to obtain the FBank features of the denoised speech.

[0132] Step S4: The speech recognition module is used to encode the noise-reduced speech frequency features using a shared encoder, decode them using a connected temporal classification decoder, and decode them using an attention-based encoder-decoder to obtain the output speech recognition transcribed text.

[0133] The speech recognition module of the present invention includes a shared encoder, a connectionist temporal classification (CTC) decoder, and an attention-based encoder (AED).

[0134] In this step, the FBank features of the denoised speech are input into a shared encoder. The shared encoder transforms the input sequence into a set of high-dimensional representations, consisting of multiple Conformer layers. The structure of the Conformer layers can be represented by the following equation:

[0135]

[0136]

[0137] in, This is the FBank feature vector of the input denoised speech, FFN is a fully connected feedforward network, MHSA is a multi-head self-attention mechanism, Conv is a convolution operation, and Layernorm is a layer normalization operation. It is the intermediate feature vector of the Conformer layer, y c This represents the high-dimensional representation of the output.

[0138] The present invention's CTC decoder processes the high-dimensional representation of the shared encoder's output. The CTC decoder receives a feature sequence from the shared encoder and maps it to a CTC activation, expressed as:

[0139]

[0140] Where p(l|x) is the probability of generating an output sequence l given an input feature sequence x, π is the alignment path, representing the sequence output by the network, which includes repeated characters and empty characters, and P(π|x) is the probability of the alignment path π, usually expressed as the product of the predicted probabilities at each time step along the path. Where p(π) t |x) is the character π predicted at time step t. t The probability, B(π), is the probability of mapping from the alignment path π to the final label sequence l. -1 (l) represents the set of all possible paths that can generate the label sequence l.

[0141] After completing CTC decoding, the AED decoder re-scores the CTC activations and generates the final recognition result. The AED decoder's process involves first processing the input denoised speech signal into hidden representations that capture the contextual information of the input sequence. Then, an attention mechanism calculates the correlation between the decoder's current time step and the encoder's hidden state, assigning weights based on these correlations to generate a weighted context vector. The decoder uses this context vector, along with the previously generated output, to produce the next output label. This process is repeated until a complete output sequence is generated. The expression for this process is:

[0142]

[0143] Where K is the hidden state of the encoder, x is the output of the encoder, and N is the input of the decoder, where N is the number of output vectors of the encoder, and the weights α are... i This represents the correlation between the decoder's current time step and the encoder's i-th hidden state, thus helping the model generate a more accurate output sequence, v. i It is the i-th context vector.

[0144] Through the above steps, the shared encoder converts the input sequence into a set of high-dimensional representations. The CTC decoder receives the high-dimensional representations from the shared encoder and maps them to CTC activations. After completing CTC decoding, the AED decoder re-scores the CTC activations and generates the final speech recognition transcribed text.

[0145] The present invention also includes the step of training the adaptive attention noise reduction module in the end-to-end speech recognition model and the neural network in the speech recognition module, including constructing a dataset and training individually and jointly, such as... Figure 4 As shown, it specifically includes:

[0146] Step S5: Construct an air traffic control environment noise-clean speech simulation dataset and an air traffic control instruction speech-text dataset.

[0147] In this embodiment, the air traffic control environment noise-clean speech simulation dataset is divided into a noise dataset and a clean speech dataset. The noise dataset uses the open-source dataset Noise-92, and the clean speech data uses the open-source dataset Aishell-1 Chinese dataset.

[0148] For the air traffic control command voice-text dataset, the data used is production data from the operation of the air traffic control system. The raw data is collected from the air traffic management system of an airport tower and other units, which records the voice data of control calls. It is mainly Chinese voice data of control calls, and includes some English voice data of control calls.

[0149] The raw data is processed to form an air traffic control command speech-text dataset. The processing procedure for the raw data within the air traffic control command speech-text dataset is as follows:

[0150] 1) Use existing automatic speech recognition models to segment and recognize human voice segments.

[0151] The raw data was divided into segments of 5 minutes each, and the range containing human voice segments was marked according to the changes in the audio spectrum. The segments were then identified using an existing pre-trained automatic speech recognition model, and the transcribed text obtained was used as a reference for the next step.

[0152] 2) Construct annotation and proofreading platforms, with manual segmentation, proofreading, and text annotation. Develop annotation rules, including audio annotation difficulty classification, segmentation annotation standards, and text annotation standards. Visualize the audio segments, annotations, and transcribed text obtained in the previous step. Professionals then further process the segmentation and annotation according to the annotation rules, accurately defining the scope and correcting incorrect annotations. After annotation is complete, the results of the professional segmentation and annotation are randomly assigned to quality control personnel for review and to improve the accuracy of the annotation.

[0153] 3) Filter the text content, remove unclear annotations, and divide the raw data into speech and text sets for single-sentence instructions. Export the manually reviewed annotations, clean the annotation data, and remove data with invalid annotations, incomplete start and end times, or unknown characters in the transcribed text. Based on the annotations, divide the text set into a format of "transcribed text content corresponding to the speech ID".

[0154] Step S6: Train the end-to-end speech recognition model using the air traffic control environment noise-clean speech simulation dataset and the air traffic control command speech-text dataset, including:

[0155] The adaptive attention noise reduction module is pre-trained using the simulated noise-clean speech dataset.

[0156] The recognition module was trained separately using an air traffic control command speech-text dataset;

[0157] The end-to-end speech recognition model was jointly trained using an air traffic control command speech-text dataset.

[0158] The specific steps include: first, pre-training the adaptive attention denoising module using a simulated noise-clean speech dataset; and then using the mean square error (MSE) of the exponent of the bandwidth gain as the loss function for the RNN network in the denoising module, expressed as:

[0159]

[0160] in, The loss function for the adaptive attention denoising module. For the gain estimate of the b-th frequency band, the exponent ζ is a sensing parameter that controls the degree of noise suppression.

[0161] Next, the recognition module is trained separately. A partial air traffic control command speech-text dataset is input to train the CTC encoder and AED encoder. The loss function consists of two parts, expressed as follows:

[0162] L c (x,y)=γL CTC (x,y)+(1-γ)L AED (x,y)

[0163] Among them, L c (x, y) is the loss function of the speech recognition module, where x and y are the input speech features and the target text sequence, respectively. CTC (x,y),L AED (x,y) represent the losses of the CTC encoder and the AED encoder, respectively, and γ is the first balancing weight coefficient.

[0164] Finally, the end-to-end model is jointly trained using an air traffic control command speech-text dataset as input, through backpropagation. The training loss is as follows:

[0165] L total (x,y)=L c (x,y)+μWER

[0166] Among them, L total (x,y) is the loss function of the end-to-end model, WER is the word error rate of the recognition result, and μ is the second balancing weight coefficient.

[0167] In the above steps, the adaptive attention noise reduction module is trained by backpropagation with the goal of minimizing the loss function of the adaptive attention noise reduction module, the speech recognition module, and the end-to-end model.

[0168] Compared with existing technologies, the present invention provides an end-to-end speech recognition method for air traffic control instructions under high noise conditions. This method addresses the problem of inaccurate recognition caused by the lack of noise reduction design tailored to the characteristics of air traffic control speech and the inability to jointly optimize the various parts of the hybrid recognition model. The present invention designs a noise reduction module specifically for air traffic control calls, improving the speech quality and recognition accuracy of air traffic control instructions. Furthermore, it constructs an end-to-end speech recognition model by integrating the noise reduction module design and designs an adaptive attention module to optimize the noise reduction module, achieving overall model optimization and improving the model's recognition capability.

[0169] While the specific embodiments of the present invention depict actions or steps in a particular order, this should be understood as requiring such actions or steps to be performed in the shown specific order or sequential order, or requiring all illustrated actions or steps to be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations. The above descriptions are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention.

[0170] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An end-to-end speech recognition method for air traffic control commands under high noise conditions, characterized in that, Speech recognition is performed using a trained end-to-end speech recognition model, including the following steps: Step S1: Preprocess the speech to be recognized to extract the original speech frequency features; Step S2: The adaptive attention noise reduction module is used to perform noise reduction intensity control, frequency band gain control and pitch tracking processing on the original speech frequency features to obtain the noise-reduced speech; Step S3: Preprocess the denoised speech to extract the frequency features of the denoised speech; Step S4: The speech recognition module is used to encode the noise-reduced speech frequency features using a shared encoder, decode them using a connected temporal classification decoder, and decode them using an attention-based encoder-decoder to obtain the output speech recognition transcribed text. Step S2 specifically includes: Step S2-1: Use an adaptive controller to process the original speech frequency features using a convolutional neural network and a gated recurrent unit to obtain the noise reduction effect; Step S2-2: The original speech frequency features are processed using an RNN-GRU network, and the frequency band gain of each frequency band of the audio frequency domain signal is obtained through smooth interpolation. Step S2-3: Detect the pitch period in the speech signal, and determine the gain signal based on the pitch period using a comb filter; Step S2-4: The gain signal is controlled by the noise reduction intensity and the frequency band gain to obtain the noise-reduced frequency domain signal, and then the noise-reduced speech is obtained by IFFT inverse fast Fourier transform. In step S2-1, the expression for the processing procedure of the convolutional neural network and the gated recurrent unit is as follows: in, It is the FBank feature vector of the input original speech. This represents the convolution operation. Indicates batch normalization. This indicates max pooling. It is a gated loop unit. It is a fully connected layer. It is the intermediate feature vector of the adaptive controller network. Indicates the noise reduction intensity parameter; In step S2-2, the expression for the processing procedure of the RNN-GRU network is as follows: in, Indicates the speech signal at the 1st The energy of the frequency band Indicates the noisy speech signal at the 1st Energy in the frequency band Indicates the dynamically adjusted noise reduction intensity; Steps S2-3 specifically include: obtaining the frequency band-dependent filter coefficients using a comb filter based on the pitch period. , where the filter coefficients The expression is: in, This indicates taking the minimum value. For the signal in the frequency band The complex value of, For frequency band The complex conjugate value of the fundamental tone signal. It is frequency band The power of the spectral value, It is frequency band The power of the fundamental tone signal, It is a frequency band index. This is the total number of frequency bands; The end-to-end speech recognition model is trained and then used for air traffic control command speech recognition. The specific training steps include: Step S5: Construct an air traffic control environment noise-clean speech simulation dataset and an air traffic control instruction speech-text dataset; Step S6: Pre-train the adaptive attention noise reduction module using the air traffic control environment noise-clean speech simulation dataset; train the speech recognition module separately using the air traffic control command speech-text dataset; and jointly train the end-to-end speech recognition model using the air traffic control command speech-text dataset.

2. The end-to-end speech recognition method for air traffic control commands under high noise conditions according to claim 1, characterized in that, Step S1 specifically includes: Step S1-1: Perform pre-emphasis, frame windowing, fast Fourier transform, and frequency band division on the speech to be recognized to obtain filter bank features. Step S1-2: Obtain the pitch period and spectral nonstationarity measure of the speech to be recognized; Steps S1-3: The first and second derivatives of the filter bank features, the pitch period, and the spectral nonstationarity measure are used together as the original speech frequency features.

3. The end-to-end speech recognition method for air traffic control commands under high noise conditions according to claim 2, characterized in that, In step S1: The expression for the pre-emphasis processing method is: in, Indicates the first speech signal The value of each sampling point, Indicates the first speech signal The sampled values ​​at each point, 0.97 is the coefficient of the high-pass filter, which determines the proportion of high-frequency compensation. This indicates the first pre-emphasis processed speech signal. The values ​​of each sampling point; The expression for the window function of the frame-by-frame windowing is: in, Indicates the first in the frame Window function values ​​at each sampling point It is the total number of sampling points in each frame; The calculation expression for the filter bank characteristics is as follows: in, It is the first Filter bank characteristics for each frequency band For the first speech signal Energy of each frequency band It is a small constant. For frequency band In frequency Amplitude at that point Represents frequency The amplitude of the corresponding spectral value.

4. The end-to-end speech recognition method for air traffic control commands under high noise conditions according to claim 3, characterized in that, Step S4 specifically includes: Step S4-1: Input the noise-reduced speech frequency features into the shared encoder to obtain a high-dimensional representation; Step S4-2: Input the high-dimensional representation into the temporal classification decoder to obtain the activation value; Step S4-3: The attention-based encoder-decoder re-scores the activation values ​​to generate the final speech recognition transcribed text.

5. The end-to-end speech recognition method for air traffic control commands under high noise conditions according to claim 4, characterized in that, In step S4-1, the shared encoder consists of multiple Conformer layers, and the structural expression of the Conformer layer is as follows: in, It is the FBank feature vector of the input denoised speech. It is a fully connected feedforward network. This is a multi-head self-attention mechanism. For convolution operations, For layer normalization operation, These are the intermediate feature vectors of the Conformer layer. This represents the high-dimensional representation of the output. In step S4-2, the expression for the calculation process of the connection time-series classification decoder is: in, Given an input feature sequence Generate output sequence The probability, To align the path, Indicates the generation of label sequences The set of all possible paths, Indicates alignment path The probability of; In step S4-3, the expression for the calculation process of the attention-based encoder-decoder is: in, This is the encoder's hidden state. It is the output of the encoder, and That is the input to the decoder. The number of output vectors of the encoder, weights Represents the current time step of the decoder and the encoder's... The correlation between hidden states helps the model generate more accurate output sequences. It is the first A context vector.

6. The end-to-end speech recognition method for air traffic control commands under high noise conditions according to claim 1, characterized in that, Step S5 specifically includes: Step S5-1: Use open-source noise dataset and open-source clean speech Chinese dataset to construct the noise dataset and clean speech dataset in the air traffic control environment noise-clean speech simulation dataset, respectively. Step S5-2: Using the air traffic control system operation production data as raw data, process the raw data to form an air traffic control command voice-text dataset; Step S5-2 specifically includes: Step S5-2-1: Divide the original data into speech segments, mark the range of human voice segments according to the changes in the audio spectrum, and use the existing automatic speech recognition model to identify the human voice segments to obtain the transcribed text of each human voice segment. Step S5-2-2: Manually annotate the voice-containing segments and the corresponding transcribed text; Step S5-2-3: Filter the manually labeled results to remove unclear parts, and divide them into speech and text of single-sentence instructions to form the air traffic control instruction speech-text dataset.

7. The end-to-end speech recognition method for air traffic control commands under high noise conditions according to claim 6, characterized in that, Step S6 specifically includes: Determine the loss function of the adaptive attention denoising module: in, The loss function for the adaptive attention denoising module. For the first Gain estimation for each frequency band, exponential It is a sensing parameter that controls the degree of noise suppression; Determine the loss function of the speech recognition module: in, Let be the loss function of the speech recognition module. The input speech features and the target text sequence are respectively. The losses of the CTC encoder and the AED encoder are respectively. This is the first balancing weight coefficient; Determine the loss function for the end-to-end model: in, The loss function for the end-to-end model. The word error rate of the recognition results, This is the second balancing weight coefficient; With the goal of minimizing the loss function of the adaptive attention noise reduction module, the speech recognition module, and the end-to-end model, the pre-training of the adaptive attention noise reduction module, the individual training of the speech recognition module, and the joint training of the end-to-end speech recognition model are completed.

Images