A speech recognition method based on air traffic control speech recognition model with speech rate perception

Abstract

The present invention discloses a speech recognition method for an air traffic control speech recognition model based on speech rate perception, comprising: step 1: counting the normal speech rate range of air traffic control speech; step 2: constructing an air traffic control speech recognition model including the speech rate estimation layer; step 3: defining a joint loss function, training the air traffic control speech recognition model, and dynamically adjusting the CTC loss weight coefficient and the speech rate perception loss weight coefficient in the joint loss function during the training process; step 4: using the trained speech recognition model to recognize air traffic control speech data. The present invention introduces speech rate perception loss constraints to form a multi-task learning mechanism that combines the main task of speech recognition with the auxiliary task of speech rate regularity learning, which significantly improves the robustness and recognition accuracy of the air traffic control land-to-air call speech recognition model in complex environments, and provides more reliable technical support for the safe and efficient operation of air traffic control.

Description

Technical Field

[0001] The present invention relates to the field of aviation air traffic control voice communication technology, and in particular to a speech recognition method of an air traffic control voice recognition model based on speech rate perception. Background Art

[0002] As the core vehicle for transmitting air traffic control instructions, the accuracy and robustness of speech recognition for ground-to-air conversations are directly related to aviation operational safety. Currently, speech recognition models based on end-to-end deep learning (such as sincnet, LSTM, and Transformer architectures) are widely used in the air traffic control field for real-time translation of ground-to-air conversations. However, in complex air traffic control environments (such as those with high noise, strong interference, and mixed multilingual scenarios), fluctuations in speech rate become a key bottleneck limiting recognition accuracy, manifesting as:

[0003] 1. Character loss due to excessive speaking speed: When the controller or pilot speaks at a speed exceeding the normal range, the model cannot match the time alignment between the audio and character sequences, resulting in missing key command characters (such as waypoints and altitude values);

[0004] 2. Redundant insertion caused by slow speech: Slow speech can easily cause the model to output redundant characters (such as repeated numbers and redundant prepositions), interfering with the accuracy of command parsing.

[0005] 3. Sensitivity to cross-language speech speed differences: The speech speed distribution of Chinese and English land and air calls varies significantly (Chinese is measured in characters per second, English is measured in words per second). The existing single speech speed constraint mechanism is difficult to adapt to multilingual scenarios.

[0006] The above defects make it difficult for existing air traffic control-oriented speech recognition models to cope with the dynamic changes in basic aviation data.

[0007] The main methods and limitations of existing technologies that attempt to solve the above problems include: 1. Acoustic feature enhancement: improving input quality through noise suppression and speech enhancement, but cannot solve the problem of sequence alignment misalignment caused by changes in speaking rate; 2. Dynamic Time Warping (DTW) optimization: improving the CTC alignment path search strategy, but with high computational complexity and limited adaptability to extreme speaking rates; 3. Rule post-processing: correcting recognition results based on heuristic rules, requiring manual design of the speaking rate-error mapping relationship, poor generalization ability and high maintenance cost.

[0008] The root cause is that traditional speech recognition models optimize solely for minimizing recognition error (e.g., CTC loss) and lack explicit constraints on the physical plausibility of the output sequence. Speech rate, an essential physical law of speech production (the number of characters / words per unit time has physiological limits), is not effectively incorporated into the model training process. This results in the model outputting character sequences that violate physical laws when the speech rate is abnormal.

[0009] Therefore, there is an urgent need for a method that can embed the physical laws of speech rate as prior knowledge into model training, and simultaneously optimize the recognition accuracy and speech rate rationality through a multi-task learning mechanism, so as to fundamentally improve the robustness of the air traffic control speech recognition system in complex environments. Summary of the Invention

[0010] Purpose of the invention: The technical problem to be solved by the present invention is to provide a speech recognition method for an air traffic control speech recognition model based on speech rate perception in response to the shortcomings of the existing technology.

[0011] In order to solve the above technical problems, the present invention discloses a speech recognition method based on an air traffic control speech recognition model with speech rate perception, comprising the following steps:

[0012] Step 1: Count the normal speaking speed range of air traffic control speech;

[0013] Step 2: Constructing an air traffic control speech recognition model including the speech rate estimation layer;

[0014] Step 3: Define a joint loss function, train the air traffic control speech recognition model, and dynamically adjust the CTC loss weight coefficient and the speech rate perception loss weight coefficient in the joint loss function during the training process;

[0015] Step 4: Use the trained speech recognition model to recognize air traffic control speech data.

[0016] The normal speaking speed range of the air traffic control speech in step 1 specifically includes:

[0017] Step 1-1: Define a Chinese and English speech rate statistics algorithm. For Chinese land and air calls, count the number of characters recognized per unit time as the speech rate indicator; for English land and air calls, count the number of words recognized per unit time as the speech rate indicator;

[0018] Step 1-2: Count the Chinese and English speaking speeds based on the previously collected air traffic control speech recognition training dataset;

[0019] Step 1-3: Set the normal speaking speed range of Chinese voice to characters per second, where and Determined by calculating the character rate in the Chinese air traffic control speech recognition training data set;

[0020] Step 1-4: Set the normal speaking speed range of English voice to words / second, where and The rate of words in the English air traffic control speech recognition training data set is determined.

[0021] Step 2 describes the construction of an air traffic control speech recognition model, specifically including:

[0022] Step 2-1: Build the backbone network of the end-to-end speech recognition model, using the FBANK feature sequence of air traffic control speech as input, and adopting an architecture based on a multi-layer Transformer network;

[0023] Step 2-2: Add a speech rate estimation layer after the network output layer to estimate the character / word rate of the current speech segment ;

[0024] Step 2-3: Before inputting the ATC speech recognition model, the speech segment is input into the language detection network, the language is automatically determined based on the FBANK features of the input speech, and the language label is passed to the joint loss function;

[0025] Step 2-4: Get the speech segment duration from the input audio metadata and pass the speech segment duration to the joint loss function.

[0026] The joint loss function in step 3 is the weighted sum of the CTC loss and the speech rate perception loss function. The joint loss function is:

[0027]

[0028] in, is the CTC loss, is the speech rate perception loss, is the dynamic weight coefficient of CTC loss, ranging from [0,1], is the dynamic weight coefficient of speech rate perception loss, ranging from [0,1].

[0029] The speech rate perception loss function specifically includes:

[0030] The speech rate loss function is defined as:

[0031]

[0032] Where R is the character / word rate currently output by the ATC speech recognition model; is the dynamic target speaking speed range; An adjustable positive hyperparameter that controls the penalty intensity for speech rates below the target range. An adjustable positive hyperparameter that controls the penalty for speaking rates above the target range.

[0033] The dynamic target speech speed interval automatically selects the Chinese normal speech speed interval determined in steps 1-3 according to the input speech language. Or the normal English speaking speed range determined in steps 1-4 .

[0034] The speech rate estimation layer described in step 2-2 specifically includes:

[0035] Step 2-2-1: Receive the original logits sequence output by the backbone network of the air traffic control speech recognition model, recorded as ,in is the number of time steps, is the vocabulary size;

[0036] Step 2-2-2: Sharpen the logits: multiply the original logits by the preset magnification factor , get the sharpened logits, recorded as ;

[0037] Step 2-2-3: Calculate the weighted character probability distribution: perform a softmax operation on the sharpened logits along the vocabulary dimension to generate a probability distribution matrix ,in, is the time dimension of the model output, is the vocabulary dimension size of the model output;

[0038] Step 2-2-4: Estimate the number of valid characters based on the probability distribution matrix ;

[0039] Step 2-2-5: Calculate the effective character rate based on the input speech duration , output rate ;

[0040]

[0041] For English speech, the vocabulary V is based on complete words, so Characterizes the estimated number of words, output rate The unit is word / second. For Chinese speech, the vocabulary V is based on characters, so Characterizes the estimated number of characters; output rate The unit is characters per second.

[0042] Step 2-2-4 is as follows:

[0043] Step 2-2-4-1, the probability distribution matrix along the time dimension Sum, that is, calculate the probability and of each vocabulary item at all time steps t, and get the vector ;

[0044] Step 2-2-4-2, based on vector Counting whitespace <pad>Number ;

[0045] Step 2-2-4-3, calculate the number of non-whitespace characters: .

[0046] The training described in step 3 specifically includes:

[0047] Step 3-1: In the initial stage of training, set the weight coefficient of speech rate perception loss and only use CTC loss for model training;

[0048] Step 3-2: During training, monitor the character error rate (CER) on the validation set.

[0049] Step 3-3: When the verification set error rate CER drops to a first preset threshold, the speech rate perception loss weight coefficient is started to be executed Adjustment;

[0050] Step 3-4: Continue training the air traffic control speech recognition model until the preset maximum number of training rounds is reached and then stop training.

[0051] Step 3-3 The adjustment method is to increase linearly , each time a training cycle is completed, Add a fixed increment , until Reach the second preset value, and according to Adjustment ,when After reaching the second preset value, keep The second preset value remains unchanged.

[0052] The steps for speech recognition based on the model in step 4 include:

[0053] Step 4-1: Obtain the air traffic control voice segment to be recognized from the civil aviation air traffic control voice communication exchange system;

[0054] Step 4-2: Preprocess the speech segment to obtain an FBANK feature sequence;

[0055] Step 4-3: Use the language detection network to automatically determine the language (Chinese or English) of the current speech segment based on the FBANK feature sequence;

[0056] Step 4-4: Inputting the FBANK feature sequence into the trained air traffic control speech recognition model;

[0057] In step 4-5, the air traffic control speech recognition model outputs the recognition result text and the effective character / word rate R to the subsequent processing system.

[0058] The air traffic control speech recognition model performs the following operations:

[0059] Step 4-4-1: Process the input feature sequence through the end-to-end speech recognition model backbone network to generate the original logits sequence;

[0060] Step 4-4-2: Use the greedy decoding algorithm to obtain the recognition result text according to the original logits sequence;

[0061] Step 4-4-3: Process the original logits sequence through the speech rate estimation layer to calculate the effective character / word rate R of the current speech segment;

[0062] Step 4-5: Output the recognition result text and effective character / word rate R to the subsequent processing system.

[0063] Beneficial effects:

[0064] 1. Addressing recognition errors caused by speech rate mismatching in traditional methods: The speech rate-aware loss function explicitly penalizes outputs outside the normal range, effectively suppressing missing characters and redundant insertions.

[0065] 2. Enhanced model robustness: A dynamic weight adjustment strategy ensures the model first focuses on the main task and converges, then gradually introduces speech rate constraints to avoid training oscillations.

[0066] 3. Deployment Practicality: The speech rate estimation layer infers speech rate based on probability distribution, without the need for additional labeled data, and is compatible with existing end-to-end model architectures. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0068] Figure 2a This is a schematic diagram for determining the normal speaking speed range of Chinese.

[0069] Figure 2b This is a schematic diagram for determining the normal speaking speed range of English.

[0070] Figure 3 Schematic diagram of the end-to-end speech recognition model of the present invention. DETAILED DESCRIPTION

[0071] To address the problems of missing characters, redundant insertions, and poor cross-language adaptability caused by fluctuating speech rates in existing land-to-air speech recognition technologies, the present invention discloses a speech recognition method based on a speech rate-aware air traffic control speech recognition model. By constructing a speech rate estimation layer, designing a speech rate-aware loss function, and combining it with a dynamic weight adjustment strategy, the physical laws of speech rate are embedded as prior knowledge in the model training process, forming a multi-task learning mechanism. This method forces the model to simultaneously optimize recognition accuracy and speech rate rationality, fundamentally improving recognition robustness in complex air traffic control environments.

[0072] The embodiment of the present invention is aimed at the land-air call speech recognition scenario. In order to solve the problems of missing characters, redundant insertions and poor cross-language adaptability caused by speech rate fluctuations, a speech recognition method for an air traffic control speech recognition model based on speech rate perception is designed. By constructing a speech rate estimation layer, designing a speech rate perception loss function, and combining a dynamic weight adjustment strategy, the physical laws of speech rate are embedded as prior knowledge in a multi-task learning framework, significantly improving the robustness of the model in complex air traffic control environments. The overall flow diagram of the present invention is shown in the attached figure. Figure 1 The specific implementation steps are as follows:

[0073] A speech recognition method based on an air traffic control speech recognition model with speech rate perception comprises the following steps:

[0074] Step 1: Count the normal speaking speed range of air traffic control speech;

[0075] Step 2: Constructing an air traffic control speech recognition model including the speech rate estimation layer;

[0076] Step 3: Define a joint loss function, train the air traffic control speech recognition model, and dynamically adjust the CTC loss weight coefficient and the speech rate perception loss weight coefficient in the joint loss function during the training process;

[0077] Step 4: Use the trained speech recognition model to recognize air traffic control speech data.

[0078] The normal speaking speed range of the air traffic control speech in step 1 specifically includes:

[0079] Step 1-1: Define a Chinese and English speech rate statistics algorithm. For Chinese land and air calls, count the number of characters recognized per unit time as the speech rate indicator; for English land and air calls, count the number of words recognized per unit time as the speech rate indicator;

[0080] Step 1-2: Count the Chinese and English speaking speeds based on the previously collected air traffic control speech recognition training dataset;

[0081] Step 1-3: Set the normal speaking speed range of Chinese voice to characters per second, where and Determined by calculating the 10%-90% percentile of the character rate in the Chinese air traffic control speech recognition training data set;

[0082] Step 1-4: Set the normal speaking speed range of English voice to words / second, where and The rate is determined by calculating the 10%-90% percentile of the word rate in the English air traffic control speech recognition training data set.

[0083] like Figure 2a As shown, in this embodiment, the 10% percentile of the character rate in the Chinese air traffic control speech recognition training data set is 3.46, and the 90% percentile is 7.10;

[0084] like Figure 2b As shown, in this embodiment, the 10% percentile of the character rate in the English air traffic control speech recognition training data set is 1.99, and the 90% percentile is 3.43.

[0085] Step 2 describes the construction of an air traffic control speech recognition model, specifically including:

[0086] Step 2-1: Build the backbone network of the end-to-end speech recognition model, using the FBANK feature sequence of air traffic control speech as input, and adopting an architecture based on a multi-layer Transformer network;

[0087] Step 2-2: Add a speech rate estimation layer after the network output layer to estimate the character / word rate of the current speech segment ;

[0088] Step 2-3: Before inputting into the ATC speech recognition model, the speech segment is fed into the language detection network, which automatically determines the language (Chinese / English) based on the FBANK features of the input speech and passes the language label to the joint loss function;

[0089] Step 2-4: Obtain the duration of the speech segment from the input audio metadata and pass the duration of the speech segment to the speech rate loss calculation module.

[0090] The joint loss function in step 3 is the weighted sum of the connection temporal classification (CTC) loss and the speech rate perception loss function. The joint loss function is:

[0091]

[0092] in, is the Connectionist Temporal Classification (CTC) loss, is the speech rate perception loss, is the dynamic weight coefficient of CTC loss, ranging from [0,1], is the dynamic weight coefficient of speech rate perception loss, ranging from [0,1].

[0093] The speech rate perception loss function specifically includes:

[0094] The speech rate loss function is defined as:

[0095]

[0096] Where R is the character / word rate currently output by the ATC speech recognition model (calculated in characters / second for Chinese and words / second for English). The dynamic target speaking speed range is automatically selected according to the input language, which is the normal speaking speed range of Chinese determined in steps 1-3. Or the normal English speaking speed range determined in steps 1-4 ; An adjustable positive hyperparameter that controls the penalty intensity for speech rates below the target range. An adjustable positive hyperparameter that controls the penalty for speaking rates above the target range.

[0097] In this embodiment Take 3, The value 2 is mainly used to prevent the model from outputting missing words, which is more likely to cause information loss.

[0098] The speech rate estimation layer described in step 2-2 specifically includes:

[0099] Step 2-2-1: Receive the original logits sequence output by the backbone network of the air traffic control speech recognition model, recorded as ,in is the number of time steps, is the vocabulary size (including whitespace) <pad>); where the size of T is related to the speech duration of the input model and the structure of the model. In this embodiment, one time step corresponds to a speech duration of 25 ms, and the size of V is 4000;

[0100] Step 2-2-2: Perform a sharpening operation on the logits: Multiply the original logits by a preset amplification factor (i.e., much greater than 1) to obtain the sharpened logits, denoted as ; To make the logits sharper, K is generally taken as 20;

[0101] Step 2-2-3: Calculate the weighted character probability distribution: Perform a softmax operation on the sharpened logits along the vocabulary dimension to generate a probability distribution matrix , where is the size of the time dimension output by the model, and is the size of the vocabulary dimension output by the model; After sharpening, the probability of the word with the highest probability output by the model at each time step is very close to 1, and in actual tests, it is not less than 0.9999;

[0102] Step 2-2-4: Estimate the number of valid characters :

[0103] Step 2-2-5: Calculate the valid character or word rate, according to the input speech duration (seconds), and the output rate ;

[0104]

[0105] Among them, for English speech, the vocabulary V uses complete words as the basic unit, so represents the estimated number of words; for Chinese speech, the vocabulary V uses characters (including Chinese characters, letters, numbers, etc.) as the basic unit, so represents the estimated number of characters; the output rate The unit is correspondingly words / second or characters / second.

[0106] In this embodiment, for the audio corresponding to the transcribed text "Dongfang 3984, turn left, course 060", the length is 2.80 seconds, and the rate is 4.64 characters per second.

[0107] Step 2-2-4 is specifically:

[0108] Step 2-2-4-1, sum the probability distribution matrix along the time dimension, that is, calculate the sum of the probabilities of each vocabulary item at all time steps t, and obtain a vector ; <00003Step 2-2-4-2, based on vector Counting whitespace <pad>Number ;

[0110] Step 2-2-4-3, calculate the number of non-blank characters (i.e., model output characters): Because the probability of the word with the highest probability output by the model at each time step is close to 1, we only need to subtract the blank character from T <pad>The cumulative sum can be used to obtain an estimate of the output characters of the model.

[0111] In this embodiment, for the audio corresponding to the transcribed text "Dongfang Sanjiuba Si turn left to course 0606", the estimated value of the number of non-blank characters output is 13.0003.

[0112] As Figure 3 shown, the training described in step 3 specifically includes:

[0113] Step 3-1: At the initial stage of training, set the weight coefficient of the speech rate perception loss , and only use the CTC loss for model training, where the weight coefficient of the CTC loss ;

[0114] Step 3-2: During the training process, monitor the character error rate CER on the validation set; the calculation method of the character error rate is as follows: CER = (S + D + I) / N × 100%, where S represents the number of positions where the characters do not match after alignment (substitution), D represents the number of positions that exist in the true text but are missing in the recognition result (deletion), and I represents the number of positions that exist in the recognition result but are missing in the true text (insertion).

[0115] For example, assume the true text is "Dongfang Sanjiuba Si", and the model recognition result is "Dongfang Sanjiuba San". After alignment analysis, S = 1, D = 0, I = 0, N = 6, so CER = 1 / 6.

[0116] Step 3-3: When the error rate CER of the validation set drops to the first preset threshold, start to perform the adjustment of the weight coefficient of the speech rate perception loss ; in this embodiment, the first preset threshold is taken as 0.05;

[0117] Step 3-4: Continue to train the air traffic control speech recognition model until the preset maximum number of training epochs is reached and then stop training.

[0118] The adjustment method of the in step 3-3 is: increase in a linear manner. For each completed training cycle (epoch), increase by a fixed increment until reaches the second preset value (for example, 0.5), and adjust according to . When reaches the second preset value, keep equal to the second preset value and remain unchanged. In this embodiment, the second preset threshold is 0.5.

[0119] The steps for speech recognition based on the model in step 4 include:

[0120] Step 4-1: Obtain the air traffic control voice segment to be recognized from the civil aviation air traffic control voice communication exchange system;

[0121] Step 4-2: Preprocess the speech segment to obtain an FBANK feature sequence;

[0122] Step 4-3: Use the language detection network to automatically determine the language (Chinese or English) of the current speech segment based on the FBANK feature sequence;

[0123] Step 4-4: Inputting the FBANK feature sequence into the trained air traffic control speech recognition model;

[0124] The air traffic control speech recognition model performs the following operations:

[0125] Step 4-4-1: Process the input feature sequence through the end-to-end speech recognition model backbone network to generate the original logits sequence;

[0126] Step 4-4-2: Use the greedy decoding algorithm to obtain the recognition result text according to the original logits sequence;

[0127] Step 4-4-3: Process the original logits sequence through the speech rate estimation layer to calculate the effective character / word rate R of the current speech segment;

[0128] Step 4-5: Output the recognition result text and effective character / word rate R to the subsequent processing system.

[0129] Example 1: Specific implementation of the air traffic control speech recognition model training and application method based on speech rate perception:

[0130] This example uses an air traffic control speech recognition system development project as an example to illustrate the implementation of the method. The hardware environment includes an Intel Xeon Gold 6248R processor and an NVIDIA RTX3080 GPU. The software environment includes Python 3.8, PyTorch 1.12, and CUDA 11.6.

[0131] Step 1: Determine the normal speaking speed range of air traffic control speech

[0132] a) Prepare a Chinese dataset: 10,000 Chinese land and air conversations (total duration 20 hours), containing Chinese characters, numbers, and letters; prepare an English dataset: 8,000 English land and air conversations (total duration 18 hours)

[0133] Audio sampling rate: 16kHz, PCM format

[0134] b) Speech rate statistics calculation:

[0135] Chinese: For each voice recording, count the total number of characters in the transcribed text (including Chinese characters / numbers / letters) and divide it by the audio duration (in seconds).

[0136] English: Count the total number of words in the transcript (space-delimited), divided by the audio duration

[0137] c) Statistical results:

[0138] Chinese speaking speed 10% percentile: 3.46 characters / second, 90% percentile: 7.10 characters / second, normal range [3.46, 7.10]

[0139] English speaking speed 10th percentile: 1.99 words / second, 90th percentile: 3.43 words / second, normal range [1.99, 3.43]

[0140] Step 2: Constructing an air traffic control speech recognition model including the speech rate estimation layer;

[0141] Step 3: Define a joint loss function, train the air traffic control speech recognition model, and dynamically adjust the CTC loss weight coefficient and the speech rate perception loss weight coefficient in the joint loss function during the training process.

[0142] The speech rate perception loss function is constructed as follows:

[0143] Write the following Python function to implement speech rate perception loss:

[0144] def speed_loss(R, S_min, S_max, lambda1=3, lambda2=2):

[0145] if R >= S_min and R <= S_max:

[0146] return 0.0

[0147] elif R < S_min:

[0148] return lambda1 * (S_min - R)**2 # Focus on penalizing missing words (λ1=3)

[0149] else:

[0150] return lambda2 * (R - S_max)**2 # Penalize redundancy (λ2=2)

[0151] Step 1: Construct the speech rate estimation layer

[0152] The input of this neural network layer is the logits sequence output by the backbone network (T = time step, V = 4000 vocabulary), followed by a sharpening operation (amplification factor K = 20), then calculate the probability distribution: P = softmax(Z', dim=-1), and finally estimate the number of valid characters, s = torch.sum(P, dim=0), N_pad = s[pad_index], N_valid = T - N_pad

[0153] For an audio with a duration of 2.80 seconds and a transcribed text of "Dongfang Sanjiuba Si Zuo Zhuan Hang Xiang Dong Liu Dong" (13 characters), the output is N_valid = 13.0003, so the speech rate R = 13.0003 / 2.80 ≈ 4.64 characters per second

[0154] The joint training model

[0155] Construct a speech recognition backbone network, which consists of 12 layers of Transformer. The audio features input to the network are 28-dimensional FBANK features with a frame length of 25ms. (Reference paper: Pardede H F, Zilvan V, Krisnandi D, et al.Generalized filter-bank features for robust speech recognition againstreverberation[C] / / 2019 International Conference on Computer, Control,Informatics and its Applications (IC3INA). IEEE, 2019: 19-24.)

[0156] The joint loss function is , and the input of the speech rate loss calculation module includes three parts: the logits sequence output by the backbone network, the language label (from the language classification model, reference paper: Snyder D,Garcia-Romero D, McCree A, et al. Spoken language recognition using x-vectors[C] / / Odyssey. 2018, 2018: 105-111.), and the audio duration (read from the metadata of the audio file).

[0157] The dynamic weight training strategy

[0158] In the initial stage, only CTC loss is used and the weights are fixed. , .

[0159] After the 30th round (one round refers to the training of all data in the training data), the validation set CER was found to be 0.049 on the validation set, which met the condition of being less than 0.05 and entered the dynamic adjustment stage.

[0160] The adjustment strategy is β = min(β + 0.05, 0.5), which means increasing by 0.05 per round, with an upper limit of 0.5, and .

[0161] When the training reaches 100 rounds, the training is stopped and the trained speech recognition model is obtained.

[0162] Finally, the speech recognition model using only the CTC loss function is compared on the test set. The results are shown in Table 1:

[0163] Table 1. Comparison of test results

[0164] Loss Function CER after 100 epochs of training CER reduction percentage CTC 0.053 - CTC+Speed ​​Perception 0.045 17.7%

[0165] The above experimental results show that the air traffic control speech recognition model training method based on speech rate perception of the present invention can effectively reduce the recognition error rate and improve the recognition accuracy.

[0166] Step 4: Use the trained model for air traffic control speech recognition

[0167] This step uses the model trained in step 3 to recognize actual air traffic control speech:

[0168] Obtaining the speech to be recognized: Obtaining the air traffic control speech segment to be recognized from the civil aviation air traffic control voice communication exchange system (for example, real-time recording or historical recording file).

[0169] The acquired speech segment is preprocessed by resampling to 16kHz and extracting a 28-dimensional FBANK feature sequence (frame length 25ms, frame shift 10ms) using the same method as step 2 in the training phase. The duration D of the speech segment is then read from the audio file metadata.

[0170] Language detection: The FBANK feature sequence extracted in step 2 is input into the language detection network to automatically determine the language of the current speech segment (Chinese or English) and obtain the language label.

[0171] Model inference: Input the preprocessed FBANK feature sequence into the air traffic control speech recognition model trained in step 3. The model performs the following operations:

[0172] Step 4-4-1: Feature processing and logits generation: The input feature sequence is processed by the backbone network of the end-to-end speech recognition model (12-layer Transformer) built in step 2 to generate the original logits sequence Z ∈ R^(T×4000).

[0173] Step 4-4-2: Text decoding: According to the original logits sequence Z, use the greedy decoding algorithm (GreedyDecoding) to obtain the final recognition result text.

[0174] Step 4-4-3: Speech Rate Estimation: Input the original logits sequence Z generated in Step 4-4-1 into the speech rate estimation layer constructed in Step 2. Calculate the number of valid characters / words N_valid in the current speech segment according to the process described in Step 2 (sharpening Z' = 20*Z -> softmax -> calculating s -> N_pad -> N_valid ). Combined with the speech duration D obtained from the metadata, calculate the speech rate R = N_valid / D.

[0175] Example: Consider a 3.5-second audio clip of English control instructions. The model outputs N_valid=8.2, so R=8.2 / 3.5 ≈ 2.34 words / second.

[0176] Output results: The recognition result text obtained in step 4-4-2 and the effective character / word rate R calculated in step 4-4-3 are output to the subsequent processing system (such as the air traffic control instruction parsing system, the controller workload assessment system or the voice data quality monitoring system).

[0177] The present invention provides a speech recognition method for an air traffic control speech recognition model based on speech rate perception. There are many methods and approaches for implementing this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.< / pad> < / pad> < / pad> < / pad>

Claims

1. A speech recognition method based on an air traffic control speech recognition model with speech rate perception, characterized in that: The steps include: Step 1: Count the normal speaking speed range of air traffic control speech; Step 2: Build an air traffic control speech recognition model that includes a speech rate estimation layer; Step 3: Define a joint loss function, train the air traffic control speech recognition model, and dynamically adjust the CTC loss weight coefficient and the speech rate perception loss weight coefficient in the joint loss function during the training process; Step 4: Use the trained ATC speech recognition model to recognize ATC speech; The normal speaking speed range of the air traffic control speech in step 1 specifically includes: Step 1-1: Define a Chinese and English speech rate statistics algorithm. For Chinese land and air calls, count the number of characters recognized per unit time as the speech rate indicator; for English land and air calls, count the number of words recognized per unit time as the speech rate indicator; Step 1-2: Count the Chinese and English speaking speeds based on the previously collected air traffic control speech recognition training dataset; Step 1-3: Set the normal speaking speed range of Chinese voice to characters per second, where is the lowest normal speaking speed for Chinese, and The highest normal speaking speed for Chinese and Determining by calculating the character rate in the air traffic control speech recognition training data set; Step 1-4: Set the normal speaking speed range of English voice to words / second, where This is the lowest normal speaking speed in English. This is the highest normal speaking speed for English. and Determined by calculating the word rate in the air traffic control speech recognition training data set; The speech rate estimation layer described in step 2-2 specifically includes: Step 2-2-1: Receive the original logits sequence output by the backbone network of the air traffic control speech recognition model, recorded as ,in, is the original logits sequence corresponding to the time step T, is the number of time steps, is the vocabulary size; Step 2-2-2: Sharpen the original logits sequence: multiply the original logits by the preset magnification factor , get the sharpened logits, recorded as ; Step 2-2-3: Calculate the weighted character probability distribution: perform a softmax operation on the sharpened logits along the vocabulary dimension to generate a probability distribution matrix ,in, is the activation function, is the time dimension of the model output, is the vocabulary dimension size of the model output; Step 2-2-4: Estimate the number of valid characters based on the probability distribution matrix ; Step 2-2-5: Calculate the effective character rate based on the input speech duration , output rate ; ； For English speech, the word table dimension size V is based on complete words as the basic unit, so Characterizes the estimated number of words, output rate The unit is word / second. For Chinese speech, the word table dimension size V is based on characters, so Characterizes the estimated number of characters; output rate The unit is characters per second.

2. The speech recognition method of the air traffic control speech recognition model based on speech rate perception according to claim 1 is characterized in that: Step 2 describes the construction of an air traffic control speech recognition model, specifically including: Step 2-1: Build the backbone network of the end-to-end air traffic control speech recognition model, using the FBANK feature sequence of air traffic control speech as input and adopting a multi-layer Transformer network-based architecture; Step 2-2: Add a speech rate estimation layer after the network output layer to estimate the current output character / word rate ; Step 2-3: Before inputting the ATC speech recognition model, the speech segment is input into the language detection network, the language is automatically determined based on the FBANK feature sequence of the input ATC speech, and the language label is passed to the joint loss function; Step 2-4: Get the speech segment duration from the input audio metadata and pass the speech segment duration to the joint loss function.

3. The speech recognition method of the air traffic control speech recognition model based on speech rate perception according to claim 2 is characterized in that: The joint loss function in step 3 is the weighted sum of the CTC loss and the speech rate perception loss function. The joint loss function is: ； in, is the CTC loss, is the speech rate perception loss, is the dynamic weight coefficient of CTC loss, ranging from [0,1], is the dynamic weight coefficient of speech rate perception loss, ranging from [0,1].

4. The speech recognition method of the air traffic control speech recognition model based on speech rate perception according to claim 3 is characterized in that: The speech rate perception loss function specifically includes: The speech rate loss function is defined as: ； Where R is the character / word rate currently output by the ATC speech recognition model; is the dynamic target speaking speed interval, is the dynamic target minimum speaking speed, The maximum speaking speed of the dynamic target; An adjustable positive hyperparameter that controls the penalty intensity for speech rates below the target range. An adjustable positive hyperparameter that controls the penalty intensity for speech rates above the target range; The dynamic target speech speed interval automatically selects the normal speech speed interval of the Chinese speech determined in steps 1-3 according to the input speech language. Or the normal speaking speed range of English speech determined in steps 1-4 .

5. The speech recognition method of the air traffic control speech recognition model based on speech rate perception according to claim 4 is characterized in that: The training described in step 3 specifically includes: Step 3-1: In the initial stage of training, set the weight coefficient of speech rate perception loss and only use CTC loss for model training; Step 3-2: During the training process, monitor the character error rate (CER) on the validation set. Step 3-3: When the character error rate CER on the verification set drops to a first preset threshold, the dynamic weight coefficient of the speech rate perception loss is started. Adjustment; Step 3-4: Continue training the air traffic control speech recognition model until the preset maximum number of training rounds is reached and then stop training.

6. The speech recognition method of the air traffic control speech recognition model based on speech rate perception according to claim 5 is characterized in that: Dynamic weight coefficient described in step 3-3 The adjustment method is to increase linearly , each time a training cycle is completed, Add a fixed increment , until Reach the second preset value, and according to Adjust the dynamic weight coefficient of CTC loss ,when After reaching the second preset value, keep The second preset value remains unchanged.

7. The speech recognition method of the air traffic control speech recognition model based on speech rate perception according to claim 6 is characterized in that: The steps of recognizing the air traffic control speech based on the trained air traffic control speech recognition model in step 4 include: Step 4-1, obtaining the air traffic control voice segment to be recognized from the civil aviation air traffic control voice communication exchange system; Step 4-2, pre-processing the air traffic control voice segment to obtain an FBANK feature sequence; Step 4-3: Automatically determine the language of the current speech segment based on the FBANK feature sequence through the language detection network; Step 4-4, inputting the FBANK feature sequence into the trained air traffic control speech recognition model; In step 4-5, the air traffic control speech recognition model outputs the recognition text results and character / word rate R to the subsequent processing system.

Images