A speech recognition method based on incremental learning for land-air communication scenarios

Abstract

The present invention discloses a speech recognition method based on incremental learning for land-air communication scenarios, comprising the following steps: Step 1: constructing a vocabulary for the speech recognition model and reserving slots in the vocabulary; Step 2: calculating the model parameter importance matrix and generating a model parameter training mask; Step 3: collecting and annotating a small number of new call segments as training and test data based on the addition of new proper nouns as needed; Step 4: pre-training the speech recognition model and loading the speech recognition model parameter training mask, performing incremental training with a cross-entropy loss, and regularly testing the recognition accuracy on the test data during training; Step 5: replacing the original speech recognition model with the newly trained speech recognition model parameters to complete the model's incremental learning; and Step 6: using the speech recognition model that has completed incremental learning to recognize speech. This method effectively overcomes the drawback of models in field deployment environments, which often cannot quickly adapt to new terminology while retaining old knowledge.

Description

Technical Field

[0001] The present invention relates to the field of aviation control communication technology, and in particular to a speech recognition method based on incremental learning for land-to-air communication scenarios. Background Art

[0002] As the core vehicle for transmitting air traffic control instructions, the accuracy and real-time performance of speech recognition in ground-to-air conversations are directly related to aviation operational safety. Currently, deep learning-based speech recognition systems are widely used in the air traffic control field for real-time translation and analysis of ground-to-air conversations. However, existing technologies have the following key bottlenecks:

[0003] 1. Lack of real-time adaptability to newly added proper nouns:

[0004] In aviation scenarios, updates to the Aeronautical Information Publication (AIP) occasionally introduce new proper nouns (such as new waypoints, new airport names, new navigation stations, and new general aviation airline call signs). Traditional speech recognition models have a rigid vocabulary structure. When encountering new terms outside the vocabulary, the system must redesign the vocabulary and fully train the model, which takes days to weeks and is unable to meet the needs of on-site air traffic control.

[0005] 2. Catastrophic forgetting problem in incremental learning:

[0006] Existing incremental learning methods, such as replay-based algorithms, rely on historical training data to adjust model parameters. However, in field deployments, original training data is often unavailable due to privacy or storage restrictions. Fine-tuning the model solely with new data can lead to significant forgetting of historical knowledge, potentially causing errors in land-air call recognition.

[0007] The above defects make it difficult for existing air traffic control-oriented speech recognition models to cope with the dynamic changes in basic aviation data. Summary of the Invention

[0008] Purpose of the invention: The technical problem to be solved by the present invention is to address the shortcomings of the existing technology and provide a speech recognition method based on incremental learning for land-air call scenarios.

[0009] In order to solve the above technical problems, the present invention discloses a speech recognition method based on incremental learning for land-air communication scenarios, comprising the following steps:

[0010] Step 1: Build a vocabulary for the speech recognition model and reserve slots in the vocabulary;

[0011] Step 2: Calculate the model parameter importance matrix and generate the model parameter training mask;

[0012] Step 3: Based on the new proper nouns that need to be added, collect and annotate a small number of new call segments as training data and test data;

[0013] Step 4: Pre-train the speech recognition model and load the speech recognition model parameter training mask. Perform incremental training with cross-entropy loss. During the training process, regularly test the recognition accuracy on the test data.

[0014] Step 5: Use the latest trained speech recognition model parameters to replace the original speech recognition model to complete the incremental learning of the model;

[0015] Step 6: Use the speech recognition model that has completed incremental learning to recognize speech.

[0016] The reserved slots in the vocabulary described in step 1 include:

[0017] The speech recognition model records the mapping between text and the one-hot encoding of the model output in its vocabulary. Generally speaking, the vocabulary size of a speech recognition model is equal to the total number of words the model needs to recognize and the number of neurons in the vocabulary output layer. During initial training, the vocabulary size is fixed. However, when new proper nouns are added, the vocabulary size changes, altering the structure of the final vocabulary output layer of the speech recognition model and requiring retraining. When constructing the speech recognition model's vocabulary, N empty slots (N = 10% × vocabulary size) are reserved. When encountering new proper nouns, they are added to the empty slots without changing the vocabulary size.

[0018] The calculation of the model parameter importance matrix described in step 2 to generate the model parameter training mask specifically includes:

[0019] Step 2-1: After the initial training of the speech recognition model is completed, the parameter importance of each model parameter is calculated using the initial training data to form a model parameter importance matrix;

[0020] Step 2-2: Binarize the importance matrix based on a preset importance threshold to generate a model parameter training mask: mark parameters with importance scores higher than the threshold as important parameters (mask value is 1), and mark the remaining parameters as adjustable parameters (mask value is 0); wherein the mask is used to lock key parameters that are sensitive to historical knowledge in the incremental training stage.

[0021] Step 2-1 specifically includes:

[0022] Step 2-1-1: For each sample in the initial training data, calculate the cross entropy loss between the model output and the true label;

[0023] Step 2-1-2: Calculate the gradient of the cross entropy loss with respect to each model parameter;

[0024] Step 2-1-3: For each model parameter, calculate the expectation of its squared gradient value on the initial training data as the parameter importance of the model parameter. For the i-th model parameter The importance of parameters Defined as:

[0025]

[0026] in, represents the initial training data set, represents the number of samples in the initial training data set, represents the set of all model parameters trained with the initial training data, is the input value of each sample, It's a model The output of x is is the partial differential, is a logarithmic function, Indicates that for a certain sample (including input x and label y), the i-th parameter Gradient in the cross entropy loss.

[0027] Step 2-1-1 specifically includes:

[0028] Step 2-1-1-1: Divide the initial training data into batches (mini-batches) of size M, where M ≥ 32;

[0029] Step 2-1-1-2: Perform parallel computation for each batch: (a) model output for all samples in the batch; (b) batch-calculate the cross entropy loss for all samples in the batch based on the model output and the true labels;

[0030] Step 2-1-1-3: Repeat step 2-1-1-2 until all initial training data are traversed.

[0031] Step 2-2 is specifically implemented as follows:

[0032] Step 2-2-1: Setting the importance threshold threhold. The importance threshold threhold may be determined by:

[0033] (1) Calculate the Kth percentile of the score distribution of all parameters in the importance matrix as threshold, where K∈[30,39]; (2) Determine the balance between anti-forgetting effect and model plasticity through experiments on the validation set;

[0034] Step 2-2-2: Binarize the importance matrix element by element:

[0035] If the parameter importance score threshold, the output mask value is 1; otherwise, the output mask value is 0;

[0036] For the i-th model parameter, it corresponds to the i-th mask value for:

[0037]

[0038] Step 2-2-3: Store the binarization result as a structured binary matrix whose dimension is consistent with the model parameter tensor dimension, and the 1-value position corresponds to the model parameter to be locked.

[0039] Step 3 specifically includes:

[0040] Step 3-1: When receiving a user instruction or the system detects that a new proper noun needs to be added, original land-air call audio segments containing the new proper noun are collected to form an unlabeled segment set;

[0041] Step 3-2: Manually screen and transcribe the unlabeled segment set, select the audio segments containing the new proper nouns, and generate text labels corresponding to the audio segments;

[0042] Step 3-3: Divide the annotated segment set into a training dataset and a test dataset according to a preset ratio, with the training dataset accounting for 80% and the test dataset accounting for 20%.

[0043] The system in step 3-1 detects that a new proper noun needs to be added, specifically including:

[0044] The output results of the speech recognition model are monitored in real time. When a ground-to-air call segment is identified with a confidence level lower than a threshold and does not comply with the ground-to-air call specifications, it is determined that a new proper noun needs to be added.

[0045] Step 4 specifically includes:

[0046] Step 4-1: Load the pre-trained speech recognition model and the model parameter training mask generated in step 2;

[0047] Step 4-2: training the mask according to the model parameters, freezing the model parameters corresponding to the mask value of 1, and retaining only the model parameters with the mask value of 0 as trainable, that is, only updating the trainable model parameters when using gradient to update the model parameters;

[0048] Step 4-3: Use the training dataset collected in step 3 to perform incremental training with the cross entropy loss function. During the training process, the mini-batch stochastic gradient descent method is used, and the batch size is set to B, where B ≥ 16;

[0049] Step 4-4: During incremental training, after each E batch of training, evaluate the recognition accuracy of the current model on the test dataset collected in step 3, where E ≥ 100;

[0050] Step 4-5: If the preset early stopping conditions are met, the early stopping mechanism is triggered and step 5 is executed.

[0051] The early stopping conditions in steps 4-5 include:

[0052] The absolute value of the improvement in accuracy for P consecutive tests is less than the threshold δ; the maximum number of training rounds is reached; and the accuracy on the test set reaches the target threshold.

[0053] Beneficial effects:

[0054] 1. Overcoming the bottleneck of rapidly adapting to new proper nouns at deployment sites: Dynamically embedding new terms through reserved slots in the vocabulary avoids the full model retraining required by traditional vocabulary reconstruction, shortening model update time from days to weeks to hours, meeting the requirements of on-site air traffic control deployments.

[0055] 2. Addressing catastrophic forgetting in scenarios without historical data: Generate a binary mask based on the parameter importance matrix (a mask value of 1 locks key parameters, and a mask value of 0 opens adjustable parameters). This allows for forgetting-resistant incremental training with only a small amount of newly annotated data. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0057] Figure 2a This is a schematic diagram of the original vocabulary slot reservation mechanism.

[0058] Figure 2b This is a schematic diagram of the slot reservation mechanism for the vocabulary table.

[0059] Figure 2c This is a schematic diagram of the vocabulary slot reservation mechanism after the addition.

[0060] Figure 3 This is a schematic diagram of mask-driven incremental training. DETAILED DESCRIPTION

[0061] The present invention relates to a method for rapid and efficient incremental learning of speech recognition models in a field deployment environment, leveraging vocabulary slot reservation, model parameter importance assessment, and mask generation techniques, combined with a small amount of newly collected annotated data. This method aims to address the difficulty faced by air-to-ground speech recognition systems in field operations, which often struggle to quickly adapt to new aviation terminology (such as new flight numbers, call signs, and airport codes) while preserving existing knowledge. This method significantly improves the robustness, adaptability, and recognition accuracy of the speech recognition system in a dynamically changing air traffic control environment, providing technical support for safe and efficient air traffic control operations. To address the challenges of existing air-to-ground speech recognition technologies, which lack real-time adaptability to new terminology and suffer from catastrophic forgetting in scenarios without historical data, the present invention discloses a speech recognition method based on incremental learning for air-to-ground conversations. Through vocabulary slot reservation and a parameter importance-driven mask training mechanism, the model can rapidly adapt to new terminology while maintaining the stability of historical knowledge in field deployments.

[0062] The embodiment of the present invention is aimed at the land-to-air call speech recognition scenario. In order to solve the problems of rapid adaptation of newly added proper nouns and catastrophic forgetting in the absence of historical data, a speech recognition method based on incremental learning for the land-to-air call scenario is designed. New terms are dynamically embedded by reserving vocabulary slots, and the mask training mechanism driven by parameter importance is combined to lock key historical knowledge. Only a small amount of new annotated data is needed to achieve anti-forgetting incremental learning. At the same time, the present invention optimizes the incremental training process, adopts cross-entropy loss combined with early stopping mechanism, and improves training efficiency while ensuring model stability. The present invention can provide on-site model update capabilities for land-to-air call speech recognition systems deployed on-site, significantly improving the adaptability and safety of air traffic control environments. The overall process diagram of the present invention is shown in the attached figure. Figure 1 As shown, the following steps are included:

[0063] Step 1: Build a vocabulary for the speech recognition model and reserve slots in the vocabulary;

[0064] Step 2: Calculate the model parameter importance matrix and generate the model parameter training mask;

[0065] Step 3: Based on the new proper nouns that need to be added, collect and annotate a small number of new call segments as training data and test data;

[0066] Step 4: Pre-train the speech recognition model and load the speech recognition model parameter training mask. Perform incremental training with cross-entropy loss. During the training process, regularly test the recognition accuracy on the test data.

[0067] Step 5: Use the latest trained speech recognition model parameters to replace the original speech recognition model to complete the incremental learning of the model;

[0068] Step 6: Use the speech recognition model that has completed incremental learning to recognize speech.

[0069] The reserved slots in the vocabulary described in step 1 include:

[0070] The speech recognition model records the mapping between text and the one-hot encoding of the model output in its vocabulary. Generally speaking, the vocabulary size of a speech recognition model is equal to the total number of words the model needs to recognize and the number of neurons in the vocabulary output layer. During initial training, the vocabulary size is fixed. However, when new proper nouns are added, the vocabulary size changes, altering the structure of the final vocabulary output layer of the speech recognition model and requiring retraining. When constructing the speech recognition model's vocabulary, N empty slots (N = 10% × vocabulary size) are reserved. When encountering new proper nouns, they are added to the empty slots without changing the vocabulary size.

[0071] like Figure 2a-2c As shown, the original vocabulary size is 4120, the reserved vocabulary size is 4532, and the vocabulary size after the addition is 4532. In this embodiment, when the speech recognition model is trained, the original vocabulary originally contains 4120 words, and 412 empty slots are added later (filled <unk>placeholder), set the new vocabulary size to 4532, and train the speech recognition model on this vocabulary (that is, the number of neurons in the output layer of the speech recognition model vocabulary is 4532). When encountering a word that is not in the vocabulary, just append it to the empty slot without changing the vocabulary size.

[0072] The calculation of the model parameter importance matrix described in step 2 to generate the model parameter training mask specifically includes:

[0073] Step 2-1: After the initial training of the speech recognition model is completed, the parameter importance of each model parameter is calculated using the initial training data to form a model parameter importance matrix;

[0074] In this embodiment, the speech recognition model consists of a 12-layer transformer neural network, which contains a total of 110M trainable parameters. The initial training data contains 1.3 million speech clips and corresponding text annotations. The mean of the squared gradient value of each model parameter is calculated on these 1.3 million training data to obtain the model parameter importance matrix, whose size is consistent with the trainable parameters of the speech recognition model, both of which are 110M. In this embodiment, the importance threshold threhold is calculated based on the 39th percentile of the score distribution and is taken as 0.03. After binarization by the importance threshold, the model parameter importance matrix is ​​converted into a model parameter training mask, which contains 110M binary (1 or 0) masks.

[0075] Step 2-2: Binarize the importance matrix based on a preset importance threshold to generate a model parameter training mask: mark parameters with importance scores higher than the threshold as important parameters (mask value is 1), and mark the remaining parameters as adjustable parameters (mask value is 0); wherein the mask is used to lock key parameters that are sensitive to historical knowledge in the incremental training stage.

[0076] Step 2-1 specifically includes:

[0077] Step 2-1-1: For each sample in the initial training data, calculate the cross entropy loss between the model output and the true label;

[0078] Step 2-1-2: Calculate the gradient of the cross entropy loss with respect to each model parameter;

[0079] Step 2-1-3: For each model parameter, calculate the expectation of its squared gradient value on the initial training data as the parameter importance of the model parameter. For the i-th model parameter The importance of parameters Defined as:

[0080]

[0081] in, represents the initial training data set, represents the number of samples in the initial training data set, represents the set of all model parameters trained with the initial training data, is the input value of each sample, It's a model The output of x is is the partial differential, is a logarithmic function, Indicates that for a certain sample (including input x and label y), the i-th parameter Gradient in the cross entropy loss.

[0082] Step 2-1-1 specifically includes:

[0083] Step 2-1-1-1: Divide the initial training data into batches (mini-batches) of size M, where M ≥ 32. In this embodiment, M = 64 is selected based on the hardware conditions of the training environment. For step 2-1-1, the larger M is, the faster the calculation speed;

[0084] Step 2-1-1-2: Perform parallel computation for each batch: (a) model output for all samples in the batch; (b) batch-calculate the cross entropy loss for all samples in the batch based on the model output and the true labels;

[0085] Step 2-1-1-3: Repeat step 2-1-1-2 until all initial training data are traversed.

[0086] Step 2-2 is specifically implemented as follows:

[0087] Step 2-2-1: Setting the importance threshold threhold. The importance threshold threhold may be determined by:

[0088] (1) Calculate the Kth percentile of the score distribution of all parameters in the importance matrix as threshold, where K∈[30,39]; (2) Determine the balance between anti-forgetting effect and model plasticity through experiments on the validation set;

[0089] Step 2-2-2: Binarize the importance matrix element by element:

[0090] If the parameter importance score threshold, the output mask value is 1; otherwise, the output mask value is 0;

[0091] For the i-th model parameter, it corresponds to the i-th mask value for:

[0092]

[0093] Step 2-2-3: Store the binarization result as a structured binary matrix whose dimension is consistent with the model parameter tensor dimension, and the 1-value position corresponds to the model parameter to be locked.

[0094] Step 3 specifically includes:

[0095] Step 3-1: When receiving a user instruction or the system detects that a new proper noun needs to be added, original land-air call audio segments containing the new proper noun are collected to form an unlabeled segment set;

[0096] When speech recognition models are used for real-time air traffic management monitoring, the Aeronautical Information Publication (AIP) is constantly updated, which leads to the inclusion of new proper nouns (such as new waypoints, new airport names, new navigation stations, and new general aviation airline call signs). These new terms do not exist in the initial training data and cannot be recognized by the speech recognition model.

[0097] In specific implementation, because these newly added proper nouns are all nouns, the confidence of the transcribed text output when the model cannot recognize them is low and will obviously not comply with the ground-air communication standards. In this embodiment, the correct transcribed text should be "Dongfang 3983 direct flight DAMEG", and the DAMEG waypoint is renamed from P38. There is no data on the pronunciation of DAMEG during the initial training of the speech recognition model. Therefore, the transcribed text output by the speech recognition model is "Dongfang 3983 direct flight", and the model output confidence after "direct flight" is 0.3, which meets the conditions of the confidence level being lower than the threshold and not complying with the ground-air communication standards. It is determined that a new proper noun needs to be added.

[0098] On the other hand, when updating the AIP, staff at the speech recognition model application site can use their experience to determine whether incremental training is needed. In particular, incremental training can generally be considered necessary when adding new waypoints.

[0099] Step 3-2: Manually screen and transcribe the unlabeled segment set, select the audio segments containing the new proper nouns, and generate text labels corresponding to the audio segments;

[0100] Step 3-3: Divide the annotated segment set into a training dataset and a test dataset according to a preset ratio, with the training dataset accounting for 80% and the test dataset accounting for 20%.

[0101] For the DAMEG waypoint in this example, audio clips containing DAMEG were manually screened from routine air-ground traffic calls and annotated by professional air traffic controllers. In this example, a total of 200 DAMEG audio clips were annotated, all with text annotations. 160 of these clips were randomly assigned to form the training dataset, and 40 of these clips were used to form the test dataset.

[0102] The system in step 3-1 detects that a new proper noun needs to be added, specifically including:

[0103] The output results of the speech recognition model are monitored in real time. When a ground-to-air call segment is identified with a confidence level lower than a threshold and does not comply with the ground-to-air call specifications, it is determined that a new proper noun needs to be added.

[0104] In this embodiment, if Figure 3 As shown in the figure, the specific freezing method is to modify the parameter update process during model training, that is, only update the model parameters with a mask value of 0, while the model parameters with a mask value of 1 remain unchanged. The formula is as follows:

[0105]

[0106] in, refers to the i-th mask of the parameter training mask, represents the i-th model parameter, represents the updated i-th model parameter, Represents the gradient value of the i-th model parameter for the current training sample, which is calculated by the gradient descent method based on the cross entropy loss function.

[0107] Step 4 specifically includes:

[0108] Step 4-1: Load the pre-trained speech recognition model and the model parameter training mask generated in step 2; in this embodiment, it contains 110M binary data (1 or 0);

[0109] Step 4-2: training the mask according to the model parameters, freezing the model parameters corresponding to the mask value of 1, and retaining only the model parameters with the mask value of 0 as trainable, that is, only updating the trainable model parameters when using gradient to update the model parameters;

[0110] Step 4-3: Use the training data set collected in step 3 to perform incremental training with the cross entropy loss function. During the training process, the mini-batch stochastic gradient descent method is used, and the batch size is set to B, where B=16;

[0111] The batch size is calculated based on the hardware environment of speech recognition at the air traffic control deployment site. The larger B is, the faster the training speed is. In this embodiment, B is set to 16 based on the on-site deployed hardware.

[0112] Step 4-4: During the incremental training process, after each training batch E, evaluate the recognition accuracy of the current model on the test dataset collected in step 3, where E = 100;

[0113] Step 4-5: If the preset early stopping conditions are met, the early stopping mechanism is triggered and step 5 is executed.

[0114] In this example, the accuracy of the early stopping condition selection test set reaches 96%, which is achieved after 1000 batches of training. Because the training set has a total of 160 samples and each batch contains 16 samples, each sample is trained 16*1000 / 160=100 times.

[0115] The early stopping conditions in steps 4-5 include:

[0116] The absolute value of the improvement in accuracy for P consecutive tests is less than the threshold δ; the maximum number of training rounds is reached; and the accuracy on the test set reaches the target threshold.

[0117] In this embodiment, after stopping the speech recognition service, the speech recognition model is replaced with the newly trained speech recognition model, and then the speech recognition service is restarted. Testing has found that the newly trained speech recognition model can accurately transcribe the land-air call segment containing DAMEG.

[0118] Example: Incremental learning for a new general aviation airline call sign scenario

[0119] This embodiment takes the case of adding a flight of Jiangnan General Aviation Airlines at an air traffic control site as an example to illustrate the implementation process of the method of the present invention:

[0120] Step 1: Vocabulary slot reservation

[0121] The initial speech recognition model vocabulary contains 4,120 aviation terms (including standard waypoints, call signs, etc.).

[0122] Reserve slots at a rate of 10%: add 412 empty slots (fill <unk>placeholders), and construct a new vocabulary (4,532 words in total).

[0123] The number of neurons in the model's output layer has been simultaneously expanded to 4,532, ensuring that the model structure does not need to be reconstructed when new terms are added.

[0124] Step 2: Parameter importance matrix and mask generation

[0125] Speech recognition model architecture: 12-layer Transformer network, 110M parameters.

[0126] Using 1.3 million initial training data, the parameter importance is calculated in parallel with batch size M=64: the cross entropy loss is calculated for each batch of samples: , back propagation obtains all parameter gradients .

[0127] Calculate the parameter importance matrix:

[0128] Threshold setting: take the 39th percentile of the importance score distribution (threhold = 0.03).

[0129] Mask generation: perform binarization operation:

[0130] Generate a 110M-dimensional binary mask to mark the key parameters that need to be locked (accounting for 39%).

[0131] Step 3: New terminology data collection and annotation

[0132] When the system detected the speech recognition output "Yaodong VX transponders, three or two, listening to your command" (confidence level 0.1), the missing airline name did not conform to the air-to-ground call specification, confirming that the current speech recognition model could not accurately identify air-to-ground calls containing Jiangnan General Aviation. 200 raw air-to-ground call audio clips containing "Jiangnan" were collected and annotated by air traffic controllers (e.g., "Jiangnan Yaodong VX transponders, three or two, listening to your command"). The dataset was split into 160 training and 40 test audio clips in an 8:2 ratio.

[0133] Step 4: Mask-driven incremental training

[0134] Optimizer: Mini-batch SGD (batch size 16), loss function: cross entropy. Parameter update rule:

[0135] The accuracy of the test set was evaluated after every 100 training batches. Training was stopped when the test accuracy was ≥ 96% (achieved after 800 batches).

[0136] Step 5: Model Hot Update

[0137] The speech recognition service was suspended, and the model parameters were replaced before restarting. Verification showed that the new model's recognition accuracy for commands containing the airline "Jiangnan" increased to 96.5%. The recognition accuracy for historical terms (such as "Oriental") remained at 98.2%.

[0138] The present invention provides a speech recognition method based on incremental learning for land-air communication scenarios. There are many methods and approaches to implement this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.< / unk> < / unk>

Claims

1. A speech recognition method based on incremental learning for land-air communication scenarios, characterized by: The steps include: Step 1: Build a vocabulary for the speech recognition model and reserve slots in the vocabulary; Step 2: Calculate the model parameter importance matrix and generate the model parameter training mask; Step 3: Based on the new proper nouns that need to be added, collect and annotate a small number of new call segments as training data and test data; Step 4: Pre-train the speech recognition model and load the speech recognition model parameter training mask. Perform incremental training with cross-entropy loss. During the training process, regularly test the recognition accuracy on the test data. Step 5: Use the latest trained speech recognition model parameters to replace the original speech recognition model to complete the incremental learning of the model; Step 6: Use the speech recognition model that has completed incremental learning to recognize speech; The calculation of the model parameter importance matrix described in step 2 to generate the model parameter training mask specifically includes: Step 2-1: After the initial training of the speech recognition model is completed, the parameter importance of each model parameter is calculated using the initial training data to form a model parameter importance matrix; Step 2-2: Binarize the importance matrix based on a preset importance threshold to generate a model parameter training mask: mark parameters with importance scores higher than the threshold as important parameters, and mark the remaining parameters as adjustable parameters; Step 2-1 specifically includes: Step 2-1-1: For each sample in the initial training data, calculate the cross entropy loss between the model output and the true label; Step 2-1-2: Calculate the gradient of the cross entropy loss with respect to each model parameter; Step 2-1-3: For each model parameter, calculate the expectation of its squared gradient value on the initial training data as the parameter importance of the model parameter. For the i-th model parameter The importance of parameters Defined as: ； in, represents the initial training data set, represents the number of samples in the initial training data set, represents the set of all model parameters, Indicates the current parameter value of the model, is the input value of each sample, It's a model The output of x is is the partial differential, is a logarithmic function, Indicates that for a certain sample, the i-th parameter Gradient in cross entropy loss; Step 2-2 is specifically implemented as follows: Step 2-2-1: Setting the importance threshold threhold. The importance threshold threhold may be determined by: (1) Calculate the Kth percentile of the score distribution of all parameters in the importance matrix as threshold, where K∈[30,39]; (2) Determined through anti-forgetting effect and model plasticity balance experiments on the validation set; Step 2-2-2: Binarize the importance matrix element by element: If the parameter importance score threshold, the output mask value is 1; otherwise, the output mask value is 0; For the i-th model parameter, it corresponds to the i-th mask value for: ； Step 2-2-3: Store the binarization result as a structured binary matrix whose dimension is consistent with the model parameter tensor dimension, and the 1-value position corresponds to the model parameter to be locked; Step 4 specifically includes: Step 4-1: Load the pre-trained speech recognition model and the model parameter training mask generated in step 2; Step 4-2: training the mask according to the model parameters, freezing the model parameters corresponding to the mask value of 1, and retaining only the model parameters with the mask value of 0 as trainable, that is, only updating the trainable model parameters when using gradient to update the model parameters; Step 4-3: Use the training dataset collected in step 3 to perform incremental training with the cross entropy loss function. During the training process, the mini-batch stochastic gradient descent method is used, and the batch size is set to B; Step 4-4: During the incremental training process, after each training batch E, the recognition accuracy of the current model is evaluated on the test dataset collected in step 3; Step 4-5: If the preset early stopping conditions are met, the early stopping mechanism is triggered and step 5 is executed.

2. The speech recognition method based on incremental learning for land-air communication scenarios according to claim 1 is characterized in that: The reserved slots in the vocabulary described in step 1 include: The speech recognition model records the mapping relationship between the text in the vocabulary and the one-hot encoding of the model output; When constructing the vocabulary of the speech recognition model, N empty slots are reserved. When new proper nouns are encountered, they are added to the empty slots without changing the vocabulary size.

3. The speech recognition method based on incremental learning for land-air communication scenarios according to claim 1 is characterized in that: Step 2-1-1 specifically includes: Step 2-1-1-1: Divide the initial training data into batches of size M, where M ≥ 32; Step 2-1-1-2: Perform parallel computation for each batch: (a) model output for all samples in the batch; (b) batch-calculate the cross entropy loss for all samples in the batch based on the model output and the true labels; Step 2-1-1-3: Repeat step 2-1-1-2 until all initial training data are traversed.

4. The speech recognition method based on incremental learning for land-air communication scenarios according to claim 1 is characterized in that: Step 3 specifically includes: Step 3-1: When a request for adding a new proper noun is received, the original land-air call audio clip containing the new proper noun is collected to form an unlabeled clip set; Step 3-2: Manually screen and transcribe the unlabeled segment set, select the audio segments containing the new proper nouns, and generate text labels corresponding to the audio segments; Step 3-3: Divide the labeled segment set into a training dataset and a test dataset according to a preset ratio.

5. The speech recognition method based on incremental learning for land-air communication scenarios according to claim 4 is characterized in that: Step 3-1 The system detects that new proper nouns need to be added, specifically including: The output results of the speech recognition model are monitored in real time. When a ground-to-air call segment is identified with a confidence level lower than a threshold and does not comply with the ground-to-air call specifications, it is determined that a new proper noun needs to be added.

6. The speech recognition method based on incremental learning for land-air communication scenarios according to claim 1 is characterized in that: The early stopping conditions in steps 4-5 include: The absolute value of the improvement in accuracy for P consecutive tests is less than the threshold δ; the maximum number of training rounds is reached; and the accuracy on the test set reaches the target threshold.

Images