A Loss Weight Adaptive Meta-Learning Method for Low-Resource Speech Recognition

Abstract

A meta-learning method for adaptive loss weights in low-resource speech recognition relates to the field of speech recognition. This method addresses existing issues such as MAML algorithm instability and difficulty in precisely adjusting training loss weights. This method addresses these issues to a certain extent by fine-tuning weights using homoscedastic uncertainty. This method is then applied to speech recognition, and the VGG-CNN network and Adapter module are introduced to improve overall recognition performance. Compared to other meta-learning methods, this method exhibits less volatility and greater stability. It eliminates the need for manual or costly precision adjustment of loss weights and can automatically and efficiently adjust training loss weights to appropriate values. It can be used on any model, offering high flexibility, and the added VGG-CNN and Adapter modules add minimal computational cost.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition, and in particular to a loss weight adaptive meta-learning method in low-resource speech recognition. Background Art

[0002] With the rapid development of machine learning, end-to-end models have achieved excellent results in the field of speech recognition. However, this excellent performance often relies on a large amount of labeled speech data. However, in most scenarios, many languages ​​do not have abundant and correctly labeled data. The high annotation costs and scarce data further limit the effectiveness and generalization performance of the model. Although the application of data augmentation (such as speed perturbation, noise addition, and speech synthesis) and some special training methods (such as transfer learning, multilingual joint learning, and adversarial learning) can alleviate these problems to a certain extent, these methods still suffer from overfitting and weak results.

[0003] To address these issues, meta-learning has been introduced to speech recognition in low-resource settings. Meta-learning, also known as learning to learn, effectively absorbs prior knowledge from other tasks and efficiently applies this knowledge to learn new tasks. The model-agnostic meta-learning algorithm MAML (Model Agnostic Meta-Learning) is one of the most flexible and effective meta-learning algorithms, based on gradient descent and eliminating redundant parameters (Chelsea Finn, Pieter Abbeel, and Sergey Levine, "Model agnostic meta-learning for fast adaptation of deep networks," in International conference on machine learning, PMLR, 2017, pp. 1126–1135). It trains on different tasks to obtain the optimal initialization parameters for the model. During the fine-tuning phase, the model can learn new tasks with only minimal sample data.

[0004] Although the MAML algorithm performs better than other low-resource methods, due to the two-layer loss backpropagation update method used during training, the losses corresponding to different tasks during the training phase are unstable, which further leads to large fluctuations in the overall recognition performance of the model. In addition, different training tasks have different impacts on the target task to be recognized, and these impacts are controlled by the corresponding loss weights. Appropriate loss weights can further improve the performance of the target model, while conversely, they will reduce the performance of the model. However, finding the optimal weights by setting hyperparameters is very expensive and relies on complex manual adjustments.

[0005] Based on this, the method of using homoscedastic uncertainty to adjust the loss weight has been adopted in the MAML algorithm (A. Boiarov, K. Khabarlak, and I. Yastrebov, Multi-task meta learning modification with stochastic approximation, 2021). Homoscedastic uncertainty, which can capture the relative confidence between different tasks, can serve as a basis for weighing the loss weight in multi-task learning problems. Borrowing this idea, the MAML algorithm treats different languages ​​as different tasks and adjusts the weights between different tasks based on homoscedastic uncertainty to stabilize the loss, thereby improving the overall performance of the model. The improved MAML algorithm based on this has been applied to image recognition (Wenfeng Shen Lin Ding, Peng Liu and Shengbo Chen, Gradient-based meta-learning using uncertainty to weigh loss for few-shot learning, arXiv:2208.0813, 2022), which evaluates the corresponding weight in the form of a single hyperparameter. Although the hyperparameters are updated as the model is trained, the individual parameters cannot well learn the effective information in the task and cannot be further utilized.

[0006] How to more effectively use homoscedastic uncertainty in speech recognition to adjust the loss weights between different tasks in the MAML algorithm, and how to enable it to learn more effective feature generalization knowledge to assist model training and improve the overall model's recognition performance for the target language are issues that are worth studying.

[0007] Chinese Patent 113178190A uses meta-learning methods such as matching networks and twin networks for speech recognition. However, these methods are highly model-dependent and lack the flexibility of the MAML algorithm used in this invention. Furthermore, while Patent 112560904A applies the MAML algorithm to machine learning, it does not adjust the loss weights, and the instability issue of the MAML algorithm persists. Summary of the Invention

[0008] This paper aims to address existing technical issues such as MAML algorithm instability and difficulty in accurately adjusting training loss weights by providing a meta-learning method for adaptive loss weighting in low-resource speech recognition. This method addresses these issues to a certain extent by fine-tuning weights with homoscedastic uncertainty. This method is then applied to speech recognition, and the VGG-CNN network and adapter module are further introduced to improve overall recognition performance.

[0009] The present invention provides a loss weight adaptive meta-learning method for low-resource speech recognition, comprising the following steps:

[0010] 1) Classify the multiple languages ​​used in training according to their language and divide them into training sets and validation sets;

[0011] 2) Split the speech data of the training set and validation set of each language category into two parts: support set and query set;

[0012] 3) Collect statistics on the text of all speech data and create a training dictionary;

[0013] 4) Write code to build a speech recognition model consisting of a Transformer model, a VGG-CNN network, and an Adapter module;

[0014] 5) Carry out model pre-training stage;

[0015] 5.1) In each training iteration, for each language's training set speech data, k data points are first extracted from the training set support set. After extracting features through the VGG-CNN network, these features are fed into the Transformer model for training, and the model parameters for the current iteration are updated.

[0016] 5.2) We then extract k data points from the query set in the corresponding language. After extracting features through the VGG-CNN network, we feed these features into the Transformer model to test the model's recognition performance and obtain the corresponding query set loss. We also feed these features into the Adapter module, where we take the mean and logarithm of the values ​​to obtain the variance required to adjust the loss weight.

[0017] 5.3) Use the obtained variance to adjust the query set loss of the corresponding language in the current round through the formula;

[0018] 5.4) After completing the above training operations for a single language, initialize the model parameters to the values ​​at the beginning of the current training round;

[0019] 6) Repeat steps 5.1) to 5.4) for the training and validation sets of the other languages ​​to obtain the adjusted query set loss for all training languages ​​in the current iteration, and sum and average them.

[0020] 7) Using the mean loss of the query set after the operation in step 6), perform gradient descent update on all model parameters at the beginning of the current iteration of the model;

[0021] 8) After updating the model parameters, the current model parameters will be used as the model initialization parameters for the next iteration, and steps 5) to 7) will be repeated;

[0022] 9) After each training epoch, the currently trained model is tested using the speech data of the validation set, and the model parameters with the best performance after the test are saved. If the saved model parameters are the same optimal values ​​after multiple tests, the above training process ends;

[0023] 10) Prepare training sets, validation sets, and test sets for the languages ​​to be recognized;

[0024] 11) Using the training set of the language to be recognized, train the model saved in step 9);

[0025] 11.1) During each training iteration, a batch of speech data is extracted from the training set. After extracting features through the VGG-CNN network, these features are fed into the Transformer model to test the model's recognition performance and obtain the corresponding loss. Simultaneously, these features are fed into the Adapter module, where the mean and logarithm operations are performed to obtain the variance required to adjust the loss weight.

[0026] 11.2) Use the obtained variance to adjust the training loss of the desired language in the current round through the formula;

[0027] 11.3) Update the model parameters using the adjusted loss obtained in step 11.2);

[0028] 12) After each epoch of training, use the speech data of the target language verification set to test the currently trained model and save the model with the best performance;

[0029] 13) After repeatedly training for multiple epochs, the above training process for the target language is completed, and a speech recognition model with excellent recognition performance can be obtained.

[0030] Compared with the prior art, the present invention has the following technical effects:

[0031] This invention can effectively improve speech recognition performance for the target language in low-resource environments. It has also been verified that the algorithmic performance of this invention has less volatility and is more stable than other meta-learning methods. It eliminates the need for manual or costly precision adjustment of loss weights. This invention can automatically and efficiently adjust the training loss weights to appropriate values. Furthermore, this invention does not modify the model and can be used on any model, offering high flexibility. The added VGG-CNN and Adapter modules increase computational costs to an almost negligible level. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a structural diagram of the Adapter module.

[0033] Figure 2 Schematic diagram of the structure of the VGG-CNN module.

[0034] Figure 3 Schematic diagram of the training process of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the following embodiments will be further described in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] The model-independent meta-learning algorithm MAML can be applied to all models that rely on gradient descent learning due to its advantage of not introducing redundant parameters. The overall process of the MAML algorithm is similar to the pre-training-fine-tuning process. Based on the learning of pre-training data, it can more effectively learn the target data in the fine-tuning stage. However, unlike ordinary model training tasks, in addition to the source speech data D used for training, source And the target speech data D for subsequent fine-tuning tar get In addition, the MAML algorithm divides different speech data into different tasks according to their corresponding languages, and each task i is further divided into a support set for simulating model training during the training process. and the validation set used to evaluate the performance of this training

[0037] In the training phase, the MAML algorithm is divided into two phases: meta-training and meta-testing. In the meta-training phase, for task i, the algorithm will select the task i from its support set. Randomly select k data to perform gradient update on the current model, and use θ to represent the parameters of the model. The meta-training process is shown in Formula 1:

[0038]

[0039] Among them, θ′ i are the parameters of the updated model for training task i; is the corresponding cross entropy loss; α is the corresponding learning rate. Under the same model parameters, all tasks are trained synchronously. Then, in the meta-test phase, the corresponding The parameters after training with the last k data evaluations are θ′ i The model performance is calculated, and then the model parameters are recharged to the initial value θ of the current round. Then, the sum of the losses of all task validation sets is used to update the initialization parameters of the model in the current round, as shown in Formula 2:

[0040]

[0041] Where β is the corresponding learning rate. The model parameters obtained after updating through Formula 2 will be used as the model initialization parameters for the next round and the above update process will be repeated. The overall process can be summarized as the following Formula 3:

[0042]

[0043] From the above formula, we can find that in the updating process, the quadratic derivative of the model parameter θ is involved, which greatly increases the computational complexity of the algorithm. The subsequent first-order derivative approximation method is proposed, as shown in Formula 4, which takes the outer layer for θ′ i The derivation of is replaced by the derivation of θ, which greatly reduces the computational cost while having almost no effect on the effect. The MAML algorithm used in the present invention is a further extension of this basis.

[0044]

[0045] After repeated iterations of the above process, very good initialization parameters can be obtained. On this basis, even if the target data D target It is low-resource and the model can achieve very good performance after a small amount of gradient updates.

[0046] During the meta-test phase of the present invention, the MAML algorithm adds the evaluation losses of all tasks to update the model's initialization parameters. However, different tasks have different sensitivities to the same loss function, and simply adding the losses will further amplify the negative impact of this sensitivity. Furthermore, the MAML algorithm's unique dual-layer gradient update method and the small amount of data used in a single update have been shown, after extensive experiments, to lack good stability. Due to these two reasons, the algorithm's updates to the model's overall parameters occasionally produce large fluctuations, even impairing the model's performance during the fine-tuning phase.

[0047] This invention aims to achieve adaptive loss weights during the meta-test phase by leveraging variance uncertainty, automatically adjusting the corresponding weights to achieve optimal results. This adaptive adjustment process also enables the corresponding adaptive module to learn more useful generalized deep representations, which are then applied to the fine-tuning phase to assist in training the target data, resulting in even better model performance.

[0048] Homoscedastic uncertainty, also known as task-dependent uncertainty, is a quantity that remains constant for the same task but varies across different tasks. Homoscedastic uncertainty can capture the relative confidence between different tasks in multi-task learning, reflecting the uncertainty of each task. Homoscedastic uncertainty was first used in multi-task learning as a weighting for different task losses to enhance loss stability. Therefore, similar to multi-task learning, meta-learning can use it as a basis for evaluating the weighting of different task losses.

[0049] Speech recognition can generally be considered as a regression problem based on Gaussian distribution:

[0050]

[0051] in, is based on the model parameters θ and the model input The model output is L, which represents the length of the output vector, T represents the time domain dimension of the input audio data, and F represents the frequency domain dimension of the audio. is the true label of the corresponding input, σ 2 ∈ It is a variance scalar related to homoscedastic uncertainty. Based on the fact that this scalar can indirectly reflect the relative distribution between different tasks, this paper uses it as the basis for measuring the loss weights between different tasks. Based on maximum likelihood estimation and cross entropy loss, we can get:

[0052]

[0053] The fluctuation of MAML algorithm performance is mainly concentrated in the validation set loss in the meta-test phase. This paper uses homoscedastic uncertainty to adjust the loss weight in the meta-test phase. Therefore, based on the above formula and the formula of the meta-test phase of the MAML algorithm, we can further obtain the validation set loss corresponding to N tasks in the meta-test phase:

[0054]

[0055] Among them, θ′ i is the model parameter obtained after the meta-training phase for task i, as well as is the validation set loss of task i in the meta-test phase and the variance scalar used to fine-tune the loss weight. Model input It is composed of B input audio data x. is the corresponding true label.

[0056] Compared with the common MAML algorithm, the present invention Based on the original simple addition, the uncertainty σ based on homoscedasticity is i Fine-tune the model to enhance the differences between different tasks and the stability of the loss, so that the model can learn more generalized speech features during training and improve the overall recognition performance of the model.

[0057] On this basis, the present invention further updates with training and serves as a hyperparameter σ for fine-tuning parameters and regularization terms. i Further replaced by a shared Adapter module (Pfeiffer, J., I., Gurevych, I., & Ruder, S. (2020). MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer (arXiv: 2005.00052). arXiv) is calculated based on the validation set. The Adapter module structure is as follows Figure 1 As shown in the figure: the module first averages the input data, then downsamples it, restores it to its original dimension through upsampling after passing it through the ReLU activation function, and finally performs a residual link between the operated data and the initial input data to obtain the final output result.

[0058] The structural diagram of the VGG-CNN module is as follows Figure 2 As shown; take the logarithm of the obtained value to prevent it from being negative, use g(X) to represent the model in VGG-CNN, Adpater, and the process of taking the mean and logarithm, and further get σ i The calculation formula and weight adjustment formula are as follows:

[0059]

[0060]

[0061] Compared to hyperparameter tuning, this method leverages the CNN network and Adapter module to acquire more useful generalized knowledge during the training phase and learn more detailed deep representations. Furthermore, by treating the target language as a single task, the current method is further used to evaluate and adjust the loss during the fine-tuning phase, thereby improving recognition performance for the target language. The overall formula is as follows, where the Conv and Adapter in g(X) are both trained during the pre-training phase.

[0062]

[0063] Based on Equation 10, the present invention automatically and accurately adjusts the weight of the training loss during MAML training and effectively updates the model parameters based on the adjusted weights. This makes the MAML algorithm in this invention more stable and offers better performance than other meta-learning methods during training. Despite the introduction of the VGG-CNN network and Adapter module into the model, the computational cost is barely increased, and the speed is essentially the same as the original MAML algorithm.

[0064] The technical solution of the present invention and its alternatives:

[0065] 1. Basic MAML algorithm

[0066] The model-independent meta-learning algorithm MAML can be applied to all models that rely on gradient descent learning due to its advantage of not introducing redundant parameters. The overall process of the MAML algorithm is similar to the pre-training-fine-tuning process. Based on the learning of pre-training data, it can more effectively learn the target data in the fine-tuning stage. However, unlike ordinary model training tasks, in addition to the source speech data D used for training, source And the target speech data D for subsequent fine-tuning tar get In addition, the MAML algorithm divides different speech data into different tasks according to their corresponding languages, and each task i is further divided into a support set for simulating model training during the training process. and the validation set used to evaluate the performance of this training

[0067] In the training phase, the MAML algorithm is divided into two phases: meta-training and meta-testing. In the meta-training phase, for task i, the algorithm will select the task i from its support set. Randomly select k data to perform gradient update on the current model, and use θ to represent the parameters of the model. The meta-training process is shown in Formula 1:

[0068]

[0069] Among them, θ′ i are the parameters of the updated model for training task i; is the corresponding cross entropy loss; a is the corresponding learning rate. Under the same model parameters, all tasks are trained synchronously. Then, in the meta-test phase, the corresponding The parameters after training with the last k data evaluations are θ′ i The model performance is calculated, and then the model parameters are recharged to the initial value θ of the current round. Then, the sum of the losses of all task validation sets is used to update the initialization parameters of the model in the current round, as shown in Formula 2:

[0070]

[0071] Where β is the corresponding learning rate. The model parameters obtained after updating through Formula 2 will be used as the model initialization parameters for the next round and the above update process will be repeated. The overall process can be summarized as the following Formula 3:

[0072]

[0073] From the above formula, we can find that in the updating process, the quadratic derivative of the model parameter θ is involved, which greatly increases the computational complexity of the algorithm. The subsequent first-order derivative approximation method is proposed, as shown in Formula 4, which takes the outer layer for θ′ i The derivation of is replaced by the derivation of θ, which greatly reduces the computational cost while having almost no effect on the effect. The MAML algorithm of the present invention is further innovated on this basis.

[0074]

[0075] After repeated iterations of the above process, very good initialization parameters can be obtained. On this basis, even if the target data D target It is low-resource and the model can achieve very good performance after a small amount of gradient updates.

[0076] 2. The weight-adaptive MAML algorithm of the present invention

[0077] During the meta-test phase, the MAML algorithm sums the evaluation losses of all tasks to update the model's initialization parameters. However, different tasks have varying degrees of sensitivity to the same loss function, and simply summing the losses would amplify the negative impact of this sensitivity. Furthermore, extensive experiments have shown that the MAML algorithm's unique dual-layer gradient update method and the small amount of data used in a single update do not guarantee good stability. Due to these two factors, the algorithm's updates to the overall model parameters can occasionally experience significant fluctuations, even compromising model performance during the fine-tuning phase.

[0078] This invention aims to achieve adaptive loss weights during the meta-test phase by leveraging variance uncertainty, automatically adjusting the corresponding weights to achieve optimal results. This adaptive adjustment process also enables the corresponding adaptive module to learn more useful generalized deep representations, which are then applied to the fine-tuning phase to assist in training the target data, resulting in even better model performance.

[0079] Homoscedastic uncertainty, also known as task-dependent uncertainty, is a quantity that remains constant for the same task but varies across different tasks. Homoscedastic uncertainty can capture the relative confidence between different tasks in multi-task learning, reflecting the uncertainty of each task. Homoscedastic uncertainty was first used in multi-task learning as a weighting for different task losses to enhance loss stability. Therefore, similar to multi-task learning, meta-learning can use it as a basis for evaluating the weighting of different task losses.

[0080] Speech recognition can generally be considered as a regression problem based on Gaussian distribution:

[0081]

[0082] in, is based on the model parameters θ and the model input The model output is L, which represents the length of the output vector, T represents the time domain dimension of the input audio data, and F represents the frequency domain dimension of the audio. is the true label of the corresponding input, It is a variance scalar related to homoscedastic uncertainty. Based on the fact that this scalar can indirectly reflect the relative distribution between different tasks, this paper uses it as the basis for measuring the loss weights between different tasks. Based on maximum likelihood estimation and cross entropy loss, we can get:

[0083]

[0084] The fluctuation of MAML algorithm performance is mainly concentrated in the validation set loss in the meta-test phase. This paper mainly uses homoscedastic uncertainty to adjust the loss weight in the meta-test phase. Therefore, based on the above formula and the formula of the meta-test phase of the MAML algorithm, we can further obtain the validation set loss corresponding to N tasks in the meta-test phase:

[0085]

[0086] Among them, θ′ i is the model parameter obtained after the meta-training phase for task i, as well as is the validation set loss of task i in the meta-test phase and the variance scalar used to fine-tune the loss weight. Model input It is composed of B input audio data x. is the corresponding true label. Compared with the ordinary MAML algorithm, the Based on the original simple addition, the uncertainty σ based on homoscedasticity is iFine-tune the model to enhance the differences between different tasks and the stability of the loss, so that the model can learn more generalized speech features during training and improve the overall recognition performance of the model.

[0087] On this basis, the present invention further updates with training and serves as a hyperparameter σ for fine-tuning parameters and regularization terms. i Further replaced by a shared Adapter module (Pfeiffer, J., I., Gurevych, I., & Ruder, S. (2020). MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer (arXiv: 2005.00052). arXiv) is calculated based on the validation set. The module first averages the input data, then downsamples it, and restores its original dimension through upsampling after the ReLU activation function. Finally, the operated data is residually linked with the initial input data to obtain the final output result.

[0088] Take the logarithm of the obtained value to prevent it from being negative, and use g(X) to represent the process of the model in VGG-CNN, Adpater, and taking the mean and logarithm, and further get σ i The calculation formula and weight adjustment formula are as follows:

[0089]

[0090]

[0091] Compared to hyperparameter tuning, this method leverages the CNN network and Adapter module to acquire more useful generalized knowledge during the training phase and learn more detailed deep representations. Furthermore, by treating the target language as a single task, the current method is further used to evaluate and adjust the loss during the fine-tuning phase, thereby improving recognition performance for the target language. The overall formula is as follows, where the Conv and Adapter in g(X) are both trained during the pre-training phase.

[0092]

[0093] Based on formula 10, the present invention can automatically and accurately adjust the weight of the training loss during the training process of the MAML algorithm, and effectively update the parameters of the model based on the adjusted weight, so that the MAML algorithm in the present invention is more stable and has better performance during the training process than other meta-learning methods. Figure 3 .

Claims

1. A loss weight adaptive meta-learning method for low-resource speech recognition, characterized by The following steps are involved: 1) Classify the multiple languages ​​used in training according to their language and divide them into training sets and validation sets; 2) Split the speech data of the training set and validation set of each language category into two parts: support set and query set; 3) Collect statistics on the text of all speech data and create a training dictionary; 4) Write code to build a speech recognition model consisting of a Transformer model, a VGG-CNN network, and an Adapter module; 5) Carry out model pre-training stage; 5.1) In each training iteration, for each language's training set speech data, k data points are first extracted from the training set support set. After extracting features through the VGG-CNN network, these features are fed into the Transformer model for training, and the model parameters for the current iteration are updated. 5.2) We then extract k data points from the query set in the corresponding language. After extracting features through the VGG-CNN network, we feed these features into the Transformer model to test the model's recognition performance and obtain the corresponding query set loss. We also feed these features into the Adapter module, where we take the mean and logarithm of the values ​​to obtain the variance required to adjust the loss weight. 5.3) Use the obtained variance to adjust the query set loss of the corresponding language in the current round through the formula; 5.4) After completing the above training operations for a single language, initialize the model parameters to the values ​​at the beginning of the current training round; 6) Repeat steps 5.1) to 5.4) for the training and validation sets of the other languages ​​to obtain the adjusted query set loss for all training languages ​​in the current iteration, and sum and average them. 7) Using the mean loss of the query set after the operation in step 6), perform gradient descent update on all model parameters at the beginning of the current iteration of the model; 8) After updating the model parameters, the current model parameters will be used as the model initialization parameters for the next iteration, and steps 5) to 7) will be repeated; 9) After each training epoch, the currently trained model is tested using the speech data of the validation set, and the model parameters with the best performance after the test are saved. If the saved model parameters are the same optimal values ​​after multiple tests, the above training process ends; 10) Prepare training sets, validation sets, and test sets for the languages ​​to be recognized; 11) Using the training set of the language to be recognized, train the model saved in step 9); 11.1) During each training iteration, a batch of speech data is extracted from the training set. After extracting features through the VGG-CNN network, these features are fed into the Transformer model to test the model's recognition performance and obtain the corresponding loss. Simultaneously, these features are fed into the Adapter module, where the mean and logarithm operations are performed to obtain the variance required to adjust the loss weight. 11.2) Use the obtained variance to adjust the training loss of the desired language in the current round through the formula; 11.3) Update the model parameters using the adjusted loss obtained in step 11.2); 12) After each epoch of training, use the speech data of the target language verification set to test the currently trained model and save the model with the best performance; 13) After repeated training for multiple epochs, the above training process for the target language is completed, and a speech recognition model with excellent recognition performance is obtained.

Images