Task-based polynomial loss boosting multilingual meta-learning speech recognition method

Abstract

This invention provides a task-based multinomial loss-enhanced multilingual meta-learning speech recognition method. The method includes: Step 1: Initializing the speech recognition model by inputting the original speech feature sequence; Step 2: Extracting task T from the multilingual dataset. i , will T i Divided into support set and query set; Step 3: Calculate T i The ASR loss is obtained by using gradient descent to get the updated parameters θ on the support set. i Step 4: Use the updated parameters θ on the support set. i Step 5: Calculate the query loss on the query set based on T. i The calculation yields T i Step 6: Repeat steps 2 to 5 N times to calculate the polynomial loss for each of the N tasks; Step 7: Update the speech recognition model parameters θ based on the polynomial losses for all tasks; Step 8: Repeat steps 2 to 7 until the updated speech recognition model f is obtained. θ Meets the given requirements.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a task-based multinomial loss-enhanced multilingual meta-learning speech recognition method. Background Technology

[0002] In the pre-training process of multilingual speech recognition, an imbalance in difficulty and data volume often exists between different languages. Furthermore, in multilingual learning, the overall loss function is affected by the losses from all single-language tasks. Designing a better loss function for each task that effectively addresses the imbalance problem without being excessively influenced by noise signals is a pressing issue.

[0003] On the one hand, a good loss function should take a more flexible form, tailored to different tasks and datasets. Therefore, we aim to design a loss function that can flexibly adjust according to changes in the training environment. On the other hand, due to the differences in learning paradigms between meta-learning and machine learning, a loss function applicable to machine learning may not be suitable for meta-learning. Therefore, we aim to research a loss function suitable for the meta-learning paradigm. Summary of the Invention

[0004] To address the imbalance between language difficulty and data volume in multilingual speech recognition, this invention provides a task-based multinomial loss-enhanced multilingual meta-learning speech recognition method.

[0005] This invention provides a task-based multinomial loss-enhanced multilingual meta-learning speech recognition method. The method employs an end-to-end speech recognition network architecture, comprising:

[0006] Step 1: Initialize the speech recognition model f θ Input the original speech feature sequence (x1, x2, ..., x...) T );

[0007] Step 2: From the multilingual dataset Extraction task T i The task T i Divided into support sets and query set Represents data in the i-th language;

[0008] Step 3: Calculate task T i The ASR loss is obtained using gradient descent on the support set. The updated parameter θ i ;

[0009] Step 4: Use in the support set The updated parameter θ iIn query set Calculate query loss

[0010] Step 5: Based on task T i query loss Calculate task T i The polynomial loss;

[0011] Step 6: Repeat steps 2 to 5 N times to calculate the polynomial loss for each of the N tasks;

[0012] Step 7: Update the speech recognition model f based on the multinomial loss corresponding to all N tasks. θ The meta-parameter θ;

[0013] Step 8: Repeat steps 2 through 7 until the updated speech recognition model f is achieved. θ Meets the given requirements.

[0014] Furthermore, in step 5, the polynomial loss L TPL The calculation formula is:

[0015]

[0016] Where l represents the query loss j represents the order of the polynomial, α j Let represent the coefficients of the j-th order polynomial.

[0017] Furthermore, the coefficients of the nth-order polynomial of the task's polynomial loss are adjusted to adapt to different learning environments, i.e., the polynomial loss L... TPL Adjusted to L TPL-n ;

[0018] L TPL-n =l+ε(1-e -l ) n (15)

[0019] Where ε represents the adjustment factor, n≥1, and n is an integer.

[0020] Furthermore, in step 7, the update formula for the meta-parameter θ is:

[0021]

[0022] Where β represents the learning rate, Indicates task T i The polynomial loss.

[0023] Furthermore, in step 7, the update formula for the meta-parameter θ is:

[0024]

[0025] Where β represents the learning rate, Indicates task T i The polynomial loss.

[0026] Furthermore, the end-to-end speech recognition network architecture specifically adopts a CTC-attention joint architecture;

[0027] Correspondingly, in step 3, task T i The formula for calculating ASR loss is:

[0028] L=λL ctc +(1-λ)L att

[0029] Among them, L ctc For CTC loss, L att For decoding loss, the hyperparameter λ represents L ctc The weight.

[0030] The beneficial effects of this invention are:

[0031] This invention transforms the probabilities in the polynomial basis of the polynomial loss function into a form where the loss function is the polynomial basis. Since task loss measures the learning state of a task, a task-based polynomial loss is proposed. This loss function can be expressed as a weighted sum of polynomial bases with the loss function as the function. The coefficients of the polynomial basis can be adjusted for different learning scenarios, allowing the loss function to better adapt to the meta-learning paradigm. Furthermore, the task-based polynomial loss is also compatible with task-based polynomial focus loss, effectively focusing on difficult tasks and adapting to meta-learning by adjusting the coefficients of the polynomial basis. The setting of this meta-loss function allows for flexible adjustment of the focus on difficult and easy tasks, reducing noise interference while focusing on difficult tasks, effectively addressing the imbalance problem in multilingual learning. Attached Figure Description

[0032] Figure 1 A flowchart illustrating a task-based multinomial loss-enhanced multilingual meta-learning speech recognition method provided in an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of the multilingual learning model structure based on the joint CTC-Attention architecture provided in an embodiment of the present invention;

[0034] Figure 3 Gradient analysis diagram of polynomial loss provided in an embodiment of the present invention;

[0035] Figure 4This diagram illustrates the comparison between sample-based loss and task-based loss pre-training under different methods provided in the embodiments of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0037] On the one hand, a good loss function should take a more flexible form to be tailored to different linguistic tasks. Therefore, we aim to design a loss function that can flexibly adjust according to changes in the training environment. On the other hand, due to the differences in learning paradigms between meta-learning and machine learning, a loss function suitable for machine learning may not be suitable for meta-learning. Therefore, we aim to design a loss function suitable for the meta-learning paradigm. We propose a loss function that uses task loss as a metric, effectively measuring the learning state of the task and adapting to the meta-learning paradigm.

[0038] Based on the above two considerations, this invention proposes a task-based multinomial loss, using task loss as a metric to effectively measure the learning state of the task, adapting to the meta-learning paradigm, and applying the value to the multilingual meta-learning speech recognition process, such as... Figure 1 As shown, the speech recognition method specifically includes the following steps:

[0039] S101: Initialize the speech recognition model f θ Input the original speech feature sequence (x1, x2, ..., x...) T );

[0040] S102: Multilingual meta-learning treats each language as a task, drawing from multilingual datasets. Extraction task T i The task T i Divided into support sets and query set This represents data in the i-th language.

[0041] S103: Computation Task T i The ASR loss is obtained using gradient descent on the support set. The updated parameter θ i ;

[0042] Specifically, this embodiment adopts the following... Figure 2The CTC-attention joint architecture shown is an end-to-end speech recognition network architecture. This architecture consists of three parts: the encoder and decoder in the Transformer model, and the Connectionist Temporal Classification (CTC) module. The CTC module guides the model to learn good alignment and improves convergence speed. Multilingual speech recognition based on the CTC-attention joint architecture is used for task T. i The formula for calculating ASR loss is:

[0043] L=λL ctc +(1-λ)L att (1)

[0044] Among them, L ctc For CTC loss, L att For decoding loss, the hyperparameter λ represents L ctc The weight.

[0045] In the inner loop, the base learner is derived from the initial meta-learner f. θ Learn the model for each task. For task T... i In other words, utilizing task T i Support set Through the Gradient descent calculation to update model parameters θ i :

[0046]

[0047] in, This is the loss function of the base learner, α is the learning rate of the inner loop, and f is the learning rate of the inner loop. θ This represents a multilingual model, where θ represents the model parameters.

[0048] S104: Used in support sets The updated parameter θ i In query set Calculate query loss

[0049] S105: According to task T i query loss Calculate task T i Polynomial loss or polynomial focus loss;

[0050] Specifically, in the outer loop of the multilingual meta-learning method, the meta-learner integrates the knowledge of each base learner in the inner loop. The updated model parameters θ i By calculating the query set The model parameters are updated by querying the loss of all tasks on the dataset. Specifically, the update method for model parameters θ is as follows:

[0051]

[0052] Where α and β are the learning rates of the inner and outer loops, respectively; from equations (2) and (3), it can be seen that the meta-learning optimization process requires calculating the second derivative of θ, which is computationally expensive. Therefore, it is recommended to use the first-order MAML algorithm (FOMAML) for model parameter updates, and correspondingly, equation (3) is adjusted to equation (4):

[0053]

[0054] Let p(T) denote the distribution of the task, and the meta-learning process can be expressed as formula (5):

[0055]

[0056] In this embodiment, query loss Using cross-entropy loss L ce ,make Combining the Taylor expansion formula, the cross-entropy loss L ce (For convenience, "l" will be used to represent L in the following formulas.) ce It can also be expressed as a weighted sum with the loss function as the polynomial basis, as shown in formula (6):

[0057]

[0058] In this embodiment of the invention, the meta-learning object is the task, and the task loss can measure the task learning state. Based on equation (6), the inventors adjusted the coefficients of the polynomial basis in equation (6) to α. j This leads to the task-based polynomial loss (TPL), which is expressed as formula (7):

[0059]

[0060] Using formula (7) as the loss function to measure task characteristics can effectively adapt to the meta-learning learning paradigm.

[0061] Furthermore, task-based multinomial loss can be combined with focus loss. The focus loss function can be expressed as formula (8):

[0062]

[0063] Therefore, the task-based focus loss can be expressed as formula (9):

[0064]

[0065] Compared to Equation (6), Equation (9) is equivalent to horizontally shifting all polynomial bases using the modulation factor γ. Similarly, using the same weighting coefficients does not adapt well to different learning scenarios. Therefore, task-based polynomial loss can also integrate focus loss, thus yielding task-based polynomial focus loss:

[0066]

[0067] In formulas (7) and (10), the polynomial coefficients α can be adjusted according to different learning scenarios. j However, since it is impossible to adjust an infinite number of α values... j Preferably, the coefficients of the lower-order polynomials are adjusted to adapt to different learning scenarios.

[0068] As one possible implementation, the coefficients of the first polynomial can be adjusted to suit different learning scenarios. Correspondingly, formulas (7) and (10) are adjusted to formulas (11) and (13), respectively:

[0069]

[0070] Combining formula (6), we can know that L TPL-1 With cross-entropy loss L ce The relationship between them is:

[0071] L TPL-1 =l+ε(1-e -l (12)

[0072] Similarly, the task-based polynomial focus loss can be expressed as Equation (13):

[0073]

[0074] Combining formula (9), formula (13) can be further expressed as:

[0075] L TPFL-1 =L TFL +ε(1-e -l ) 1+γ (14)

[0076] Where ε represents the adjustment factor.

[0077] Based on the above, more generally, the coefficients of the nth-order polynomial can be adjusted to adapt to different learning scenarios. Correspondingly, formula (12) is adjusted to formula (15):

[0078] L TPL-n =l+ε(1-e -l ) n (15)

[0079] Where ε represents the adjustment factor, n≥1, and n is an integer.

[0080] S106: Repeat steps S102 to S105 N times to calculate the polynomial loss or polynomial focus loss corresponding to N tasks.

[0081] S107: Update the speech recognition model f based on the polynomial loss or polynomial focus loss corresponding to all N tasks. θ The meta-parameter θ;

[0082] As one possible implementation, when polynomial loss is applied to the MAML algorithm to update the meta-parameter θ, the update formula for the meta-parameter θ is:

[0083]

[0084] Where β represents the learning rate of the outer loop. Represents task T i The polynomial loss (understandably, when using formula (7) to calculate the polynomial loss, It refers to task T. i L TPL When formula (15) is used to calculate polynomial loss, It refers to task T. i L TPL-n );

[0085] Similarly, when the polynomial focus loss is applied to the MAML algorithm to update the meta-parameter θ, the update formula for the meta-parameter θ is:

[0086]

[0087] Where β represents the learning rate of the outer loop. Represents task T i Polynomial focus loss;

[0088] In practical applications, to reduce computational load, as another feasible implementation, it is preferable to apply polynomial loss or polynomial focus loss to the FOMAML algorithm to update the meta-parameter θ. In this case, the update formula for the meta-parameter θ is:

[0089] When the loss is polynomial... or

[0090] When the loss is polynomial focus Where β represents the learning rate of the outer loop. Represents task T i The polynomial loss (understandably, when using formula (7) to calculate the polynomial loss, It refers to task T. i L TPL When formula (15) is used to calculate polynomial loss, It refers to task T. i L TPL-n ); Represents task T i The polynomial focus loss.

[0091] S108: Repeat steps S102 to S107 until the updated speech recognition model f is obtained. θ Meets the given requirements.

[0092] This invention transforms the probabilities in the polynomial basis of the polynomial loss function into a form where the loss function is the polynomial basis. Since task loss measures the learning state of a task, a task-based polynomial loss is proposed. This loss function can be expressed as a weighted sum of polynomial bases with the loss function as the function. The coefficients of the polynomial basis can be adjusted for different learning scenarios, allowing the loss function to better adapt to the meta-learning paradigm. Furthermore, the task-based polynomial loss is also compatible with task-based polynomial focus loss, effectively addressing difficult tasks and adapting to meta-learning by adjusting the coefficients of the polynomial basis.

[0093] In this embodiment, the above content provides two adjustment methods (i.e., adjusting the coefficients of the first polynomial and adjusting the coefficients of the nth polynomial) to make the polynomial loss adapt to different learning scenarios. The following is a brief theoretical support for the reason why it can adapt to different learning scenarios to demonstrate the effectiveness of the method of the present invention.

[0094] (1) Analyze the adjustment of the first polynomial coefficients. Calculate the gradient of the polynomial loss relative to the query loss, as given by formula (16):

[0095]

[0096] From formula (16), we can see that when ε>0, the gradient value g PF The gradient value g is inversely proportional to the query loss l, indicating that the model focuses on tasks with smaller losses. When ε < 0, the gradient value g... PF The loss is proportional to the query loss *l*, and in this case, the model will focus on tasks with higher losses. However, when *ε* is too small (*ε* < - *e*), the model will focus on tasks with higher losses. l At this point, a negative gradient will be generated, therefore it needs to satisfy -e l <ε<0.

[0097] (2) Analyze the coefficients of the second-order and higher polynomials. When adjusting the basis of the j-th order polynomial, calculate the gradient of the polynomial loss relative to the query loss, as given by formula (17):

[0098]

[0099] Let \(1 + \varepsilon j(1 - e\) -l ) j-1 e -l be a function of the variable "l", as shown in Equation (18):

[0100] f(l) = 1 + \varepsilon j(1 - e -l ) j-1 e -l (18)

[0101] The pole of the function \(l_0=\ln j\) and the maximum value can be obtained

[0102] When \(l < l_0\), the gradient value \(g\) PF is proportional to the query loss \(l\). When \(l > l_0\), the gradient value \(g\) PF is inversely proportional to the query loss \(l\); and when \(j = \infty\), \(l_0=\infty\).

[0103] More generally, we can adaptively adjust \(j\) to adjust the attention to the task. As Figure 3 shown, when the loss value is less than \(l_0\), the attention increases, and when it is greater than \(l_0\), the attention decreases. This also implies that tasks near \(l_0\) are given higher attention.

[0104] To further verify the effectiveness of the method of the present invention, the present invention also gives the following experimental data.

[0105] (1) Dataset

[0106] The OpenSLR dataset is adopted. For OpenSLR, 9 languages are selected as the source languages for pre-training, and 5 target languages are fine-tuned. 80% of the data is used for training, and 20% of the data is used to test each language.

[0107] (2) Experimental conditions

[0108] Feature extraction was performed using the Kaldi toolkit to obtain 40-dimensional Mel-frequency cepstral coefficients (MFCC) features and 3-dimensional pitch features computed every 10 milliseconds within a 25-millisecond window. The model employs a 6-layer VGG convolutional network, with the transformer consisting of 4 encoder blocks and 2 decoder blocks. Each block comprises 512 hidden units, 4 attention heads, and 2048 feedforward hidden units. For each task, it contains 64 examples, with 32 examples assigned to the support set and 32 to the query set. The weight of the CTC loss λ was set to 0.3. During inference, beam search was used to obtain the optimal sequence. When using BPE, the vocabulary size was set to 9000. SGD was used except for the inner loop in meta-learning. Adam was used for the remaining model optimizations. For OpenSLR, the warm-up steps were set to 1000, and k was 0.5. In the experiments, the character error rate (CER) was used as the criterion, and five best models were retained. Finally, the five best model parameters are averaged to obtain the final model. During training, spectral augmentation and velocity perturbation are used.

[0109] (3) Algorithm Flow

[0110] The model structure used is as follows Figure 1 As shown, based on this model structure, the training method adopts steps S101 to S108 as described above:

[0111] In addition, step S109 is included: evaluating the accuracy of the model on the validation set and retaining the models corresponding to the best 5 validation set accuracies for testing.

[0112] (4) Experimental results:

[0113] The performance of random initialization, multilingual transfer learning ASR (MTL-ASR), multilingual meta-learning ASR (MML-ASR), and the TPL-ASR and TPFL-ASR of the present invention for fine-tuning the target language on OpenSLR was compared, as shown in Table 1.

[0114] Table 1. Fine-tuning performance of each method in different languages

[0115] language Telugu Nigerian Venezuelan Burmese Yoruba No pre-training 48.61 73.08 52.23 57.92 55.45 MTL-ASR 80.47 84.12 82.42 81.70 96.12 MML-ASR 30.86 54.00 30.98 37.56 43.60 The TPL-ASR of the present invention 30.94 53.02 28.79 37.73 42.32 The TPFL-ASR of the present invention 28.94 50.90 27.07 36.48 41.56

[0116] As can be seen from Table 1, the TPL-ASR of the present invention is generally better than the MML-ASR, while the TPFL-ASR of the present invention is the best.

[0117] Furthermore, the pre-training performance of task-based loss and sample-based loss was compared, such as... Figure 4As shown, task-based loss significantly outperforms sample-based loss in the meta-learning paradigm, demonstrating the effectiveness of the proposed method.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A task-based multinomial loss-enhanced multilingual meta-learning speech recognition method, characterized in that, The method employs an end-to-end speech recognition network architecture, specifically including: Step 1: Initialize the speech recognition model Input the original speech feature sequence ; Step 2: From the multilingual dataset Extraction task The task Divided into support sets and query set ; Represents data in the i-th language; Step 3: Calculate the task The ASR loss is obtained using gradient descent on the support set. Updated parameters ; Step 4: Use in the support set Updated parameters In query set Calculate query loss ; Step 5: According to the task query loss Calculate the task Polynomial loss; polynomial loss The calculation formula is: （7） Where l represents the query loss j represents the order of the polynomial. represents the coefficients of the j-th order polynomial; By adjusting the coefficients of the nth-order polynomial of the polynomial loss for the task to adapt it to different learning environments, the polynomial loss is... Adjusted to ; （15） in, This represents the adjustment factor, where n ≥ 1 and n is an integer; The method of adjusting the coefficients of the nth-order polynomial of the task's polynomial loss to adapt it to different learning environments includes: When n=1, the process of adjusting the coefficients of the first polynomial to adapt to different learning environments is as follows: When, gradient value The loss is inversely proportional to the query loss l, causing the model to focus on tasks with lower losses; when When, gradient value The loss is proportional to the query loss l, causing the model to focus on tasks with higher losses; When n≥2, the process of adjusting the coefficients of the second-order and higher polynomials to adapt to different learning environments is as follows: When adjusting the basis of the j-th order polynomial, the gradient of the polynomial loss relative to the query loss is calculated, and there exists formula (17): （17） Will As a function of the variable "l", as shown in formula (18): （18） Find the poles of the function and maximum value ; when When, gradient value It is proportional to the query loss l, when When, gradient value It is inversely proportional to the query loss l; and when Sometimes Thus, by adaptively adjusting To adjust the focus on the task: when the loss value is less than Attention increased, greater than The decrease in attention at that time led to Nearby tasks are given higher attention; Step 6: Repeat steps 2 to 5 N times to calculate the polynomial loss for each of the N tasks; Step 7: Update the speech recognition model based on the multinomial loss corresponding to all N tasks. meta-parameters ; Step 8: Repeat steps 2 through 7 until the updated speech recognition model is achieved. Meets the given requirements.

2. The task-based multinomial loss-enhanced multilingual meta-learning speech recognition method according to claim 1, characterized in that, In step 7, meta-parameters The update formula is: in, Indicates the learning rate. Indicates task The polynomial loss.

3. The task-based multinomial loss-enhanced multilingual meta-learning speech recognition method according to claim 1, characterized in that, In step 7, meta-parameters The update formula is: in, Indicates the learning rate. Indicates task The polynomial loss.

4. The task-based multinomial loss-enhanced multilingual meta-learning speech recognition method according to claim 1, wherein the end-to-end speech recognition network architecture specifically adopts a CTC-attention joint architecture; Correspondingly, in step 3, the task The formula for calculating ASR loss is: in, For CTC losses, For decoding loss, hyperparameters express The weight.

Images