Progressive language code conversion method for cross-language migration

Abstract

The invention provides a progressive speech code conversion method for cross-language migration, which comprises the following steps: designing a difficulty measurer to measure the influence of each word replacement on a sentence, and then generating speech code conversion data with gradually increased difficulty based on a controllable temperature variable, finally, determining when to sample more difficult speech code conversion data for model training through a training scheduler; the method overcomes the defects that in the prior art, speech code conversion is prone to causing loss of original context information, insufficient representation cross-language alignment and limited data change, and the ability of model cross-language learning and knowledge migration is limited, and effective utilization of speech code conversion data and model generalization are achieved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computers, and more specifically, to a progressive code-switching method for cross-lingual transfer. BACKGROUND

[0002] Zero-shot cross-lingual transfer learning aims to train an adaptable model on a source language, which can effectively perform on a target language without target language labeled data. This research is particularly valuable in scenarios where the target language has limited or no annotations. In recent years, multilingual pre-trained language models such as mBERT, XLM and XLM-R have achieved remarkable performance improvement by fine-tuning on source language data and directly applying to target language data (e.g. Figure 1 (a) shown). In addition, the performance of zero-shot cross-lingual transfer can be further improved by replacing some tokens in the source text with target language words through the use of bilingual dictionaries, which is called code-switching (CS). The common practice (e.g. Figure 1 (b) shown) is to randomly select replacement words, and has been significantly improved in many zero-shot cross-lingual tasks.

[0003] On the one hand, as a data augmentation technique, code-switching inevitably leads to the loss of original context information when too many other language synonyms in the sentence are replaced with original words. On the other hand, code-switching sentences with too few replacements result in insufficient representation of cross-lingual alignment and limited data variation, which may limit the model's ability to learn and transfer knowledge across languages. Existing research shows that such uncontrolled samples may not necessarily benefit model learning. For example, an original English sentence is “All the services were beautiful.” Its corresponding multilingual code-switching sentence is “todas (ES) les (FR) services waren (DE) Great.”, where the capital letters in the brackets represent the target language abbreviation (e.g. ES for Spanish). Since the expression difference between the sentence before and after code-switching is large, such code-switching sentence may not help the alignment of multilingual word representation, and in some extreme cases, it may even harm the model's cross-lingual generalization ability. Therefore, the present invention should design a certain code-switching scheme to skillfully control the degree of conversion, for example, by the number of word replacements or the impact of code-switching sentences on model training. SUMMARY

[0004] The purpose of the embodiments of the present disclosure is to provide a progressive code switching method for cross-lingual transfer, a difficulty measurer is designed to measure the impact of each word replacement on the sentence, then based on the controllable temperature variable to generate code switching data with gradually increasing difficulty, and finally through the training scheduler to determine when to sample more difficult code switching data for model training.

[0005] In one general aspect, there is provided a progressive code switching method for cross-lingual transfer, comprising: after inputting a natural language, constructing a progressive code switching framework by introducing a difficulty evaluator, a code switcher, a training scheduler and a model trainer in the learning process of the code switching data by the guided model, achieving the ordered model training of the code switching data in the cross-lingual transfer task, and obtaining a multi-language model serving artificial intelligence; wherein:

[0006] The difficulty evaluator is based on a trained model, and uses layer-wise relevance propagation to assign a relevance score to each word as a basis for estimating code switching difficulty;

[0007] The difficulty evaluator is based on a trained model, and uses layer-wise relevance propagation to assign a relevance score to each word as a basis for estimating code switching difficulty;

[0008] The code switcher is based on introducing a code switcher containing a variable temperature parameter to score the relevance of sentences and words in the original source language; wherein

[0009] Given a trained model f, which has learned a prediction function that outputs a scalar value as a prediction result, f c (x) represents the predicted probability of the input sequence x belonging to class c in the classification task,

[0010] Using the BERT-based LRP algorithm, for linear layers, given the relevance of the output neuron rj, the relevance of the input neuron ri is calculated by the following equation, where ε is a very small positive number as a stabilizer to prevent division by zero errors;

[0011]

[0012] Starting from the output neuron whose relevance is set to the prediction function value, i.e. f c (x), LRP iteratively reallocates relevance from the last layer f c (x) to the input layer layer by layer, and follows the relevance conservation principle, and represents r(d)(x) as the relevance of the dth dimension of the word x, which is derived as follows:

[0013]

[0014] where z (l) is the column matrix of the hidden state in the l-th layer, the derivative of the nonlinear activation function a'(·) can be ignored here; for nonlinear layers, such as self-attention layers and residual layers, z (l) is approximated as the first term in the Taylor expansion, as follows

[0015]

[0016] The relevance score of the word r(x) with respect to the c-th class is derived using the absolute sum of r(d)(x), i.e.

[0017] r(x) = ∑ d r (d) (x).

[0018] Further, the training scheduler starts from a temperature (0), which is equivalent to sampling only source language data. Then, each time the validation loss converges, the temperature is linearly increased by an increment (e.g. 0.1) until the final temperature (1).

[0019] Further, as the temperature increases, the present application will generate more difficult code-switching data.

[0020] Further, the model trainer is a pre-trained multilingual model mBERT, which is used as an encoder to obtain the representation, and then the model predicts the task-specific probability distribution; specifically, at the k-th curriculum stage, the scheduler selects code-switching data D(i) CS for model training according to the following probability:

[0021]

[0022] The technical effects to be achieved by the embodiments of the present application are:

[0023] The present application proposes a progressive code-switching method to address the problem that uncontrollable code-switching has a negative impact on model training in cross-lingual transfer tasks. By introducing code-switching data with gradually increasing difficulty, the model is guided to gradually learn multilingual knowledge to improve the performance of cross-lingual transfer. In practical application scenarios, the present application overcomes the shortcomings of the prior art that code-switching easily leads to loss of original context information, insufficient representation of cross-lingual alignment, and limited data variation, which limits the model's ability to learn and transfer knowledge across languages. The present application realizes effective utilization of code-switching data and model generalization, and has strong application value. BRIEF DESCRIPTION OF DRAWINGS

[0024] The above and other objects and features of the present disclosure will become more apparent from the following description made with reference to the accompanying drawings.

[0025] Figure 1 is a schematic diagram illustrating a problem of transcoding across languages in the prior art according to the present disclosure;

[0026] Figure 2 is a transcoding framework according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following detailed description is provided to help the reader obtain a thorough understanding of the methods, devices, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be clear to those skilled in the art after understanding the present disclosure. For example, the order of the operations described herein is merely an example and is not limited to those set forth herein, but can be changed as will be clear to one of ordinary skill in the art after understanding the present disclosure, except for operations that must occur in a particular order. Also, descriptions of features known in the art can be omitted for the sake of clarity and conciseness.

[0028] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, these examples have been provided so that this disclosure will be thorough and complete, and will fully convey the scope of the methods, devices, and / or systems to be implemented.

[0029] As used herein, the term “and / or” includes any one of the associated listed items, as well as any combination of any two or more of the associated listed items.

[0030] Although terms such as “first,” “second,” and “third” can be used herein to describe various components, assemblies, regions, layers, or sections, these components, assemblies, regions, layers, or sections should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or section from another component, assembly, region, layer, or section. Thus, a component, assembly, region, layer, or section referred to as a first component, a first assembly, a first region, a first layer, or a first section in the examples described herein can also be referred to as a second component, a second assembly, a second region, a second layer, or a second section without departing from the teachings of the examples.

[0031] In the specification, when an element (such as a layer, a region, or a substrate) is referred to as being “on” another element, “connected to” or “coupled to” another element, it can be directly on, directly connected to, or directly coupled to the other element, or one or more other elements can be interposed therebetween. In contrast, when an element is referred to as being “directly on” another element, “directly connected to” or “directly coupled to” another element, there are no other elements interposed therebetween.

[0032] The terminology used herein is for the purpose of describing various examples only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises," "comprising," "includes," "including" and "has," "having" as used herein, are specifically intended to be construed as open-ended terms, i.e., to mean that the described features, numbers, operations, components, elements, and / or combinations thereof, are not limited to the listed items.

[0033] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs when read in light of the present disclosure. Unless explicitly defined otherwise, terms such as "comprise," "comprising," "including," "including," and "have" should be interpreted as specifying the presence of the stated features, numbers, operations, components, elements, and / or combinations thereof, but not precluding the presence or addition of one or more other features, numbers, operations, components, elements, and / or combinations thereof.

[0034] In addition, in the description of the examples, detailed descriptions of related structures or functions which are considered to cause obscuring of the present disclosure will be omitted.

[0035] Figure 1 is a schematic diagram showing a problem of progressive transcoding across languages in the prior art according to an embodiment of the present disclosure.

[0036] The application scenario of the present application scheme is that with the acceleration of globalization process, more and more artificial intelligence technologies need to provide services for users of different languages. For example, in the legal field, for a legal intelligent question and answer model, not only accurate understanding for English or Chinese is needed, but also services for languages such as Thai and French are needed. When users ask legal related questions in different languages, such as contract terms, legal responsibilities or regulation interpretation, the model must be able to understand and accurately answer these questions to meet the needs of users. Transcoding is an effective method to expand the training set of the model and improve the multilingual ability of the model, but uncontrollable transcoding will produce data that is not conducive to the model, causing negative transfer problem. By strategically generating appropriate transcoding data, the learning effect of the model can be guaranteed, the multilingual ability of the model can be improved, and the service audience range of artificial intelligence technology can be promoted. The transcoding data is orderly trained in the cross-language migration task to obtain a multilingual model serving artificial intelligence

[0037] In the present invention, the present invention assumes that simple code-switching samples can be used as pre-learned knowledge to guide the model to learn more difficult code-switching data. For example, the present invention first considers the simple code-switching sentence "All les(FR) services were Great.", which has a large amount of context overlap with the original sentence. In this case, the model can easily correctly align the word pair "the(EN)-les(FR)". Then, for the more difficult code-switching sentence "All les(FR) services waren(DE) Great.", the previously aligned "the(EN)-les(FR)" can be used as the pivot to align the new word pair "is(EN)-waren(DE)". By adopting this progressive strategy, the model can align new word pairs based on previously identified word pairs. Therefore, even sentences with a higher degree of code-switching, such as "todas(ES) les(FR) services waren(DE) great", will become easier to understand, thereby gradually improving the model's multilingual alignment capabilities.

[0038] To pursue the effective utilization of code-switching data and model generalization, the present invention draws on the idea of curriculum learning and proposes a progressive code-switching framework called PCS (Progressive Code-Switching) (as shown in Figure 1 (c). However, it is challenging to measure the difficulty of code-switching data, as the importance of each word is different for different tasks. Inspired by the interpretable learning method, the present invention develops a difficulty measurer to estimate the difficulty of code-switching sentences based on the contribution of the replacement word pair prediction results. Then, the present invention introduces a code-switching generator with an adjustable temperature parameter to generate appropriate code-switching sentences that match the current curriculum difficulty level. In addition, to alleviate the problem of catastrophic forgetting in curriculum learning, the present invention designs a scheduler that can dynamically adjust the difficulty level to review previously acquired knowledge. The progressive code-switching framework in the present invention is shown in Figure 2 ,

[0039] PCS framework

[0040] Interpretable difficulty evaluator

[0041] The idea of the present invention PCS is the strategy of "training model from easy-to-difficult code-switched data". Since the code-switched data is generated from the original data, there is no available pre-defined difficulty score label, so the present invention first needs to measure which code-switched data is more difficult than others. The existing common difficulty measures in natural language tasks include: sentence length, word rarity and substitution rate. However, these measures ignore the fact that the same word has different effects under different tasks. Therefore, these common difficulty measures cannot guarantee the "easy-to-difficult" order, because for the model, the code-switched sentence that replaces a few important words is more difficult than the code-switched sentence that replaces a large number of irrelevant words. Here, the present invention considers that the code-switched sentence with more important words replaced has greater semantic distortion to the original sentence and can be regarded as a more difficult training sample. Inspired by interpretable learning, the present invention uses layer-wise relevance propagation (LRP) to assign a relevance score to each word, indicating its contribution to a specific prediction, which serves as the basis for estimating the difficulty of CS. In other words, LRP can quantify the degree of influence of each word in the sentence on the prediction result of the model.

[0042] The present invention assumes that given a trained model, for example a BERT-based classification model, consisting of standard forward pass and specific backward pass. In practice, starting from the neuron that finally outputs the prediction function value, LRP iteratively redistributes the relevance from the last layer to the input layer using the above formula, and obeys the relevance conservation property. Through this backward pass, the present invention can observe which words really contribute to the output. As shown in Figure 2 "service" and "great" make significant contributions to the prediction of "positive".

[0043] Assume that given a trained model f, it has learned a prediction function that outputs a scalar value as the prediction result. For example f c (x) represents the prediction probability of input sequence x belonging to class c in the classification task.

[0044] The present invention adopts the LRP algorithm based on BERT, which consists of standard forward pass and specific backward pass. For linear layers, given the relevance of output neuron r j , the relevance of input neuron r i is calculated by the following equation, where ε is a very small positive number as a stabilizer to prevent division by zero error.

[0045]

[0046] Starting from the output neurons whose relevance is set to the predicted function value, i.e. f c (x), the LRP iteratively redistributes the relevance from the last layer f c (x) to the input layer layer by layer and follows the principle of relevance conservation. The present invention represents r (d) (x) as the relevance of the d-th dimension of word x, it can be derived as follows:

[0047]

[0048] where z (l) is the column matrix of hidden states in the l-th layer, and the derivative of the non-linear activation function a`(·) can be ignored here. For non-linear layers, such as self-attention layers and residual layers, z (l) is approximated as the first term in the Taylor expansion as follows:

[0049] The present invention uses the absolute sum of r (d) (x) to derive the relevance score of word r(x) with respect to the c-th class, i.e.

[0050] r(x) =∑ d r (d) (x). Through this backward pass, the present invention can observe which words have really contributed to the output.

[0051] Code-switcher

[0052] To generate code-switched sentences that match the current lesson difficulty level, the present invention introduces a code-switcher that contains a variable temperature parameter. This temperature parameter represents the proportion of words that are replaced and is linearly increased as the lesson progresses. Given the original source language sentence and the word relevance scores, the code-switcher selects words in ascending order of word relevance. Afterwards, the target language is randomly selected from the bilingual dictionary. It is worth noting that a source language word can have multiple translations in the target language. In this case, one of the multiple translations is randomly selected as the target word. Although this selection can not guarantee an accurate word-for-word translation in context, the present invention observes that this situation is not common in most bilingual dictionaries. Therefore, the randomness introduced by this process has little impact on the code-switching of the present invention.

[0053] Training scheduler

[0054] The purpose of the scheduler is to sample data and send it to the model trainer for training. The scheduler decides when to sample harder training data based on the training progress. For the PCS of the present invention, the present invention starts from a temperature (0), which is equivalent to sampling only source language data. Then, every time the validation loss converges, the temperature is increased linearly by an increment (e.g. 0.1) until the final temperature (1). As the temperature increases, the present invention will generate harder code-switching data. To encourage the model to pay more attention to harder data, the present invention sets a larger early stopping patience parameter for harder courses than for easier courses. However, the present invention finds that training the model on a series of CS datasets faces the problem of catastrophic forgetting. That is, as the course phase progresses, the code-switching training datasets with different levels of augmentation are sequentially input into the model. This leads to the modification of the parameter weights learned in the initial course once the model encounters the target of the new course, resulting in the occurrence of catastrophic forgetting. To alleviate this problem, the present invention designs a dynamic course scheduler for the model to review previous courses.

[0055] As the course phase progresses, the code-switching training datasets with different levels of augmentation are sequentially input into the model, which leads to the modification of the weight parameters learned in the initial course once the model encounters the target of the new course, resulting in the occurrence of catastrophic forgetting. To alleviate this problem, the present invention designs a dynamic course scheduler for the model to review previous courses. Specifically, at the kth course phase, the scheduler selects code-switching data D according to the following probability (i) CS Model training:

[0056]

[0057] Model trainer

[0058] The model trainer uses the training data given by the scheduler to train the downstream task-specific model step by step. This model has the same network architecture as the pre-trained model in the difficulty measurer. The present invention uses the traditional “pre-training-fine-tuning” paradigm. Specifically, the present invention uses a pre-trained multilingual model mBERT as an encoder to obtain representations, and then the model predicts a task-specific probability distribution.

[0059] Based on the above framework, the following experiments are conducted to evaluate the effect of the present invention:

[0060] To comprehensively evaluate the proposed method, the present application conducts experiments on three different types of cross-lingual transfer tasks using three widely used datasets. (1) For paraphrase recognition, the present application uses the PAWS-X dataset, which contains seven languages. The label has two possible values: 0 indicates that the pair of sentences has different meanings, while 1 indicates that the pair of sentences is a paraphrase relationship. The evaluation index is classification accuracy (ACC). (2) For document classification, the present application uses MLDoc as the experimental dataset of the present application, which includes seven different target languages. The evaluation index is classification accuracy (ACC). (3) For spoken language understanding, the present application uses the cross-lingual task-oriented dialogue dataset (XTOD), which includes English, Spanish, and Thai languages. This corpus includes 12 intent types and 11 slot types, and the model must detect the intent of the user's utterance and perform slot filling for each word in the utterance. The performance of intent detection is evaluated using classification accuracy (ACC), while slot filling is evaluated as a sequence labeling task using F1-score.

[0061]

[0062] Table 1: PAWS-X results.

[0063]

[0064] Table 2: MLDoc results.

[0065]

[0066] Table 3: XTOD results.

[0067] As shown in Tables 1, 2, and 3, PCS shows its superiority and versatility in different tasks under the zero-shot setting compared to strong code-switching baseline methods. In MLDoc and XTOD, the present application conducts experiments according to the reported XLM-R setting. In Table 1, the performance of PCS based on mBERT is superior to SCOPA by 0.5%, and the performance of PCS based on XLMR is superior to IECC by 0.5%. For Table 2, based on mBERT, the performance of PCS is superior to SCOPA by 1.1%, and superior to the XLM-R replicated by the present application by 0.7%. In Table 3, compared to randomly selecting (CoSDA) or individually selecting keywords (MLT) to construct code-switching sentences, the method of the present application shows superior performance.

[0068] While some embodiments of the present disclosure have been shown and described, it will be understood by those skilled in the art that modifications can be made to these embodiments without departing from the principles and spirit of the present disclosure, which are defined by the scope of the claims and their equivalents.

Claims

1. A gradual code-switching method for cross-language transfer, characterized in that: include: After natural language is input, in the process of guiding the model to learn code-conversion data, a progressive code-conversion framework is constructed by introducing a difficulty estimator, a code converter, a training scheduler, and a model trainer. This allows the code-conversion data to be sequentially trained on the model in cross-language transfer tasks, resulting in a multilingual model serving artificial intelligence. Among them: The difficulty estimator is based on a trained model and uses layer-by-layer relevance propagation to assign a relevance score to each word as a basis for estimating the difficulty of code switching; The code converter is to score the relevance of sentences and words in the original source language by introducing a code converter containing a variable temperature parameter; in Given a trained model f, a prediction function has been learned that outputs a scalar value as the prediction result, f c (x) represents the predicted probability that the input sequence x belongs to class c in the classification task, Using the BERT-based LRP algorithm, for the linear layer, given the correlation of the output neuron rj, the correlation of the input neuron ri is calculated by the following equation, where ε is a small positive number as a stabilizer to prevent division by zero errors; Starting with the output neuron whose relevance is set to the value of the prediction function, i.e. f c (x), LRP uses the above two formulas to iteratively transfer the correlation from the last layer f c (x) is redistributed to the input layer layer by layer, and following the principle of relevance conservation, r(d)(x) is expressed as the relevance of the d-th dimension of word x, which is derived as follows: where z (l) is the column matrix of the hidden state in the lth layer. The derivative of the nonlinear activation function a`(·) can be ignored here. For nonlinear layers, such as self-attention layers and residual layers, z (l) It is approximately the first term in Taylor's expansion, as follows The absolute sum of r(d)(x) is used to derive the relevance score of word r(x) with respect to category c, i.e. r(x)=∑ d r (d) (x)。 2. A gradual code-switching method for cross-language transfer according to claim 1, characterized in that: The training scheduler starts at temperature (0), which is equivalent to sampling only the source language data. Then, each time the validation loss converges, the temperature is linearly increased by an increment (e.g., 0.1) until the final temperature (1).

3. A gradual code-switching method for cross-language transfer according to claim 2, characterized in that: As the temperature increases, the present invention will generate more difficult transcoding data.

4. A gradual code-switching method for cross-language transfer according to claim 3, characterized in that: The model trainer is a pre-trained multilingual model mBERT, which is used as an encoder to obtain representations, and then the model predicts a task-specific probability distribution. Specifically, in the k-th course stage, the scheduler selects the code-switching data D(i)CS for model training according to the following probability:

Images