A zero-shot multilingual understanding algorithm based on unsupervised pseudo semantic data enhancement

Abstract

The application discloses a zero-shot multilingual understanding algorithm based on unsupervised pseudo semantic data enhancement, and utilizes an unsupervised pseudo semantic data enhancement mechanism to enhance training data without manual intervention. The application designs a sequence clusterer from single language to multilingual to obtain multilingual similar semantic tokens. In addition, the application utilizes obtained multilingual clustering result representation to replace the representation of key components in a sentence, so as to achieve the purpose of injecting multilingual knowledge into training corpus. The application obtains good effects on three types of zero-shot natural language understanding tasks (sequence classification, information extraction and question and answer), and proves the effectiveness of the algorithm.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and relates to a data augmentation multilingual algorithm, specifically a zero-shot multilingual understanding algorithm based on unsupervised pseudo-semantic data augmentation. Background Technology

[0002] Multilingual representation learning leverages abundant linguistic information to improve the accuracy of resource-poor language representations in natural language understanding tasks, and is experimentally validated in downstream tasks such as text classification, sentiment analysis, information extraction, and question answering. Existing multilingual work relies on supervised parallel corpora and shallow unsupervised data augmentation mechanisms to explicitly share semantic information among multiple languages. These data augmentation methods can be divided into two main categories: parallel corpus data augmentation and unsupervised shallow data augmentation.

[0003] a. Parallel corpus data augmentation works use bilingual dictionaries and translation tools to align the semantics between different languages, but these corpora are time-consuming and rely on manual annotation.

[0004] b. Unsupervised shallow data augmentation work utilizes unsupervised EDA operations (back-translation, random deletion, random replacement, etc.) to introduce a large amount of contextual semantic information into the training data.

[0005] However, these methods only focus on surface-level string information to align data across multiple languages, without considering the deeper, contextual representations of sentences. For example... Figure 1 As shown.

[0006] Related existing technologies:

[0007] a. Multilingual pre-trained model

[0008] Existing multilingual pre-trained models fall into two main categories:

[0009] (1) Monolingual training data: Pre-trained models such as m-BERT and XLM-R use monolingual text to train the model through the masked language model task, and obtain the final representation of multilingual input through the knowledge transfer ability from high-resource semantics to low-resource language.

[0010] (2) Multilingual training data: Some works extend XLM-R by using alignment techniques to map and align the representations of different languages ​​before training. Other works use alignment tools to operate on bilingual corpora to encourage the model to learn more linguistic knowledge.

[0011] b. Multilingual data augmentation

[0012] Existing multilingual data augmentation work can be divided into two main categories:

[0013] (1) Supervised data augmentation: CoSDA-ML proposed an encoding transformation framework that uses pre-mapped parallel multilingual dictionaries to inject multilingual knowledge into training statements. MulDA introduces a multilingual named entity recognition model, where the multilingual part uses a translator to incorporate information from other languages. Other works also use alignment tools to integrate the representation spaces of the source and target languages.

[0014] (2) Unsupervised data augmentation: Some works use adversarial training to introduce augmented knowledge into the training data. Other works use generative models to generate multilingual samples and import the knowledge contained in these samples into the model. At the same time, data augmentation techniques such as back-translation, word substitution, and low-resource language sentence selection mechanisms are also frequently used to improve the model's representation ability in multiple languages.

[0015] c.mBERT pre-trained model

[0016] mBERT is a multilingual pre-trained model derived from BERT, capable of representing multiple languages ​​in the same semantic space. It first segments the input text using a pre-trained vocabulary, then maps each token to a unique index position in the vocabulary. Based on these index positions, it constructs three input information for the model: semantic embedding, fragment embedding, and positional embedding. Next, it uses a self-attention mechanism to calculate the degree of mutual information interaction between each token, globally modeling the contextual semantics of the sentence. A multi-head attention mechanism is then added to further capture the multi-layered semantics of the representation space. Finally, a non-linear activation function is introduced into the fully connected feedforward network, transforming the space of the attention output and increasing the model's performance. Its model structure is as follows: Figure 2 As shown. Summary of the Invention

[0017] To address the issues of reliance on parallel corpora and focus on shallow data augmentation, the present invention aims to provide a zero-shot multilingual understanding algorithm based on unsupervised pseudo-semantic data augmentation. This algorithm reduces the reliance on multilingual parallel corpora through an unsupervised clustering approach and can fully utilize deep semantic relationships to augment low-resource languages. Ultimately, the algorithm enables the model to perform well on downstream tasks.

[0018] A zero-shot multilingual understanding algorithm based on unsupervised pseudo-semantic data augmentation, the specific steps of which are as follows:

[0019] S1: Encode the text using a text encoder;

[0020] S2: Clustering is performed using a domino chain clustering process to group similar semantic words from different languages ​​together. The domino chain clustering process consists of three successive stages: a monolingual clustering stage, a language family clustering stage, and a multilingual clustering stage. In each clustering stage, the GMM algorithm is used to group multilingual words with similar semantics into one class.

[0021] S3: The pseudo-semantic data augmentation process first treats context-aware semantic representations as pseudo-semantic word units; the sentence is parsed into a tree structure using a multilingual parser, and the subject, verb, and object are regarded as the main components of the sentence; the original representations of the subject, verb, and object are replaced by any multilingual element representation of the same class as the subject, verb, and object words; in addition, a bias-removing optimal transfer affine regularization module is designed to solve the problem of spatial deviation between the sentence representations before and after the replacement, so as to obtain a faster and more stable convergence process;

[0022] S4: The model parameters are updated using a combination of task loss and bias-reduction optimal transmission regularization loss; ultimately, a more accurate multilingual model is obtained; where:

[0023] The text encoder:

[0024] Let the training statement S i It consists of l words.

[0025] S i =(w i1 w i2 ,…,w ij , ..., w il )

[0026] We use m-BERT as the text encoder and take the average of the first and last layer representations as the final text representation; therefore, the word w ij The representation of is in The first layer of the model represents the word w. ij The result after processing This indicates that the last layer of the model represents the word w. ij The result after processing

[0027]

[0028] Similarly, sentence S i The representation of is

[0029]

[0030] The domino chain clustering process is specifically as follows:

[0031] A multilingual clustering process is obtained by using three successive clustering stages;

[0032] a. Monolingual clustering stage: Using both the part-of-speech tag and the encoded representation of words as clustering indicators. To perform clustering, we obtain monolingual clustering results; where |sin m | represents the total number of words in the m-th monolingual language participating in the cluster. This represents the number of clusters after the m-th monolingual clustering. This represents the first set of classes after monolingual clustering.

[0033]

[0034] b. Language family clustering stage: Representing the cluster centers of each language as H sin Assign expert weight A sin The data is then fed into the GMM clusterer to obtain language clustering results, among which... This represents the number of clusters after the nth language family is clustered. This represents the first set of classes after language family clustering.

[0035]

[0036] c. Multilingual clustering stage: Representing the cluster centers of language family clusters as H fam Assign expert weight B fam The data is then fed into the GMM clusterer to obtain multilingual clustering results, among which... This represents the first set of classes after multilingual clustering.

[0037]

[0038] The pseudo-semantic data enhancement process is specifically as follows:

[0039] a. Obtaining sentence representation using an encoder And a multilingual parser is used to obtain the subject, verb, and object components of the sentence, where... It is the embedded representation of the first token in the sentence. and These are the embedded representations of the subject, verb, and object components in the sentence, |L S | represents the total number of tokens for the sentence.

[0040]

[0041] Select any multilingual representation of the cluster containing the subject, verb, and object components. and Replace the subject, verb, and object components in the sentence representation. The sentence representation after replacement is as follows:

[0042]

[0043] Then, the replaced sentence representations are integrated using Transformers;

[0044] b. Partial-optimal transport regularization

[0045] To address the issue of differences in sentence representation space before and after replacement, the optimal transport Wasserstein distance contraction loss is employed. OT eigenvalue shrinkage loss eig Bias reduction distance loss dis Together they constitute the regularization loss. Reg Where ρ1, ρ2, and ρ3 are trainable scale hyperparameters.

[0046]

[0047] Model training objective:

[0048] Overall model training optimization target loss total task loss task and bias-reduction optimal transport regularization loss Reg The structure consists of proportional control parameters, where λ1 and λ2 are the control parameters.

[0049] loss total =λ1loss task +λ2loss Reg .

[0050] The beneficial effects of this invention are as follows: This invention uses unsupervised clustering to align words with the same semantic meaning in different languages, reducing the model's dependence on parallel corpora or translators in multilingual contexts; it performs data augmentation on low-resource languages ​​from a deep semantic perspective, rather than the traditional simple augmentation of the string surface; the pre-trained model under the algorithm has achieved good experimental results in text classification, sentiment analysis, information extraction, and question answering tasks, and the ablation experiment results have also verified the effectiveness of each component of the algorithm. Attached Figure Description

[0051] Figure 1 Example diagram of traditional data augmentation algorithms;

[0052] Figure 2 This is a diagram of the BERT network structure in existing technologies;

[0053] Figure 3 This is a diagram of the algorithm structure of the present invention;

[0054] Figure 4A schematic diagram illustrating the experimental results of the model on a text classification task;

[0055] Figure 5 A schematic diagram illustrating the experimental results of the model on the sentiment analysis task;

[0056] Figure 6 A schematic diagram illustrating the experimental results of the model on the information extraction task;

[0057] Figure 7 A schematic diagram illustrating the experimental results of the model on the question-answering task;

[0058] Figure 8 Schematic diagram of the model ablation experiment results;

[0059] Figure 9 A diagram comparing word representations after dimensionality reduction by different clustering algorithms;

[0060] Figure 10 A diagram comparing the performance of different data augmentation methods in information extraction tasks;

[0061] Figure 11 A diagram comparing the performance of different clustering algorithms on information extraction tasks;

[0062] Figure 12 A diagram illustrating the impact of different hyperparameters (learning rate, number of epochs, k, eta) on model performance;

[0063] Figure 13 The distribution of improvement and degradation of the model on the sample. Detailed Implementation

[0064] The present invention will now be described in detail with reference to the accompanying drawings.

[0065] See Figure 3 The specific steps of the algorithm of this invention are as follows:

[0066] S1: Encode the text using a text encoder.

[0067] S2: Clustering is performed using a domino chain process to group semantically similar words from different languages ​​together. The domino clustering module consists of three successive stages: a monolingual clustering stage, a language family clustering stage, and a multilingual clustering stage. Each clustering stage uses the GMM algorithm to group multilingual words with similar semantics into one class.

[0068] S3: Context-aware semantic representation is used as pseudo-semantic word units. The sentence is parsed into a tree structure using a multilingual parser, and the subject, verb, and object are considered as the main components of the sentence. The original representations of the subject, verb, and object are replaced with any multilingual element representation belonging to the same class as the subject, verb, and object words. Furthermore, this invention designs a bias-reduction optimal transport affine regularization module to address the spatial deviation problem of the sentence representations before and after replacement, thereby achieving a faster and more stable convergence process.

[0069] S4: The model parameters are updated using a combination of task loss and bias-reduced optimal transport regularization loss, ultimately resulting in a more accurate multilingual model. Text Encoder:

[0070] Assume training statement S i Composed of l words

[0071] S i =(w i1 w i2 , ..., w ij , ..., w il )

[0072] We use m-BERT as the text encoder and take the average of the first and last layer representations as the final text representation. Therefore, the word w... ij The representation of is in The first layer of the model represents the word w. ij The result after processing This indicates that the last layer of the model represents the word w. ij The result after processing

[0073]

[0074] Sentence S i The representation of is

[0075]

[0076] Domino clustering process:

[0077] a. Monolingual Clustering: In the clustering process, this invention combines word part-of-speech and the encoded representation of words as clustering indicators. The clustering algorithm incorporating part-of-speech information prevents words with the same semantics but different parts of speech from being clustered together.

[0078] Monolingual clustering first uses a multilingual lexical analysis tool to obtain the part-of-speech tag (17 types) for each word, and then maps the word's part-of-speech tag into a one-hot vector v. POSThe vector is then transformed into the same dimension as the word representation using a linear equation, and finally concatenated with the word semantic representation to form the final word representation. Where w POS Let b represent the transformation matrix. POS This indicates conversion bias.

[0079]

[0080] The GMM clustering algorithm uses the final representation of words... Clustering is performed on the indicators to obtain monolingual clustering results, where |sin m | represents the total number of words in the m-th monolingual language participating in the cluster. This represents the number of clusters after the m-th monolingual clustering. This represents the first set of classes after monolingual clustering.

[0081]

[0082] b. Language family clustering: Representing the cluster centers of a single language Assign expert weight α mt The expert weights are calculated using the Gate mechanism, based on the percentage of each element within a single language cluster relative to the total number of words in the entire single language, N. sin proportion The final weighted representation is fed into the GMM clusterer to obtain the language clustering results, where... This represents the number of clusters after the nth language family is clustered. Let A represent the first set of clusters after language family clustering. sin H represents the set of all expert weights within a single language. sin It is the set of all cluster center representations within a single language.

[0083]

[0084]

[0085]

[0086] c. Multilingual clustering: Representing the cluster centers of language families Assign expert weight β ng The expert weight calculation method is the same as that for language family clustering. Finally, the weighted representations are fed into the GMM clusterer to obtain multilingual clustering results, where B... fam H represents the set of weights of all experts within a language family. fam It is the set of all cluster center representations within a language family:

[0087]

[0088]

[0089]

[0090] Pseudo-semantic data augmentation:

[0091] a. Pseudo-semantic substitution: Two semantic paths are designed for the training sentences. One path encodes the sentence using m-BERT to obtain a sentence representation, and the other path obtains the sentence's parse tree using a multilingual parser. To enable the model to have a more accurate representation of key sentence components, this invention focuses on the key components of the sentence, namely, performing subsequent operations on the subject-verb-object components. This invention first annotates the subject-verb-object components in the sentence representation.

[0092]

[0093] Then, the domino clustering results containing the subject-verb-object components of the sentence are retrieved, and an element from the clustering result containing the subject-verb-object components is randomly selected to replace the subject-verb-object component representation. The sentence representation after replacement is as follows:

[0094]

[0095] Then, the replaced sentence representations are integrated using Transformers.

[0096] b. Partial-optimal transport regularization:

[0097] 1. Wasserstein distance contraction

[0098] This invention utilizes an optimal transmission mechanism to integrate and align the sentence representation spaces before and after data augmentation. The optimal transmission loss is determined by separately calculating the transmission plan P and the transmission cost C.

[0099]

[0100] The transmission overhead C is measured by the Euclidean distance between the sentences before and after the substitution.

[0101]

[0102] This invention optimizes the entire computation process by minimizing the 1-Wasserstein distance. Since calculating the optimal transmission plan P is computationally very complex, this invention utilizes the Sinkhorn algorithm for iterative approximation. Here, ε represents a trainable hyperparameter.

[0103]

[0104]

[0105] Where u and v can The initial values ​​are used for iterative calculations, and the iterative formula is as follows, where a and b are distribution mapping equations, and l is the iteration number index.

[0106]

[0107] 2. Partial eigenvalue shrinkage:

[0108] This invention utilizes a linear orthogonal weight matrix W to map and align the replaced sentence representation with the original sentence representation. W is obtained by decomposing the sentence representations before and after replacement using the SVD algorithm, where U and V represent the two components after decomposition.

[0109]

[0110] W = VU T

[0111] Since smaller eigenvalues ​​are redundant in the computation process, this invention penalizes the k smallest eigenvalues ​​after decomposition. The eigenvalue shrinkage loss can be defined as:

[0112]

[0113] Where δ r η represents the r-th smallest eigenvalue, and η is the penalty control parameter.

[0114] 3. Distance reduction:

[0115] This invention constructs a distance loss by reducing the distance between the sentence representations before and after the replacement, in order to adjust the learning direction of the model, where sim(·,·) represents the similarity:

[0116]

[0117] This invention combines the optimal transmission loss, eigenvalue shrinkage loss, and distance loss to form the optimal transmission affine regularization loss, where ρ1, ρ2, and ρ3 are trainable proportional hyperparameters:

[0118]

[0119] Overall training objective of the model:

[0120] The model training objective consists of two parts: task loss and optimal transfer affine regularization loss, where λ1 and λ2 are proportional control parameters.

[0121] loss total =λ1loss task+λ2loss Reg

[0122] Experimental verification:

[0123] Finally, this invention validates the model on sequence classification, information extraction, and question answering tasks, such as... Figure 4-7 As shown, the models all achieved good results under the algorithm, verifying its effectiveness. Ablation experiments also verified the effectiveness of each component of the algorithm, such as... Figure 8-11 As shown. Figure 12 The four subgraphs in the diagram illustrate the selection process of the experimental hyperparameters (learning rate, number of epochs, k, eta). Figure 13 The model's ability to improve the accuracy of text representation was verified using the lifting distribution of subgraph (a) and the degenerate distribution of subgraph (b).

Claims

1. A zero-shot multilingual understanding algorithm based on unsupervised pseudo-semantic data augmentation, characterized in that, The specific steps are as follows: S1: Encode the text using a text encoder; S2: Clustering is performed using a domino chain clustering process to group similar semantic words from different languages ​​together. The domino chain clustering process consists of three successive stages: a monolingual clustering stage, a language family clustering stage, and a multilingual clustering stage. In each clustering stage, the GMM algorithm is used to group multilingual words with similar semantics into one class. S3: The pseudo-semantic data augmentation process first treats context-aware semantic representations as pseudo-semantic word units; the sentence is parsed into a tree structure using a multilingual parser, and the subject, verb, and object are regarded as the main components of the sentence; the original representations of the subject, verb, and object are replaced by any multilingual element representation of the same class as the subject, verb, and object words; in addition, a bias-removing optimal transfer affine regularization module is designed to solve the problem of spatial deviation between the sentence representations before and after the replacement, so as to obtain a faster and more stable convergence process; S4: The model parameters are updated using a combination of task loss and bias-reduction optimal transmission regularization loss; ultimately, a more accurate multilingual model is obtained; where: The text encoder: Let the training statement S i It consists of l words. S i =(w i1 ,w i2 ,…,w ij ,…,w il ) We use m-BERT as the text encoder and take the average of the first and last layer representations as the final text representation; therefore, the word w ij The representation of is in The first layer of the model represents the word w. ij The result after processing This indicates that the last layer of the model represents the word w. ij The result after processing Similarly, sentence S i The representation of is The domino chain clustering process is specifically as follows: A multilingual clustering process is obtained by using three successive clustering stages; a. Monolingual clustering stage: Using both the part-of-speech tag and the encoded representation of words as clustering indicators. To perform clustering, we obtain monolingual clustering results; where |sin m | represents the total number of words in the m-th monolingual language participating in the cluster. This represents the number of clusters after the m-th monolingual clustering. This represents the first set of classes after monolingual clustering. b. Language family clustering stage: Representing the cluster centers of each language as H sin Assign expert weight A sin The data is then fed into the GMM clusterer to obtain language clustering results, among which... This represents the number of clusters after the nth language family is clustered. This represents the first set of classes after language family clustering. c. Multilingual clustering stage: Representing the cluster centers of language family clusters as H fam Assign expert weight B fam The data is then fed into the GMM clusterer to obtain multilingual clustering results, among which... This represents the first set of classes after multilingual clustering. The pseudo-semantic data enhancement process is specifically as follows: a. Obtaining sentence representation using an encoder And a multilingual parser is used to obtain the subject, verb, and object components of the sentence, where... It is the embedded representation of the first token in the sentence. and These are the embedded representations of the subject, verb, and object components in the sentence, |L S | represents the total number of tokens for the sentence. Select any multilingual representation of the cluster containing the subject, verb, and object components. and Replace the subject, verb, and object components in the sentence representation. The sentence representation after replacement is as follows: Then, the replaced sentence representations are integrated using Transformers; b. Partial-optimal transport regularization To address the issue of differences in sentence representation space before and after replacement, the optimal transport Wasserstein distance contraction loss is employed. Biased eigenvalue shrinkage loss eig Bias reduction distance loss dis Together they constitute the regularization loss. Reg Where ρ1, ρ2, and ρ3 are trainable scale hyperparameters. Model training objective: Overall model training optimization target loss total task loss task and bias-reduction optimal transport regularization loss Reg The structure consists of proportional control parameters, where λ1 and λ2 are the control parameters. loss total =λ1loss task +λ2loss Reg 。

Images