A bilingual vocabulary mapping learning method based on axis term weighting retrieval criterion

Abstract

The application relates to the technical field of unsupervised word translation, and discloses a bilingual vocabulary mapping learning method based on an axis core word weighted retrieval standard, which comprises the following steps: initializing t=0, setting the iteration number T, setting hyperparameters mu and k, setting a score array R={}, for all candidate word pairs l x and l y from dictionaries L x and L y , obtaining corresponding word vectors x and y according to the dictionaries. The bilingual vocabulary mapping learning method based on the axis core word weighted retrieval standard combines a weighted moving average idea into an alignment iteration process, so that the optimization effect is more stable. Compared with existing bilingual dictionary retrieval technologies, the bilingual dictionary and the optimized multilingual word vector generated by the application are more suitable for specific field downstream tasks, and the application combines the weighted moving average idea, so that the problem of violent fluctuation caused by too large difference between new and old dictionaries can be alleviated.

Description

Technical Field

[0001] The present invention relates to the technical field of unsupervised word translation, and in particular to a bilingual vocabulary mapping learning method based on an axis word weighted retrieval standard. Background Art

[0002] Multilingual word vectors have developed rapidly in recent years. Among the various methods for generating multilingual word vectors, the offline mapping method has received particular attention. This method can align two monolingual word vectors, which can be trained and generated offline separately. Therefore, this offline mapping-based method does not require or only requires a small target training dictionary to achieve alignment. In the above unsupervised multilingual word vector generation methods, a key point is the mapping learning of the bilingual dictionary. Since there is no or only a small target training dictionary, how to summarize a high-quality bilingual dictionary based on the already preliminarily aligned multilingual word vectors will in turn affect the quality of the multilingual word vectors.

[0003] In 2017, Conneau et al. proposed MUSE (Multilingual Unsupervised and Supervised Embedding), an unsupervised multilingual word embedding alignment framework based on generative adversarial networks. Their alignment strategy is to find an orthogonal transformation matrix W that minimizes the distance between the word embeddings X and Y in the two languages. Specifically, this is expressed as ||XW-Y|| F →0, where X is aligned to Y and the Euclidean distance is used. Their approach has two main stages. In the first stage, a generative adversarial network is used to achieve a basic word vector alignment. They train a discriminator D and a generator W, where the discriminator D aims to distinguish between the authenticity of XW and Y (XW is a fake sample). When the generator W is optimized to ||XW-Y|| 2 →0, the discriminator D will have difficulty distinguishing XW and Y. In the second stage, after obtaining a roughly aligned orthogonal matrix W, this step will further optimize the alignment effect. They proposed a cross-domain similarity local scaling mapping learning method: CSLS (cross-domain similarity local scaling), which is used to calculate the matching score of given candidate aligned word vectors x and y:

[0004]

[0005] After obtaining the matching scores for all candidate aligned word vectors, Conneau et al. aligned and sorted them in descending order, obtaining the top k word vector matching results (k is a hyperparameter). Based on the words or characters corresponding to the word vectors, they inferred the bilingual dictionary L. To this end, we can further optimize the alignment of multilingual word vectors in a supervised manner based on the bilingual dictionary generated by this unsupervised method. By continuously iterating the above two processes of "bilingual dictionary induction" and "supervised alignment," we can further improve the alignment effect.

[0006] Joulin et al. introduced a convexly relaxed objective training function based on CSLS. This objective function can be more efficiently optimized using projected subgradient descent. They proposed the following convexly relaxed RCSLS:

[0007]

[0008] Joulin et al. conducted experiments on word translation benchmarks and demonstrated that RCSLS achieved state-of-the-art results across 25 languages. Furthermore, it significantly improved word translation for geographically distant languages, such as Chinese and English.

[0009] Isomorphism is the basis for offline mapping. Isomorphism specifically means that no matter what language family these languages ​​come from or what cultural background they belong to, all language spaces have similar structures. In the above-mentioned existing technologies, as isomorphism is gradually questioned, it is difficult for offline mapping methods to further improve the quality of alignment. However, in the actual application of multilingual natural language processing, the current mapping learning method based on global alignment may not be able to generate bilingual dictionaries that are suitable for the actual needs of downstream tasks. For example, in sentiment analysis tasks, if we use English corpus with sufficient corpus to improve the situation where Chinese corpus is insufficient, then intuitively, we get<happy,快乐> and<sad,伤心> Translation pairs related to emotions are more accurate than alignment<mercaptan,硫醇> or<green,绿油油> On the other hand, during the continuous iteration of "bilingual dictionary induction" and "supervised alignment," the bilingual dictionary induction process is unstable. This is because after the "supervised alignment" process, we cannot guarantee how much the inferred new dictionary will differ from the bilingual dictionary used in the previous iteration. If the difference is too large, it may cause drastic update fluctuations and lead to unstable results.

[0010] In response to the shortcomings of existing methods such as being unsuitable for downstream tasks and being unstable, this paper proposes a bilingual vocabulary mapping learning method PCSLS (Pivot-based CSLS) based on the pivot-based weighted retrieval criterion.

[0011] Based on CSLS, the specific improvements are as follows:

[0012] 1. Based on the pivot words required by the downstream task and the given weights, a bilingual translation dictionary that is more suitable for use in the downstream task is generated.

[0013] 2. Propose a more stable optimization method combined with weighted moving average (WMA). Summary of the Invention

[0014] The purpose of the present invention is to provide a bilingual vocabulary mapping learning method based on an axis word weighted retrieval standard to solve the problems raised in the above background technology.

[0015] To achieve the above object, the present invention provides the following technical solutions:

[0016] A bilingual vocabulary mapping learning method based on an axis word weighted retrieval standard comprises the following steps:

[0017] S1

[0018] Initialize t=0, set the number of iterations T, and set the hyperparameters μ and k.

[0019] S2

[0020] Set the score array R = {}.

[0021] S3

[0022] For all the x and L y Candidate word pair l x and l y , obtain the corresponding word vectors x and y according to the dictionary.

[0023] S4

[0024] When t=0, PCSLS0=PCSLS(xW,y).

[0025] S5

[0026] When t≠0, PCSLSt=μ×PCSLS(xW,y)+(1-μ)×PCSLSt.

[0027] S6

[0028] Append the PCSLS scores PCSLSt of the candidate word pairs Lx and Ly to the array R.

[0029] S7

[0030] The steps S3, S4, S5 and S6 are repeated until the PCSLS scores of all candidate word pairs are calculated.

[0031] S8

[0032] The score array R in step S2 is sorted in descending order, the top-k word pairs with the highest scores are obtained, and the dictionary L is updated as L←L′.

[0033] S9

[0034] The orthogonal alignment matrix W is updated as W←svd(X L Y L T ), where svd denotes singular value decomposition, X L and Y L are the word vectors corresponding to the new dictionaries L, and t is updated as t←t+1.

[0035] Preferably, the steps S2-S9 are repeated until t=T, and the optimized orthogonal matrix W and the bilingual dictionary L are obtained.

[0036] Preferably, in step S6, when there is no candidate word pair, the score array is sorted in descending order, the top-k candidate word pairs with the highest scores are obtained, a new word pair dictionary is formed, t is incremented by 1, and steps S2-S9 are repeated until t=T, and the optimized orthogonal matrix W and the bilingual dictionary L are obtained.

[0037] Given the word vectors of two languages as and , the word dictionaries L x and L y corresponding to the word vectors, and the weight vectors of the pivot words of the two languages as S(x)=[α1,α2,...,α n ] and S(y)=[β1,β2,...,β n ], a given orthogonal matrix preliminarily satisfies ||XW-Y|| 2 ≈0, where n is the number of word vectors, d is the dimension of the word vectors, and k is the size of the alignment dictionary. Our goal is to further optimize W to improve the quality and stability of translation (especially the pivot words) and output a bilingual dictionary more suitable for downstream tasks.

[0038] First, for two vectors x and y and for the weights S(x) and S(y), we give the specific expression of PCSLS as follows:

[0039]

[0040] Compared with the prior art, the bilingual vocabulary mapping learning method based on the axis word weighted retrieval standard provided by the present invention has the following beneficial effects:

[0041] 1. This bilingual vocabulary mapping learning method based on the axis word weighted retrieval standard can generate a bilingual translation dictionary that is more suitable for downstream tasks based on the axis words required by downstream tasks and the given weights.

[0042] 2. This bilingual vocabulary mapping learning method, based on the weighted retrieval standard for pivot words, incorporates the concept of weighted moving average into the iterative alignment process, making the optimization effect more stable. Compared with existing bilingual dictionary retrieval technologies, the bilingual dictionary and optimized multilingual word vectors generated by this invention are more suitable for downstream tasks in specific fields. At the same time, the weighted moving average concept is combined with this invention to make the iterative process more stable, which helps alleviate the problem of large differences between the new and old dictionaries leading to severe fluctuations.

[0043] 3. This bilingual vocabulary mapping learning method based on the pivot word weighted retrieval standard can generate a bilingual translation dictionary that is more suitable for use in downstream tasks based on the pivot words required by downstream tasks and the given weights. The present invention incorporates the idea of ​​weighted moving average into the iterative alignment process, making the optimization effect more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work.

[0045] Figure 1 Schematic diagram of the flow of a bilingual vocabulary mapping learning method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] In the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "connect," "fixed," etc. should be understood broadly. For example, they may refer to fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0048] See also Figure 1 , the present invention provides the following embodiments:

[0049] Example 1

[0050] The bilingual vocabulary mapping learning method based on the axis word weighted retrieval standard provided in this embodiment includes the following steps:

[0051] S1

[0052] Initialize t=0, set the number of iterations T, and set the hyperparameters μ and k.

[0053] S2

[0054] Set the score array R = {}.

[0055] S3

[0056] For all the x and L y Candidate word pair l x and l y , obtain the corresponding word vectors x and y according to the dictionary.

[0057] S4

[0058] When t=0, PCSLS0=PCSLS(xW,y).

[0059] S5

[0060] When t≠0, PCSLSt=μ×PCSLS(xW,y)+(1-μ)×PCSLSt.

[0061] S6

[0062] Append the PCSLS scores PCSLSt of the candidate word pairs Lx and Ly to the array R.

[0063] S7

[0064] Steps S3, S4, S5, and S6 are repeated until the PCSLS scores of all candidate words are calculated.

[0065] S8

[0066] In step S2, the score array R is sorted in descending order, the bilingual dictionaries L′ with the top k scores are obtained, and the dictionary L←L′ is updated.

[0067] S9

[0068] Update the orthogonal alignment matrix W←svd(XLYLT), where svd refers to the singular value decomposition, XL and YL refer to the word vectors corresponding to the new dictionary L, and update t=t+1.

[0069] Steps S2 to S9 are repeatedly calculated until t=T, and the optimized orthogonal matrix W and bilingual dictionary L are obtained.

[0070] Example 2

[0071] A bilingual vocabulary mapping learning method based on an axis word weighted retrieval standard comprises the following steps:

[0072] S1

[0073] Initialize t=0, set the number of iterations T, and set the hyperparameters μ and k.

[0074] S2

[0075] Set the score array R = {}.

[0076] S3

[0077] For all the x and L y Candidate word pair l x and l y , obtain the corresponding word vectors x and y according to the dictionary.

[0078] S4

[0079] When t=0, PCSLS0=PCSLS(xW,y).

[0080] S5

[0081] When t≠0, PCSLSt=μ×PCSLS(xW,y)+(1-μ)×PCSLSt.

[0082] S6

[0083] Append the PCSLS scores PCSLSt of the candidate word pairs Lx and Ly to the array R.

[0084] S7

[0085] Steps S3, S4, S5, and S6 are repeated until the PCSLS scores of all candidate words are calculated.

[0086] S8

[0087] In step S2, the score array R is sorted in descending order, the bilingual dictionaries L′ with the top k scores are obtained, and the dictionary L←L′ is updated.

[0088] S9

[0089] Update the orthogonal alignment matrix W←svd(XLYLT), where svd refers to the singular value decomposition, XL and YL refer to the word vectors corresponding to the new dictionary L, and update t=t+1.

[0090] Steps S2 to S9 are repeated until t = T, and the optimized orthogonal matrix W and bilingual dictionary L are obtained. If there are no candidate word pairs in step S6, the score array is sorted in descending order, and the candidate word pairs of the top K scores are obtained to form a new dictionary. t is incremented by 1. When t <T时,重复步骤S2~S9重复进行计算,直到t=T,获得优化后的正交矩阵W以及双语词典L。

[0091] Given two language word vectors and Dictionary L corresponding to the word vector x and L y , the weight vectors of the axis words corresponding to the two languages ​​are S(x)=[α1,α2,...,α n ] and S(y)=[β1,β2,...,β n ]. Given an orthogonal matrix Initial satisfaction ||XW-Y|| 2 ≈ 0. Where n is the number of word vectors, d is the dimension of the word vector, and k is the size of the alignment dictionary. Our goal is to further optimize W to improve the quality and stability of translation (especially the axis words) and output a bilingual dictionary that is more suitable for downstream tasks.

[0092] First, for two vectors x and y and for weights S(x) and S(y), we give the specific expression of PCSLS as follows:

[0093]

[0094] The method provided in the above embodiment can generate a bilingual translation dictionary that is more suitable for use in downstream tasks based on the pivot words required by the downstream tasks and the given weights, and incorporates the idea of ​​weighted moving average into the iterative process of alignment, making the optimization effect more stable. Compared with existing bilingual dictionary retrieval technology, the bilingual dictionary generated by the present invention and the optimized multilingual word vector are more suitable for downstream tasks in specific fields. At the same time, the present invention incorporates the idea of ​​weighted moving average to make the iterative process more stable, which is conducive to alleviating the problem of drastic fluctuations caused by the large difference between the new and old dictionaries. This method can generate a bilingual translation dictionary that is more suitable for use in downstream tasks based on the pivot words required by the downstream tasks and the given weights. The present invention incorporates the idea of ​​weighted moving average into the iterative process of alignment, making the optimization effect more stable, which is conducive to alleviating the problem of drastic fluctuations caused by the large difference between the new and old dictionaries.

[0095] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0096] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A bilingual vocabulary mapping learning method based on the axis word weighted retrieval standard, characterized by: It includes the following steps: S1： Initialize t = 0, set the number of iterations T, and set the hyperparameters μ and k; S2： Set the score array R = {}; S3： For all the x and L y Candidate word pair l x and l y , obtain the corresponding word vectors x and y according to the dictionary; S4： When t = 0, PCSLS0 = PCSLS(xW, y); Among them, S(x) and S(y) are the weight vectors of the axis words corresponding to the two languages, S(x)=[α1,α2,...,α n ]; S(y)=[β1,β2,...,β n ]; S5： When t≠0, PCSLS t =μ×PCSLS(xW,y)+(1-μ)×PCSLS t-1 ; S6： Add the PCSLS scores of candidate word pairs Lx and Ly t Into array R; S7： Repeat the calculations in steps S3, S4, S5, and S6 until the PCSLS scores of all candidate words are calculated; S8： Sort the score array R in step S2 in descending order, obtain the bilingual dictionary L' with the top k scores, and update the dictionary L ← L'; S9： Update the orthogonal alignment matrix W←svd(X L Y L T ), svd refers to singular value decomposition, X L and Y L They refer to the word vectors corresponding to the new dictionary L, and are updated at t=t+1.

2. The bilingual vocabulary mapping learning method based on the axis word weighted retrieval standard according to claim 1 is characterized in that: Repeat the calculations in steps S2 to S9 until t = T to obtain the optimized orthogonal matrix W and the bilingual dictionary L; 3. The bilingual vocabulary mapping learning method based on the axis word weighted retrieval standard according to claim 1 is characterized in that: When there are no candidate word pairs in step S6, sort the score array in descending order, obtain the candidate word pairs with the top K scores, form a new pair dictionary, increment t by 1, and when t < T, repeat the calculations in steps S2 to S9 until t = T to obtain the optimized orthogonal matrix W and the bilingual dictionary L.

Images