A multi-language word vector weighting alignment method based on orthogonal pach analysis

Abstract

This invention relates to the field of natural language processing technology and discloses a multilingual word vector weighted alignment method based on orthogonal Protodyakonov analysis. The method includes the following steps: S1: Randomly initialize an orthogonal matrix R, and randomly rotate the Chinese word vectors, i.e., X < -XR; substitute the weight matrix A, calculate the weighted orthogonal transformation matrix W according to the provided weighted Protodyakonov analysis, and obtain the weighted aligned Chinese word vector XW2. The Chinese word vectors are downloaded through a given word vector link to obtain Chinese and English word vectors X and Y. A given alignment dictionary L is used to weight sentiment words, with the goal of weighted alignment from X to Y. The method provided by this invention can perform weighted alignment according to the needs of downstream tasks, further improving the performance of downstream tasks, realizing a shared word vector space, and providing a mathematical proof of the weighted orthogonal Protodyakonov analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and specifically to a multi - language word vector weighted alignment method based on orthogonal Procrustes analysis. Background Art

[0002] Natural language processing has achieved rapid development in most languages of the world (such as Chinese, English, etc.). Due to the lack of resources and the imbalance of labels, research on minority languages has been facing difficulties. Usually, a paradigm for improving low - resource languages is: by constructing a shared language feature space, low - resource corpora can perform cross - language transfer learning based on rich - resource corpora in this space, thereby improving the performance of various downstream tasks. Among them, multi - language word vectors are a method to achieve a shared language feature space at the word level. In order to realize this shared word vector space, it is necessary to consider multi - language word vector alignment methods.

[0003] In orthogonal linear transformation, due to the limitation of isomorphism, it is difficult to further improve the alignment quality. For non - orthogonal linear transformation methods, since the restriction on W is relaxed, the alignment quality has improved, but at the same time, some of the original word vector structure information has been sacrificed, and the alignment has become more complex and difficult to understand. Generally speaking, the above research, whether it is based on orthogonal linear transformation or non - orthogonal linear transformation methods, is pursuing more extreme alignment quality. However, in the actual application of multi - language natural language processing, multi - language word vectors with higher alignment quality are not necessarily suitable for the actual needs of downstream tasks. For example, in the sentiment analysis task, if we use English corpora with sufficient corpus quantity to improve the situation of insufficient Chinese corpora, then intuitively, aligning the word vectors corresponding to <happy, 快乐> and <sad, 伤心> is more meaningful than aligning <mercaptan, 硫醇> or <green, 绿油油>, etc. On the other hand, due to the limitation of isomorphism, we know that there is a theoretical upper limit to the alignment space. Aligning some meaningless word vectors may even damage the alignment effect of some key word vectors. Summary of the Invention

[0004] The purpose of the present invention is to design a weighted alignment algorithm for multi - language word vectors, which can perform weighted alignment according to the needs of downstream tasks according to the set weights, and further improve the performance of downstream tasks.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A multi - language word vector weighted alignment method based on orthogonal Procrustes analysis, comprising the following steps:

[0007] S1: Randomly initialize an orthogonal matrix R, and randomly rotate the Chinese word vectors, i.e., X < -XR.

[0008] S2: Substitute the weight matrix A, calculate the weighted orthogonal transformation matrix W based on the provided weighted Protodyakonov analysis, and obtain the weighted aligned Chinese word vector XW2.

[0009] Preferably, the Chinese word vectors are downloaded through a given word vector link to obtain Chinese and English word vectors X and Y.

[0010] Preferably, the given alignment dictionary L: L = {sun: sun, moon: moon, king: king, queen: queen, male: male, female: female, afraid: afraid, angry: angry, excited: excited}.

[0011] Preferably, the given alignment dictionary L is weighted for emotion words, corresponding to {afraid: afraid, angry: angry, excited: excited} in L.

[0012] Preferably, the objective is to perform weighted alignment from X to Y and compare it with unweighted alignment.

[0013] Compared with the prior art, the beneficial effects of the present invention are:

[0014] 1. This invention proposes a weighted alignment algorithm for multilingual word vectors, which can perform weighted alignment according to the needs of downstream tasks and set weights, thereby improving the performance of downstream tasks. Furthermore, a mathematical proof of the weighted orthogonal Protodyakonov analysis is presented. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a projection diagram of S1PCA according to an embodiment of the present invention;

[0017] Figure 2 This is an S2PCA projection diagram of an embodiment of the present invention;

[0018] Figure 3 This is an S3PCA projection diagram of an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0021] Example

[0022] Please see Figure 1-3 The multilingual word vector weighted alignment method based on orthogonal Protodyakonov analysis provided by this invention includes the following steps:

[0023] S1: Randomly initialize an orthogonal matrix R, and randomly rotate the Chinese word vectors, i.e., X < -XR. The alignment effect is projected onto a 2D plane by PCA (Principal Component Analysis). Figure 1 If we calculate the orthogonal transformation matrix W without weighting, we obtain the Chinese word vector XW1 under unweighted alignment. The alignment effect is projected onto the 2D plane by PCA (Principal Component Analysis). Figure 2 .

[0024] S2: Substitute the weight matrix A, calculate the weighted orthogonal transformation matrix W based on the provided weighted Prototype analysis, and obtain the weighted aligned Chinese word vector XW2. The alignment effect is projected onto the 2D plane by PCA (Principal Component Analysis). Figure 3 .

[0025] The Chinese word vectors can be downloaded via the provided link: https: / / fasttext.cc / docs / en / crawl-vectors.html.

[0026] Obtain word vectors X and Y for Chinese and English, respectively. Given an alignment dictionary L: L = {sun: sun, moon: moon, king: king, queen: queen, male: male, female: female, afraid: afraid, angry: angry, excited: excited}, in this embodiment, the alignment dictionary L is weighted for emotion words, corresponding to {afraid: afraid, angry: angry, excited: excited} in L. The goal is to perform weighted alignment from X to Y and compare it with unweighted alignment.

[0027] In early research, Mikolov et al. discovered in 2013 that languages ​​with closer language families tend to have more similar word vector space structures, such as Chinese and Tibetan, and English and German. This similarity in spatial structure provides a possibility for aligning word vectors between different languages. Following this discovery, a series of related studies emerged. However, these studies can be summarized into two categories: orthogonal linear transformation methods and non-orthogonal linear transformation methods.

[0028] The idea behind orthogonal linear transformation is to find an orthogonal transformation matrix W such that the distance between the word vectors X and Y corresponding to the two languages ​​is minimized. Specifically, this is expressed as ||XW-Y|| F→0 Here, X is aligned to Y. Given that W is an orthogonal matrix and the distance uses Euclidean distance, this problem can be reduced to an orthogonal Protodyakonov analysis problem. Mathematically, Protodyakonov analysis is a method of finding an affine transformation from shape X to shape Y using the least squares method. Specifically, given shape X∈R... n×d and Y∈R n ×d Find an orthogonal matrix W∈R d×d For any pair to be aligned {x i y i}, x i ∈X and y i ∈Y have:

[0029]

[0030] Minimizing the above expression is equivalent to maximizing: Since Z is also an orthogonal matrix, therefore we have And Z i，i ≤1, the "=" holds only if Z = I. Therefore, Z = V T WU = I, further solving for W, we get: * =UV T U, ∑, V T =svd(XY T ).

[0031] The problem of this invention and related definitions: Given word vectors X∈R in two languages.n×d and Y∈R n×d The alignment dictionary for the target is L = {x1:y1, x2:y2, ..., x...} n :y n The weight matrix for the target aligned word pairs is:

[0032]

[0033]

[0034] Where n is the number of word vectors, d is the dimension of the word vectors, and k is the size of the alignment dictionary. The specific problem is to find an orthogonal matrix: W∈R d×d ,have

[0035]

[0036] The proof process of this invention is as follows: For the above formula, the expansion is:

[0037]

[0038] Minimizing the above expression is equivalent to maximizing: Since Z is also an orthogonal matrix, therefore we have And Z i，i ≤1, the "=" holds only if Z = I. Therefore, Z = V T WU = I, further solving for W, we get: * =UV T U, ∑, V T =svd(XAY) T ).

[0039] After the above mathematical proof, we obtained the weighted aligned word vectors XW and Y.

[0040] The multilingual word vector weighted alignment method based on orthogonal Protodyakonov analysis provided in the above embodiments of the present invention can perform weighted alignment according to the set weights as needed by downstream tasks, thereby further improving the performance of downstream tasks, realizing a shared word vector space, and providing a mathematical proof of weighted orthogonal Protodyakonov analysis.

[0041] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0042] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multilingual word vector weighted alignment method based on orthogonal Protodyakonov analysis, characterized in that, Includes the following steps: S1: Randomly initialize an orthogonal matrix R, and randomly rotate the Chinese word vectors, i.e., X < -XR; S2: Substitute the weight matrix A, calculate the weighted orthogonal transformation matrix W based on the provided weighted Protodyakonov analysis, and obtain the weighted aligned Chinese word vector XW; Given word vectors for two languages and The alignment dictionary for the target is The weight matrix for the target aligned word pairs is: ， Where n is the number of word vectors, d is the dimension of the word vectors, and k is the size of the alignment dictionary; Find an orthogonal matrix: ,have ； Expanding the above equation, we get: Minimizing the above expression is equivalent to maximizing: Since Z is also an orthogonal matrix, therefore we have ,and Only when When, "=" is true; Therefore: Further solving yields: ; After the above mathematical proof, the weighted aligned word vectors XW and Y were obtained.

2. The multilingual word vector weighted alignment method based on orthogonal Protodyakonov analysis according to claim 1, characterized in that: The Chinese word vectors are downloaded through a given word vector link to obtain Chinese and English word vectors X and Y.

3. The multilingual word vector weighted alignment method based on orthogonal Protodyakonov analysis according to claim 1, characterized in that: Given an alignment dictionary L: L={sun: sun, moon: moon, king: king, queen: queen, male: male, female: female, afraid: afraid, angry: angry, excited: excited}.

4. The multilingual word vector weighted alignment method based on orthogonal Protodyakonov analysis according to claim 3, characterized in that: The given alignment dictionary L is weighted for emotion words, corresponding to {afraid: afraid, angry: angry, excited: excited} in L.

5. The multilingual word vector weighted alignment method based on orthogonal Protodyakonov analysis according to claim 1, characterized in that: The objective is to perform weighted alignment from X to Y and compare it with unweighted alignment.

Images