A morphologically enhanced tensorized word embedding compression system

Abstract

The application discloses a morphological enhancement based tensorized word embedding compression system, which comprises a morpheme segmentation module, a morpheme index and embedding module and a word embedding generation module; the morpheme segmentation module divides each word in a word table of a text task into morphemes; the morpheme index and embedding module firstly generates a morpheme table by counting the division result of the morpheme segmentation module, then defines a morpheme index matrix and a plurality of trainable morpheme embedding matrices, each row of the morpheme index matrix represents the position of the morpheme of a corresponding word in the word table in the morpheme table, and each row of the morpheme embedding matrix represents an embedding vector of a corresponding morpheme in the morpheme table; the word embedding generation module indexes the morpheme vector from the morpheme embedding matrix and performs a tensor product on each word in the word table, adds the results of a plurality of tensor products to generate a word embedding vector; and the application overcomes the problems of a large amount of parameters and storage space occupation in general word embedding technology, and the problem of task effect loss when high compression word embedding is performed.

Description

Technical fields:

[0001] This invention relates to the field of natural language processing, and more particularly to a tensor word embedding compression system based on morphological enhancement. Background technology:

[0002] Word embedding technology is the foundation of various natural language processing (NLP) tasks such as machine translation, text classification, information retrieval, and question answering. It can represent words in text data as fixed-length vectors so that computers can process the text. All different word vectors in the task will form a |V|×d word embedding matrix W, where |V| represents the size of the vocabulary V, that is, the number of different words contained in the text of the task, and d represents the dimension of the word vectors. The word embedding matrix W is a parameter of the NLP model and needs to be trained and optimized. When the vocabulary size and word vector dimension are large, the word embedding matrix will occupy a large number of model parameters. For example, the vocabulary size |V| of the One Billion Word[1] language modeling task can reach 800,000, while the word vector dimension d is usually 300~1024[2,3]. Depending on the task and model settings, the parameters of the word embedding matrix usually account for 20%~90% of the total parameters of the model. Therefore, storing and accessing word embedding parameters using word embedding technology requires a large amount of disk and memory space. This limits the deployment of NLP models on resource-constrained devices, hindering their practical implementation and application.

[0003] To address the issue of large numbers of parameters in word embeddings, numerous researchers have dedicated themselves to word embedding compression techniques. These studies can be broadly categorized into two types: methods based on product quantization and methods based on decomposition.

[0004] The core idea of ​​word embedding compression methods based on product quantization [4,5,6] is to learn a unique set of compact codes for each word, where each code value points to a vector in the codebook. That is, the vector representation of each word can be synthesized from the vectors in the codebook that its compact code points to. Therefore, the focus of this type of word embedding compression method is the learning of word compact codes. For example, Raphael Shu et al. [4] used the Gumbel-Softmax technique [7] to implement end-to-end word compact code learning based on neural networks, where each word is represented as the sum of the codebook vectors pointed to by the compact code. Since only the word compact code and codebook vector need to be stored, this method has achieved a compression of more than 10 times the word embedding layer parameters in tasks such as machine translation. However, the additional task of learning word compact codes introduced by this type of method has some potential risks. On the one hand, the task of learning compact codes depends on the already trained word embeddings; on the other hand, the quality of learning compact codes will also directly affect the quality of word embedding learning, thus causing a loss of task performance.

[0005] The word embedding compression method based on decomposition [8,9,10] mainly utilizes the inverse process of low-rank matrix decomposition and tensor decomposition to replace the original large word embedding matrix with a series of small matrices or tensors. The model only needs to store the parameters of these small matrices or tensors, thereby achieving the compression of word embedding parameters. Among them, the word embedding compression method based on low-rank matrix decomposition usually uses the product of two small matrices to approximate the original word embedding matrix. For example, ALBERT [8] uses this method to compress word embedding parameters by about 6 times on a pre-trained language model. The word embedding compression method TT Embeddings

[10] based on TensorTrain decomposition [9] uses the product of a series of second-order or third-order tensors to approximate the original large word embedding matrix. The word embedding compression method Word2ket

[11] based on quantum entanglement states means that a single word vector is the sum of the tensor direct product of several low-dimensional vectors, which is essentially similar to the rank decomposition in tensor decomposition

[12] . TT Embeddings and Word2ket, two tensor decomposition-based word embedding compression methods, possess powerful compression capabilities, achieving tens or even hundreds of times the compression of word embedding parameters in machine translation tasks. However, they often struggle to maintain task performance when achieving high-compression ratios. Therefore, we consider improving these methods with strong compression potential to maintain task performance even at high compression ratios. Summary of the Invention:

[0006] To address the problems of existing technologies, this invention proposes a tensor-based word embedding compression system based on morphological enhancement. This system divides words into smaller morpheme units with specific semantic or syntactic functions, and then represents each word as the sum of tensor direct products of its morpheme vectors based on rank decomposition. This invention only needs to store morpheme vectors with a small number of parameters, eliminating the need to store the word embedding matrix, thus significantly reducing the number of word embedding parameters. Simultaneously, using morphemes to model word vectors injects prior morphological knowledge into the learning of word embedding representations, improving the quality of word embeddings and helping to maintain task performance.

[0007] The present invention solves its practical problem by adopting the following technical solution:

[0008] A tensor-based word embedding compression system based on morphological enhancement, the model includes a morpheme segmentation module, a morpheme indexing and embedding module, and a word embedding generation module;

[0009] The morpheme segmentation module segments each word in the vocabulary V of the text task into morphemes of length n and constructs a morpheme table M.

[0010] The morpheme indexing and embedding module constructs a morpheme index matrix I with dimension |V|×n based on the morpheme table M and the word table V; each row of the morpheme index matrix represents the position of the morpheme of the corresponding word in the morpheme table M; where |V| is the size of the word table V;

[0011] The morpheme indexing and embedding module constructs r trainable morpheme embedding matrices f of dimension |M|×q based on the morpheme table M. i Each row vector of the morpheme embedding matrix represents the q-dimensional embedding vector of the corresponding morpheme in the morpheme table; where |M| is the size of the morpheme table M, 1≤i≤r;

[0012] The word embedding generation module generates words based on the morpheme index vectors of the morpheme index matrix I. j and the morpheme embedding matrix f i The morpheme vectors indexed are used to construct a word embedding vector model by performing tensor product operations.

[0013] Furthermore, the process of constructing the morpheme table M by the morpheme segmentation module is as follows:

[0014] The morpheme segmentation module statistically analyzes the different words contained in the text data of the task to form a vocabulary V, and uses morphological morpheme segmentation methods to segment each word in the vocabulary; wherein:

[0015] A word consisting of l morphemes will be divided into a sequence of morphemes [m1, m2, ..., m]. l ];

[0016] The morpheme sequences of different words in vocabulary V are post-processed to the same fixed length n:

[0017] (1) For words with fewer than n morphemes l, fill them with filler characters;

[0018] (2) For words with more than n morphemes l, concatenate the nth morpheme and all the morphemes following it ([m n ,…,m l ]) is a morpheme;

[0019] The morpheme segmentation module statistically analyzes the results of the above morpheme segmentation and post-processing, and constructs a morpheme table M based on the different morphemes generated.

[0020] Furthermore, the word embedding generation module constructs the word embedding vector model as follows:

[0021] For the j-th word in the vocabulary V, first find the morpheme index vector I of that word from the morpheme index matrix I. j , that is [I j1 ,I j2 ,…,I jn ];

[0022] Based on the morpheme index vector from the single morpheme embedding matrix f i The n morpheme vectors are indexed and their tensor products are performed to obtain a q. n A dimensional vector;

[0023] q is obtained from the embedding matrix of r morphemes n The word q is obtained by adding the dimensional vectors. n 3D word vector generation word embedding vector model:

[0024]

[0025] Beneficial effects:

[0026] This invention provides a morphology-enhanced tensor-based word embedding compression system. The system segments each word in the text into smaller morphemes, then constructs a morpheme index matrix and a morpheme embedding matrix based on the morpheme segmentation results. Next, for each word in the vocabulary, the system indexes the vectors of its contained morphemes from the morpheme embedding matrix based on its morpheme index vector, and performs tensor product and summation operations to generate the word vector. This invention combines the powerful compression capabilities of tensor product with morphological knowledge based on morphemes to achieve high-quality word embedding compression.

[0027] This invention is applicable to parameter compression of word embedding layers in various natural language processing models. It combines the powerful compression capability of tensor products with morphological knowledge based on morphemes to achieve high-quality compressed word embedding representation.

[0028] This invention only requires storing and training the parameters contained in the morpheme embeddings. Since the number of morphemes is much smaller than the number of words, and the vector dimension of morpheme embeddings is much smaller than that of word embeddings, this technique can significantly reduce the number of parameters required for word embeddings. This saves natural language processing models a significant amount of memory and disk space, facilitating their deployment and application on resource-constrained devices. Furthermore, the word embedding learning process utilizes morpheme-based linguistic morphology knowledge, which helps improve the performance of compressed word embedding models. Attached image description:

[0029] Figure 1 This is a flowchart of the tensor word embedding compression technology based on morphological enhancement of the present invention;

[0030] Figure 2 Examples of word morphemes in different languages. Detailed Implementation

[0031] This invention provides a tensor-based word embedding compression technique based on morphological enhancement. Figure 1Shows the flow chart of the technology. The technology includes a morpheme segmentation module, a morpheme index and embedding module, and a word embedding generation module.

[0032] (1) The morpheme segmentation module is used to segment each word in the vocabulary of the text task into morphemes.

[0033] In linguistics, a word can usually be divided into morphemes with specific semantic or grammatical functions. For languages such as English, a word can be split into smaller units such as roots and affixes. For example, "unkindly" can be split into the negative "un", the "kind" with meanings such as "friendly", and the adverbial "ly". For Chinese, a Chinese character can also be split into smaller units such as radicals, e.g., "沐" can be split into the radical "氵" representing water and "木". Figure 2 Shows that words can be morphemically segmented, and such phenomena are common in different languages. In addition, a small number of morphemes can combine to form more words.

[0034] The morpheme segmentation module first counts the different words contained in the text data of the task to form a vocabulary V, and then uses a morphological morpheme segmentation tool to segment each word in the vocabulary into morphemes. A word consisting of l morphemes will be divided into a morpheme sequence [m1, m2, …, m l . In order to make the morpheme sequences of different words in the vocabulary have the same fixed length n, the following post-processing needs to be performed on the results of morpheme segmentation. (1) For words with fewer than n (n≥2) morphemes, padding symbols are used for padding. For example, a single-morpheme word needs to be padded with n - 1 padding symbols. (2) For words with more than n morphemes, the nth morpheme and all the morphemes following it are concatenated into one morpheme. For example, a word "unfeelingly" consisting of 4 morphemes, which is segmented into [un, feel, ing, ly], when n = 3, it is processed as [un, feel, ingly]. The above post-processing process can be formalized as shown in the following formula.

[0035]

[0036] The morpheme segmentation module counts the results segmented in the above way and forms a morpheme table M according to the different morphemes generated.

[0037] (2) The morpheme index and embedding module is used to construct a morpheme index matrix and a morpheme embedding matrix.

[0038] The morpheme indexing and embedding module constructs a morpheme index vector of length n for each word in the vocabulary V. The j-th element of the vector is the position of the j-th morpheme of the word in the morpheme table M. The morpheme vectors of all words form a morpheme index matrix I with dimension |V|×n, where |V| is the size of the vocabulary V.

[0039] Next, the morpheme indexing and embedding module will construct r morpheme embedding matrices f with dimensions |M|×q. i , where |M| is the size of the morpheme table M, 1≤i≤r. Each row vector of the morpheme embedding matrix represents the q-dimensional embedding vector of the corresponding morpheme in the morpheme table. These r morpheme embedding matrices are the model parameters that need to be trained and optimized in this word embedding compression technique. Before model training, these morpheme embedding matrices can be randomly initialized.

[0040] (3) The word embedding generation module is used to generate word vectors for each word in the vocabulary.

[0041] For the j-th word in the vocabulary V, first find the morpheme index vector I of that word from the morpheme index matrix I. j , that is [I j1 ,I j2 ,…,I jn Then, based on the morpheme index vector, from the single morpheme embedding matrix f i The n morpheme vectors are indexed and their tensor products are performed to obtain a q. n A vector of dimension, then q obtained from the embedding matrix of r morphemes. n The word q is obtained by adding the dimensional vectors. n Word vectors of dimension 1. This process is formalized as shown in the following formula:

[0042]

[0043] This invention has demonstrated its effectiveness on multiple tasks, including translation, question answering, and natural language inference. Furthermore, we compare this technique with related factorization-based word embedding compression methods, including methods based on low-rank matrix factorization (Matrix Factor.), methods based on Tensor Train, and rank factorization-based methods Word2ket and Word2ketXs. Additionally, this invention denotes the original, uncompressed word embedding method as Original.

[0044] (1) For the translation task, we selected four different language pair translation datasets: IWSLT'14 German-English (German-English), OPUS-100 English-Italian (English-Italian), OPUS-100 English-Spanish (English-Spanish), and OPUS-100 English-Russian (English-Russian). For the German-English dataset, the training set contains approximately 160,000 samples, and the validation and test sets contain approximately 7,000 samples. We also preprocessed the encoding with 10,000 token bytes (BPE). For the three OPUS-100 datasets, the training set contains approximately 1 million samples, and the validation and test sets contain approximately 2,000 samples. We also preprocessed the encoding with 40,000 token bytes and shared the same vocabulary for both the source and target languages. The translation task was implemented using a Transformer model with a 6-layer encoder and a 6-layer decoder. All word embedding methods used the same model configuration, differing only in the word embedding layers. We used BLEU to evaluate the model's translation performance; a higher BLEU score indicates better performance. All word embedding compression methods were compared under the condition of compressing the Original word embedding parameters by approximately 20 times.

[0045] Table 1. Experimental results of the translation task

[0046] method German-English English-Italian English-Spanish English-Russian Original 34.5 32.9 39.1 31.6 Matrix Factor. 32.7 31.0 38.0 28.9 Tensor Train 34.3 32.4 38.7 31.2 Word2ketXs 34.3 32.6 38.4 31.5 Word2ket 34.2 32.3 39.1 31.3 Our method 34.9 32.9 39.1 31.9

[0047] As shown in Table 1, under a compression ratio of approximately 20x, our method maintains the performance of the original uncompressed word embedding method on four different translation datasets. Furthermore, it achieves higher BLEU scores than the original uncompressed word embedding method on the German-English and English-Russian datasets, respectively. However, other word embedding methods based on matrix factorization, tensor train decomposition, and rank decomposition-based Word2ket and Word2ketXs methods almost entirely fail to maintain the translation performance of the original word embeddings.

[0048] (2) For the question-answering task, we use the WikiQA dataset, a question-answering dataset based on Wikipedia retrieval. It contains questions and corresponding candidate answers, with 0 and 1 indicating whether the answer and question match. Its training set contains approximately 20,400 samples, the validation set approximately 2,700 samples, and the test set approximately 6,200 samples. Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) are used as evaluation metrics for this task. For the natural language inference task, we use the SNLI benchmark dataset, which contains 50,000 pairs of sentences with four types of manually annotated labels. Accuracy is used as the evaluation metric for this task. We use the RE2 model to implement both the question-answering and natural language inference tasks. For the WikiQA dataset, all word embedding compression methods compress word embedding parameters by approximately 80 times. For the SNLI dataset, all word embedding compression methods compress word embedding parameters by approximately 38 times. The Word2ket method is an exception, as it cannot achieve the aforementioned compression ratio.

[0049] Table 2. Experimental results of question answering and natural language reasoning tasks.

[0050]

[0051] As shown in Table 2, for the question-answering task based on the WikiQA dataset, our method achieves better results than the original uncompressed word embeddings with a compression factor of approximately 80, and outperforms other word embedding compression methods under the same conditions, even matching the Word2ket method with a compression factor of 21. For the natural language inference task based on the SNLI dataset, our method achieves better results than the original uncompressed word embeddings and other compressed word embedding methods with a compression factor of approximately 38.

[0052] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate and explain the technical solutions of this invention. The specific embodiments described above are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.

Claims

1. A tensor-based word embedding compression system based on morphological enhancement, the model comprising a morpheme segmentation module, a morpheme indexing and embedding module, and a word embedding generation module; characterized in that: The morpheme segmentation module will use the vocabulary of the text task. Each word in the text is segmented into Morphemes, and construct a morpheme table ; The morpheme indexing and embedding module is based on the morpheme table. and word list Construction dimension morpheme index matrix Each row of the morpheme index matrix represents the morpheme of the corresponding word in the morpheme table. The position in; where: It is a vocabulary list Size; The morpheme indexing and embedding module is based on the morpheme table. Build There are trainable dimensions. morpheme embedding matrix Each row vector of the morpheme embedding matrix represents the corresponding morpheme in the morpheme table. dimensional embedding vector; where: It is a morpheme table Size, ; The word embedding generation module is based on the morpheme index matrix. index vector and the morpheme embedding matrix The word embedding vector model is constructed by performing tensor product operations on the morpheme vectors indexed from the source. The word embedding generation module constructs the word embedding vector model as follows: vocabulary The first in Each word, first from the morpheme index matrix Find the morpheme index vector of the word ,Right now ; Based on the morpheme index vector from the single morpheme embedding matrix Index A vector of morphemes is obtained by performing a tensor product operation on the vector of morphemes. A dimensional vector; from The morpheme embedding matrix obtained The word is obtained by adding the dimensional vectors. 3D word vector generation word embedding vector model: 。 2. The tensor-based word embedding compression system based on morphological enhancement according to claim 1, characterized in that: The morpheme segmentation module constructs a morpheme table. process: The morpheme segmentation module statistically analyzes the different words contained in the text data to form a vocabulary. Morphological morpheme segmentation is used to segment each word in the vocabulary; where: A by Words composed of individual morphemes will be divided into a sequence of morphemes. ; vocabulary The morpheme sequences of different words in the text are post-processed to the same fixed length. : (1) Regarding the number of morphemes less than Fill in the blanks with filler characters; (2) Regarding the number of morphemes Exceed The word, spelled out The morpheme and all the morphemes following it ( ) is a morpheme; The morpheme segmentation module statistically analyzes the results of the above morpheme segmentation and post-processing, and constructs a morpheme table based on the different morphemes generated. .

Images