Vocabulary sparse distributed coding method based on word embedding

Abstract

The invention discloses a word embedding-based sparse distributed coding method for vocabularies, which relates to the field of computer science and natural language processing and comprises a quantile conversion module, a nonlinear mapping module and a binarization processing module. The word vector sparse distributed coding method is characterized in that the word vector sparse distributed coding method and a specific structure form thereof are realized through connection and interaction among a quantile conversion module, a nonlinear mapping module and a binarization processing module; according to the method, the calculation complexity and the storage cost are reduced, the high-dimensional dense continuous word vector is converted into the sparse binary vector, the method is excellent in the aspects of reducing the calculation complexity and the storage cost, resources needed for calculating the similarity or carrying out other mathematical operations are greatly reduced, and therefore the processing efficiency is improved; the method is suitable for various machine learning models, keeps semantic information of word vectors, reduces coding precision loss, is wide in applicability, and can be widely applied to different corpora and application scenes.

Description

Technical Field

[0001] The present invention relates to the fields of computer science and natural language processing (NLP), and in particular to a vocabulary sparse distributed coding method based on word embedding. Background Art

[0002] As is known, word embedding technology is widely used in the field of natural language processing (NLP) to represent and process text data. Currently, commonly used word embedding technologies include Word2Vec, GloVe and FastText, which provide important tools for the field of NLP. Word2Vec is based on the continuous bag of words model (CBOW) and the skip-gram model, which can effectively capture the semantic relationship between words and perform well in natural language processing tasks. GloVe uses the global word co-occurrence matrix to generate word vectors, accurately reflects the semantic relationship between words, and provides a more comprehensive expression for the overall structure of the corpus. FastText uses a collection of character n-grams to represent words, processes unregistered words and the internal structure of words, and enhances the expressive power of word vectors. In addition to these mainstream word embedding technologies, some new word embedding methods have emerged in recent years. For example, ELMo generates context-related word vectors by performing bidirectional LSTM modeling on the context of the entire sentence, which better captures the changes in the meaning of words in different contexts. BERT uses a bidirectional Transformer architecture to pre-train text and generate context-related word vectors, which not only captures the semantic relationship between words, but also understands the structure and context information of the sentence.

[0003] Although these word embedding technologies perform well in semantic representation, they also have some limitations. One of the main problems is that it is difficult to directly adapt to specific format requirements. For example, when word vectors need to be applied to binary input scenarios such as HTM models, the generated continuous value vectors are difficult to apply directly. In addition, word vectors generated based on word embedding technology usually have high dimensions and density, which leads to increased computing and storage costs. When processing large-scale corpora, the computational complexity is also high and requires a lot of computing resources.

[0004] Therefore, a new encoding method is needed at this stage to convert these dense continuous word vectors into sparse binary vectors. This method can not only retain the semantic information of the word vectors, but also reduce the computational complexity and improve the processing efficiency to adapt to machine learning models with specific requirements such as HTM models.

[0005] In addition, this encoding method also has certain application potential. It can be used for efficient storage and processing in neural networks, solve large-scale text classification and clustering problems, and be applied to large-scale analysis scenarios such as gene sequence analysis in bioinformatics that require improved computing efficiency. In practical applications, combining these excellent word embedding technologies with encoding methods suitable for specific scenarios will more effectively meet the needs of various natural language processing tasks. Therefore, we propose a lexical sparse distributed encoding method based on word embedding. Summary of the invention

[0006] In order to solve the problems existing in the prior art, the purpose of the present invention is to provide a vocabulary sparse distributed coding method based on word embedding.

[0007] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0008] A vocabulary sparse distributed encoding method based on word embedding includes the following steps:

[0009] Step 1: Get the word vectors pre-trained by the word embedding model

[0010] The word embedding models pre-trained using Word2Vec, GloVe or FastText technology can map words or phrases in the text into a high-dimensional vector space. These word embedding models are trained on a large corpus to capture the semantic relationship and contextual information between words, providing a strong foundation for subsequent natural language processing tasks. The word vectors of the required vocabulary can be obtained through the existing pre-trained word vector library.

[0011] Step 2: Based on non-parametric quantile transformation method

[0012] Quantile transformation is a non-parametric data transformation method that maps data to the quantiles of a specified distribution (such as a normal distribution). The quantile transformation method is used to standardize word vectors. The mapping formula used inside Quantile Transformer involves multiple steps, including calculating the quantiles corresponding to the original data, and then applying the inverse cumulative distribution function of the normal distribution to map the quantiles to the quantiles of the normal distribution.

[0013] Compute the quantiles of the original data:

[0014] The cumulative distribution function F(x) is the probability that the value at point x is less than or equal to x. In the estimation process, the empirical cumulative distribution function (ECDF) can be used to approximate the true CDF. The ECDF is used to calculate the quantile of the original data. For a given data point x, i ∈X, its quantile F X (x i) can be calculated through ECDF, and the formula is:

[0015]

[0016] Among them, I(x i ≤x) is the indicator function, when x i 1 if ≤x, 0 otherwise.

[0017] Map the original data quantiles to standard normal distribution quantiles:

[0018] The original data quantile F X (x i ) is mapped to the quantile formula of the target distribution N(0,1):

[0019] z i =F -1 (F X (x i ))

[0020]

[0021] Where F(x) is the cumulative distribution function of the standard normal distribution, F -1 It is the inverse function of the cumulative distribution function F(x) of the standard normal distribution.

[0022] Randomly extract the word vector "inner requirements" from the pre-trained word vector library and process them using the above-mentioned quantile transformation method. The Shapiro-Wilk normality test shows that the original data does not conform to the normal distribution. After the non-parametric quantile transformation, the word vector data distribution conforms to the normal distribution, and the data distribution range is wider. This processing method is conducive to the subsequent nonlinear mapping step;

[0023] Step 3: Nonlinear mapping method based on sigmoid function

[0024] After completing the quantile transformation and obtaining the word vectors of the standard normal distribution, we need to perform nonlinear mapping on these word vectors of the standard normal distribution. The sigmoid function is used for nonlinear mapping. The definition of the sigmoid function is as follows:

[0025]

[0026] After the word vector "inner requirements" are quantile-transformed, they are processed using the above-mentioned nonlinear mapping method. After nonlinear mapping based on the Sigmoid function, the word vector data distribution is approximately normal, showing a distribution form with more in the middle and less at both ends, which is similar to the original word vector data distribution. Therefore, the nonlinear mapping basically retains the form of data distribution. In addition, the value range of the mapped data is between (0,1), which is helpful for the subsequent averaging operation and will not be affected by negative data.

[0027] Step 4: Binarization method based on dynamic threshold

[0028] Calculate the dynamic threshold of the mapped word vector and binarize it to generate a sparse binary vector (i.e., a 0-1 vector). The specific method is to compare the mapped word vector value with its mean, set the value higher than or equal to the mean to 1, and set the value lower than the mean to 0, thereby generating a sparse binary vector. The calculation formula is as follows:

[0029]

[0030] Where B(y i ) is the 0-1 sparse coding vector obtained after scaling by Sigmoid function and binarization. Sigmoid(x i ) is a nonlinear scaling of the code after quantile conversion, mean(Sigmoid(x i )) is the averaged processing after nonlinear scaling.

[0031] Due to the adoption of the above technical solution, the present invention has the following beneficial effects:

[0032] 1. Reduce computational complexity and storage cost: By converting high-dimensional, dense continuous word vectors into sparse binary vectors, the present invention performs well in reducing computational complexity and storage cost, greatly reducing the resources required for calculating similarity or performing other mathematical operations, thereby improving processing efficiency;

[0033] 2. Adaptability to various machine learning models: The encoding method of the present invention is particularly suitable for machine learning models that require binary input, such as the Hierarchical Temporal Memory (HTM) model. At the same time, the word vector after nonlinear mapping and binarization is also suitable for other models and application scenarios that require sparse representation;

[0034] 3. Maintaining the semantic information of word vectors: The present invention can effectively maintain the semantic information of word vectors through quantile transformation and nonlinear mapping. The experimental results show that the Pearson correlation coefficient calculated by the proposed encoding method and the standard value of the word vector in the benchmark set is high and the correlation is significant, indicating that the method is effective in maintaining the semantic information of word vectors;

[0035] 4. Reduce coding accuracy loss: On different benchmark sets, the Pearson correlation coefficient of the proposed coding method is close to the calculation result of the original word vector, indicating that the method loses less accuracy during the coding process. Compared with the traditional semantic folding method, the proposed coding method has greater advantages in maintaining coding accuracy;

[0036] 5. Wide applicability: The encoding method of the present invention is not only applicable to Chinese and English word vectors, but can also be extended to other languages ​​and fields. By using a pre-trained word vector library, combined with quantile transformation and nonlinear mapping, the encoding method of the present invention can be widely applied to different corpora and application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the flow structure of the vocabulary sparse distributed coding method based on word embedding of the present invention;

[0038] Figure 2 A schematic diagram of the distribution of the word vector "inner requirements" of the present invention based on non-parametric quantile transformation;

[0039] Figure 3 This is a schematic diagram of the nonlinear mapping distribution of the word vector "inner requirements" of the present invention based on the sigmoid function; DETAILED DESCRIPTION

[0040] The present invention can be explained in detail by the following examples, and the purpose of disclosing the present invention is to protect all technical improvements within the scope of the present invention.

[0041] Combined with Figures 1 to 3 The word embedding-based vocabulary sparse distributed coding method includes a quantile conversion module, a nonlinear mapping module and a binarization processing module to implement the word vector sparse distributed coding method and its specific structural form.

[0042] Quantile transformation module: Use the pre-trained word vector library to obtain the word vector of the target word, and use the non-parametric quantile transformation method to map the data distribution of the original word vector to the target distribution (such as normal distribution).

[0043] The specific implementation of this module involves calculating the empirical cumulative distribution function (ECDF) of each data point in the word vector and mapping its quantile position to the same quantile position of the target distribution.

[0044] Non-linear mapping module: A non-linear activation function is applied to the word vector after quantile transformation to map the data value of the word vector to the (0,1) interval. The specific implementation of this module includes non-linear mapping using the Sigmoid function, and its formula is: Make sure that the word vector values ​​after mapping are between (0,1).

[0045] Binarization processing module: calculates the dynamic threshold of the word vector after nonlinear mapping, and binarizes the word vector to generate a sparse binary vector. The specific implementation method of this module includes calculating the mean of the word vector as the dynamic threshold, setting the value above or equal to the mean to 1, and setting the value below the mean to 0, to generate a 0-1 sparse code.

[0046] Connection and interaction between modules. The quantile transformation module outputs the standardized word vector and inputs it into the nonlinear mapping module. The nonlinear mapping module performs nonlinear mapping on the standardized word vector to generate data in the (0,1) interval and passes it to the binarization processing module. The binarization processing module binarizes the mapped word vector according to the dynamic threshold to generate the final sparse binary vector.

[0047] The specific structural forms of the quantile conversion module, nonlinear mapping module and binarization processing module to realize word vector sparse distributed coding are as follows: the specific structure of the quantile conversion module includes a data input interface, a quantile calculation unit and a mapping unit; the specific structure of the nonlinear mapping module includes a standardized data receiving unit, a nonlinear activation function unit and a mapping output interface; the specific structure of the binarization processing module includes a mapping data receiving unit, a dynamic threshold calculation unit and a binarization processing unit.

[0048] The implementation steps are as follows: the first step is to obtain word vectors generated by training based on word embedding technology, and use existing word embedding technology (such as Word2Vec, GloVe or FastText) to train and generate word vectors. For example, Tencent AI Laboratory combined the training technology of Word2Vec and the sub-word feature processing technology of FastText to train and obtain a word vector library containing more than 10 million Chinese words, and the dimension of each word vector is 200 dimensions; another example is based on FastText word embedding technology, using Common Crawl (containing 600 billion tags) text data for training, and obtained word vectors for 2 million English words, and the dimension of each word vector is 300 dimensions; these pre-trained word vector libraries provide high-quality word vector representations for various natural language processing tasks.

[0049] The second step is to perform quantile transformation on the obtained word vectors based on the non-parametric quantile transformation method. The purpose of quantile transformation is to map the word vector data to the quantile of the target distribution (such as normal distribution), thereby expanding the distribution range of the data. This step is conducive to subsequent nonlinear mapping, making data processing more efficient.

[0050] The third step is to use the nonlinear mapping method based on the sigmoid function to apply a nonlinear activation function (such as the sigmoid function) to perform nonlinear mapping on the standardized word vector. The sigmoid function is a commonly used nonlinear activation function, and its formula is Able to map input data to the (0,1) interval.

[0051] The fourth step is to calculate the dynamic threshold of the mapped word vector based on the binarization method, and binarize it to generate a sparse binary vector (i.e., 0-1 vector). The specific method is to compare the mapped word vector value with its mean, set the value higher than or equal to the mean to 1, and set the value lower than the mean to 0, thereby generating a sparse binary vector.

[0052] According to the whole process of the above-mentioned word embedding-based lexical sparse distributed coding method, a flowchart of the coding method is drawn, see the attached Figure 1 , in order to intuitively show the various steps and transformations of the method

[0053] In Example 1, a lexical sparse distributed coding method based on word embedding is described. In order to evaluate the effect of the proposed coding method, this paper uses multiple benchmark data sets for testing, including WordSim353 (English benchmark set), Words-240 (Chinese benchmark set) and Geo-Terminology Relatedness Dataset (GTRD). The evaluation process is as follows:

[0054] First, this paper uses the pre-trained word vector to calculate the cosine similarity of the words contained in the benchmark set as a control. In addition, the proposed encoding method and semantic folding method are used to encode the corresponding words in the benchmark set, and the cosine similarity of the encoded corresponding words is calculated respectively. Then, according to the cosine similarity calculated by the proposed encoding method and semantic folding method and the benchmark value of the corresponding words in the benchmark set, the Pearson correlation coefficient is calculated to evaluate the encoding effect of the proposed method. At the same time, by comparing the size of the Pearson correlation coefficient after encoding using the proposed encoding method, the advantages and disadvantages of the encoding method are further evaluated. Finally, in order to determine whether the correlation is statistically significant, the Pearson correlation coefficient is tested for P value. The calculation formulas of cosine similarity, Pearson correlation coefficient and P value are as follows:

[0055] Cosine similarity calculation:

[0056] The proposed encoding method is used to calculate the cosine similarity of the benchmark words contained in the WordSim353 benchmark set (English benchmark set), the Words-240 benchmark set (Chinese benchmark set) and the Geographical Term Similarity Dataset (GTRD). The above benchmark sets all contain two columns of benchmark terms. Let the encoding vector corresponding to the first column of terms be Let the encoding vector corresponding to the term in the second column be The corresponding cosine similarity calculation formula is:

[0057]

[0058] Pearson correlation coefficient calculation:

[0059] Let α i The cosine similarity of the terms in the benchmark set is calculated for the proposed encoding method, β j The benchmark value corresponding to the term in the benchmark set, P ij is the calculated Pearson correlation coefficient.

[0060]

[0061] P value calculation:

[0062] Let n be the number of corresponding test samples, T p is the test statistic corresponding to the Pearson correlation coefficient, P1 is the P value corresponding to the two-sided test of the Pearson correlation coefficient, t(T p ) is the cumulative probability value of the t distribution, and the corresponding Pearson correlation coefficient test statistic and P value calculation formula are as follows:

[0063]

[0064] P1=2*(1-t(T p ))

[0065] The WordSim353 benchmark set (English benchmark set), Words-240 benchmark set (Chinese benchmark set) and Geographical Term Similarity Dataset (GTRD) are used as benchmark sets. The similarity of the benchmark words calculated by the proposed encoding method and semantic folding method is compared with the benchmark value of the words in the benchmark set to obtain the Pearson correlation coefficient. At the same time, by comparing the Pearson correlation coefficient of the original word embedding technology encoding method with the proposed encoding method, the advantages and disadvantages of the encoding method's accuracy loss are further evaluated. In addition, a statistical test is performed on the correlation coefficient, and the significance of the correlation coefficient is judged based on whether the P value is greater than 0.05. The following are the experimental results:

[0066] Table 1 Correlation coefficients and P-value test table between the original word embedding technology encoding method, the proposed encoding method, the semantic folding method and the benchmark set

[0067]

[0068] The experimental results show that the Pearson correlation coefficients of the proposed encoding method and the word embedding technology encoding method on multiple benchmark sets are higher than those of the semantic folding method, and all P values ​​are less than 0.05, indicating a significant correlation. The proposed encoding method performs better than the semantic folding method on all benchmark sets. Compared with the original word embedding technology encoding method, the Pearson correlation coefficients of the proposed encoding method on different benchmark sets are close, and the encoding accuracy loss is less. Therefore, the proposed encoding method has significant advantages in improving the accuracy of word vector similarity calculation.

[0069] In Example 2, the sparse distributed coding method based on word embedding randomly extracts the word vector "inner requirements" from the pre-trained word vector library, and processes it using the above-mentioned quantile conversion method, combined with the attached Figure 2 As shown in the figure, the Shapiro-Wilk normality test shows that the original data does not conform to the normal distribution. After the non-parametric quantile transformation, the word vector data distribution conforms to the normal distribution, and the data distribution range is wider.

[0070] Embodiment 3, a vocabulary sparse distributed coding method based on word embedding, combined with the attached Figure 3 As shown in the figure, after nonlinear mapping based on the Sigmoid function, the word vector data distribution is approximately normal, showing a distribution form with more in the middle and less at both ends, which is similar to the original word vector data distribution.

[0071] Therefore, the nonlinear mapping basically retains the form of data distribution. In addition, the value range of the mapped data is between (0,1), which is helpful for the subsequent averaging operation and will not be affected by negative data.

[0072] The parts of the present invention that are not described in detail are prior art. Although the present invention is specifically demonstrated and introduced in conjunction with the preferred implementation scheme, there are many methods and approaches to specifically implement the technical solution. The above is only a preferred implementation scheme of the present invention, but technical personnel in the relevant field should understand that various changes can be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined in the attached claims, and all of them are within the scope of protection of the present invention.

Claims

1. A lexical sparse distributed coding method based on word embedding, comprising a quantile conversion module, a nonlinear mapping module and a binarization processing module, characterized in that: The connection and interaction between the quantile conversion module, the nonlinear mapping module and the binarization processing module are used to implement the method of sparse distributed coding of word vectors and its specific structural form.

2. The method for sparse distributed coding of words based on word embedding according to claim 1, characterized in that: The quantile conversion module uses a pre-trained word vector library to obtain the word vector of the target word, and uses a non-parametric quantile conversion method to map the data distribution of the original word vector to the target distribution, such as a normal distribution. The specific implementation of this module includes calculating the empirical cumulative distribution function (ECDF) of each data point in the word vector and mapping its quantile position to the same quantile position of the target distribution.

3. The method for sparse distributed coding of words based on word embedding according to claim 2, characterized in that: The nonlinear mapping module applies a nonlinear activation function to the word vector after quantile conversion to map the data value of the word vector to the (0,1) interval. The specific implementation of this module includes using the Sigmoid function for nonlinear mapping, and its formula is: Make sure that the word vector values ​​after mapping are between (0,1).

4. The method for sparse distributed coding of words based on word embedding according to claim 3, characterized in that: The binarization processing module calculates the dynamic threshold of the word vector after nonlinear mapping, and binarizes the word vector to generate a sparse binary vector. The specific implementation method of this module includes calculating the mean of the word vector as the dynamic threshold, setting the value above or equal to the mean to 1, and setting the value below the mean to 0, to generate a 0-1 sparse code.

5. The method for sparse distributed coding of words based on word embedding according to claim 1, characterized in that: The connection and interaction between the quantile conversion module, the nonlinear mapping module and the binarization processing module are that the quantile conversion module outputs the standardized word vector, and inputs it into the nonlinear mapping module. The nonlinear mapping module performs nonlinear mapping on the standardized word vector to generate data in the (0,1) interval, and passes it to the binarization processing module. The binarization processing module performs binarization on the mapped word vector according to a dynamic threshold to generate a final sparse binary vector.

6. The method for sparse distributed coding of words based on word embedding according to claim 1, characterized in that: The specific structural forms of the quantile conversion module, nonlinear mapping module and binarization processing module to realize word vector sparse distributed coding are as follows: the specific structure of the quantile conversion module includes a data input interface, a quantile calculation unit and a mapping unit, the specific structure of the nonlinear mapping module includes a standardized data receiving unit, a nonlinear activation function unit and a mapping output interface, and the specific structure of the binarization processing module includes a mapping data receiving unit, a dynamic threshold calculation unit and a binarization processing unit.

7. The method for sparse distributed coding of words based on word embedding according to claim 1, characterized in that: The following steps are involved: Step 1: Get the word vectors pre-trained by the word embedding model The word embedding models pre-trained using Word2Vec, GloVe or FastText technology can map words or phrases in the text into a high-dimensional vector space. These word embedding models are trained on a large corpus to capture the semantic relationship and contextual information between words, providing a strong foundation for subsequent natural language processing tasks. The word vectors of the required vocabulary can be obtained through the existing pre-trained word vector library. Step 2: Based on non-parametric quantile transformation method Quantile transformation is a non-parametric data transformation method that maps data to the quantiles of a specified distribution (such as a normal distribution). The quantile transformation method is used to standardize word vectors. The mapping formula used inside Quantile Transformer involves multiple steps, including calculating the quantiles corresponding to the original data, and then applying the inverse cumulative distribution function of the normal distribution to map the quantiles to the quantiles of the normal distribution. Compute the quantiles of the original data: The cumulative distribution function F(x) is the probability that the value at point x is less than or equal to x. In the estimation process, the empirical cumulative distribution function (ECDF) can be used to approximate the true CDF. The ECDF is used to calculate the quantile of the original data. For a given data point x, i ∈X, its quantile F X (x i ) can be calculated through ECDF, and the formula is: Among them, I(x i ≤x) is the indicator function, when x i 1 if ≤x, 0 otherwise. Map the original data quantiles to standard normal distribution quantiles: The original data quantile F X (x i ) is mapped to the quantile formula of the target distribution N(0,1): z i =F -1 (F X (x i )) Where F(x) is the cumulative distribution function of the standard normal distribution, F -1 It is the inverse function of the cumulative distribution function F(x) of the standard normal distribution. Randomly extract the word vector "inner requirements" from the pre-trained word vector library and process them using the above-mentioned quantile transformation method. The Shapiro-Wilk normality test shows that the original data does not conform to the normal distribution. After the non-parametric quantile transformation, the word vector data distribution conforms to the normal distribution, and the data distribution range is wider. This processing method is conducive to the subsequent nonlinear mapping step; Step 3: Nonlinear mapping method based on sigmoid function After completing the quantile transformation and obtaining the word vectors of the standard normal distribution, we need to perform nonlinear mapping on these word vectors of the standard normal distribution. The sigmoid function is used for nonlinear mapping. The definition of the sigmoid function is as follows: After the word vector "Inner Requirement" is quantile-transformed, it is processed using the above nonlinear mapping method. After nonlinear mapping based on the Sigmoid function, the word vector data distribution is approximately normal, showing a distribution form with more in the middle and less at both ends, which is similar to the original word vector data distribution. Therefore, the nonlinear mapping basically retains the form of data distribution. In addition, the value range of the mapped data is between (0,1), which is helpful for the subsequent averaging operation and will not be affected by negative data. Step 4: Binarization method based on dynamic threshold Calculate the dynamic threshold of the mapped word vector and binarize it to generate a sparse binary vector (i.e., a 0-1 vector). The specific method is to compare the mapped word vector value with its mean, set the value higher than or equal to the mean to 1, and set the value lower than the mean to 0, thereby generating a sparse binary vector. The calculation formula is as follows: Where B(y i ) is the 0-1 sparse coding vector obtained after scaling by Sigmoid function and binarization. Sigmoid(x i ) is a nonlinear scaling of the code after quantile conversion, mean(Sigmoid(x i )) is the averaged processing after nonlinear scaling.

8. The method for sparse distributed coding of words based on word embedding according to claim 7, characterized in that: The word embedding model is responsible for generating initial high-dimensional word vectors.

9. A vocabulary sparse distributed coding method based on word embedding according to any one of claims 1 to 7, characterized in that: The quantile conversion module is used to standardize word vectors, and the nonlinear activation function module performs nonlinear mapping on the standardized word vectors.

10. A vocabulary sparse distributed coding method based on word embedding according to any one of claims 1 to 7, characterized in that: The binarization processing module calculates a dynamic threshold and converts a word vector into a sparse binary vector.

Images