Graph-based word embedding optimization method

Abstract

The invention discloses a word embedding optimization method based on a graph. The word embedding optimization method comprises the steps of obtaining a word co-occurrence graph based on a large-scale corpus; wherein the word co-occurrence graph is a directed weighted word graph; obtaining a transition probability between nodes based on the word co-occurrence graph and extracting importance scores of the nodes; based on the transition probability between the nodes and the importance score of the nodes, obtaining a plurality of word sequences through random walk sampling to construct a sampling corpus; and applying the character skipping model to a sampling corpus to obtain a final word vector. The invention provides a graph-based word embedding optimization method, which comprises the following steps of: converting a large corpus into a word co-occurrence graph, randomly extracting word sequence samples, and training word vectors on a sampling corpus; the method has stable operation time on a large-scale data set, and along with the increase of a training corpus, the performance advantages of the method become more and more obvious.

Description

Technical Field

[0001] The present invention belongs to the field of modern natural language processing technology, and in particular relates to a graph-based word embedding optimization method. Background Art

[0002] One of the most prominent approaches to word-level representation is Word2vec. To date, Word2vec has been widely established for its effectiveness in achieving state-of-the-art performance on various clinical NLP tasks. GloVe is another unsupervised learning method for obtaining vector representations of individual words. Unlike Word2vec, GloVe is a statistical model that combines global matrix factorization and local context windows. Learning relies on dimensionality reduction of the co-occurrence count matrix based on the frequency of a word's appearance in a context. FastText is also an established word representation library. Unlike Word2vec and GloVe, FastText treats individual words as character n-grams. Words actually have different meanings in different contexts, and the vector representations of words in these two models are the same in different contexts. ELMO is also optimized for this purpose. ELMO can learn complex features of word usage and how these complex usages vary across different contexts. However, the structure of these pre-trained models is limited by the unidirectional language model (left-to-right or right-to-left), which also limits the model's representation capabilities and allows it to only capture unidirectional contextual information. BERT uses a masked language model (MLM) and a context-predicted language model (NSP) for pre-training, and uses a deep bidirectional transformer component to build the entire model. As a result, it ultimately generates a deep bilingual representation that can integrate left and right context information.

[0003] The above methods have achieved satisfactory speed and accuracy on terabyte-sized corpora. However, when encountering large-scale corpora with petabytes of size, the training time cost will increase significantly.

[0004] In graph analysis, traditional machine learning methods often rely on manual design and are limited by flexibility and high cost. Building on the idea of ​​representation learning and the success of Word2vec, DeepWalk, as the first graph vector method based on representation learning, applies the Skip-Gram model to generate random walks. Similarly, inspired by DeepWalk, Node2vec improves the random walk pattern in DeepWalk. Taking into account the difference between the linear structure of text and the complex structure of graphs, Node2vec introduces a heuristic method - the second-order random walk, which uses two hyperparameters p and q and a weight parameter α to control the walking strategy. Therefore, the generated sequence is a combination of DFS (depth-first search) and BFS (breadth-first search).

[0005] Modeling text as a network of word connections (word graphs) is a promising area of ​​exploration in linguistics. Word graphs can seamlessly integrate grammatical, semantic, and other information from textual language into the complex structure of the graph. The advantage of studying word graphs rather than entire texts is that they can easily represent the nonlinear and non-hierarchical structure of language.

[0006] The aforementioned graph vector techniques have been investigated and used to sample word sequences on word graphs, but these methods do not take the weights of nodes into account during the sampling process. Summary of the Invention

[0007] To solve the above technical problems, the present invention proposes a graph-based word embedding optimization method to solve the problems existing in the above-mentioned prior art.

[0008] To achieve the above objectives, the present invention provides a graph-based word embedding optimization method, comprising:

[0009] Obtaining a word co-occurrence graph based on a large-scale corpus; wherein the word co-occurrence graph is a directed weighted word graph;

[0010] Based on the word co-occurrence graph, the transition probability between nodes is obtained and the importance score of the node is extracted; based on the transition probability between nodes and the importance score of the node, a number of word sequences are obtained through random walk sampling to construct a sampling corpus;

[0011] The skip-gram model is applied to the sampled corpus to obtain the final word vector.

[0012] Optionally, the vertices of the word co-occurrence graph represent unique words, the edges represent the co-occurrence between words, the edge direction represents the word order, and the edge weights and node weights are marked.

[0013] Optionally, the number of times the words at both ends of the directed edge appear in the text is used as the weight value of the edge and stored in a weighted adjacency matrix.

[0014] Optionally, the node weight is an importance score of the corresponding word in the large-scale corpus; the importance score is calculated as follows:

[0015]

[0016] Among them, PW v is the importance score of word v, n v is the number of occurrences of word v, ∑ k n k is the number of occurrences of all words, To compute the inverse document frequency score, |C| is the number of documents in the corpus, d is the number of documents in the corpus, and |{d:v∈d}| is the number of documents containing v.

[0017] Optionally, the transition probability is calculated as follows:

[0018] π vx =α pq (t,x)×W vx ;

[0019] Among them, W vx is the weight of edge (v, x)∈e, α is the path bias parameter, p is the return probability, and q is the range parameter.

[0020] Optionally, the process of obtaining a plurality of word sequences includes:

[0021] A probabilistic sampling method based on node importance scores is used to select the root node. The second-order random walk of the Node2vec model is used to guide the walk using preset return hyperparameters and internal and external hyperparameters. The next node to walk is selected according to the adjacency matrix of the state transition probability to obtain several word sequences.

[0022] Optionally, the adjacency matrix of state transition probabilities is calculated as follows:

[0023]

[0024] Among them, c i represents the i-th node in the walk, Z is a normalization constant, and E is the set of edges.

[0025] The present invention also discloses a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned graph-based word embedding optimization method.

[0026] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned graph-based word embedding optimization method are implemented.

[0027] The present invention also discloses a computer program product, comprising a computer program, which implements the steps of the above-mentioned graph-based word embedding optimization method when executed by a processor.

[0028] Compared with the prior art, the present invention has the following advantages and technical effects:

[0029] This paper proposes a graph-based word embedding optimization method that converts a large corpus into a word co-occurrence graph, randomly samples word sequences, and trains the word embeddings on the sampled corpus. Because English vocabulary is stable and relatively fixed in terms of idioms and expressions, the size and density of the word co-occurrence graph do not change significantly as the training corpus grows. As a result, Word-Graph2vec has a stable runtime on large datasets, and its performance advantage becomes increasingly apparent as the training corpus grows. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0031] Figure 1 is an example diagram of a directed weighted graph G, an adjacency matrix W, and a node weight vector PW according to an embodiment of the present invention;

[0032] Figure 2 This is a flowchart of Word-Graph2vec according to an embodiment of the present invention;

[0033] Figure 3 Schematic diagram of the process of generating a word co-occurrence graph according to an embodiment of the present invention, (a) is a schematic diagram of directed edges, and (b) is a weighted directed graph;

[0034] Figure 4 This is an example diagram of a random walk process in a directed graph according to an embodiment of the present invention;

[0035] Figure 5 This is a diagram showing the effect of selecting various parameters for walking length according to an embodiment of the present invention;

[0036] Figure 6 Schematic diagram of time and corpus size changes in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] Example 1

[0040] like Figure 1-6As shown, this embodiment provides a graph-based word embedding optimization method, including:

[0041] A word co-occurrence graph is extracted from a large-scale corpus, where each node is a word, edges represent the co-occurrence of a word with adjacent nodes, and the direction of the edge represents the word order. Then, all words in the graph (words and nodes have a one-to-one correspondence) are mapped to a low-dimensional vector space, where each word is represented as a dense vector space. The local relations and deep connections between words have semantic connections in the vector space. Formally, this problem can be defined as:

[0042] Input: A preprocessed monolingual corpus with billions of words and hundreds of thousands of words in the vocabulary.

[0043] Output: A global representation matrix Φ∈R |V|×d , its Φ i (Line i) is a representation of word V i A d-dimensional vector.

[0044] The word co-occurrence graph is defined as follows. Let G = (V, E) be a directed weighted word graph, where V represents the set of nodes, Defines the edges between nodes. Figure 1 As shown, each edge has a non-negative weight W ij , used to describe the strength of the connection between nodes; if two nodes are disconnected, the edge weight W ij Set to zero, therefore, using the matrix W = [W ij ]∈R |V|×|V| Represents all the weights between nodes, and the vector PW = [PW i ]∈R 1×|V| Represented as the weight of each node. Figure 1 An example of a word co-occurrence map is shown, which contains four words (a, b, c, and d).

[0045] The symbols in this embodiment are defined as follows:

[0046] Table 1

[0047]

[0048]

[0049] Word-Graph2vec Algorithm

[0050] (1) Motivation

[0051] Word graphs extracted from text have been successfully applied to natural language processing (NLP) tasks such as automatic summarization, information retrieval, and text classification. The impact of word order on sentence meaning has long been a hot topic, and the relationships between words are generally considered to play a significant role in text processing. For example, the sentence "Lily is prettier than Lucy" is completely different from "Lucy is prettier than Lily." Therefore, this example uses a word co-occurrence graph representation to capture these word relationships, rather than the sliding window of traditional word vector methods.

[0052] Training word vectors on word co-occurrence graphs is an effective method. First, the number of nodes in this word co-occurrence graph is not large. At the same time, even for the large En-wikipedia dataset (see Table 1), the total number of word graph nodes is about 100,723. Therefore, these graph node vector methods, such as DeepWalk, Node2Vec, Line, etc., can be effectively used for these co-occurrence graphs. Secondly, these word graphs are highly sparse. For example, the density of En-wikipedia is only 0.64%. Therefore, a more effective sparse graph vector technology can be applied to these datasets. The last and most important problem is that as the size of the training corpus increases, the weights on the edges will change greatly, but the size of the nodes and the graph density do not change significantly. Therefore, this embodiment proposes a graph-based word embedding algorithm optimization method-Word-Graph2vec method to learn the entire corpus and train word vectors.

[0053] (2) Word-Graph2vec Method Framework

[0054] The Word-Graph2vec method framework diagram is as follows Figure 2 As shown in the figure, the process involves three steps. First, a word co-occurrence graph is constructed using the training corpus. Second, a random walk technique is applied to generate word sequences from the word graph. Third, the sequence set is used as a new corpus to train word vectors using the Skip-Gram algorithm in the Word2vec model. The following sections explain this in detail.

[0055] Generate word co-occurrence graph: A text document is presented as a word co-occurrence graph, which corresponds to a weighted directed graph whose vertices represent unique words, whose edges represent the co-occurrence between words, and whose edge directions represent the word order. Figure 3(a) Given a sentence from the original letter by Philip Dormer Stanhope: "In truth, whatever is worth doing at all, is worth doing well." Draw an edge between this sentence and the following word. A solid arrow indicates a new directed edge, while a dashed arrow indicates an existing edge in the graph. For clarity, these edges are created from the vertex corresponding to "do." Figure 3 (b) corresponds to the resulting weighted directed graph, where each vertex represents a unique word and each edge is the co-occurrence point of the two words.

[0056] Edge weight: The number of times the two words appear together in the text is used as the edge weight. The weighted adjacency matrix w is used to store the edge weight, W v,x is the weight from node v to node x.

[0057] W v，x =∑co-occurrence(v,x)(1)

[0058] Here, co-occurrence count (v, x) is the number of times words v and x appear together in the same order from left to right. For example, in Figure 3 In (b), the number of occurrences of the word pair "worth" and "do" is 2, so the weight of the edge from "worth" to "do" is equal to 2. Similarly, the relationship between each word in the entire sentence is obtained, and the generated word graph is as follows Figure 3 (b) shown.

[0059] Weight on the node: The weight of the node is used to determine the sampling time during the random walk. v As an expression of the importance of word v in the entire corpus, that is, the weight of the node in the graph, two types of methods are used to set the weight in this embodiment.

[0060] First, intuitively, term frequency (TF) can be applied to measure the importance score.

[0061]

[0062] Among them, PW v is the importance score of word v, n v is the number of occurrences of word v, ∑ k n k is the number of occurrences of all words. Figure 3 For example, (b), the total number of words is 20, then

[0063] Secondly, in order to minimize the weight of frequently occurring terms while giving less frequently occurring terms a higher influence, this embodiment adds the inverse document frequency (IDF) to the importance score. The equation is as follows:

[0064]

[0065] is the equation for calculating the IDF score, |C| is the number of documents in the corpus, d is the document in the corpus, and |{d:v∈d}| is the number of documents containing v. However, because the datasets used to train word vectors (Text8, 1bwordsbenchmark, and En-Wikipedia) are all long word sequences, this example uses a window size of 200 to identify different documents. Words in the same window mean that these words are in the same document; otherwise, these words appear in different documents.

[0066] Sampling word sequences by random walks: This random walk sampling process iteratively explores the global structural network of the object to estimate the proximity between two nodes. A random walk is generated from the current node, and random neighbors of the current node are selected as candidate nodes based on the transition probability on the word co-occurrence graph. For example, in Figure 3 In (b), a random walk sampling sequence is "whatever, is, worth, doing, well, without, attention", which goes from "whatever" to "attention". These random walk sampling word sequences are summarized as the training corpus for learning word vectors. There are two points worth emphasizing in this sampling process.

[0067] First, different nodes have different probabilities of being selected as the starting point, as opposed to the same probability of being selected for the graph vector. Different word importances should be used, and different probabilities should be chosen as the starting point. In other words, the probability of an important node being the root node should be higher than that of an unimportant node. Intuitively, a probability sampling method based on importance scores (see Formulas (2) and (3)) is used to select the root node.

[0068] For example, the sampling time of node v as the starting node is shown in formula (4):

[0069]

[0070] Where total_walks is the total number of random walk samples. For example, let total_walks = 200 and number_walks(doing) be set to This means there are 20 word sequences starting with the word "doing".

[0071] Illustration of a random walk in a directed graph. Suppose that during sampling, the process has just transitioned from node t to node v and is now evaluating its next step out of node v. The edge labels represent the search bias α.d tx is the “distance” from node t to node x, that is, the length of the shortest path from t to x, where d∈0,1,2; p is called the return (inward) hyperparameter, which controls the probability of immediately revisiting a node during a walk; q is the inward-outward hyperparameter, which controls the probability of exploring more distant parts of the graph.

[0072] Secondly, especially considering the shuttle between the similarities of the two graphs (homogeneity and structural equivalence), this embodiment selects Node2vec as the graph sampling method. In order to set different p and q values, Node2vec can choose different walking strategies (width-first or depth-first) to move the graph. The specific equation is as follows. For a given source node u, according to the second-order random walk of the Node2vec model, the parameters p and q are used to guide the walk: Consider a random walk that has just crossed the edge (t, v) and is now located on the node v. As Figure 4 As shown in Figure 2 (assuming that these nodes are connected in both directions), the calculation method of the search bias α in the directed graph is given. Let the transition probability between node v and node x be:

[0073] π vx =α pq (t, x)×W vx (5)

[0074] Where W vx is the weight of edge vx, α is the path bias parameter, p is the return probability, and q is the range parameter.

[0075] The sampling sequence of a node is determined by simulating several biased random walks of fixed length l. Two nodes are close to each other if they are several steps apart during the sampling process. More formally, let the sequence be represented by c = (c1, c2, ..., c l ); Use strategy S to simulate a walk of length l. In each step, around the center N s (c j )=(c j )={c i :jk≤i≤j+k}, select the next node according to the following formula (6). If it is not a node in the walk, c i Set to i-to and c i-1 =v. The adjacency matrix of state transition probability P∈R |V|×|V| The calculation is as follows:

[0076]

[0077] where ci Indicates the i-th node in the walk, and Z is a normalization constant. In this embodiment, the Alias ​​Table algorithm is used in the node sampling process, which can reduce the complexity of the sampling time to O(1).

[0078] Learning vectors through the Skip-Gram model: Word2vec can generate distributed representations of words using either of the following two model architectures: Continuous Bag of Words (CBOW) or Skip-Gram. In the continuous bag of words architecture, the model predicts the current word from a window of surrounding context words. The order of the context words does not affect the prediction (bag-of-words assumption). In Skip-Gram, the current word is used to predict the surrounding window of context words. CBOW is faster, while Skip-Gram performs better in NLP tasks. Therefore, this embodiment chooses to use the Skip-Gram algorithm to learn word vectors from these sampled corpora.

[0079] (3) Word-Graph2vec algorithm pseudocode

[0080] Algorithms 1 and 2 describe the Word-Graph2vec algorithm, which takes a large-scale corpus C as input and calculates word vectors φ from these corpora. The Word-Graph2vec algorithm contains 7 parameters. The first parameter C is the input corpus. The second and third parameters are the number of sampling times n and the walk length l. These two parameters are used to control the number of sampling sequences and the length of the sampling sequences. The fourth and fifth parameters q and p are used to adjust the random walk strategy. The sixth and seventh parameters are the context size k and dimension d, respectively. The context size k is used to adjust the size of the sliding window, which is used to identify whether two words are in the same context. The dimension d is used to set the dimension of the vector. In the present invention, a word co-occurrence graph is first constructed by scanning the entire corpus. Then, a bunch of word sequences are generated by randomly moving on the word co-occurrence graph. Finally, word vectors are learned from these word sequences.

[0081] In Algorithm 1, we first initialize the variables (line 1). We scan the entire corpus once and build a word co-occurrence graph (lines 2-10). Through this scan, we calculate the adjacency matrix W between words (lines 3-7) and the weight PW of word nodes (lines 8-10). The process of Algorithm 1 (building a word graph) is as follows:

[0082]

[0083] Then, in the LearnFeatures method of Algorithm 2, Equ 5 is used to calculate the non-standard transition probability π (line 2). The Word-Graph2vec method introduces a method for generating word sequences SC by traveling on the word co-occurrence graph. The nodes on the graph are scanned one by one, and the number of times each node is traversed as a root node is equal to The random walk sampling function, in lines 13-17, describes the process of generating a word sequence through random walk. AliasSample is used to speed up the movement process. Finally, in line 9, the word vector Φ is calculated using Skip-Gram based on the sampled word corpus SC. Algorithm 2 proceeds as follows:

[0084]

[0085] Time and space complexity analysis

[0086] In the Gensim Python package for Word2vec, negative sampling is used by default for word embedding learning. Gensim uses a binary search array based on the vocabulary size to sample negative examples. The execution time of a Word2vec model depends primarily on the size of the corpus and scales linearly with it. Theoretically, the runtime complexity of a Gensim model is O(|N|log(|V|)), where |N| is the total size of the corpus and |V| is the vocabulary size.

[0087] The Word-Graph2vec algorithm consists of three stages. In the first step, the corpus is scanned once to construct a word co-occurrence graph. Therefore, the time complexity is O(|N|). In the second step, a random walk is required to calculate the word co-occurrence graph. Let n be the total number of samples and l be the length of the walk. Therefore, the time complexity of the random walk process is O(nl). The third step is to use Skip-Gram to train the sampled corpus to obtain word embeddings. Since the size of the sampled corpus is equal to n×l, the time complexity should be O(nllog(|V|)). Therefore, the total time complexity should be O(|N|+nl+nllog(|V|)).

[0088] The size of the random sample corpus (nl) not only affects the running time of Word-Graph2vec, but also affects the accuracy of the method provided by the present invention. Intuitively, the larger the sample (the sequence of words that randomly walk on the word graph; see Figure 3 (b)) is more accurate but more time-consuming; however, interestingly, in studies of the Central Limit Theorem (CLT), the sample size required to produce accurate results is often surprisingly small. Therefore, in practice, we set the walk length l to 200 and the default number of times a node is used as a starting node n to 30×|V|.

[0089] Regarding space complexity, Word-Graph2vec needs to store the word graph, the generated word corpus, and the final word vectors. Therefore, the space complexity of Word-Graph2vec is around O(m|V|), where m is the average degree of the graph; the space complexity of the generated word corpus is O(nl); and the space complexity of the word vectors is O(d|V|). Therefore, the space complexity of Word-Graph2vec is O(m|V| + nl + d|V|).

[0090] Experimental verification

[0091] (1) Experimental setup

[0092] This embodiment uses different data sets to conduct experimental tests. The data sets used are shown in Table 2.

[0093] Text8: Text8 contains 100M processed Wikipedia characters. All text was converted to lowercase and all characters other than the 26 letters were removed. The Wordnet dictionary database was also used to filter correct English words. After preprocessing, Text8 contains 135,317 unique words and 3,920,065 pairs of word co-occurrence relationships. The word co-occurrence density is approximately 0.02%.

[0094] One billion words benchmark (1b words benchmark): This is a benchmark corpus of nearly 1 billion words. It was originally used to measure progress in statistical language modeling. This dataset was filtered using the same preprocessing methods as Text8. After preprocessing, the 1b words benchmark contains 82,473 unique words, 54,125,475 word co-occurrences, and a word co-occurrence density of approximately 0.80%.

[0095] English Wikipedia data set (En-Wikipedia): This is a word corpus of English articles collected from Wikipedia pages. After preprocessing (using the same method as above), En-Wikipedia contains 100,723 unique words, 64,633,532 word co-occurrence relationships, and a word co-occurrence density of approximately 0.64%.

[0096] Concatenating data sets: To test the scalability of the proposed method on large datasets, several Wikipedia datasets were processed sequentially as a single, concatenated dataset. Co-En-Wikipedia is used to represent the concatenated dataset. The capacity of Co-En-Wikipedia-2 is 16.4 GB, and that of Co-En-Wikipedia-3 is 24.6 GB.

[0097] Baseline Methods: The classic Skip-Gram is one of the baseline methods in this example. Skip-Gram parameters, such as the sliding window size and minimum count, are adjusted for different NLP tasks to achieve optimal performance. Fasttext is also used as another baseline method in this example.

[0098] All experiments in this embodiment were conducted on a PC with a 1.60GHz Intel Core 5 Duo processor, 8GB of memory, and Windows 10, and all algorithms were implemented in Python and C++. At the same time, the experiment used a compact graph data structure optimized for cache and pre-calculation / parallelization to improve the shortcomings of Node2vec. Since the constructed word graph is relatively sparse (density does not exceed 2%, see Table 2), according to Pecanpy's distinction between the operating modes of networks with different densities, the SparseOTF mode was selected, which is suitable for large and sparse networks.

[0099] Table 2

[0100]

[0101] (2) Experimental evaluation criteria

[0102] This embodiment adopts three evaluation tasks: classification, similarity, and analogy, and tests them.

[0103] Classification: The goal is to recover word clusters into different categories. Therefore, the word vectors corresponding to all words in the dataset are clustered, and the purity of the clusters is calculated based on the labeled dataset. The two evaluation datasets selected are Battig and BLESS. Battig consists of 83 concepts in 10 common concrete categories (up to 10 concepts per class), which are selected so that they are rated as highly prototypical of the class. Classification examples include land mammals (such as dogs, elephants, etc.), tools (such as screwdrivers, hammers, etc.), and fruits (such as oranges, plums, etc.). The BLESS dataset designed for evaluating distributional semantic models contains 200 different English concrete nouns as target concepts, which can be divided into 17 major categories.

[0104] Similarity: In this example, two evaluation datasets (Simlex999 and MEN) were selected. These datasets contain a list of word-pair relevance scores and an associated human-judged similarity. The cosine similarity between two words should be highly correlated with the human relevance score.

[0105] Mikolov improved this task. The goal is to find an x ​​for a given y such that x:y is most similar to the sample relation a:b. This example uses two test datasets: MSR and Semeva_2012_task 2. The MSR dataset contains only syntactic questions involving morphological variation; the Semeva_2012_task2 task requires predicting how similar the semantic relationship between x and y is to the semantic relationship between a and b.

[0106] The effectiveness of word similarity prediction is measured using Spearman's rank correlation coefficient ρ. It is a measure of the dependence of two variables. If there are no repeated values ​​in the data and the two variables are completely monotonically correlated, the value of ρ is +1 or -1. This measures the rank correlation (the higher the better) between the list of word pairs predicted by the word vector algorithm in descending order of similarity values ​​and the reference list of word pairs judged by humans. For the analogy and concept classification tasks, this example reports the accuracy of predicting the reference word and the reference word of the class, respectively.

[0107] (3) Experimental results prediction

[0108] In accuracy tasks, Word-Graph2vec does not lose accuracy due to the characteristics of its sampling method (as described by the Central Limit Theorem). In computational time tasks, because the vocabulary size and density of the word graph are relatively stable, WordGraph2vec's runtime increases very slowly as the amount of data increases. However, Skip-Gram's runtime increases linearly with the amount of data. Therefore, initially, Skip-Gram's runtime is much lower than Word-Graph2vec's, but as the amount of data increases, Skip-Gram's runtime quickly exceeds Word-Graph2vec's.

[0109] (4) Experimental parameter adjustment

[0110] Walk Length Study: The walk length l is crucial to accuracy and performance. This example sets the default value of the number of samples n to 30×|V| and l varies from 200 to 300. Qualitatively similar trends of Word-Graph2vec are observed on all datasets. Due to space limitations, we only illustrate the results of Word-Graph2vec on the Wikipedia dataset for classification tasks. Figure 5In the example, as the walk length l increases, the runtime increases rapidly from 28,759 seconds to 34,459 seconds, but the classification accuracy is significantly affected, ranging from 81.7% to 82.6%. This suggests that according to the CLT, the sample size required to produce accurate results in research is often surprisingly small. Therefore, to achieve high performance, the walk length was set to 200 in the experiments of this example.

[0111] The figure shows three different word-graph2vec models with walk lengths of 200, 250, and 300. All other parameters are the same for the three models. BLESS is used for the classification task.

[0112] P, Q research: The optimal combination of parameters p and q is explored for the three combinations [(0.001, 1), (1, 1), (1, 0.001)]. Table 3 shows the evaluation results of the three tasks. It can be seen from the results that the (p, q) combination of (1, 0.001) can achieve better accuracy. The task of predicting nodes in the network needs to solve two problems: the homogeneity of nodes and the structural equivalence problem. DFS and BFS complete these two tasks respectively. After mapping the text to a word graph, synonyms usually have almost similar network structures in the graph because they have similar contexts. Based on this feature, it is necessary to shorten the distance between synonyms through random walks with DFS bias to better capture synonyms. Therefore, setting the parameter q = 0.001 that controls the walk to explore farther parts of the graph, which is much smaller than the parameter p = 1, to control the walk tendency of DFS, can obtain better word vectors.

[0113] Table 3

[0114]

[0115] Node Weight Study: Table 4 shows a comparison of the evaluation results of word vectors obtained by setting various word weights (e.g., Pecanpy+TF, Pecanpy+TF-IDF, and Pecanpy). Method 1 (Pecanpy+TF) uses TF values ​​as node weights; Method 2 (Pecanpy+TF-IDF) uses TF-IDF values ​​as weights; Method 3 (Pecanpy) does not set node weights.

[0116] The experimental results in Table 4 show that: (i) In general, Pecanpy+TF and Pecanpy+TF-IDF have higher accuracy than + (for example, for the classification task of the dataset hit, the two methods with node weights have an accuracy improvement of 4% over the original method). The weight of the node is important for improving accuracy because important words should be sampled more times. (ii) The accuracy of Pecanpy+TF-IDF is slightly higher than Pecanpy+TF because, as we know, IDF can also be extracted from the background corpus to correct sampling bias, or in the dataset used in the current experiment. Therefore, in the following experiments, Pecanpy+TF-IDF is used as the default model.

[0117] Table 4

[0118]

[0119] In summary, all subsequent experiments were conducted with the following parameter values: p = 1, q = 0.001, dimension = 100, walk length l = 200, number of samples n = 30, and window_sizes = 10. For other parameters not mentioned, the default values ​​of Pecanpy were used in this experiment.

[0120] Accuracy test

[0121] This example compares the performance of Word-Graph2vec and Skip-Gram on three tasks. Tables 5, 6, and 7 show the experimental results for the classification task, word similarity task, and word analogy task, respectively. The experimental results show that: (i) Across all three tasks, the quality of word vectors trained by Word-Graph2vec and Skip-Gram improves with increasing datasets. The larger the corpus, the higher the quality of the word vectors. (ii) Generally speaking, Word-Graph2vec's accuracy on Text8 is slightly lower than Skip-Gram's. Word-Graph2vec's random walk process aggregates statistically frequently co-occurring words into a sequence, which is equivalent to pre-learning the relationships between words. If the corpus is too small, the walk process will focus on high-probability words and ignore the possible influence of low-probability words. As a result, the final sampling result fails to capture all words that may be semantically related in a sampled sequence. (iii) Word-Graph2vec's accuracy on the 1b-word benchmark and Wikipedia is close to or higher than Skip-Gram. As the corpus size increases, we can see that the performance of the Word-Graph2vec training word vectors for each task gradually approaches or even surpasses them. This suggests that the random walk has a certain probability of capturing marginal words with relatively few occurrences. Therefore, this makes the sampled corpus closer to the real text and can even track the distribution of similar words in advance. In addition, this may explain why the word vectors obtained by Word-Graph2vec perform better in some test tasks.

[0122] Table 5

[0123]

[0124] Table 6

[0125]

[0126] Table 7

[0127]

[0128] (6) Efficiency test

[0129] Table 8

[0130]

[0131] Table 8 shows the training time comparison of the two methods on three datasets. The experimental results show that Word-Graph2vec takes more training time in Text8 and 1b word benchmarks. However, as long as the dataset is large enough, such as Wikipedia, the performance of Word-Graph2vec improves more than twice as fast as Skip-Gram. Through experimental results, we found that the time cost of text8 in the range of 100M is not much different from that of 8Gen-Wikipedia. However, due to the increase in the corpus, the training time of Skip-Gram has changed significantly. In addition, in order to observe the time trend more intuitively, this embodiment generates two larger datasets (Con-En-Wikipedia-1, 16.4G; Con-En-Wikipedia-2, 24.6G) for testing with En-Wikipedia. As shown in Figure 6 As shown in Figure 1, the experimental results of four datasets of different sizes (2.51G, 8.22G, 16.4G, and 24.6G) are plotted as time curves (including 1bWords Benchmark and En-Wikipedia). Figure 6 The results show that the runtime of Skip-Gram increases almost linearly. While keeping the number of samples n and the walk length l constant, the runtime of Word-Graph2vec increases slowly (the slight increase in runtime is due to loading the word corpus). This can be explained by the fact that no matter how large the original corpus is, the number of word sequence samples is constant, so the training time of Word-Graph2vec increases steadily.

[0132] Word embedding maps words as vectors into a continuous real space, so that words with similar meanings have similar representations. It is widely used in modern natural language processing (NLP) tasks, including semantic analysis, sentiment analysis, information retrieval, machine translation, and question answering systems. Current word embedding methods, such as Word2vec and Glove, rely on large corpora to learn associations between words and obtain statistical correlations between different words, thereby simulating the human cognitive process of a word. The time complexity of these two methods clearly shows that the runtime increases linearly with the size of the corpus.

[0133] However, massive amounts of text data are collected daily as training data, including scientific literature, records from marketing and economic sectors, speeches from political discourse, such as presidential campaigns and inaugural addresses, and conference proceedings. Furthermore, online data sources, such as emails, web pages, blogs / microblogs, and social media posts and comments, provide a rich source of text data for training. Therefore, in the era of big data, accelerating these existing word embedding methods is becoming increasingly important.

[0134] Various graph models have been proposed to map text into word co-occurrence graphs and capture the implicit structure of text data. Experimental results show that these methods can achieve good accuracy in clustering and retrieval tasks. Inspired by these methods, this paper proposes a graph-based word embedding method, called Word-Graph2vec, to address the problem of efficiently computing word embeddings for large-scale corpora.

[0135] The main advantage of Word-Graph2vec is its performance on large-scale corpora, because the vocabulary remains relatively stable despite the continuous expansion of the corpus size. In addition, related idioms (e.g., "turn a blind eye", etc.) and fixed expressions (e.g., "on the other hand", etc.) appear frequently in large-scale texts, allowing Word-Graph2vec to effectively capture and process these language features, thereby improving its efficiency and accuracy on large-scale datasets. In English, the number of nodes and the density of the word co-occurrence graph change slightly as the training corpus increases. Therefore, Word-Graph2vec has a stable running time on large-scale datasets, and its performance advantage becomes more and more obvious as the training corpus grows. In addition, adding more training corpora only adjusts the edge weights. Therefore, the time to train the corpus increases slowly as the corpus increases. In addition, although the present invention requires a scan of the entire corpus to construct the word co-occurrence graph, this process is much faster than the training process.

[0136] This embodiment also discloses a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned graph-based word embedding optimization method.

[0137] This embodiment also discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned graph-based word embedding optimization method are implemented.

[0138] This embodiment also discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned graph-based word embedding optimization method.

[0139] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A graph-based word embedding optimization method, characterized in that: The following steps are involved: Obtaining a word co-occurrence graph based on a large-scale corpus; wherein the word co-occurrence graph is a directed weighted word graph; Based on the word co-occurrence graph, the transition probability between nodes is obtained and the importance score of the node is extracted; based on the transition probability between nodes and the importance score of the node, a number of word sequences are obtained through random walk sampling to construct a sampling corpus; The skip-gram model is applied to the sampled corpus to obtain the final word vector.

2. The graph-based word embedding optimization method according to claim 1, characterized in that The vertices of the word co-occurrence graph represent unique words, the edges represent the co-occurrence between words, the edge direction represents the word order, and the edge weight and node weight are marked.

3. The graph-based word embedding optimization method according to claim 2, characterized in that The number of times the words at both ends of the directed edge appear in the text is used as the weight value of the edge and stored in a weighted adjacency matrix.

4. The graph-based word embedding optimization method according to claim 2, characterized in that The node weight is the importance score of the corresponding word in the large-scale corpus; the importance score is calculated as follows: Among them, PW v is the importance score of word v, n v is the number of occurrences of word v, ∑ k n k is the number of occurrences of all words, To compute the inverse document frequency score, |C| is the number of documents in the corpus, d is the number of documents in the corpus, and |{d:v∈d}| is the number of documents containing v.

5. The graph-based word embedding optimization method according to claim 1, characterized in that The calculation formula for the transition probability is as follows: p vx =a pq (t,x)×W vx ; Among them, W vx is the weight of edge (v, x)∈e, α is the path bias parameter, p is the return probability, and q is the range parameter.

6. The graph-based word embedding optimization method according to claim 1, characterized in that The process of obtaining a number of word sequences includes: A probabilistic sampling method based on node importance scores is used to select the root node. The second-order random walk of the Node2vec model is used to guide the walk using preset return hyperparameters and internal and external hyperparameters. The next node to walk is selected according to the adjacency matrix of the state transition probability to obtain several word sequences.

7. The graph-based word embedding optimization method according to claim 6, characterized in that The adjacency matrix of state transition probability is calculated as follows: Among them, c i represents the i-th node in the walk, Z is the normalization constant, and E is the set of edges.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the graph-based word embedding optimization method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the graph-based word embedding optimization method described in any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the graph-based word embedding optimization method described in any one of claims 1 to 7 are implemented.

Images