A method and device for generating pre-trained word vectors

Abstract

The present invention discloses a method and device for generating pre-trained word vectors, which are used to express semantic interpretation information in words through pre-trained word vectors. The method comprises: according to the weights between nodes with connection relationships in a pre-constructed definition relationship graph, probability sampling is performed on the words corresponding to each node in the definition relationship graph to determine the word sequence, wherein the definition relationship graph comprises a plurality of nodes, each node corresponds to a word, the connection relationship between the nodes is used to characterize that there is a semantic interpretation relationship between the corresponding words, the weight is used to characterize the semantic relevance between the words with semantic interpretation relationships, and there is a semantic interpretation relationship between adjacent words in the word sequence; for any word in the word sequence, the initial word vector of the word is adjusted through the previous word and the following word of the any word to obtain the pre-trained word vector of the word.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method and device for generating pre-trained word vectors. Background Art

[0002] A word vector is understood as a vector that maps a word or phrase to a real number. A pre-trained word vector is a pre-trained word vector, which is widely used in various fields of natural language processing. For example, Word2vec proposes to use the continuous bag-of-words model and the Skip-gram model to obtain pre-trained word vectors.

[0003] The traditional algorithm for obtaining pre-trained word vectors mainly trains the model based on the general text corpus context as the input sample of the model, and uses the frequency information of the number of times different words appear together in the context (also called co-occurrence information) to represent the semantic relationship between different words. Finally, the pre-trained word vector corresponding to each word in the context is obtained, that is, the pre-trained word vector corresponding to each word contains the co-occurrence information between other words in the context. However, since the co-occurrence information does not involve the actual semantic relationship between the words, the words expressed by the obtained pre-trained word vectors lack actual semantics and cannot represent more accurate semantic information. Summary of the invention

[0004] The present invention provides a method and device for generating a pre-trained word vector, which is used to express semantic interpretation information in a word through the pre-trained word vector.

[0005] In a first aspect, an embodiment of the present invention provides a method for generating a pre-trained word vector, comprising:

[0006] According to the weights between the nodes with connection relationships in the pre-constructed definition relationship graph, probability sampling is performed on the words corresponding to each node in the definition relationship graph to determine the word sequence, wherein the definition relationship graph includes multiple nodes, each node corresponds to a word, the connection relationship between the nodes is used to represent that there is a semantic interpretation relationship between the corresponding words, the weight is used to represent the degree of semantic relevance between the words with semantic interpretation relationships, and there is a semantic interpretation relationship between adjacent words in the word sequence;

[0007] For any word in the word sequence, the initial word vector of the word is adjusted through the previous words and the following words of the word to obtain the pre-trained word vector of the word, wherein the previous words refer to a preset number of words in the word sequence that are located before the any word, and the following words refer to a preset number of words in the word sequence that are located after the any word.

[0008] The method for generating pre-selected word vectors in the embodiment of the present invention is determined based on a word sequence obtained by probability sampling based on a definition relationship graph. Since the word sequence in the embodiment of the present invention contains a semantic interpretation relationship, in the process of adjusting the initial word vector of any word in the word sequence using the context words of the word sequence, words containing accurate semantic interpretation relationships will be included in the pre-trained word vector of any word, so that the pre-trained word vector obtained in the embodiment of the present invention can contain a semantic interpretation relationship. In the application of downstream retrieval tasks, content that is semantically close or similar to the input search term can be more accurately retrieved.

[0009] As an optional implementation, the definition relationship diagram is constructed in the following manner, including:

[0010] Establish a one-to-one correspondence between words and nodes in the dictionary;

[0011] Determining words having a semantic interpretation relationship based on the words in the dictionary and the words that interpret the words;

[0012] The nodes corresponding to the words having the semantic interpretation relationship are connected to construct a definition relationship graph.

[0013] As an optional implementation, the weights between connected nodes are determined in the following manner:

[0014] The weights between connected nodes are determined based on the number of times any word appears in a semantically interpreted sentence and the number of all semantically interpreted sentences containing the any word, wherein the semantically interpreted sentence is a sentence that semantically interprets a word corresponding to a node that is connected to a node corresponding to the any word.

[0015] As an optional implementation, according to the weights between the nodes with connection relationships in the pre-constructed definition relationship graph, probabilistic sampling is performed on the words corresponding to each node in the definition relationship graph to determine the word sequence, including:

[0016] The word corresponding to the preset node is used as the first word in the word sequence;

[0017] Using a random walk algorithm, taking the preset node as a starting node, and performing probability sampling on the connected nodes according to the weights between the nodes connected to the starting node, to determine the next node of the starting node;

[0018] The word corresponding to the next node is taken as the next word in the word sequence. If the length of the word sequence does not reach the preset length, the next node is taken as the new starting node to continue probability sampling to determine the word corresponding to the next node of the new starting node until the length of the word sequence reaches the preset length.

[0019] As an optional implementation, adjusting the initial word vector of the word to obtain the pre-trained word vector of the word includes:

[0020] For any word in the word sequence, determine the minimization loss function according to the word vectors of the previous word and the following word of the any word;

[0021] The initial word vector of the word is adjusted using the minimization loss function to obtain a pre-trained word vector of the word.

[0022] In a second aspect, an embodiment of the present invention provides a device for generating a pre-trained word vector, the device comprising a processor and a memory, the memory being used to store a program executable by the processor, the processor being used to read the program in the memory and perform the following steps:

[0023] According to the weights between the nodes with connection relationships in the pre-constructed definition relationship graph, probability sampling is performed on the words corresponding to each node in the definition relationship graph to determine the word sequence, wherein the definition relationship graph includes multiple nodes, each node corresponds to a word, the connection relationship between the nodes is used to represent that there is a semantic interpretation relationship between the corresponding words, the weight is used to represent the degree of semantic relevance between the words with semantic interpretation relationships, and there is a semantic interpretation relationship between adjacent words in the word sequence;

[0024] For any word in the word sequence, the initial word vector of the word is adjusted through the previous words and the following words of the word to obtain the pre-trained word vector of the word, wherein the previous words refer to a preset number of words in the word sequence that are located before the any word, and the following words refer to a preset number of words in the word sequence that are located after the any word.

[0025] As an optional implementation manner, the processor is specifically configured to construct a definition relationship graph in the following manner:

[0026] Establish a one-to-one correspondence between words and nodes in the dictionary;

[0027] Determining words having a semantic interpretation relationship based on the words in the dictionary and the words that interpret the words;

[0028] The nodes corresponding to the words having the semantic interpretation relationship are connected to construct a definition relationship graph.

[0029] As an optional implementation manner, the processor is specifically configured to determine the weights between nodes having a connection relationship in the following manner:

[0030] The weights between connected nodes are determined based on the number of times any word appears in a semantically interpreted sentence and the number of all semantically interpreted sentences containing the any word, wherein the semantically interpreted sentence is a sentence that semantically interprets a word corresponding to a node that is connected to a node corresponding to the any word.

[0031] As an optional implementation manner, the processor is specifically configured to:

[0032] The word corresponding to the preset node is used as the first word in the word sequence;

[0033] Using a random walk algorithm, taking the preset node as a starting node, and performing probability sampling on the connected nodes according to the weights between the nodes connected to the starting node, to determine the next node of the starting node;

[0034] The word corresponding to the next node is taken as the next word in the word sequence. If the length of the word sequence does not reach the preset length, the next node is taken as the new starting node to continue probability sampling to determine the word corresponding to the next node of the new starting node until the length of the word sequence reaches the preset length.

[0035] As an optional implementation manner, the processor is specifically configured to:

[0036] For any word in the word sequence, determine the minimization loss function according to the word vectors of the previous word and the following word of the any word;

[0037] The initial word vector of the word is adjusted using the minimization loss function to obtain a pre-trained word vector of the word.

[0038] In a third aspect, an embodiment of the present invention provides a device for generating a pre-trained word vector, including:

[0039] A word sequence determination unit is used to perform probability sampling on words corresponding to each node in a pre-constructed definition relationship graph according to weights between nodes with connection relationships in the definition relationship graph to determine a word sequence, wherein the definition relationship graph includes a plurality of nodes, each node corresponds to a word, the connection relationship between the nodes is used to characterize that there is a semantic interpretation relationship between the corresponding words, the weight is used to characterize the degree of semantic relevance between the words with semantic interpretation relationships, and there is a semantic interpretation relationship between adjacent words in the word sequence;

[0040] Determine a pre-trained word vector unit, which is used to adjust the initial word vector of any word in the word sequence through the previous words and the following words of the any word to obtain the pre-trained word vector of the word, wherein the previous words refer to a preset number of words in the word sequence that are located before the any word, and the following words refer to a preset number of words in the word sequence that are located after the any word.

[0041] As an optional implementation, the determination word sequence unit is specifically used to construct a definition relationship graph in the following manner:

[0042] Establish a one-to-one correspondence between words and nodes in the dictionary;

[0043] Determining words having a semantic interpretation relationship based on the words in the dictionary and the words that interpret the words;

[0044] The nodes corresponding to the words having the semantic interpretation relationship are connected to construct a definition relationship graph.

[0045] As an optional implementation, the word sequence determination unit is specifically used to determine the weights between nodes having a connection relationship in the following manner:

[0046] The weights between connected nodes are determined based on the number of times any word appears in a semantically interpreted sentence and the number of all semantically interpreted sentences containing the any word, wherein the semantically interpreted sentence is a sentence that semantically interprets a word corresponding to a node that is connected to a node corresponding to the any word.

[0047] As an optional implementation manner, the determination word sequence unit is specifically used for:

[0048] The word corresponding to the preset node is used as the first word in the word sequence;

[0049] Using a random walk algorithm, taking the preset node as a starting node, and performing probability sampling on the connected nodes according to the weights between the nodes connected to the starting node, to determine the next node of the starting node;

[0050] The word corresponding to the next node is taken as the next word in the word sequence. If the length of the word sequence does not reach the preset length, the next node is taken as the new starting node to continue probability sampling to determine the word corresponding to the next node of the new starting node until the length of the word sequence reaches the preset length.

[0051] As an optional implementation, determining the pre-trained word vector unit is specifically used for:

[0052] For any word in the word sequence, determine the minimization loss function according to the word vectors of the previous word and the following word of the any word;

[0053] The initial word vector of the word is adjusted using the minimization loss function to obtain a pre-trained word vector of the word.

[0054] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, it is used to implement the steps of the method described in the first aspect above.

[0055] These and other aspects of the present application will be more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0057] Figure 1 A flow chart of a method for generating pre-trained word vectors provided in an embodiment of the present invention;

[0058] Figure 2 A flowchart of a specific implementation method of generating a pre-trained word vector provided by an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of a device for generating pre-trained word vectors provided in an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of a device for generating pre-trained word vectors provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] In the embodiments of the present invention, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0063] The application scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0064] Example 1

[0065] Traditional algorithms for obtaining pre-trained word vectors are mainly based on general text corpus context as the input sample of the model to train the model, and the semantic relationship between different words is represented by the frequency information of the number of co-occurrences between different words in the context (also called co-occurrence information), and finally the pre-trained word vector corresponding to each word in the context is obtained, where the context is used to represent the sentences adjacent to the target word; for example, Word2vec proposes to use the continuous bag of words model and the Skip-gram model to obtain pre-trained word vectors. Both models are trained by iterative input (context, target word) word pairs, that is, given a word, predict the target word in the context; the difference is that the target word of the continuous bag of words model is the central word in the context window, and the target word of the Skip-gram algorithm is the context word. The essence of the Word2vec algorithm is the implicit PMI (pointwise mutual information) matrix decomposition of the context co-occurrence matrix, but the information counted by the context co-occurrence matrix is ​​constrained within the context window, and the global word frequency statistics cannot be obtained. Since the traditional method of generating and training word vectors can only represent the semantic relationship between words through the frequency information of the number of times words appear together (co-occurrence information), the algorithm is essentially the decomposition of the co-occurrence information matrix. However, since the co-occurrence information does not involve the actual semantic relationship between words, the words expressed by the obtained pre-trained word vectors lack actual semantics and cannot represent more accurate semantic information.

[0066] In order to solve the above technical problems, the embodiment of the present invention provides a method for generating pre-trained word vectors according to the process of human learning vocabulary semantics. Humans learn vocabulary semantics by explaining or defining vocabulary, and external dictionary knowledge plays a vital role in the process of learning vocabulary semantics. Imitating the semantic knowledge process of human learning vocabulary, the present invention proposes a pre-trained word vector generation method based on a definition relationship graph to capture high-order neighborhood information. The embodiment of the present invention can extract a large number of semantic relationships of words by making full use of the definition knowledge of the dictionary when a word is given as input. The embodiment of the present invention determines the word sequence for training according to a pre-constructed definition relationship graph (undirected weighted graph), in which the vertices represent words, the edges between the points represent that there is a semantic interpretation relationship between the two words, and according to the semantic relevance between the words with a semantic interpretation relationship, a specific weight value is assigned to the edge between the two word nodes. Secondly, in order to fully and effectively utilize the structural information of the definition relationship graph, it is proposed to perform probability sampling of the words corresponding to each node in the definition relationship graph to determine the word sequence. Finally, each word sequence is used as a sample, and the initial word vector of any word in the word sequence is adjusted by the context word to obtain the pre-trained word vector of any word.

[0067] The present invention proposes a low-cost word vector generation method that introduces external dictionary knowledge. The present invention uses the definition knowledge of the natural language dictionary to construct a bidirectional weighted definition relationship graph, proving that more accurate semantic information can be extracted from the natural language dictionary by combining the dictionary definition knowledge with the graph representation method. The pre-trained word vector generation method proposed in the present invention can learn the lexical semantics more accurately, and the definition relationship graph also contains richer knowledge than the manually constructed knowledge base to a certain extent.

[0068] like Figure 1 As shown, the embodiment of the present invention provides a method for generating a pre-trained word vector, and the specific implementation process of the method is as follows:

[0069] Step 100: According to the weights between the nodes with connection relationships in the pre-constructed definition relationship graph, probability sampling is performed on the words corresponding to each node in the definition relationship graph to determine the word sequence;

[0070] The definition relationship graph includes a plurality of nodes, each node corresponds to a word, the connection relationship between the nodes is used to represent that there is a semantic interpretation relationship between the corresponding words, the weight is used to represent the degree of semantic relevance between the words with a semantic interpretation relationship, and there is a semantic interpretation relationship between adjacent words in the word sequence;

[0071] It should be noted that the definition relationship graph in the embodiment of the present invention is a bidirectional weighted definition relationship graph, which is generated based on the words in the dictionary and the explanatory sentences of the words. Since the connection relationship between the nodes in the definition relationship graph can represent the semantic interpretation relationship between the corresponding words, there is a semantic interpretation relationship between adjacent words in the word sequence obtained by probability sampling according to the weights between the nodes, and the pre-trained word vector in the embodiment of the present invention can be expressed according to the semantic interpretation relationship between the words.

[0072] It is easy to understand that probability sampling means that every unit in the survey population sample has an equal chance of being selected. Also known as random sampling, probability sampling is based on probability theory and random principles to select samples, which is a sampling that makes every unit in the population have a known non-zero probability of being selected.

[0073] The embodiment of the present invention can obtain multiple word sequences by probabilistically sampling each node in the definition relationship graph, and each word sequence is processed through the following steps to obtain a pre-trained word vector corresponding to each word in each word sequence. Among them, the pre-trained word vector in this embodiment is a vector of fixed length, which is convenient for mathematical processing and convenient for any downstream task system to process. For example, it is convenient for the retrieval system to retrieve words similar or close to the input words and words or knowledge points related to the semantic interpretation of the words when performing retrieval tasks.

[0074] Step 101: for any word in the word sequence, the initial word vector of the word is adjusted by the previous words and the following words of the word to obtain a pre-trained word vector of the word;

[0075] The preceding words refer to a preset number of words in the word sequence that are located before any of the words, and the following words refer to a preset number of words in the word sequence that are located after any of the words.

[0076] In the process of determining the pre-trained word vector, the embodiment of the present invention is a process of adjusting the initial word vector of any word in the word sequence through the context words of the any word. Since there is a semantic interpretation relationship between adjacent words in the word sequence, the pre-trained word vector of any word obtained by adjusting the context words of the any word also has a semantic interpretation relationship.

[0077] It should be noted that the initial word vector in the embodiment of the present invention is a vector randomly initialized for each word obtained by segmenting each word sequence, and the vectors randomly generated for different words may be the same or different, and this embodiment does not impose too many restrictions on this. Among them, the word segmentation can be performed using the Jieba word segmenter.

[0078] As an optional implementation manner, the embodiment of the present invention constructs a definition relationship graph in the following manner, including:

[0079] 1) Establish a one-to-one correspondence between words and nodes in the dictionary;

[0080] 2) determining words having a semantic interpretation relationship based on the words in the dictionary and the words that interpret the words;

[0081] 3) Connecting the nodes corresponding to the words with semantic interpretation relationships to construct a definition relationship graph.

[0082] The definition relationship graph constructed by the embodiment of the present invention has no direction between the nodes. For example, the definition relationship graph obtained is: in, Represents a collection of nodes in a defined relationship graph. Represents the set of edges connecting the nodes in the definition graph. j and v i If there is a semantic interpretation relationship between them, then there will be an edge e between the nodes corresponding to the words. ij ∈ε.

[0083] Optionally, in order to distinguish the importance of words in the definition relationship graph, weights are introduced to represent the semantic relevance between words with semantic interpretation relationships. The more important the word, the greater the semantic relevance, and vice versa.

[0084] As an optional implementation manner, the embodiment of the present invention determines the weights between nodes having a connection relationship in the following manner:

[0085] Determine the weights between nodes having a connection relationship according to the number of times any word appears in the semantically interpreted sentence and the number of all semantically interpreted sentences containing the any word;

[0086] The semantically interpreted sentence is a sentence that semantically interprets a word corresponding to a node that has a connection relationship with a node corresponding to any one of the words.

[0087] In implementation, word frequency-inverse document frequency can be introduced as weight, and the specific definition is as follows:

[0088] tf-idf(v i , v j )=tf(v i , v j )×idf(v j );

[0089] The number of times any word appears in a semantically interpreted sentence tf(*), and the number of all semantically interpreted sentences containing the word idf(*) are defined as:

[0090]

[0091]

[0092] Among them, N is the set of words in the sentence that contains all semantic interpretations, N i is the word v i The words in the explanation sentence, |N ij | indicates that it appears in the word v i The word v in the explanation sentence j The number of times;

[0093] u ij Represents edge e ij The weight on edge e ij By word v i and v j The corresponding node connection structure;

[0094] sgn represents the indicator function.

[0095] Optional, if there is a mutual interpretation relationship between the words, that is, the words v i The explanation sentence contains the word v j , word v j The explanation sentence contains the word v i , then the weight between the nodes corresponding to the words that have a mutual explanation relationship is the sum of the two weights obtained, which is as follows:

[0096] tf-idf(v i , v j )+tf-idf(v j , v i );

[0097] For the entire definition relationship graph, the embodiment of the present invention expresses the weight corresponding to each edge through the following formula:

[0098] u ij =u ji =tf-idf(v i , v j )+tf-idf(v j , v i ).

[0099] After constructing the definition relationship graph in the above manner, the embodiment of the present invention samples the word sequence based on the following method:

[0100] As an optional implementation, according to the weights between the nodes with connection relationships in the pre-constructed definition relationship graph, probabilistic sampling is performed on the words corresponding to each node in the definition relationship graph to determine the word sequence, including:

[0101] 1) The word corresponding to the preset node is taken as the first word in the word sequence;

[0102] In implementation, a starting node may be pre-set, and the word corresponding to the starting node may be used as the first word in the word sequence. Probabilistic sampling may be performed in the following steps in sequence to determine the wandering direction of the node.

[0103] 2) using a random walk algorithm to take the preset node as the starting node, and performing probability sampling on the connected nodes according to the weights of the nodes connected to the starting node to determine the next node of the starting node;

[0104] In implementation, from node n x To node n x+1 The sampling probability of is defined as:

[0105] p(n x+1 =v j |n x =v i )=u ij / Z;

[0106]

[0107] Among them, Z represents the node v i The normalization constant, n x Represents the xth node in the generated sequence.

[0108] 3) Taking the word corresponding to the next node as the next word in the word sequence; if the length of the word sequence does not reach a preset length, taking the next node as a new starting node and continuing to perform probability sampling to determine the word corresponding to the next node of the new starting node, until the length of the word sequence reaches a preset length.

[0109] The length of the word sequence in the embodiment of the present invention represents the number of words in the word sequence. The length of each word sequence determined in the embodiment of the present invention is the same.

[0110] After sampling the word sequence through the above steps, the initial word vector corresponding to each word in the word sequence is adjusted, and finally a pre-trained word vector for each word is obtained, which contains the semantic interpretation relationship of the context.

[0111] As an optional implementation manner, the embodiment of the present invention adjusts the initial word vector of the word to obtain the pre-trained word vector of the word, including:

[0112] 1) for any word in the word sequence, determine the minimization loss function according to the word vectors of the previous word and the following word of the any word;

[0113] 2) Using the minimization loss function to adjust the initial word vector of the word to obtain the pre-trained word vector of the word.

[0114] In practice, the formula for minimizing the loss function is as follows:

[0115]

[0116]

[0117]

[0118] Where L represents the minimization loss function, n represents the number of determined word sequences; w t Indicates any word currently being adjusted in the word sequence; w t+k are the previous and next words in a window of size C, where C is an integer greater than zero; σ(x) represents the sigmoid function, v t The word w t The vector representation of ; The word w t The set of previous words and following words. Word w t The Log value of the sigmoid function corresponding to the previous word and the following word; For the word w t Negative sampling.

[0119] Optionally, the method for generating word vectors provided in an embodiment of the present invention can be combined with an existing method for generating word vectors. During implementation, the word sequence determined by the embodiment of the present invention can be combined with an existing free text sequence as a new word sequence, and then for any word in the new word sequence, the initial word vector of the word can be adjusted through the previous words and the following words of the word to obtain a pre-trained word vector of the word.

[0120] The embodiments of the present invention can be applied to information retrieval systems in the field of natural language processing, such as question-answering systems, search systems, etc. By introducing external dictionary knowledge, the present invention can more accurately improve the detailed vocabulary retrieval capability of the retrieval system, and greatly improve the retrieval accuracy and precision recall rate at the sentence level. In addition, the embodiments of the present invention can retrieve the interpreted words by expressing the semantic interpretation relationship between words, using the words in the interpretation sentence. A typical example is the retrieval of knowledge points with explanations. Given a knowledge point and the explanation of the knowledge point, the pre-trained word vector obtained by the present invention can retrieve words or knowledge points related to the knowledge point.

[0121] like Figure 2 As shown, the embodiment of the present invention also provides a specific implementation method for generating a pre-trained word vector, as shown below:

[0122] Step 200, establishing a one-to-one correspondence between words in the dictionary and nodes;

[0123] Step 201, determining words having a semantic interpretation relationship according to the words in the dictionary and the words that interpret the words;

[0124] Step 202: Connect the nodes corresponding to the words having the semantic interpretation relationship to construct a definition relationship graph;

[0125] Step 203: Determine the weights between nodes having a connection relationship according to the number of times any word appears in the semantically interpreted sentence and the number of all semantically interpreted sentences containing the any word;

[0126] The semantically interpreted sentence is a sentence that semantically interprets a word corresponding to a node that has a connection relationship with a node corresponding to any one of the words.

[0127] Step 204: taking the word corresponding to the preset node as the first word in the word sequence;

[0128] Step 205: Using a random walk algorithm, taking the preset node as a starting node, and performing probability sampling on the connected nodes according to the weights of the nodes connected to the starting node, to determine the next node of the starting node;

[0129] Step 206: taking the word corresponding to the next node as the next word in the word sequence, and if the length of the word sequence does not reach the preset length, taking the next node as the new starting node and continuing to perform probability sampling to determine the word corresponding to the next node of the new starting node, until the length of the word sequence reaches the preset length;

[0130] Step 207: for any word in the word sequence, determine a minimization loss function according to the word vectors of the preceding word and the following word of the any word;

[0131] Step 208: Use the minimization loss function to adjust the initial word vector of the word to obtain the pre-trained word vector of the word.

[0132] Example 2

[0133] Based on the same inventive concept, an embodiment of the present invention also provides a device for generating pre-trained word vectors. Since the device is the device in the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0134] like Figure 3 As shown, the device includes a processor 300 and a memory 301, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0135] According to the weights between the nodes with connection relationships in the pre-constructed definition relationship graph, probability sampling is performed on the words corresponding to each node in the definition relationship graph to determine the word sequence, wherein the definition relationship graph includes multiple nodes, each node corresponds to a word, the connection relationship between the nodes is used to represent that there is a semantic interpretation relationship between the corresponding words, the weight is used to represent the degree of semantic relevance between the words with semantic interpretation relationships, and there is a semantic interpretation relationship between adjacent words in the word sequence;

[0136] For any word in the word sequence, the initial word vector of the word is adjusted through the previous words and the following words of the word to obtain the pre-trained word vector of the word, wherein the previous words refer to a preset number of words in the word sequence that are located before the any word, and the following words refer to a preset number of words in the word sequence that are located after the any word.

[0137] As an optional implementation manner, the processor is specifically configured to construct a definition relationship graph in the following manner:

[0138] Establish a one-to-one correspondence between words and nodes in the dictionary;

[0139] Determining words having a semantic interpretation relationship based on the words in the dictionary and the words that interpret the words;

[0140] The nodes corresponding to the words having the semantic interpretation relationship are connected to construct a definition relationship graph.

[0141] As an optional implementation manner, the processor is specifically configured to determine the weights between nodes having a connection relationship in the following manner:

[0142] The weights between connected nodes are determined based on the number of times any word appears in a semantically interpreted sentence and the number of all semantically interpreted sentences containing the any word, wherein the semantically interpreted sentence is a sentence that semantically interprets a word corresponding to a node that is connected to a node corresponding to the any word.

[0143] As an optional implementation manner, the processor is specifically configured to:

[0144] The word corresponding to the preset node is used as the first word in the word sequence;

[0145] Using a random walk algorithm, taking the preset node as a starting node, and performing probability sampling on the connected nodes according to the weights between the nodes connected to the starting node, to determine the next node of the starting node;

[0146] The word corresponding to the next node is taken as the next word in the word sequence. If the length of the word sequence does not reach the preset length, the next node is taken as the new starting node to continue probability sampling to determine the word corresponding to the next node of the new starting node until the length of the word sequence reaches the preset length.

[0147] As an optional implementation manner, the processor is specifically configured to:

[0148] For any word in the word sequence, determine the minimization loss function according to the word vectors of the previous word and the following word of the any word;

[0149] The initial word vector of the word is adjusted using the minimization loss function to obtain a pre-trained word vector of the word.

[0150] Example 3

[0151] Based on the same inventive concept, an embodiment of the present invention also provides a device for generating pre-trained word vectors. Since the device is the device in the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0152] like Figure 4 As shown, the device comprises:

[0153] The word sequence determination unit 400 is used to perform probability sampling on the words corresponding to each node in the definition relationship graph to determine the word sequence according to the weights between the nodes with connection relationships in the pre-constructed definition relationship graph, wherein the definition relationship graph includes a plurality of nodes, each node corresponds to a word, the connection relationship between the nodes is used to represent that there is a semantic interpretation relationship between the corresponding words, the weight is used to represent the semantic relevance between the words with semantic interpretation relationships, and there is a semantic interpretation relationship between adjacent words in the word sequence;

[0154] A pre-trained word vector determination unit 401 is used to adjust the initial word vector of any word in the word sequence through the previous words and the following words of the any word to obtain the pre-trained word vector of the word, wherein the previous words refer to a preset number of words in the word sequence that are located before the any word, and the following words refer to a preset number of words in the word sequence that are located after the any word.

[0155] As an optional implementation, the determination word sequence unit is specifically used to construct a definition relationship graph in the following manner:

[0156] Establish a one-to-one correspondence between words and nodes in the dictionary;

[0157] Determining words having a semantic interpretation relationship based on the words in the dictionary and the words that interpret the words;

[0158] The nodes corresponding to the words having the semantic interpretation relationship are connected to construct a definition relationship graph.

[0159] As an optional implementation, the word sequence determination unit is specifically used to determine the weights between nodes having a connection relationship in the following manner:

[0160] The weights between connected nodes are determined based on the number of times any word appears in a semantically interpreted sentence and the number of all semantically interpreted sentences containing the any word, wherein the semantically interpreted sentence is a sentence that semantically interprets a word corresponding to a node that is connected to a node corresponding to the any word.

[0161] As an optional implementation manner, the determination word sequence unit is specifically used for:

[0162] The word corresponding to the preset node is used as the first word in the word sequence;

[0163] Using a random walk algorithm, taking the preset node as a starting node, and performing probability sampling on the connected nodes according to the weights between the nodes connected to the starting node, to determine the next node of the starting node;

[0164] The word corresponding to the next node is taken as the next word in the word sequence. If the length of the word sequence does not reach the preset length, the next node is taken as the new starting node to continue probability sampling to determine the word corresponding to the next node of the new starting node until the length of the word sequence reaches the preset length.

[0165] As an optional implementation, determining the pre-trained word vector unit is specifically used for:

[0166] For any word in the word sequence, determine the minimization loss function according to the word vectors of the previous word and the following word of the any word;

[0167] The initial word vector of the word is adjusted using the minimization loss function to obtain a pre-trained word vector of the word.

[0168] Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the following steps are implemented:

[0169] According to the weights between the nodes with connection relationships in the pre-constructed definition relationship graph, probability sampling is performed on the words corresponding to each node in the definition relationship graph to determine the word sequence, wherein the definition relationship graph includes multiple nodes, each node corresponds to a word, the connection relationship between the nodes is used to represent that there is a semantic interpretation relationship between the corresponding words, the weight is used to represent the degree of semantic relevance between the words with semantic interpretation relationships, and there is a semantic interpretation relationship between adjacent words in the word sequence;

[0170] For any word in the word sequence, the initial word vector of the word is adjusted through the previous words and the following words of the word to obtain the pre-trained word vector of the word, wherein the previous words refer to a preset number of words in the word sequence that are located before the any word, and the following words refer to a preset number of words in the word sequence that are located after the any word.

[0171] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0172] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that has the functions specified in one or more boxes.

[0173] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0175] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for generating a pre-trained word vector, characterized in that: The method includes: According to the weights between nodes with connection relationships in a pre-constructed definition relationship graph, probability sampling is performed on the words corresponding to each node in the definition relationship graph to determine a word sequence, wherein the definition relationship graph includes multiple nodes, each node corresponds to a word, the connection relationship between the nodes is used to characterize that there is a semantic interpretation relationship between the corresponding words, the weight is used to characterize the degree of semantic relevance between the words with semantic interpretation relationships, and there is a semantic interpretation relationship between adjacent words in the word sequence; the definition relationship graph is constructed in the following manner, including: Establishing a one-to-one correspondence between words in the dictionary and nodes; determining words with a semantic interpretation relationship based on the words in the dictionary and the words that interpret the words; connecting the nodes corresponding to the words with the semantic interpretation relationship to construct a definition relationship graph; For any word in the word sequence, the initial word vector of the word is adjusted through the previous words and the following words of the word to obtain the pre-trained word vector of the word, wherein the previous words refer to a preset number of words in the word sequence that are located before the any word, and the following words refer to a preset number of words in the word sequence that are located after the any word.

2. The method according to claim 1, characterized in that The weights between connected nodes are determined as follows: The weights between connected nodes are determined based on the number of times any word appears in a semantically interpreted sentence and the number of all semantically interpreted sentences containing the any word, wherein the semantically interpreted sentence is a sentence that semantically interprets a word corresponding to a node that is connected to a node corresponding to the any word.

3. The method according to claim 1, characterized in that According to the weights between the nodes with connection relationships in the pre-constructed definition relationship graph, probabilistic sampling is performed on the words corresponding to each node in the definition relationship graph to determine the word sequence, including: The word corresponding to the preset node is used as the first word in the word sequence; Using a random walk algorithm, taking the preset node as a starting node, and performing probability sampling on the connected nodes according to the weights between the nodes connected to the starting node, to determine the next node of the starting node; The word corresponding to the next node is taken as the next word in the word sequence. If the length of the word sequence does not reach the preset length, the next node is taken as the new starting node to continue probability sampling to determine the word corresponding to the next node of the new starting node until the length of the word sequence reaches the preset length.

4. The method according to claim 1, characterized in that: The initial word vector of the word is adjusted to obtain the pre-trained word vector of the word, including: For any word in the word sequence, determine the minimization loss function according to the word vectors of the previous word and the following word of the any word; The initial word vector of the word is adjusted using the minimization loss function to obtain a pre-trained word vector of the word.

5. A device for generating pre-trained word vectors, characterized in that: The device includes a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps: According to the weights between nodes with connection relationships in a pre-constructed definition relationship graph, probability sampling is performed on the words corresponding to each node in the definition relationship graph to determine the word sequence, wherein the definition relationship graph includes multiple nodes, each node corresponds to a word, the connection relationship between the nodes is used to characterize that there is a semantic interpretation relationship between the corresponding words, the weight is used to characterize the degree of semantic relevance between the words with semantic interpretation relationships, and there is a semantic interpretation relationship between adjacent words in the word sequence; the processor is specifically used to construct the definition relationship graph in the following manner: Establishing a one-to-one correspondence between words in the dictionary and nodes; determining words with a semantic interpretation relationship based on the words in the dictionary and the words that interpret the words; connecting the nodes corresponding to the words with the semantic interpretation relationship to construct a definition relationship graph; For any word in the word sequence, the initial word vector of the word is adjusted through the previous words and the following words of the word to obtain the pre-trained word vector of the word, wherein the previous words refer to a preset number of words in the word sequence that are located before the any word, and the following words refer to a preset number of words in the word sequence that are located after the any word.

6. The device according to claim 5, characterized in that The processor is specifically used to determine the weights between nodes having a connection relationship in the following manner: The weights between connected nodes are determined based on the number of times any word appears in a semantically interpreted sentence and the number of all semantically interpreted sentences containing the any word, wherein the semantically interpreted sentence is a sentence that semantically interprets a word corresponding to a node that is connected to a node corresponding to the any word.

7. The device according to claim 5, characterized in that The processor is specifically used for: The word corresponding to the preset node is used as the first word in the word sequence; Using a random walk algorithm, taking the preset node as a starting node, and performing probability sampling on the connected nodes according to the weights between the nodes connected to the starting node, to determine the next node of the starting node; The word corresponding to the next node is taken as the next word in the word sequence. If the length of the word sequence does not reach the preset length, the next node is taken as the new starting node to continue probability sampling to determine the word corresponding to the next node of the new starting node until the length of the word sequence reaches the preset length.

8. The device according to claim 5, characterized in that The processor is specifically used for: For any word in the word sequence, determine the minimization loss function according to the word vectors of the previous word and the following word of the any word; The initial word vector of the word is adjusted using the minimization loss function to obtain a pre-trained word vector of the word.

Images