A word vector training method and device

Abstract

The present invention provides a word vector training method and device, the method comprising: pre-training an initialization word vector of a word to obtain a pre-trained word vector; training the pre-trained word vector using a semantically fused interpretation autoencoder to obtain a target word vector; wherein the semantically fused interpretation autoencoder fuses the semantic relationship of the word and the dictionary interpretation of the word; the embodiment of the present invention fuses the semantic relationship and the dictionary interpretation in the semantically fused interpretation autoencoder, significantly improving the modeling ability of the word vector for semantic similarity; the target word vector trained by using the semantically fused interpretation autoencoder has strong universality, and the word vector training method also has the advantages of short training time, low resource consumption, and the like.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a word vector training method and device. Background Art

[0002] Converting natural language "symbols" that humans can understand into "digital" vectors that computers can process is a necessary step in natural language processing. This method of using vectors to express words is called word vector (or word embedding) technology. Before the rise of neural network technology, the commonly used word vector was to encode words in binary, distinguishing different words according to the different positions of 1. However, due to the large number of words, this vector has the characteristics of high dimensionality, high sparsity, and independence from each other, making it difficult for subsequent models to extract effective semantic information from it.

[0003] Subsequently, a neural language model was proposed, which opened the precedent of using real number vectors as word vector expressions. This type of word vector has the advantages of low dimensionality, strong expressiveness, and modeling semantics, making it possible to perform natural language processing based on a neural network model. It was further proposed to use contextual information for word vector training, and word vector identifiers containing semantic information were obtained. For example, after the word vectors trained on a large-scale corpus are visualized through dimensionality reduction, it can be seen that words with similar semantics are more concentrated in the vector space, indicating that there is a strong correspondence between the learned word vector space and the semantic space. This method of word vector training using a large amount of corpus for unsupervised training has received widespread attention. These classic word vectors have been widely used in downstream tasks of natural language processing as a series of lightweight and effective word embedding models.

[0004] Although the above-mentioned word vector models learn semantic relevance through the statistical co-occurrence information of words in large-scale corpora, due to the lack of supervision of human language knowledge, such methods are difficult to effectively model semantic similarity. For example, "coffee" and "cup" often appear in the same context. They have a high correlation, but they are very different in semantics; on the contrary, "cheap" and "low price" are semantically similar, but they are rarely used at the same time. This lack of semantic similarity modeling will seriously affect downstream tasks that rely on semantic similarity, such as natural language understanding, and limit the overall completion level of the task. Using external linguistic knowledge to semantically enhance classic word vectors can effectively alleviate the above problems and increase the semantic similarity modeling ability of word vectors. Existing methods are mainly divided into two categories: the first category uses semantic relationship information such as synonyms and antonyms to adjust the position of word vectors, for example, making the word vectors of synonyms close to each other and the word vectors of antonyms far away from each other; the second category uses the semantic knowledge provided by the dictionary to improve the training of word vectors. These methods have observed performance improvements in semantic similarity benchmarks and multiple downstream tasks, verifying the effectiveness of injecting external linguistic knowledge in improving the semantic similarity modeling of word vectors.

[0005] There are two problems with existing methods of semantically enhancing word embeddings using external linguistic knowledge: (1) They are all based on the use of a single external linguistic knowledge source. The single knowledge source has narrow coverage and incomplete semantic knowledge, which greatly limits the further improvement of word embedding performance. (2) Existing enhancement methods are not suitable for fusing dictionary information and semantic relationships because the models and training methods used by different methods are very different. Direct combination has poor results and low efficiency. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide a word vector training method and device to solve the problem of narrow coverage of a single knowledge source in the word vector training method of the prior art.

[0007] In order to solve the above problems, an embodiment of the present invention provides a word vector training method, including:

[0008] Pre-train the initial word vector of the word to obtain the pre-trained word vector;

[0009] The pre-trained word vector is trained using a semantically fused interpretation autoencoder to obtain a target word vector;

[0010] The semantically fused interpretation autoencoder fuses the semantic relationship of the words and the dictionary interpretation of the words.

[0011] Wherein, the semantically fused interpretation autoencoder comprises an encoder and a decoder;

[0012] The method of training the pre-trained word vector using the semantically fused interpretation autoencoder to obtain a target word vector includes:

[0013] Determine the interpretation sentence of the word according to the dictionary interpretation of the word;

[0014] Using the encoder to encode the interpretation sentence of the word to obtain a interpretation vector;

[0015] Using the decoder to decode the interpretation vector to reconstruct the word vector of the word in the interpretation sentence;

[0016] Reconstructing the word vectors of the synonyms and antonyms of the word according to the semantic relationship of the word using the decoder;

[0017] The target word vector includes: the word vector of the word in the interpretation, the word vector of the synonym of the word, and the word vector of the antonym of the word.

[0018] Among them, the loss function of the semantic fusion interpretation autoencoder is:

[0019] J(θ,θ′)=J r (θ,θ′)+αJ p (θ)+βJ s (θ′)

[0020] Among them, J(θ,θ′) is the loss function of the semantic fusion interpretation autoencoder; J r (θ,θ′) is the first loss function constraining the reconstruction of the paraphrase; J p (θ) is the second loss function that constrains the interpretation vector and the pre-trained word vector; J s (θ′) is a third loss function for constraining the word vectors of the synonyms and antonyms of the word; α and β are pre-set parameters.

[0021] Among them, the first loss function is:

[0022]

[0023] Among them, J r (θ,θ′) is the first loss function; V K are all the defined words in the dictionary corresponding to the word, w is the word; f θ (D w ) is the interpretation vector; D w is the definition sentence of the word; log P θ ′(Dw |f θ (D w )) represents the paraphrase reconstructed by the decoder; θ represents a trainable parameter; θ′ represents the set of all parameters in the decoder.

[0024] Wherein, the method further comprises:

[0025] Determine the paraphrase sentence log P reconstructed by the decoder θ ′(D w |f θ (D w ));

[0026] in,

[0027]

[0028] h=f θ (D w )

[0029] Where E′ is a dimension of |V D |×m parameter matrix, where m is the word vector dimension and b′ is the dimension |V D | parameter vector; E′ k , and b′ k They are the parameters corresponding to the words indicated by the subscripts; t represents the sequence number of the word in the interpretation sentence.

[0030] Among them, the second loss function is:

[0031]

[0032] Among them, J p (θ) is the second loss function; V K are all the defined words in the dictionary corresponding to the word, w is the word; V D is all the words that appear in the dictionary corresponding to the word; E w , is the word vector corresponding to word w; f θ (D w ) is the interpretation vector.

[0033] Wherein, the third loss function is:

[0034]

[0035] Among them, J s (θ′) is the third loss function; S w is a synonym set of word w; A w The antonym of word w is urgent; E wis the pre-trained word vector of word w; logP θ′ (c|E w ) represents a given input word vector E w The decoder model outputs the probability of word c when .

[0036] The embodiment of the present invention also provides a word vector training device, including:

[0037] The first training module is used to pre-train the initialization word vector of the word to obtain the pre-trained word vector;

[0038] A second training module is used to train the pre-trained word vector using a semantically fused interpretation autoencoder to obtain a target word vector;

[0039] The semantically fused interpretation autoencoder fuses the semantic relationship of the words and the dictionary interpretation of the words.

[0040] An embodiment of the present invention also provides a word vector training device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the word vector training method as described above when executing the program.

[0041] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the word vector training method as described above.

[0042] The above technical solution of the present invention has at least the following beneficial effects:

[0043] In the word vector training method and device of the embodiment of the present invention, semantic relations and dictionary definitions are integrated in the semantically fused interpretation autoencoder, which significantly improves the modeling ability of word vectors for semantic similarity; the target word vectors trained using the semantically fused interpretation autoencoder have strong universality, and the word vector training method also has the advantages of short training time and low resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart showing the steps of the word vector training method provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram showing the principle of the word vector training method provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram showing the structure of a semantically fused interpretation autoencoder in a word vector training method provided in an embodiment of the present invention;

[0047] Figure 4A schematic diagram showing encoding and decoding of a paraphrase sentence in a word vector training method provided by an embodiment of the present invention;

[0048] Figure 5 A schematic diagram showing decoding of semantic relations in the word vector training method provided in an embodiment of the present invention;

[0049] Figure 6 A schematic diagram showing the structure of a word vector training device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0051] like Figure 1 As shown, an embodiment of the present invention provides a word vector training method, including:

[0052] Step 11, pre-train the initialization word vector of the word to obtain the pre-trained word vector.

[0053] Step 12: Use a semantically fused interpretation autoencoder to train the pre-trained word vector to obtain a target word vector; wherein the semantically fused interpretation autoencoder fuses the semantic relationship of the word and the dictionary interpretation of the word.

[0054] The semantically fused interpretation autoencoder in the embodiment of the present invention uses two parts of external linguistic knowledge, namely dictionary interpretation and synonym / antonym semantic relations, to enhance word vectors, thus overcoming the problem that existing methods can only utilize a single knowledge source.

[0055] Specifically, Figure 2 As shown in the figure, the word vector is pre-trained on a large-scale corpus using the classic method to obtain the pre-trained word vector; the semantic fusion-based interpretation autoencoder adds semantics to the pre-trained word vector to obtain the target word vector; the optimized target word vector can be applied to various downstream tasks of natural language processing. This word vector training method can make full use of the two parts of information, semantic relationship and dictionary interpretation, to improve the semantic similarity modeling ability of word vectors.

[0056] In this step, the semantic fusion interpretation autoencoder needs to use two external linguistic knowledge sources, dictionary interpretation and semantic relationship (semantic relationship is specifically synonymous and antonym semantic relationship), which can be obtained by downloading the electronic version of the Modern Chinese Dictionary, Synonym and Antonym Cilin and other open source databases. The acquired knowledge source is preprocessed, such as word segmentation, polysemous splicing, truncation, etc. of the dictionary interpretation.

[0057] The embodiment of the invention proposes a semantically fused definition autoencoder (DASI) method. In the semantically fused definition autoencoder, a unified model (i.e., a semantically fused definition autoencoder) is used to fuse the two parts of information, semantic relationship and dictionary definition, which significantly improves the modeling ability of word vectors for semantic similarity. The word vectors trained using this method not only surpass the existing methods that use a single external linguistic knowledge, improve the performance on the semantic similarity benchmark dataset, but also significantly outperform the direct combination of existing methods. At the same time, the word vectors trained using the semantically fused definition autoencoder method have strong universality, and have achieved significant performance improvements when optimizing different word vector spaces and applying to different models of natural language understanding in downstream tasks-dialogue systems. And this method also has the advantages of short training time and low resource consumption, and has important application value for natural language processing models based on word vectors.

[0058] As an optional embodiment, the semantically fused interpretation autoencoder includes an encoder and a decoder; step 12 includes:

[0059] Determine the interpretation sentence of the word according to the dictionary interpretation of the word;

[0060] Using the encoder to encode the interpretation sentence of the word to obtain a interpretation vector;

[0061] Using the decoder to decode the interpretation vector to reconstruct the word vector of the word in the interpretation sentence;

[0062] Reconstructing the word vectors of the synonyms and antonyms of the word according to the semantic relationship of the word using the decoder;

[0063] The target word vector includes: the word vector of the word in the interpretation, the word vector of the synonym of the word, and the word vector of the antonym of the word.

[0064] In an embodiment of the present invention, the encoder is used to encode the interpretation sentence of a word into an interpretation vector, the decoder is used to decode the interpretation vector and reconstruct it into the interpretation sentence, and the decoder is also used to reconstruct the semantic relationship of a word into the word vector of its synonyms and the word vector of its antonyms.

[0065] For example, Figure 3 As shown in the figure, the word "happy" can be reconstructed from its word vector into the corresponding sentence "feeling happy and relaxed", as well as the synonym "happy" and antonym "sad".

[0066] As an optional embodiment, the optimization objective of the semantically fused interpretation autoencoder consists of three parts: a loss function for a given reconstructed interpretation sentence (i.e., a first loss function J constraining the reconstructed interpretation sentence); r (θ,θ′)), one is the L2 regular constraint term between the interpretation vector and the word vector (that is, the second loss function J that constrains the interpretation vector and the pre-trained word vector p (θ)), one is the loss function of reconstructing the corresponding synonyms and antonyms from the word vector (i.e., the third loss function J constrains the word vectors of the synonyms and antonyms of the word s (θ′)); Under the joint guidance of the above three loss functions, the post-processed word vector can contain more semantic information.

[0067] Optionally, the loss function of the semantically fused interpretation autoencoder is:

[0068] J(θ,θ′)=J r (θ,θ′)+αJ p (θ)+βJ s (θ′)

[0069] Among them, J(θ,θ′) is the loss function of the semantic fusion interpretation autoencoder; J r (θ,θ′) is the first loss function constraining the reconstruction of the paraphrase; J p (θ) is the second loss function that constrains the interpretation vector and the pre-trained word vector; J s (θ′) is a third loss function for constraining the word vectors of the synonyms and antonyms of the word; α and β are pre-set parameters.

[0070] Combining the target loss function of the above three loss functions, the two parts of external language information (dictionary interpretation and semantic relationship) can be jointly optimized in a unified training process, so that the optimized word vector contains rich semantic similarity information, and the performance is significantly better than the direct combination of existing semantic enhancement.

[0071] Assume V D are all the words that appear in the dictionary, V K are all the words defined in the dictionary. Let word be w, w∈V K ; The definition of word w is D w =(D w,1 ,D w,2 ,…,D w,T ), where D w,T is the vocabulary V D The encoder in the embodiment of the present invention adopts a long short-term memory unit (LSTM) to decode the meaning sentence D wEncoding, each word is passed through the parameter matrix E to obtain the corresponding word vector, and then sent to LSTM for calculation one by one. The hidden layer output of the last step is subjected to a linear transformation to obtain the sentence vector representation of the interpretation sentence, which is called the interpretation vector h or f θ (D w ).

[0072] h=f θ (D w )=W.LSTM(D w )+b

[0073] Among them, θ represents the trainable parameters, including the parameters of LSTM and the linear layer parameters W and b.

[0074] Furthermore, based on the paraphrase vector, the decoder can reconstruct the paraphrase sentence using the bag-of-words model, and the training criterion is to maximize the posterior probability of the words in the paraphrase sentence.

[0075] As an optional embodiment, the first loss function is:

[0076]

[0077] Among them, J r (θ,θ′) is the first loss function; V K are all the defined words in the dictionary corresponding to the word, w is the word; f θ (D w ) is the interpretation vector; D w is the definition sentence of the word; log P θ′ (D w |f θ (D w )) represents the paraphrase reconstructed by the decoder; θ represents a trainable parameter; θ′ represents the set of all parameters in the decoder.

[0078] Optionally, the method further includes:

[0079] Determine the paraphrase sentence log P reconstructed by the decoder θ ′(D w |f θ (D w ));

[0080] in,

[0081]

[0082] h=f θ (D w )

[0083] Where E′ is a dimension of |V D|×m parameter matrix, where m is the word vector dimension and b′ is the dimension |V D | parameter vector; E′ k , and b′ k They are the parameters corresponding to the words indicated by the subscripts; t represents the sequence number of the word in the interpretation sentence.

[0084] As another optional embodiment, the second loss function is an L2 regularization term, also known as a consistency penalty term, which requires that the distance between the meaning vector and the word vector of a word is close. Let the word vector corresponding to word w be E w ; Then the second loss function is:

[0085]

[0086] Among them, J p (θ) is the second loss function; V K are all the defined words in the dictionary corresponding to the word, w is the word; V D is all the words that appear in the dictionary corresponding to the word; E w , is the word vector corresponding to word w; f θ (D w ) is the interpretation vector. Where d(·) is the Euclidean distance.

[0087] As another optional embodiment, the third optimization function of the semantic fusion interpretation autoencoder is a constraint item for the semantic relationship. A good word vector should satisfy the requirement that the word vectors of synonyms are close and the word vectors of antonyms are far apart. Assume that the synonym set of word w is S w , the antonym set is A w , then the word vector E of the word is required w Synonyms and antonyms can be reconstructed using the bag-of-words model in the same way as the autoencoder; that is, the third loss function is:

[0088]

[0089] Among them, J s (θ′) is the third loss function; S w is a synonym set of word w; A w The antonym of word w is urgent; E w is the pre-trained word vector of word w; logP θ′ (c|E w ) represents a given input word vector E w The decoder model outputs the probability of word c when .

[0090] The semantic relationship decoder and the decoder for the interpretation vector share parameters, both of which are θ′.

[0091] In order to more clearly describe the word vector training method provided by an embodiment of the present invention, an example and accompanying drawings are used for illustration below.

[0092] First, obtain the common word vectors that have been trained on a large corpus, then prepare the corresponding dictionary with word definitions and a vocabulary of synonyms and antonyms, and then use the semantically fused interpretation autoencoder to fine-tune the common word vectors, and then determine whether to terminate the training based on the performance of the word benchmark dataset.

[0093] like Figure 4 As shown in the figure, the dictionary definition of the word "good quality and cheap" is "cheap things with good quality". First, the word vector of each word is extracted from the definition sentence through the embedding matrix, and then input into the long short-term unit (LSTM) one by one to obtain the definition vector. The definition vector and the word vector corresponding to the word "good quality and cheap" are constrained by the consistency penalty term. At the same time, the definition vector can predict the words in the definition sentence, and the optimization method is the maximum likelihood criterion. At the same time, the word vector corresponding to "good quality and cheap" can also use the same decoder to predict the antonym and synonym of "good quality and cheap". The antonym adopts the minimum likelihood criterion, and the synonym adopts the maximum likelihood criterion, as shown in the figure. Figure 5 shown.

[0094] In the experiment, since there is no dataset for Chinese, the total number of iterations is directly set to 5 for model training. The hyperparameters of the loss function of the semantic fusion interpretation autoencoder are set to α=10 and β=0.1. The following is a quantitative explanation of the above advantages based on the experimental results obtained on the international standard dataset. The experimental results are shown in Table 1, Table 2, and Table 3, where Attract-Repel and CPAE are the best existing methods using a single knowledge source. From the evaluation results in Table 1, it can be seen that the semantic fusion interpretation autoencoder method (hereinafter referred to as DASI) has achieved significant improvements in three different evaluation indicators when optimizing different pre-trained word vectors, and has achieved an improvement of up to 10 points (23%) over the existing optimization methods. Table 2 shows the comparison results of DASI directly combined with existing methods, and its improvement over the best existing methods on the SV-test evaluation dataset is still as high as 12%. Finally, in the downstream natural language understanding task, DASI has achieved a performance improvement of more than 1% in the "target accuracy" of both dialogue state tracking models, which is significant in this task.

[0095]

[0096]

[0097] Table 1 Performance improvement on word similarity benchmark dataset

[0098]

[0099] Table 2 Comparison results of direct combination of the embodiment of the present invention (DASI) and the existing method

[0100]

[0101] Table 3 Performance improvement on downstream natural language understanding tasks

[0102] In order to better solve the above problems, Figure 6 As shown, the embodiment of the present invention also provides a word vector training device, including:

[0103] The first training module 61 is used to pre-train the initialization word vector of the word to obtain the pre-trained word vector;

[0104] A second training module 62 is used to train the pre-trained word vector using a semantically fused interpretation autoencoder to obtain a target word vector;

[0105] The semantically fused interpretation autoencoder fuses the semantic relationship of the words and the dictionary interpretation of the words.

[0106] Optionally, in the above embodiment of the present invention, the semantically fused interpretation autoencoder includes an encoder and a decoder;

[0107] The second training module includes:

[0108] The first submodule is used to determine the interpretation sentence of the word according to the dictionary interpretation of the word;

[0109] A second submodule is used to encode the interpretation sentence of the word using the encoder to obtain a interpretation vector;

[0110] A third submodule is used to use the decoder to decode the interpretation vector to reconstruct the word vector of the word in the interpretation sentence;

[0111] A fourth submodule is used to reconstruct the word vector of the synonym and the word vector of the antonym of the word using the decoder according to the semantic relationship of the word;

[0112] The target word vector includes: the word vector of the word in the interpretation, the word vector of the synonym of the word, and the word vector of the antonym of the word.

[0113] Optionally, in the above embodiment of the present invention, the loss function of the semantically fused interpretation autoencoder is:

[0114] J(θ,θ′)=J r (θ,θ′)+αJ p (θ)+βJ s (θ′)

[0115] Among them, J(θ,θ′) is the loss function of the semantic fusion interpretation autoencoder; J r (θ,θ′) is the first loss function constraining the reconstruction of the paraphrase; J p (θ) is the second loss function that constrains the interpretation vector and the pre-trained word vector; J s (θ′) is a third loss function for constraining the word vectors of the synonyms and antonyms of the word; α and β are pre-set parameters.

[0116] Optionally, in the above embodiment of the present invention, the first loss function is:

[0117]

[0118] Among them, J r (θ,θ′) is the first loss function; V K are all the defined words in the dictionary corresponding to the word, w is the word; f θ (D w ) is the interpretation vector; D w is the definition sentence of the word; log P θ ′(D w |f θ (D w )) represents the paraphrase reconstructed by the decoder; θ represents a trainable parameter; θ′ represents the set of all parameters in the decoder.

[0119] Optionally, in the above embodiment of the present invention, the device further includes:

[0120] A determination module is used to determine the paraphrase sentence logP reconstructed by the decoder θ′ (D w |f θ (D w ));

[0121] in,

[0122]

[0123] h=f θ (D w )

[0124] Where E′ is a dimension of |V D |×m parameter matrix, where m is the word vector dimension and b′ is the dimension |V D| parameter vector; E′ k , and b′ k They are the parameters corresponding to the words indicated by the subscripts; t represents the sequence number of the word in the interpretation sentence.

[0125] Optionally, in the above embodiment of the present invention, the second loss function is:

[0126]

[0127] Among them, J p (θ) is the second loss function; V K are all the defined words in the dictionary corresponding to the word, w is the word; V D is all the words that appear in the dictionary corresponding to the word; E w is the word vector corresponding to word w; f θ (D w ) is the interpretation vector.

[0128] Optionally, in the above embodiment of the present invention, the third loss function is:

[0129]

[0130] Among them, J s (θ′) is the third loss function; S w is a synonym set of word w; A w The antonym of word w is urgent; E w is the pre-trained word vector of word w; logP θ′ (c|E w ) represents a given input word vector E w The decoder model outputs the probability of word c when .

[0131] In summary, the embodiments of the present invention fuse semantic relations and dictionary definitions in a semantically fused interpretation autoencoder, significantly improving the modeling capability of word vectors for semantic similarity; the target word vectors trained using the semantically fused interpretation autoencoder have strong universality, and the word vector training method also has the advantages of short training time and low resource consumption.

[0132] It should be noted that the word vector training device provided in the embodiment of the present invention is a device capable of executing the above-mentioned word vector training method. All embodiments of the above-mentioned word vector training method are applicable to the device and can achieve the same or similar beneficial effects.

[0133] An embodiment of the present invention also provides a word vector training device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the various processes in the word vector training method embodiment described above are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be described here.

[0134] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, each process in the above-mentioned word vector training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0136] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A device that specifies functions in one or more processes and / or one or more blocks.

[0137] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable storage medium produce a paper product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that the computer or other programmable device executes a series of operating steps to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0139] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A word vector training method, characterized in that: include: Pre-train the initialization word vector of the word to obtain the pre-trained word vector; The pre-trained word vector is trained using a semantically fused interpretation autoencoder to obtain a target word vector; Wherein, the semantically fused interpretation autoencoder fuses the semantic relationship of the word and the dictionary interpretation of the word; Wherein, the semantically fused interpretation autoencoder comprises an encoder and a decoder; The method of training the pre-trained word vector using the semantically fused interpretation autoencoder to obtain a target word vector includes: Determine the interpretation sentence of the word according to the dictionary interpretation of the word; Using the encoder to encode the interpretation sentence of the word to obtain a interpretation vector; Using the decoder to decode the interpretation vector to reconstruct the word vector of the word in the interpretation sentence; Reconstructing the word vectors of the synonyms and antonyms of the word according to the semantic relationship of the word using the decoder; The target word vector includes: the word vector of the word in the interpretation, the word vector of the synonym of the word, and the word vector of the antonym of the word; Among them, the loss function of the semantic fusion interpretation autoencoder is: J(θ,θ′)=J r (θ,θ′)+αJ p (θ)+βJ s (θ′) Among them, J(θ,θ′) is the loss function of the semantic fusion interpretation autoencoder; J r (θ,θ′) is the first loss function constraining the reconstruction of the paraphrase; J p (θ) is the second loss function that constrains the interpretation vector and the pre-trained word vector; J s (θ′) is a third loss function for constraining the word vectors of the synonyms and antonyms of the word; α and β are pre-set parameters; θ represents a trainable parameter; θ′ represents the set of all parameters in the decoder.

2. The method according to claim 1, characterized in that The first loss function is: Among them, J r (θ,θ′) is the first loss function; V K are all the defined words in the dictionary corresponding to the word, w is the word; f θ (D w ) is the interpretation vector; D w is the definition sentence of the word; log P θ′ (D W |f θ (D w )) represents the paraphrase reconstructed by the decoder; θ represents a trainable parameter; θ′ represents the set of all parameters in the decoder.

3. The method according to claim 2, characterized in that The method further comprises: Determine the paraphrase sentence log P reconstructed by the decoder θ′ (D w |f θ (D w )); in, h=f θ (D w ) Where E′ is a dimension of |V D |×m parameter matrix, where m is the word vector dimension and b′ is the dimension |V D | parameter vector; E′ k , and b′ k are the parameters corresponding to the words indicated by the subscripts; t represents the sequence number of the word in the interpretation sentence; D w,T is the vocabulary V D A word in .

4. The method according to claim 1, characterized in that: The second loss function is: Among them, J p (θ) is the second loss function; V K are all the defined words in the dictionary corresponding to the word, w is the word; V D is all the words that appear in the dictionary corresponding to the word; E w is the word vector corresponding to word w; f θ (D w ) is the interpretation vector.

5. The method according to claim 1, characterized in that The third loss function is: Among them, J s (θ′) is the third loss function; S w is a synonym set of word w; A w is the antonym set of word w; E w is the pre-trained word vector of word w; logP θ′ (c|E w ) represents a given input word vector E w The decoder model outputs the probability of word c when .

6. A word vector training device, characterized in that: include: The first training module is used to pre-train the initialization word vector of the word to obtain the pre-trained word vector; A second training module is used to train the pre-trained word vector using a semantically fused interpretation autoencoder to obtain a target word vector; Wherein, the semantically fused interpretation autoencoder fuses the semantic relationship of the word and the dictionary interpretation of the word; The semantically fused interpretation autoencoder includes an encoder and a decoder; and the second training module is further used for: Determine the interpretation sentence of the word according to the dictionary interpretation of the word; Using the encoder to encode the interpretation sentence of the word to obtain a interpretation vector; Using the decoder to decode the interpretation vector to reconstruct the word vector of the word in the interpretation sentence; Reconstructing the word vectors of the synonyms and antonyms of the word according to the semantic relationship of the word using the decoder; The target word vector includes: the word vector of the word in the interpretation, the word vector of the synonym of the word, and the word vector of the antonym of the word; Among them, the loss function of the semantic fusion interpretation autoencoder is: J(θ,θ′)=J r (θ,θ′)+αJ p (θ)+βJ s (θ′) Among them, J(θ,θ′) is the loss function of the semantic fusion interpretation autoencoder; J r (θ,θ′) is the first loss function constraining the reconstruction of the paraphrase; J p (θ) is the second loss function that constrains the interpretation vector and the pre-trained word vector; J s (θ′) is a third loss function for constraining the word vectors of the synonyms and antonyms of the word; α and β are pre-set parameters; θ represents a trainable parameter; θ′ represents the set of all parameters in the decoder.

7. A word vector training device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the word vector training method as described in any one of claims 1-5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the word vector training method as described in any one of claims 1 to 5 are implemented.

Images