A coding method for word vectorization using point-wise mutual information between words

Abstract

The invention discloses a coding method for word vectorization using point-wise mutual information between words, comprising: step S0, traversing a corpus, segmenting all texts, and obtaining a word set; step S1, initializing word embedding parameters and counting the word frequency of each word; step S2, traversing the corpus, extracting positive and negative sample pairs for training word embedding parameters; step S3, calculating the loss function of the positive and negative sample pairs, and finding and updating the gradient of the corresponding word embedding parameters; step S4, after traversing the corpus for several times, saving the context word vector of each word, and using it as the desired word embedding. The invention can achieve more accurate fitting of the point-wise mutual information between words.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for vectorizing and encoding words (including English words or Chinese words). Background Art

[0002] Word quantization is a very critical step in natural language processing. The goal of word quantization is to convert the symbolic information of natural language into digital information in the form of vectors, and use a continuous vector to represent each word to facilitate subsequent information processing. This process is also called word encoding or word embedding. Currently, common word quantization methods are divided into two categories: one-hot representation model and distributed representation model. The former uses a sparse vector to represent words. The vector length is the size of the dictionary N. Each vector has only one dimension of 1, indicating the position of the word in the dictionary, and the rest of the dimensions are all 0. The latter uses a low-dimensional dense vector to represent words. The vector encodes the contextual information of the word, and the words that appear in the same context have similar encodings. The one-hot encoding method often encounters the problem of dimensionality disaster, and semantic information cannot be represented, making it difficult to reveal the potential connection between different words.

[0003] Word2Vec is currently the most commonly used word encoding method. It has been proven to be able to capture the semantic information of words well and is widely used in a series of fields such as document classification, sentiment analysis, and part-of-speech tagging. In order to obtain word embeddings, traditional methods represented by Word2Vec mainly use the co-occurrence matrix that can describe the frequency of common occurrences between words to estimate word vectors, but there is a problem of poor interpretability of the encoding model, and there is still room for improvement in performance. Summary of the invention

[0004] Based on the problems existing in the prior art, the purpose of the present invention is to provide a coding method for word vectorization using point-wise mutual information between words. The word vectorization coding method proposed in the present invention belongs to a type of distributed representation model. The method proposed in the present invention estimates the word vector by estimating the point-wise mutual information matrix between words. The model has better interpretability and can better describe the semantic similarity of words.

[0005] The embodiment of the present invention provides a coding method for word vectorization using point-wise mutual information between words, wherein the input is an original expected (text) set, and the main steps of the method include:

[0006] Step S0, segmenting each prediction into word sets;

[0007] Step S1, initialize word embedding parameters and count the frequency of each word;

[0008] Step S2, traverse the corpus and extract positive and negative sample pairs for training word embedding parameters;

[0009] Step S3, calculate the loss function of the positive and negative sample pairs, find the gradient of the corresponding word embedding parameters and update them.

[0010] Step S4, after traversing the corpus several times, save the context word vector of each word and use it as the desired word embedding.

[0011] Wherein, the step S0 includes:

[0012] Sub-step S001: traverse the prediction and segment all texts. For Chinese, the text segmentation can be performed using the hidden Markov model method to obtain a set of Chinese words. For English, it can be directly expressed as a set of English words.

[0013] Wherein, the step S1 comprises:

[0014] Sub-step S101: traverse the corpus, count the frequency of each word in the corpus, and use the frequency as noise distribution for subsequent random extraction of words;

[0015] Sub-step S102: randomly initialize a center word vector and a context word vector for each word, where the center word vector is a unit vector and the context word vector is the final word embedding;

[0016] Wherein, the step S2 comprises:

[0017] Sub-step S201: Take each word as the central word in the order in which it appears in the corpus, and extract words from the limited context of the central word as context words. The central word and context words constitute a pair of positive samples;

[0018] Sub-step S202: randomly extracting a number of words from the noise distribution, and taking their context word vectors, together with the current center word, to form a number of pairs of negative samples;

[0019] Sub-step S203: Within the limited range of the current central word vector, two words are sampled from the corpus again and are called additional context words. The additional context word vectors are added to obtain a summed word vector, and the summed word vector and the central word vector constitute a pair of positive samples. The corresponding negative samples are extracted in the manner described in sub-step S202.

[0020] Wherein, the step S3 comprises:

[0021] Sub-step S301: Calculate the inner product of the central word vector and the positive sample context word vector, use the inner product as the input of a binary classification model, and calculate the positive sample loss function when the label is positive; calculate the inner product of the central word vector and the negative sample context word vector, use the inner product as the input of a binary classification model, and calculate the negative sample loss function when the label is negative;

[0022] Sub-step S302: Calculate the gradient of the center word vector and the context word vector for the positive and negative sample loss functions of the context words. The gradient of the center word vector needs to be saved, while the gradient of the context word vector is updated immediately;

[0023] Sub-step S303: Calculate the gradients of the sum and context word vectors respectively and update the parameters of the corresponding additional context word embeddings, as well as the gradient of the center word vector, and save them in the manner described in sub-step S302.

[0024] Sub-step S304: Calculate the average gradient of the accumulated gradient of the central word vector and the corresponding Riemann gradient. Finally, use the Riemann gradient to update the central word vector.

[0025] It can be seen from the technical solution provided by the present invention that the method for word embedding estimation provided by an embodiment of the present invention has the beneficial effect of being able to more effectively fit the point-wise mutual information between words, and to estimate word vectors by estimating the point-wise mutual information matrix between words. The model has better interpretability and can better describe the semantic similarity of words. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 A flow chart of the optimization steps of a method for solving word embedding using noise contrast estimation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the specific content of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in this field.

[0029] like Figure 1As shown, the embodiment of the present invention provides an encoding method for word vectorization using point-wise mutual information between words, which can achieve more accurate fitting of point-wise mutual information between words, including:

[0030] Step S0, traverse the forecast, segment all texts to obtain a word set;

[0031] Step S1, initialize word embedding parameters and count the frequency of each word;

[0032] Step S2, traverse the corpus and extract positive and negative sample pairs for training word embedding parameters;

[0033] Step S3, calculate the loss function of the positive and negative sample pairs, find the gradient of the corresponding word embedding parameters and update them.

[0034] Step S4, after traversing the corpus several times, save the context word vector of each word and use it as the desired word embedding.

[0035] Wherein, the step S0 comprises:

[0036] Sub-step S001: traverse the corpus and segment all the texts in the corpus to obtain a word set. For Chinese, the hidden Markov model segmentation method can be used to obtain the Chinese word segmentation of the text. For English, the word can be directly used as the segmentation result;

[0037] Wherein, the step S1 comprises:

[0038] Sub-step S101: traverse the corpus, count the frequency of each word in the corpus, and use the frequency as noise distribution for subsequent random extraction of words;

[0039] Sub-step S102: randomly initialize a center word vector and a context word vector for each word, where the center word vector is a unit vector and the context word vector is the final word embedding;

[0040] In the sub-step S101 of the method, the result of counting the frequency of each word in the corpus is:

[0041] Each word is labeled with a serial number 1≤i≤V, where V is the size of the vocabulary. The frequency of word i is labeled with 0≤p(i)≤1 and satisfies ∑ i p(i)=1.

[0042] In sub-step S102 of the method, the result of randomly initializing a center word vector and a context word vector for each word is:

[0043] Let d represent the dimension of the word vector. When word i is the central word, use the vector As the central word vector of the word, and always constrain the vector to be a unit vector, that is, ||I i ||2=1; when word i is used as a context word, use vector As the context word vector of the word, and the context word vector will be used as the final word embedding.

[0044] Wherein, the step S2 comprises:

[0045] Sub-step S201: Take each word as the central word in the order in which it appears in the corpus, and extract words from the limited context of the central word as context words. The central word and context words constitute a pair of positive samples;

[0046] Sub-step S202: randomly extracting a number of words from the noise distribution, and taking their context word vectors, together with the current center word, to form a number of pairs of negative samples;

[0047] Sub-step S203: Within the limited range of the current central word vector, two words are sampled from the corpus again and are called additional context words. The additional context word vectors are added to obtain a summed word vector, and the summed word vector and the central word vector constitute a pair of positive samples. The corresponding negative samples are extracted in the manner described in sub-step S202.

[0048] In the sub-step S201 of the method, the result of extracting words as context words as positive samples in the context within the limited range of the central word is:

[0049] Let 1≤t≤T represent the order in which a word appears in the corpus, where T is the length of the corpus. Mark the tth word in the corpus and indicate that its sequence number in the vocabulary is i. Let L represent a fixed integer, then in the corpus, the position is close to the word Words that do not exceed L in length are called Context words. At this time, use represents these context words, and the subscript l satisfies -L≤l≤L and l≠0; It is called the central word. is called a positive sample, and it is considered that the word and words Appear once in the corpus.

[0050] In sub-step S202 of the method, the result of randomly extracting negative samples from the noise distribution is:

[0051] use represents the word with sequence number k in the vocabulary, and the subscript n represents sampling from the noise distribution. Here, the noise distribution is selected as the frequency of each word, that is, k~p(k). For each central word Sample N noise words is called a negative sample.

[0052] In sub-step S203 of the method, additional context words are sampled from the corpus again to form positive samples and negative samples, and the result of calculating and adding the context word vector is:

[0053] exist Randomly select two words w from the expected range again. a With w b As context words. Look up the table for each of the two context words to get the corresponding context word vector O a and O b . a , w b ) as a new positive sample and considers word w a With w b by The condition appears once in the corpus. a +O b Same as I i Find the inner product as (w a , w b )and The score is recorded as a +O b , I i 〉.

[0054] Similarly, 2N noise samples are independently drawn from the noise distribution again and used Represents a negative sample. Query the parameter matrix to get the corresponding context word vector O p With O q After that, use O p +O q Same as I i Find the inner product as and The score is recorded as p +O q , I i 〉.

[0055] Wherein, the step S3 comprises:

[0056] Sub-step S301: Calculate the inner product of the central word vector and the positive sample context word vector, use the inner product as the input of a binary classification model, and calculate the positive sample loss function when the label is positive; calculate the inner product of the central word vector and the negative sample context word vector, use the inner product as the input of a binary classification model, and calculate the negative sample loss function when the label is negative;

[0057] Sub-step S302: Calculate the gradient of the center word vector and the context word vector for the positive and negative sample loss functions of the context words. The gradient of the center word vector needs to be saved, while the gradient of the context word vector is updated immediately;

[0058] Sub-step S303: Calculate the gradients of the sum and context word vectors respectively and update the parameters of the corresponding additional context word embeddings, as well as the gradient of the center word vector, and save them in the manner described in sub-step S302.

[0059] Sub-step S304: Calculate the average gradient of the accumulated gradient of the central word vector and the corresponding Riemann gradient. Finally, use the Riemann gradient to update the central word vector.

[0060] In the sub-step S301 of the method, the inner product of the central word vector and the positive and negative sample context word vector is calculated. The inner product is used as the energy function of a binary classification model, and the result of the positive and negative sample loss function is calculated as follows:

[0061] The central word Look up the table from the parameter matrix to get its central word vector I i ; for each context word in the positive sample Look up the corresponding context word vector O from the parameter matrix j Calculation I i With O j The inner product of and The score is recorded as j , I i 〉. Finally, j , I i 〉Substitute the following logistic function:

[0062]

[0063] The output σ( <O j , I i >) is the word embedding model pair is the probability prediction of the positive sample. Finally, taking the logarithm of the probability, we get the following positive sample binary classification loss:

[0064] log(σ( <O j , I i >)).

[0065] For each context word in the negative sample Query the corresponding context word vector O from the parameter matrix k . Calculation I i With O k The inner product of and The score is recorded as <Ok , I i >. Finally, <O k , I i >Substitute the above logistic function. Because is a negative sample, so its negative sample binary classification loss is:

[0066] log(1-σ( <O k , I i >)).

[0067] In sub-step S302 of the method, the result of calculating the gradient of the center word vector and the context word vector for the positive and negative sample loss functions is:

[0068] use Represents the loss log(σ( <O j , I i >)) For the center word vector I i The Euclidean gradient (1-σ(j, i))O j . All positive samples The resulting Euclidean gradient It is stored and not updated, and recorded as use Represents the loss of each positive sample for the context word vector O j The gradient of (1-σ(j, i))I i , and immediately update each

[0069]

[0070] where η t The current central word The step length of η t As the corpus is traversed, it continues to decrease from the initial value to 0. Assuming that the algorithm only traverses the corpus once, then η t The update formula is:

[0071]

[0072] use Represents the loss of each negative sample log(1-σ( <O k , I i >)) For the center word vector I i The Euclidean gradient -σ(k,i)O i . All negative samples The resulting Euclidean gradient It is stored and not updated, and recorded as use Represents the loss of each negative sample for the context word vector O kThe gradient of -σ(k,i)I i , and update immediately.

[0073] In sub-step S303 of the method, the result of updating the word embedding parameters with the additional context word vector is:

[0074] Will <O a +O b , I i >Substitute the above logistic function and take the logarithm to get the new positive sample loss function:

[0075] log(σ( <O a +O b , I i >)).

[0076] Will <O p +O q , I i >Substitute the above logistic function and take the logarithm to get the new negative sample loss function:

[0077] log(1-σ( <O p +O q , I i >)).

[0078] use represents log(σ( <O a +O b , I i >)) For the center word vector I i The Euclidean gradient (1-σ(a, b))(O a +O b ). The positive sample (w a , w b ) produces the Euclidean gradient It is stored and not updated. and represents log(σ( <O a +O b , I i >)) For the context word vector O a With O b The gradient of (1-σ(a+b, i))I i , and update immediately and use Represents the loss of each negative sample log(1-σ( <O p +O q , I i >)) For the center word vector I i The Euclidean gradient -σ(p+q,i)(O p+O q ). All negative samples The resulting Euclidean gradient It is stored and not updated, and recorded as use and Represents the loss of each negative sample for the context word vector O p With O q The gradient is updated immediately.

[0079] In the sub-step S304 of the method, the average gradient of the accumulated gradient of the central word vector and the corresponding Riemann gradient are calculated. Finally, the result of updating the central word vector using the Riemann gradient is:

[0080] use Represents the center word embedding I i The average Euclidean gradient is calculated as follows:

[0081]

[0082] in Represents the set of the aforementioned accumulated gradients, the specific elements include

[0083] Get the average Euclidean gradient After that, we need to further convert Projection onto the unit sphere at I i The corresponding Riemann gradient grad(I i ):

[0084]

[0085] Where Identity is the identity matrix with only the diagonal elements being 1 and all other elements being 0.

[0086] Finally, update the central word vector I according to the following formula i :

[0087]

[0088] in The calculation method is as follows:

[0089]

[0090] The advantage of the present invention is that it constructs a network model that combines a probability distribution model with a traditional sparse coding solution method, compared with the existing solution methods.

[0091] So far, the present invention has been described in detail with reference to the accompanying drawings. According to the above description, those skilled in the art should have a clear understanding of the present invention.

[0092] It should be noted that the implementation methods not shown or described in the drawings or the text of the specification are all forms known to ordinary technicians in the relevant technical field and are not described in detail. In addition, the above definitions of each element are not limited to the various specific structures, shapes or methods mentioned in the embodiments, and ordinary technicians in the field can simply change or replace them, for example:

[0093] (1) Directional terms mentioned in the embodiments, such as "upper", "lower", "front", "back", "left", "right", etc., are only used to refer to the directions of the drawings and are not intended to limit the scope of protection of the present invention;

[0094] (2) The above embodiments can be used in combination with each other or with other embodiments based on design and reliability considerations. That is, the technical features in different embodiments can be freely combined to form more embodiments.

[0095] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A coding method for word vectorization using point-wise mutual information between words, characterized in that: include: Step S0, segmenting the corpus to obtain a word set; Step S1, initialize word embedding parameters and count the frequency of each word; Step S2, traverse the corpus and extract positive and negative sample pairs for training word embedding parameters; Step S2 of the method comprises: Sub-step S201: taking each word as a central word in the order in which it appears in the corpus, and extracting words from the limited context of the central word as context words, the central word and the context words forming a pair of positive samples; Sub-step S202: randomly extracting a number of words from the noise distribution, and taking their context word vectors, together with the current center word, to form a number of pairs of negative samples; Sub-step S203: Within the limited range of the current central word vector, two words are sampled from the corpus again and are called additional context words. The additional context word vectors are added to obtain a summed word vector. The summed word vector and the central word vector constitute a pair of positive samples. The corresponding negative samples are extracted in the manner described in sub-step S202. Step S3, calculate the loss function of the positive and negative sample pairs, find the gradient of the corresponding word embedding parameters and update them; Step S4, after traversing the corpus several times, save the context word vector of each word and use it as the desired word embedding; The sub-step S203: make ,exist About Randomly select two words from the expected range again and As context words, look up the table for each of the two context words to get the corresponding context word vector and ,Will As a new positive sample, and consider the word and by The condition appears once in the corpus, using same Find the inner product as and The score is recorded as ; Let N be a constant. Similarly, 2N noise samples are independently drawn from the noise distribution again and used Represents a negative sample, query the parameter matrix to get the corresponding context word vector and After that, use same Find the inner product as and The score is recorded as ; Step S3 of the method comprises: Sub-step S301: Calculate the inner product of the central word vector and the positive sample context word vector, use the inner product as the input of a binary classification model, and calculate the positive sample loss function when the label is positive; calculate the inner product of the central word vector and the negative sample context word vector, use the inner product as the input of a binary classification model, and calculate the negative sample loss function when the label is negative; Sub-step S302: Calculate the gradient of the center word vector and the context word vector based on the positive and negative sample loss functions of the context words. The gradient of the center word vector needs to be saved, while the gradient of the context word vector is updated immediately. Sub-step S303: Calculate the gradients of the summed and context word vectors respectively and update the parameters of the corresponding additional context word embeddings and the gradient of the center word vector in the same manner as described in sub-step S302 and save them; Sub-step S304: Calculate the average gradient of the accumulated gradient of the central word vector and the corresponding Riemann gradient, and finally use the Riemann gradient to update the central word vector; The sub-step S303: make represents the logistic function, represents the input of the logistic function, then The calculation formula is as follows: ， Will Substituting the aforementioned logistic function and taking the logarithm value, we get a new positive sample loss function: ； Will Substituting the aforementioned logistic function and taking the logarithm value, we get a new negative sample loss function: ； use express For the center word vector The Euclidean gradient , the positive sample The resulting Euclidean gradient Stored and not updated, use and express For context word vector and Gradient , and update immediately and ,use Represents the loss of each negative sample For the center word vector The Euclidean gradient ; All negative samples The resulting Euclidean gradient It is stored and not updated, and recorded as ;use and Represents the loss of each negative sample for the context word vector and The gradient of , and update it immediately; The sub-step S304: use Represents the center word embedding The average Euclidean gradient is calculated as follows: ， in Represents the set of the aforementioned accumulated gradients, the specific elements include ; Get the average Euclidean gradient After that, we need to further convert Projected onto the unit sphere The corresponding Riemann gradient is obtained on the tangent plane at : ， in is the identity matrix with only diagonal elements being 1 and other elements being 0; make Indicates the current central word The step size is , and finally the center word vector is updated according to the following formula : ， in The calculation method is as follows: 。 2. According to claim 1, a coding method for word vectorization using point-wise mutual information between words is characterized in that: Step S1 of the method comprises: Sub-step S101: traverse the corpus, count the frequency of each word in the corpus, and use the frequency as noise distribution for subsequent random extraction of words; Sub-step S102: randomly initialize a central word vector and a context word vector for each word, where the central word vector is a unit vector and the context word vector is the final word embedding.

3. A coding method for word vectorization using point-wise mutual information between words as claimed in claim 2, characterized in that: The sub-step S102: make represents the length of the vector, let Indicates the Euclidean space to which the vector belongs. When the word As the central word, use vector As the central word vector of the word, and always constrain the vector to be a unit vector, that is, ; When the word As a context word, use vector As the context word vector of the word, and the context word vector will be used as the final word embedding.

Images