A text abnormal word recognition method and system based on a neural language model

Abstract

The present application relates to the field of text semantic understanding, and more particularly to a text abnormal word recognition method and system based on a neural language model, which is used for recognizing network language, new words and wrong words, and the method comprises the following steps: step 1) collecting text to be recognized; step 2) determining the context word sequence of each word in the text to be recognized in a sliding window manner; step 3) inputting the context word sequence of each word into a pre-established and trained recognition model respectively to obtain the probability of occurrence of each word under the context word sequence; and step 4) comparing the probability of occurrence of each word with a set threshold value to determine whether the word is an abnormal word. The present application does not need to mark abnormal words and can be trained on a large amount of unsupervised data; the prediction probability threshold value of abnormal words and normal words determined by using a large number of positive and negative samples to predict probability statistics has universality; and the selection of negative samples is performed in a negative sampling manner, so that the training speed is accelerated.

Description

Technical Field

[0001] The present invention relates to the field of text semantic understanding, and particularly to a method and system for identifying abnormal words in text based on a neural language model. Background Art

[0002] With the development of the Internet and the continuous increase in the participation of Internet users, a large amount of text data has been published by Internet users. On online platforms such as Weibo and forums, the content published by Internet users has很强的随意性 (a high degree of randomness). Internet terms, new words, and misspelled words emerge in an endless stream, such as "老铁" (buddy), "打call" (cheer for), "稀饭 (homophone for 喜欢, like)", etc. These words that do not express their original meanings in the text are called abnormal words in the text. The appearance of these words has brought great challenges to the work of understanding text semantics. Therefore, identifying abnormal words in text is an important technology for text semantic understanding. By identifying abnormal words in the text, special semantic understanding and selection can be carried out for these abnormal words, and the true meaning of the word in the text can be found, so as to more accurately understand the semantics of the text.

[0003] Currently, the mainstream methods for text semantic understanding are all based on distributed word vectors, that is, each word is mapped to a word vector, and the similarity of the word vectors represents the semantic similarity of the words. Then, for new Internet words, no corresponding word vectors can be found for the existing word vectors; for misspelled words caused by homophony or editing errors, the semantics of the existing word vectors are not the same as the semantics of the words in the text. Therefore, directly using the method of existing word vectors will cause a huge deviation in semantic understanding when encountering abnormal words. Identifying abnormal words in the text and learning and selecting correct word vectors are of great significance for correctly understanding text semantics. However, the current identification of new Internet words still remains at the level of keyword matching constructed manually, and there is no relevant research on the identification of misspelled words in the text, which makes the semantic analysis of social network text always a major challenge.

[0004] Therefore, how to accurately identify various abnormal words that appear in the text is the basis for correctly understanding text semantics and an urgent problem to be solved in text semantic understanding. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and propose a method and system for identifying abnormal words in text based on a neural language model.

[0006] A method for identifying abnormal words in text based on a neural language model, used to identify Internet terms, new words, and misspelled words, the method includes:

[0007] Step 1) Collect the text to be identified;

[0008] Step 2) Determine the context word sequence of each word in the text to be identified by using a sliding window method;

[0009] Step 3) Input the context word sequence of each word into the pre-built and trained recognition model to obtain the probability of each word appearing in the context word sequence;

[0010] Step 4) Compare the probability of each word with the set threshold to determine whether the word is an abnormal word.

[0011] As an improvement to the above method, the input of the recognition model is a word, and the output is the recognition probability of the word. The recognition model includes a word vector layer, a fully connected layer, a hidden layer and a prediction layer connected in sequence.

[0012] As an improvement to the above method, step 2) specifically includes:

[0013] For the text to be recognized, T = [w1, w2, ..., w i ,...,w |T| There are |T| words in total, and the context window length is 2k+1. k is a non-negative integer, and the context word sequence of each word in T is obtained using a sliding window method; where the i-th word w i Using the word as the center, obtain information about the word w. i The context word sequence [w i-k ,...w i-1 ,w i+1 ...,w i+k ].

[0014] As an improvement to the above method, step 3) specifically includes:

[0015] The word w i The context word sequence [w i-k ,...w i-1 ,w i+1 ...,w i+k Input a pre-built and trained recognition model, where,

[0016] The word vector layer uses the word2vec model to convert words into vectors. i The context word sequence [w i-k ,...w i-1 ,w i+1 ...,w i+k ] is mapped to the corresponding word vector [e i-k ,...,e i-1 ,e i+1 ,...,e i+k ];

[0017] The fully connected layer will use word vectors [e] i-k ,...,e i-1 ,ei+1 ,...,e i+k A fully connected vector v is formed. i :

[0018]

[0019] The hidden feature h is calculated using the following formula:

[0020] h = f(Wv) i +b)

[0021] Where W is the hidden layer weight parameter matrix, b is the hidden layer bias parameter vector, and f is the activation function;

[0022] The prediction layer calculates the probability vector y of each word appearing at the position of the i-th word using the following formula:

[0023] y = softmax(Uh + b) h )

[0024] Where U is the prediction layer weight parameter matrix, b h Here is the prediction layer bias parameter vector, and softmax represents the normalized exponential function;

[0025] The word w is obtained from y i The probability y of the i-th word appearing at position i wi y wi ∈[0,1].

[0026] As an improvement to the above method, step 4) specifically includes:

[0027] The word w i The probability y of the i-th word appearing at position i wi It is compared with the set threshold l(m), when y wi When the value is greater than l(m), the word w i If it is a normal word; otherwise, the word w i This is an abnormal word.

[0028] As an improvement to the above method, the method further includes a training step for the recognition model; specifically including:

[0029] The sample set was constructed by using the Wiki Chinese dataset, the Peking University Chinese People's Daily dataset, and / or the Sohu News dataset, and preprocessing the text using a word segmentation tool.

[0030] For each text in the sample set, the word at each position is taken as the center word and recorded as a positive sample. Words other than the center word obtained by a fixed-size context window are recorded as negative samples, forming a training set, which is then input into the recognition model in sequence.

[0031] Set the learning rate and number of training iterations, and use the loss function of maximizing the probability of the word appearing while minimizing the probability of abnormal words to train the model parameters until the training requirements are met, thus obtaining a well-trained recognition model.

[0032] As an improvement to the above method, the method further includes setting a threshold l(m), specifically including:

[0033] Each word in the training set is sequentially input into the trained recognition model to obtain the predicted probability distribution of each center word and negative sampled word. A boundary probability value m is set, and the boundary probability points p of all positive samples are determined based on the boundary probability value m. pos (m) and all negative sample boundary probability points p neg (m):

[0034]

[0035] Where z is the probability of the word appearing as predicted by the recognition model. It is the probability set of all correct words included in the statistics, where count(x≥z) represents the probability of all correct words in the statistics. The number of probability values ​​greater than or equal to z. express The total number of probability values;

[0036]

[0037] in, It is the probability set of all negatively sampled words participating in the statistics, where count(x≤z) represents the probability set of all negatively sampled words. The number of probability values ​​less than or equal to z. express The total number of probability values;

[0038] The threshold l(m) is determined according to the following formula:

[0039] l(m=(p pos (m)+p neg (m)) / 2.

[0040] A text anomaly word recognition system based on a neural language model is disclosed for identifying internet slang, new words, and misspelled words. The system includes: a recognition model, a data acquisition module, a sliding window module, a probability output module, and an anomaly word recognition module.

[0041] The acquisition module is used to acquire the text to be recognized;

[0042] The sliding window module is used to determine the context word sequence of each word in the text to be recognized using a sliding window method;

[0043] The probability output module is used to input the context word sequence of each word into the pre-established and trained recognition model to obtain the probability of each word appearing in the context word sequence.

[0044] The abnormal word identification module is used to compare the probability of each word's occurrence with a set threshold to determine whether the word is an abnormal word.

[0045] As an improvement to the above system, the input of the recognition model is a word, and the output is the recognition probability of the word. The recognition model includes a word vector layer, a fully connected layer, a hidden layer and a prediction layer connected in sequence.

[0046] Compared with the prior art, the advantages of the present invention are:

[0047] 1. This invention borrows the idea of ​​neural language models to detect abnormal words in text. It does not require the labeling of abnormal words and can be trained on a large amount of unsupervised data.

[0048] 2. This invention uses a large number of positive and negative samples to predict probability statistics to determine the prediction probability thresholds for abnormal words and normal words. These thresholds are based on a large amount of data and have universality.

[0049] 3. This invention uses negative sampling to select negative samples, resulting in fast training speed. Attached Figure Description

[0050] Figure 1 This is a flowchart of the method of the present invention;

[0051] Figure 2 This is a schematic diagram of a fully connected neural network neural language model. Detailed Implementation

[0052] To overcome the shortcomings mentioned above, this invention aims to provide a method and system for identifying abnormal words in text based on a neural language model.

[0053] The method includes the following steps:

[0054] Step 1: Prepare the text dataset, including both unlabeled and labeled datasets;

[0055] The unlabeled dataset in step 1 contains a large amount of unlabeled text for training character and word vectors, and can be a large dataset such as Wiki Chinese; the labeled dataset is text with category labels, where each text has a category label, which is used for training the classification model.

[0056] Step 2: Pre-train character vectors and word vectors from the text on the unlabeled dataset;

[0057] In step 2, the character vector refers to representing each character in the text dataset with a unique vector, and all character vectors have the same dimension; the word vector refers to representing each word in the text dataset with a unique vector, and all word vectors have the same dimension. A word is a word obtained by segmenting the text using a word segmentation tool.

[0058] Step 3: For each word in the text, input all the character vectors contained in this word into a Long Short-Term Memory network, and take the output vector of the last node of the Long Short-Term Memory network as a feature vector of this word;

[0059] Step 4: For each word in the text, connect the word vector obtained in Step 2 and the feature vector calculated in Step 3 to form the combined feature vector of that word;

[0060] Step 5: Input the combined feature vector of all words in the text into a Long Short-Term Memory network, and take the output vector of the last node of the Long Short-Term Memory network as the feature vector of the text;

[0061] In steps 3 and 5, characters and words that did not appear in the pre-training are initialized with random values.

[0062] Step 6: Input the text feature vector obtained in Step 5 into a fully connected neural network to build a text classification model;

[0063] Step 7: Train the classification model built in Steps 2 to 5 on the labeled dataset to obtain the parameters of two long short-term memory networks and a fully connected neural network, as well as the updated word vectors and character vectors;

[0064] In step 7, the classification model can be trained on the labeled dataset using the stochastic gradient descent algorithm.

[0065] Step 8: Use the trained model to classify the new text and obtain the classification results.

[0066] The construction and training of the Long Short-Term Memory network and fully connected neural network models can use common frameworks such as Tensorflow.

[0067] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0068] Example 1

[0069] like Figure 1 As shown, Embodiment 1 of the present invention provides a method for identifying abnormal words in text based on a neural language model, comprising the following steps:

[0070] Step 1: Prepare and preprocess the text dataset. The text dataset contains a large amount of text with standardized language expressions. Wiki Chinese, Peking University Chinese People's Daily dataset, Sohu News dataset, etc., can be used. Preprocessing mainly involves using word segmentation tools to segment the Chinese text.

[0071] In this specific implementation, the unlabeled dataset used is the publicly released Wikipedia Chinese dataset, which contains 236,000 text entries after removing text organization information. Word segmentation was performed using the Tsinghua University open-source toolkit, Thulac.

[0072] Step 2: Pre-train word vectors on the text dataset. Specifically, for the segmented text, separate each word with a space, and then input the text into a word vector training tool to obtain the word vector for each word.

[0073] In a specific implementation, word vectors are trained on the dataset prepared in step 1 using the word2vec model of the gensim toolkit, and the dimension of each word vector is 500.

[0074] Step 3: Construct a fully connected neural network as a neural language model, such as... Figure 2 As shown, the model consists of a text input layer, a word vector layer, a fully connected feature layer, a hidden layer, and a prediction layer. Specific construction method:

[0075] 1) Text Input Layer. The input to the text input layer is the words within a certain window size centered on a specific word in a text; that is, the context of a specific word in the text. Note that the word at the center of the window is not used as input, but as the output to be predicted. For a text T = [w1, w2, ..., w...], ... |T| [With the central word w] i For example, if the context window length is 2k+1, then the word input to the input layer is [w i-k ,...w i-1 ,w i+1 ...,w i+k ].

[0076] 2) Word Vector Layer. Using the word vector matrix trained in step 2, all words input to the input layer are mapped to word vectors, and w is used to denote the word vectors. i The word vector is e i So, the input word [w] i-k ,...w i-1 ,w i+1 ...,w i+k The corresponding word vector is:

[0077] [e i-k ,...,e i-1 ,ei+1 ,...,e i+k ]

[0078] 3) Feature Fully Connected Layer. The word vectors mapped in step 2) are fully connected into a single vector:

[0079]

[0080] 4) Hidden Layer. Hidden features are calculated using a linear transformation and a nonlinear function:

[0081] h = f(Wv) i +b)

[0082] Where W is the weight parameter matrix, b is the bias parameter vector, and f is the activation function.

[0083] 5) Prediction layer. A linear transformation and a softmax layer are used to calculate the probability of the word being compared appearing at that position:

[0084] y = softmax(Uh + b) h )

[0085] Where U is the weight parameter matrix, b h This is the bias parameter vector. The dimension of y is the number of words involved in the comparison. If all words are involved in the comparison, then the dimension of y is the vocabulary length. If only a certain number of negative samples are sampled, then the dimension of y is the number of negative samples plus 1. All values ​​in y are between 0 and 1, and the sum of all values ​​is 1. The word represented by the largest value is the predicted word, and the corresponding value is the predicted probability of that word appearing at that position.

[0086] In a specific implementation, the neural language model is constructed using Tensorflow, with a window context length k of 3 and a hidden layer h dimension of 300.

[0087] Step 4: Train the neural language model parameters on the text dataset. The model parameters include all the parameters in the model mentioned in Step 3.

[0088] For each text in the text dataset, the word at each position is taken as the center word, and the words in its context window are taken as the model input. The center word is taken as the positive sample, and a certain number of words are randomly sampled as negative samples. The model parameters are trained with the goal of maximizing the prediction probability of the center word at all positions in the dataset and minimizing the prediction probability of the negative samples.

[0089] In a specific implementation, 10 negative samples are sampled, Tensorflow is used for model training, stochastic gradient descent is used for each batch of 1000 texts, the learning rate is 0.001, and the number of training iterations is 300.

[0090] Step 5: Statistically analyze the probability distribution of correct and abnormal words predicted by the neural language model on the text dataset, and determine the threshold for distinguishing the probability of correct and abnormal words.

[0091] Based on step 4, with the model parameters fixed, the text dataset data is input into the model in the same way as in step 4. This yields the predicted probability distribution for each center word and negative sample word. A boundary probability value m is set, and the positive sample boundary probability point p is determined based on the boundary probability value m. pos (m) and negative sample boundary probability point p neg (m), satisfying:

[0092]

[0093] Where z is the probability of the word appearing as predicted by the recognition model. It is the probability set of all correct words participating in the statistics, where x is a set. The element count(x≥z) represents the number of elements in the x≥z range. The number of probability values ​​greater than or equal to z. express The total number of probability values;

[0094]

[0095] in, It is the set of probabilities obtained from all negatively sampled words participating in the statistics, where x is a set. The element count(x≤z) represents the number of elements in the x≤z range. The number of probability values ​​less than or equal to z. express The total number of probability values;

[0096] The threshold l(m) is determined according to the following formula:

[0097] l(m=(p pos (m)+p neg (m)) / 2.

[0098] In a specific implementation, the boundary probability value m is set to 0.95.

[0099] Step 6: Identification of Abnormal Words in New Text. For new text, a sliding window approach is used to predict the probability of each word appearing at a given position using a neural language model. The probability of the word being an abnormal word is then determined based on the threshold from Step 5. If the predicted probability is greater than l(m), the word is considered normal; if the predicted probability is less than l(m), the word is considered abnormal.

[0100] Example 2

[0101] Embodiment 2 of the present invention proposes a text abnormal word recognition system based on a neural language model for recognizing internet slang, new words, and misspelled words. It is implemented based on the method of Embodiment 1. The system includes: a recognition model, a data acquisition module, a sliding window module, a probability output module, and an abnormal word recognition module; wherein,

[0102] The acquisition module is used to acquire the text to be recognized;

[0103] The sliding window module is used to determine the context word sequence of each word in the text to be recognized using a sliding window method;

[0104] The probability output module is used to input the context word sequence of each word into the pre-established and trained recognition model to obtain the probability of each word appearing in the context word sequence.

[0105] The abnormal word identification module is used to compare the probability of each word's occurrence with a set threshold to determine whether the word is an abnormal word.

[0106] The recognition model takes words as input and outputs the recognition probability of words. The recognition model includes a word vector layer, a fully connected layer, a hidden layer and a prediction layer connected in sequence.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A text abnormal word recognition method based on a neural language model, used to identify internet slang, new words, and misspelled words, the method comprising: Step 1) Collect the text to be recognized; Step 2) Use a sliding window to determine the context word sequence for each word in the text to be recognized; Step 3) Input the context word sequence of each word into the pre-built and trained recognition model to obtain the probability of each word appearing in the context word sequence; Step 4) Compare the probability of each word with the set threshold to determine whether the word is an abnormal word; The setting of the threshold specifically includes: Each word in the training set is sequentially input into the trained recognition model to obtain the predicted probability distribution of each center word and negative sampled word, and boundary probability values ​​are set. According to the boundary probability value Determine the boundary probability points of all positive samples and all negative sample boundary probability points : in, It is the probability of a word appearing as predicted by the recognition model. It is the set of probabilities obtained from all the correct words that participated in the statistics. Indicates in The number of probability values ​​greater than or equal to z. express The total number of probability values; in, It is the set of probabilities obtained from all negatively sampled words participating in the statistics. Indicates in The number of probability values ​​less than or equal to z. express The total number of probability values; The threshold is determined according to the following formula. for: 。 2. The text abnormal word recognition method based on a neural language model according to claim 1, characterized in that, The recognition model takes words as input and outputs the recognition probability of words. The recognition model includes a word vector layer, a fully connected layer, a hidden layer, and a prediction layer connected in sequence.

3. The text abnormal word recognition method based on a neural language model according to claim 1, characterized in that, Step 2) specifically includes: For the text to be recognized There are a total of Words, context window length is , , For non-negative integers, obtain using a sliding window method. T The sequence of context words for each word; where, starting with the first word... i Word Using the central word, obtain information about the word. Context word sequence .

4. The text abnormal word recognition method based on a neural language model according to claim 3, characterized in that, Step 3) specifically includes: Words Context word sequence Input a pre-built and trained recognition model, where, The word vector layer uses the word2vec model to convert words... Context word sequence Mapped to the corresponding word vectors ; The fully connected layer will use word vectors A fully connected vector : The hidden features are calculated using the following formula: : in, This is the hidden layer weight parameter matrix. This is the hidden layer bias parameter vector. For activation functions; The prediction layer calculates the value of each word in the [number]th [order] using the following formula: i The probability vector of the position of each word : in, This is the prediction layer weight parameter matrix. For the prediction layer bias parameter vector, This represents the normalized exponential function; Depend on Get words In the i The probability of a word appearing in a position , .

5. The text abnormal word recognition method based on a neural language model according to claim 4, characterized in that, Step 4) specifically includes: Words In the i The probability of a word appearing in a position With the set threshold When comparing, Greater than Time, words It is a normal word; otherwise, it is a word. This is an abnormal word.

6. The text abnormal word recognition method based on a neural language model according to claim 5, characterized in that, The method also includes a training step for the identification model; specifically including: The sample set was constructed by using the Wiki Chinese dataset, the Peking University Chinese People's Daily dataset, and / or the Sohu News dataset, and preprocessing the text using a word segmentation tool. For each text in the sample set, the word at each position is taken as the center word and recorded as a positive sample. Words other than the center word obtained by a fixed-size context window are recorded as negative samples, forming a training set, which is then input into the recognition model in sequence. Set the learning rate and number of training iterations, and use the loss function of maximizing the probability of the word appearing while minimizing the probability of abnormal words to train the model parameters until the training requirements are met, thus obtaining a well-trained recognition model.

7. A text abnormal word recognition system based on a neural language model, used to identify internet slang, new words, and misspelled words, characterized in that, The system includes: a recognition model, a data acquisition module, a sliding window module, a probability output module, and an abnormal word recognition module; wherein, The acquisition module is used to acquire the text to be recognized; The sliding window module is used to determine the context word sequence of each word in the text to be recognized using a sliding window method; The probability output module is used to input the context word sequence of each word into the pre-established and trained recognition model to obtain the probability of each word appearing in the context word sequence. The abnormal word identification module is used to compare the probability of each word's occurrence with a set threshold to determine whether the word is an abnormal word. The setting of the threshold specifically includes: Each word in the training set is sequentially input into the trained recognition model to obtain the predicted probability distribution of each center word and negative sampled word, and boundary probability values ​​are set. According to the boundary probability value Determine the boundary probability points of all positive samples and all negative sample boundary probability points : in, It is the probability of a word appearing as predicted by the recognition model. It is the set of probabilities obtained from all the correct words that participated in the statistics. Indicates in The number of probability values ​​greater than or equal to z. express The total number of probability values; in, It is the set of probabilities obtained from all negatively sampled words participating in the statistics. Indicates in The number of probability values ​​less than or equal to z. express The total number of probability values; The threshold is determined according to the following formula. for: 。 8. The text abnormal word recognition system based on a neural language model according to claim 7, characterized in that, The recognition model takes words as input and outputs the recognition probability of words. The recognition model includes a word vector layer, a fully connected layer, a hidden layer, and a prediction layer connected in sequence.

Images