A new word discovery method, a computing device and a readable storage medium

Abstract

The present disclosure discloses a new word discovery method, a computing device and a readable storage medium. The new word discovery method comprises: processing text data to obtain a sequence containing multiple words as a word string; determining a candidate word string by counting word information and multiple word features of each word string; generating a feature set based on a feature vector of the candidate word string, wherein the feature vector contains word features and a topic vector; using machine learning to iteratively process the feature set to generate a new word set containing multiple new words; and screening the new words based on the semantic vector of each new word and the semantic vector of the sub-word constituting the corresponding new word to determine the final new word.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer networks, and particularly relates to a new word discovery method. BACKGROUND

[0002] With the development and popularity of vertical field Internet applications, social media has gradually developed into an important carrier of information transmission and is integrated into people's daily life. Social media text has the characteristics of wide distribution and high degree of colloquialization, and the segmentation standard has not been unified. In text mining, the unregistered words not included in the dictionary are collectively referred to as new words.

[0003] The emergence of new words brings great challenges to many natural language processing tasks. The segmentation tool cannot effectively identify these new words, which will directly affect the analysis effect of the upper task based on segmentation. The new word discovery technology field has no mature training corpus, and it is impossible to train a reliable new word recognition model through supervised method, so the unsupervised new word discovery method is valued by people.

[0004] At present, there have been some researches on unsupervised new word discovery at home and abroad. The first mainstream research is the new word discovery based on statistical rules, and its biggest weakness is the domain limitation, which needs a large number of domain experts to formulate rule templates matching the domain text. With the development of technology, more and more unsupervised new word discovery methods based on probability are used. The advantage of this method is not limited to a specific field, and it does not need to give training corpus in advance. The disadvantage is that there are more garbage word strings. Recently, word vectors have been applied to new word discovery tasks. The new word discovery method based on word vector pruning first uses co-occurrence frequency, information entropy and other means to obtain a new word candidate set, and then uses the trained word vector to prune the candidate words to obtain new words. This method has improved the domain generality, but the recall rate is low.

[0005] Therefore, a new new word discovery scheme is needed. SUMMARY

[0006] The present disclosure provides a new word discovery scheme to try to solve or at least alleviate at least one of the problems existing above.

[0007] According to an aspect of the present disclosure, a new word discovery method is provided, including: processing text data to obtain a sequence containing multiple words as a word string; determining candidate word strings by counting word information and multiple word features of each word string; generating a feature set based on a feature vector of the candidate word string, wherein the feature vector contains word features and a topic vector; iteratively processing the feature set using machine learning to generate a new word set containing multiple new words; and screening the new words based on a semantic vector of each new word and semantic vectors of sub-words constituting the corresponding new word to determine final new words.

[0008] Optionally, the method according to the present disclosure further includes: extracting a labeled word sample from the text data using an AC automaton, wherein the AC automaton is constructed based on a user dictionary; selecting part of the feature vectors as a first feature set from the feature set, labeling the first feature set using the labeled word sample, and training a regression model adapted to predict a probability that a candidate word string indicated by an input feature vector is a new word; selecting another part of the feature vectors as a second feature set from the feature set, inputting the second feature set into the trained regression model, and confirming that the candidate word string indicated by the feature vector is a new word when the output predicted probability is greater than a preset value; adding the confirmed new word to the user dictionary and the new word set respectively to update the user dictionary and the new word set correspondingly; and repeating the steps of extracting the labeled word sample, training the regression model, confirming the new word, and updating the user dictionary and the new word set until no new new word is confirmed for multiple times in succession, and the iteration ends.

[0009] Optionally, in the method according to the present disclosure, the step of determining the candidate word strings by counting the word information and the multiple word features of each word string includes: performing word information counting on each word string to generate a first candidate word string set; calculating word features of each candidate word string in the first candidate word string set, and generating a second candidate word string set based on the word features, wherein the candidate word strings in the second candidate word string set are the determined candidate word strings.

[0010] Optionally, in the method according to the present disclosure, the step of performing word information counting on each word string to generate a first candidate word string set includes: counting word frequencies and inverse text frequency indexes of each word string; and judging based on at least values of the word frequencies and the inverse text frequency indexes to filter out word strings less than corresponding median values to generate the first candidate word string set.

[0011] Optionally, in the method according to the present disclosure, the word features include coagulation degree, weighted left and right information entropy, and appearance probability, and the steps of calculating the word features of each candidate word string in the first candidate word string set and generating the second candidate word string set based on the word features include: calculating the coagulation degree of each candidate word string; calculating the left information entropy and the right information entropy of each candidate word string respectively, and determining the weighted left and right information entropy based on the left information entropy and the right information entropy; determining the appearance probability of each candidate word string based on the language model; determining the score of each candidate word string based on the coagulation degree, the weighted left and right information entropy, and the appearance probability, and screening the candidate word strings based on the score to generate the second candidate word string set.

[0012] Optionally, in the method according to the present disclosure, the step of calculating the coagulation degree of each candidate word string includes: counting the minimum value of the point mutual information of each subset constituting the candidate word string; and processing the minimum value by using the length of the candidate word string to determine the coagulation degree of the candidate word string.

[0013] Optionally, in the method according to the present disclosure, the step of screening the candidate word strings based on the score to generate the second candidate word string set includes: selecting the candidate word strings with a score not less than the average value of the score to constitute the second candidate word string set.

[0014] Optionally, in the method according to the present disclosure, the step of generating the feature set based on the feature vectors of the candidate word strings further includes: determining the topic vector of the candidate word string.

[0015] Optionally, in the method according to the present disclosure, the step of labeling the first feature set by using the labeled word samples includes: matching the candidate word string indicated by the feature vector in the first feature set with the labeled word samples, labeling the feature vector as a positive sample if the matching is consistent, or labeling the feature vector as a negative sample otherwise.

[0016] Optionally, in the method according to the present disclosure, the step of screening the new word based on the semantic vector of the new word and the semantic vectors of the subwords constituting the corresponding new word includes: performing a whitening operation on the new word and the subwords constituting the corresponding new word based on BERT to generate corresponding whitened semantic vectors; calculating the similarity between the semantic vector of the new word and the semantic vectors of the subwords constituting the corresponding new word; and deleting the new word from the new word set if the similarity between the semantic vector of the new word and the semantic vector of a certain subword is lower than a threshold value.

[0017] Optionally, in the method according to the present disclosure, the step of processing the text data to obtain a plurality of sequences containing a plurality of characters as word strings includes: dividing the text data into sequences with a length of N based on an N-Gram model.

[0018] According to yet another aspect of the present disclosure, a computing device is provided, comprising: one or more processors memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any of the methods above.

[0019] According to yet another aspect of the present disclosure, a computer-readable storage medium storing one or more programs is provided, the one or more programs comprising instructions which, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0020] In summary, according to the scheme of the present disclosure, the word information of the word string after segmentation is counted, and the word features of the candidate word string are calculated through multiple dimensions; then, machine learning iteration is performed to mine new words; then, the new words are output through the whitened BERT semantic vector. When calculating the word features, the calculation of the coagulation degree and the left and right entropy is improved, and a language model is introduced to more accurately evaluate the probability that a candidate word string is a new word. At the same time, the SVM model is iteratively trained to mine new words, and the topic features are introduced, which can well improve the accuracy and recall rate. In addition, the whitened BERT vector is introduced to do pruning, which effectively filters out garbage strings and reduces the false deletion rate.

[0021] The above description is only a summary of the technical scheme of the present disclosure. In order to enable a clearer understanding of the technical means of the present disclosure, the contents of the specification can be implemented, and in order to enable the above and other purposes, features and advantages of the present disclosure to be more apparent and easy to understand, the specific embodiments of the present disclosure are described below. BRIEF DESCRIPTION OF DRAWINGS

[0022] To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings. These aspects are indicative of various ways in which the principles disclosed herein can be practiced and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The foregoing and other objects, features, and advantages of the disclosure will be apparent from the following description of certain aspects as taken in connection with the accompanying drawings in which like reference characters refer to the same parts throughout the several views. The present disclosure will be described with reference to the attached drawings, of which:

[0023] Figure 1 A schematic diagram of a computing device 100 according to some embodiments of the present disclosure is shown;

[0024] Figure 2 A flowchart of a new word discovery method 200 according to some embodiments of the present disclosure is shown;

[0025] Figure 3 A flowchart of step 240 of iteratively generating a new word set according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0027] According to the new word discovery scheme of the present disclosure, first, word information statistics are performed on the N-gram segmented corpus (i.e., word string) after data cleaning; these word strings are taken as candidate words, second, word features of these candidate words are calculated in multiple dimensions; third, machine learning iteration is performed to mine new words; and finally, pruning is performed through the whitened BERT semantic vector to output a new word dictionary.

[0028] The new word discovery scheme of the present disclosure can be executed in one or more computing devices. Figure 1 is a block diagram of an example computing device 100.

[0029] In a basic configuration 102, the computing device 100 typically includes a system memory 106 and one or more processors 104. A memory bus 108 can be used for communicating between the processor 104 and the system memory 106.

[0030] Depending on the desired configuration, the processor 104 can be of any type, including but not limited to a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. The processor 104 can include one or more levels of caching, such as a level one cache 110 and a level two cache 112, a processor core 114, and registers 116. The example processor core 114 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP Core), or any combination thereof. An example memory controller 118 can be used with the processor 104 or, in some implementations, the memory controller 118 can be an internal part of the processor 104.

[0031] Depending on the desired configuration, the system memory 106 can be of any type including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.) or any combination thereof. Physical memory in a computing device typically refers to volatile memory, such as RAM, and data is typically transferred between data storage and physical memory before the data is processed by the processor 104. The system memory 106 can include an operating system 120, one or more applications 122, and program data 124. In some embodiments, the applications 122 are arranged to execute instructions from the operating system 120 using the program data 124 to perform tasks. The operating system 120, for example, can be a Linux, Windows, etc., that includes program instructions for handling basic system services and for performing hardware dependent tasks. The applications 122 include program instructions for implementing user desired functions, such as a browser, instant messaging software, software development tools (e.g., an integrated development environment (IDE), a compiler, etc.), etc., but are not limited thereto.

[0032] When the computing device 100 is in operation, the processor 104 is configured to read and execute instructions from the memory 106, such as the operating system 120. The applications 122 are executed on the operating system 120 to implement various user desired functions using the interfaces provided by the operating system 120 and underlying hardware. When a user launches an application 122, the application 122 is loaded into the memory 106 and the processor 104 reads and executes the program instructions of the application 122 from the memory 106.

[0033] The computing device 100 also includes storage devices 132, which include removable storage 136 (e.g., a CD, DVD, USB flash drive, a removable hard disk drive, etc.) and non-removable storage 138 (e.g., a hard disk drive (HDD), etc.), which are connected to the storage interface bus 134.

[0034] The computing device 100 can also include a storage interface bus 134. The storage interface bus 134 enables communication between the storage devices 132 (e.g., the removable storage 136 and the non-removable storage 138) and the basic configuration 102 via the bus / interface controller 130. The operating system 120, the applications 122, and the program data 124 are at least partially stored either in the removable storage 136 and / or the non-removable storage 138 during the manufacturing of the computing device 100, and are loaded in the system memory 106 and executed by the one or more processors 104 at the time of booting of the computing device 100 or when the applications 122 are required to be executed.

[0035] The computing device 100 can also include an interface bus 140 for facilitating communication from various interface devices (e.g., output devices 142, peripheral interfaces 144, and communication devices 146) to the basic configuration 102 via the bus / interface controller 130. Example output devices 142 include a graphics processing unit 148 and an audio processing unit 150, which can be configured to facilitate communication to various external devices such as a display or speakers via one or more A / V ports 152. Example peripheral interfaces 144 include a serial interface controller 154 or a parallel interface controller 156, which can be configured to facilitate communication to various external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripheral devices (e.g., printer, scanner, etc.) via one or more I / O ports 158. An example

[0036] Network communication links can be one example of a communication media. Communication media can typically be embodied by computer readable instructions, data structures, program modules, and the like, in a modulated data signal, such as a carrier wave or other transport mechanism, and can include any information delivery media. A "modulated data signal" can be a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), microwave, infrared (IR) and other wireless media. The term computer readable media as used herein can include both storage media and communication media. In some embodiments, computer readable media stores one or more programs implementing some of the methods described herein.

[0037] The computing device 100 can be implemented as a portion of a small- sized portable (or mobile) electronic device such as a cell phone, a personal digital assistant (PDA), a personal media player device, a wireless web browsing device, a personal headset, an application-specific device, or a hybrid device that can include any of the above functions. The computing device 100 can also be implemented as a personal computer including both desktop and notebook configurations. The computing device 100 can also be implemented as a server having the above-described configurations.

[0038] In embodiments according to the present disclosure, the computing device 100 is configured to perform a new word discovery method, wherein the computing device 100 contains a plurality of program instructions in the application 122 to perform the new word discovery method 200 according to the present disclosure, and the program data 124 can further store relevant data of the language models and machine learning models used to perform the method 200, including but not limited to training data, hyperparameter information, etc.

[0039] Figure 2 A flowchart of the new word discovery method 200 according to an embodiment of the present disclosure is shown. As shown, the method 200 starts at step 210. Figure 2

[0040] In step 210, the text data is processed to obtain a plurality of sequences of characters as character strings.

[0041] According to embodiments of the present disclosure, various text data is collected as a data set, and a training corpus is generated through a series of processing of the data set. The training corpus is usually a plurality of sequences (or fragments) of characters, such as "pick up a car", "pick up a car", "pick up a car for a week", etc. According to embodiments of the present disclosure, these sequences are referred to as "character strings".

[0042] Specifically, first, the text data is cleaned and the corpus is de-duplicated based on the spark distributed framework; then, the text data is segmented by N-Gram, and the text data is divided into a plurality of sequences of length N based on the N-Gram model. Usually, the value of N indicates the number of characters in the character string. In an embodiment, the value of N is usually 2-10, but is not limited thereto.

[0043] In addition, the processing of the data set also includes segmenting the segmentation results based on the sentence break points. The sentence break points are, for example, words in the stop word list stop_words, words with part-of-speech such as punctuation, conjunction, auxiliary word, preposition, pronoun, and general stop words (website, time, unit, etc.) extracted using regular expressions.

[0044] In step 220, the candidate character strings are determined by counting the word information and various word features of each character string.

[0045] According to some embodiments, the word information of each character string is first counted to generate a first candidate character string set CW1 based on the word information counting result. Then, the word features of each candidate character string in the first candidate character string set CW1 are calculated, and a second candidate character string set CW2 is generated based on the word features. In this way, the candidate character strings in the second candidate character string set CW2 are the final determined candidate character strings.

[0046] The generation processes of CW1 and CW2 are described below. ​

[0047] (1) Generating a first candidate word string set CW1 based on word information statistics

[0048] According to some embodiments, the word information statistics are mainly based on TF-IDF. First, the term frequency (TF) and inverse document frequency (IDF) of each word string are counted. The tf, idf, tfidf of a word string can be calculated by the following formulas (1), (2), (3).

[0049]

[0050]

[0051]

[0052] wherein N(W i ) represents the number of times the word string W i appears in a document, N represents the total number of words, T represents the total number of documents, and df i represents the number of documents containing the word string W i .

[0053] Then, at least based on the values of the term frequency and the inverse document frequency, a judgment is made to filter out word strings less than the corresponding median value, to generate the first candidate word string set CW1.

[0054] Specifically, the word string tf, idf, tfidf calculated by formulas (1), (2), (3) are respectively subjected to min-max standardization processing to obtain tf', idf', tfidf'. The medians of tf', idf', tfidf' are respectively counted and denoted as tf z , idf z , tfidf z . Then, a judgment is made according to the above values and the median values of each word string to filter out word strings that do not meet the conditions, specifically: if tf' < tf z , or idf' < idf z , or tfidf' < tfidf z , the word string corresponding to the value is filtered out. Finally, based on the filtered word strings, CW1 is generated.

[0055] (2) Generating a second candidate word string set CW2 based on word features

[0056] According to some embodiments, the word features include solidity, weighted left and right information entropy, and appearance probability. The processes of calculating the word features of the candidate word strings in CW1 are introduced below through (a)-(c).

[0057] (a) Calculating the solidity of each candidate word string.

[0058] In some embodiments, the solidity of a candidate word string is obtained by improving the point mutual information (PMI). The point mutual information is generally used to represent the degree of mutual dependence between two words. Considering that the value of the point mutual information is affected by the length of the word string W i , the longer the length of the word string W i , the greater the value of the mutual information, according to the embodiments of the present disclosure, for each candidate word string W i in CW1, the minimum value of the PMI of all possible subsets constituting the candidate word string W i is counted and then divided by a linear coefficient related to the length of the candidate word string to balance the information error caused by different lengths.

[0059] Taking a candidate word string W i composed of three subsets a, b, and c as an example, the solidity of the candidate word string is determined by the following formula (4):

[0060]

[0061] In the formula, PLMI represents the solidity, a, b, and c represent the subsets of the candidate word string, p(x) represents the probability of the occurrence of x, word.length represents the length of the candidate word string, and a1 and a2 are parameters. Optionally, a1 and a2 take values of 0.8 and 0.5, respectively.

[0062] It should be noted that the calculation process of the solidity is only taken as an example of a candidate word string composed of three subsets a, b, and c, and in actual applications, the candidate word string can be composed of four or more subsets, which is not limited by the present disclosure.

[0063] (b) calculating the left information entropy and the right information entropy of each candidate word string, respectively, and determining the weighted left and right information entropy based on the left information entropy and the right information entropy.

[0064] The left and right adjacent entropy is also called information entropy, which represents the information entropy of a word and its left and right adjacent words. In some embodiments, based on the candidate word string W i in CW1, a Parent tree of N+1 is constructed, all left and right adjacent characters of W i in the tree are searched and stored in sets S l and S r , respectively, and then the LE and RE of W i are calculated by using formulas (5) and (6).

[0065]

[0066]

[0067] In the formula, LE(W i) represents the left information entropy, RE(W i ) represents the right information entropy, P(W i ) represents the probability of occurrence of the corresponding left adjacent character (or right adjacent character). Regarding the left and right information entropy, it is known to those skilled in the art and will not be described here.

[0068] Considering that if the left and right information entropy is directly summed to represent the left and right richness of the word, the word of the garbage string xx with large right entropy and small left entropy can also get a high score. In order to better measure the richness of the left and right adjacent characters, according to the embodiments of the present disclosure, the size of the left and right information entropy (LE, RE), the absolute value of the difference between LE and RE (|LE-RE|), and the average entropy value of the boundary character are combined to smooth the left and right information entropy to obtain a weighted left and right information entropy to solve the error of the imbalance of the left and right entropy.

[0069] According to some embodiments, the weighted left and right information entropy is calculated by the following formula (7):

[0070]

[0071] In the formula, LRE(W i ), LE(W i ), RE(W i ) respectively represent the weighted left and right information entropy, the left information entropy, and the right information entropy of the candidate string W i , BE(W i ) represents the difference between the left information entropy and the right information entropy, and β1 is a parameter. Optionally, β1 is 2.

[0072] (c) determining the occurrence probability of each candidate string based on the language model.

[0073] Using the language model, the score of each W i in CW1 is calculated, which indicates the occurrence probability of the candidate string W i . In some embodiments, the language model uses kenlm, and the candidate string W i in CW1 is input into the trained kenlm language model to obtain the occurrence probability of each W i , denoted as LM(W i ), and the score of each W i in the language model can be represented by formula (8):

[0074] LM(W i ) = P(w1, w2,..., w i ) = P(w1)P(w2|w1)...P(w i |w1, w2,..., w i-1 ) (8)

[0075] It should be noted that the present disclosure aims to protect the way of calculating the word features of the candidate word string in multiple dimensions, and thus does not limit the specific language model used to predict the occurrence probability of the candidate word string.

[0076] Finally, the scores of the candidate word strings are determined based on the solidity, the weighted left and right information entropy and the occurrence probability, and the candidate word strings are screened based on the scores to generate the second candidate word string set.

[0077] In some embodiments, the solidity, the weighted left and right information entropy and the occurrence probability obtained by the above formula (4), (7) are respectively standardized (for example, normalized, not limited thereto), and then the values after the standardization are weighted to obtain the score of the candidate word string, as shown in the following formula (9):

[0078] Score(W i )=λ1PMI(W i )+λ2LRE(W i )+λ3LM(W i ) (9)

[0079] In the formula, PMI(W i ), LRE(W i ) and LM(W i ) respectively represent the solidity, the weighted left and right information entropy and the occurrence probability of the candidate word string W i , Score(W i ) represents the score of the candidate word string W i , and λ1, λ2 and λ3 are weight parameters. Optionally, the values of λ1, λ2 and λ6 are respectively 0.3, 0.3 and 0.4.

[0080] Then, the average value of the scores Score(W j ) of all candidate word strings is calculated, and the candidate word strings with scores not less than the average value are selected to form the second candidate word string set CW2.

[0081] According to the embodiments of the present disclosure, the word string is preliminarily screened by using statistical word information (word frequency, inverse text frequency index), and the comprehensive score of the candidate word string is obtained by calculating the word features of the candidate word string in multiple dimensions. In the calculation process of the word features, in addition to improving the calculation formula of the solidity and the left and right entropy, the language model is also introduced to predict the occurrence probability of the candidate word string, which has a good interpretability for the measurement of the candidate word string, so as to more accurately evaluate the probability that a candidate word string is a new word.

[0082] In step 230, a feature set is generated based on the feature vector of the candidate word string. The feature vector includes word features and topic vectors.

[0083] According to embodiments of the present disclosure, the feature vector of the candidate word string further includes a topic vector in addition to the word features mentioned in step 220. Therefore, step 230 further includes a step of determining the topic vector of the candidate word string in CW2. In some embodiments, the topic vector is generated by a topic model (e.g., a Latent Dirichlet Allocation model).

[0084] Suppose the text data contains m documents in total, involving K topics, where each document has its own topic distribution (the topic distribution is subject to a multinomial distribution), and the parameters of the multinomial distribution are subject to a Dirichlet distribution; at the same time, each topic has its own word distribution, and the word distribution is also subject to a multinomial distribution, and the parameters of the multinomial distribution are also subject to a symmetric Dirichlet distribution. The topic vector can be represented by formula (10):

[0085]

[0086] In the formula, T j represents the jth document, z represents a topic drawn from a topic subject to a multinomial distribution, and K represents the number of topics.

[0087] In step 240, the feature set is iteratively processed by machine learning to generate a new word set containing multiple new words.

[0088] Figure 3 A flowchart of step 240 of iteratively generating a new word set according to some embodiments of the present disclosure is shown. As shown in 241, an AC automatic machine (Aho-Corasick automation) is constructed based on a user dictionary. At the beginning of the iteration, the user dictionary can be selected from an existing user dictionary. The AC automatic machine is actually a multi-mode matching algorithm, which can be implemented by using the existing Python library, and therefore will not be described here. Figure 3

[0089] In 242, the AC automatic machine is used to extract a labeled word sample (denoted as ACwords) from the text data.

[0090] ​In 243, part of the feature vectors in the feature set X is selected as the first feature set X1. In some embodiments, the feature set X is randomly divided into two parts: X1 and X2. In this step, the first feature set X1 is labeled by using the labeled word samples. Specifically, the candidate string indicated by the feature vector in X1 is matched with the labeled word samples. If the match is consistent, the feature vector is labeled as a positive sample. Otherwise, the feature vector is labeled as a negative sample. Then, the regression model is trained by using the labeled X1. The regression model is implemented by using a support vector machine (SVM) to predict the probability that the candidate string indicated by the input feature vector is a new word.

[0091] In addition, when training the SVM regression model, X1 can be further divided into a training set and a test set. The SVM regression model is constructed and preliminarily trained on the training set of X1. Then, the preliminarily trained SVM regression model is tested by using the test set of X1. If the cumulative probability value calculated by the model on the test set is greater than or equal to 0.8 for 10 times, it is considered that the SVM regression model is trained and the subsequent steps can be continued. Otherwise, the iteration is ended.

[0092] In 244, another part of the feature vectors in the feature set X is selected as the second feature set X2. The second feature set is input into the trained regression model. When the output prediction probability is greater than a preset value, it is determined that the candidate string indicated by the feature vector is a new word. Optionally, the preset value is 0.9.

[0093] In 245, the confirmed new words are added to the user dictionary and the new word set, respectively, to update the user dictionary and the new word set.

[0094] In 246, it is determined whether the iteration condition is met.

[0095] If the iteration condition is met, the iteration is ended. If the iteration condition is not met, the step 241 is returned and the steps are sequentially executed to repeat the above steps of constructing the AC automaton to extract the labeled word samples, training the regression model, confirming the new words, and updating the user dictionary and the new word set until the iteration condition is met. In an embodiment, the iteration condition is that no new new word is confirmed for a plurality of times (for example, 10 times).

[0096] According to the embodiments of the present disclosure, in the process of iteratively mining new words by the SVM regression model, the topic feature of the candidate string is introduced, which can ensure sufficient semantic information of the candidate word, improve the prediction accuracy, and achieve the goal of accurately recalling new words.

[0097] In step 250, the new words are screened based on the semantic vector of each new word and the semantic vector of the subword constituting the corresponding new word, to determine the final new words.

[0098] According to the embodiments of the present disclosure, for each candidate word in the new word set, a whitening operation is performed on the BERT vector to obtain a new semantic vector, and then pruning is performed based on the new semantic vector to obtain the final new word. The processing process is specifically introduced below.

[0099] First, based on BERT, a whitening operation is performed on the new word and the subwords constituting the corresponding new word to generate the corresponding whitened semantic vector.

[0100] The coordinate basis of the sentence vector generated by BERT is not a standard orthogonal basis, that is, anisotropic. In order to solve the problem of BERT sentence embedding vector, it is necessary to convert the sentence vector into a standard orthogonal basis. According to some embodiments, the semantic vector generated by BERT is converted through a whitening operation to obtain a whitened semantic vector. At the same time, the new word as a candidate word and each subword constituting the new word are subjected to a whitening operation in the same space to expand the difference between words and improve the accuracy, also reducing the probability of false pruning.

[0101] Then, the similarity between the semantic vector of the new word and the semantic vector of each subword constituting the corresponding new word is calculated. In other words, the similarity between each pair of adjacent subwords combined into a new word and the new word is calculated. The present disclosure does not limit the specific algorithm used to calculate the similarity between vectors, such as cosine distance.

[0102] If the similarity between the semantic vector of the new word and the semantic vector of a certain subword is lower than a threshold value, the new word is deleted from the new word set. The threshold value can be selected according to actual needs, and the present disclosure does not make too many restrictions on this. That is, when the similarity between the new word and a certain adjacent subword is low, the word is removed from the new word set.

[0103] According to the present method, by introducing the whitened BERT vector for pruning, it can effectively filter out garbage strings and reduce the false deletion rate.

[0104] According to the method 200 of the present disclosure, an unsupervised new word discovery algorithm is realized based on machine learning, without the need for a large number of artificial rules and labeled data, and has better accuracy and recall rate compared to traditional algorithms, and is not limited by the professionalism of the corpus, and has good universality.

[0105] To further illustrate the advantages of the method 200, 300,000 articles were randomly selected from a website for comparison without any artificial rule intervention. The comparison results are shown in Table 1. Experiments prove that the method 200 is a more effective new word mining method.

[0106] Table 1 Comparison of results of each algorithm

[0107] Precision (%) Recall (%) F1 value Method 200 80.1 85.2 82.6 Existing method 1 28.2 66.4 39.6 Existing method 2 59.4 34.6 43.7

[0108] In summary, according to the scheme of the present disclosure, an unsupervised new word discovery algorithm combining statistics, dynamic semantic vector and machine learning is proposed. It has the following characteristics.

[0109] 1) It can truly achieve unsupervised learning without relying on labeled corpus, and has high mining efficiency and is suitable for new word discovery in various fields and data scales.

[0110] 2) Through new statistical formula, machine learning, language model, topic, dynamic semantic pruning and other multi semantic expressions, the rationality of a word is measured from multiple aspects, effectively improving the accuracy of new word mining.

[0111] 3) Through continuous unsupervised iteration and replacing static semantics (such as Word2Vec) with dynamic semantics (BERT) and compressing candidate words to a reasonable dimension through whitening technology, the recall rate is effectively improved through two technical means.

[0112] 4) The BERT semantic vector after whitening is used to prune and output a new word dictionary, which can improve the word segmentation effect and lay a foundation for subsequent upper-layer tasks.

[0113] The various techniques described herein can be implemented in connection with hardware or software or, where appropriate, with a combination of hardware / software. Thus, the methods and apparatus of the present disclosure, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embodied in tangible media, such as movable disks, USB, floppy disks, CD-ROMs, or any other machine-readable storage media, wherein, when the program code is loaded into a machine, such as a computer, the machine becomes an apparatus for practicing the present disclosure.

[0114] In the case of program code execution on a programmable computer, the computing device generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store program code; the processor is configured to execute instructions in the program code stored in the memory according to the instructions in the program code stored in the memory to perform the new word discovery method of the present disclosure.

[0115] The present disclosure also includes:

[0116] A9. The method of A5, wherein the step of screening the candidate word strings based on the scores to generate a second set of candidate word strings comprises: selecting candidate word strings with scores not less than the average value of the scores to form the second set of candidate word strings; and the score of the candidate word string is calculated by the following formula:

[0117] Score(W i )=λ1PMI(W i)+ λ2LRE(W i )+ λ3LM(W i ),

[0118] wherein PMI(W i ), LRE(W i ), LM(W i ) respectively represent the solidity, weighted left-right information entropy and appearance probability of the candidate word string W i , Score(W i ) represents the score of the candidate word string W i , and λ1, λ2 and λ3 are weight parameters.

[0119] A10. The method of any one of A1-9, wherein the step of generating the feature set based on the feature vector of the candidate word string further comprises determining a topic vector of the candidate word string.

[0120] A11. The method of any one of A1-10, wherein the step of labeling the first feature set with labeled word samples comprises matching the candidate word string indicated by the feature vector in the first feature set with the labeled word samples, and labeling the feature vector as a positive sample if the matching is consistent, or labeling the feature vector as a negative sample otherwise.

[0121] A12. The method of any one of A1-11, wherein the step of screening the new words based on the semantic vector of each new word and the semantic vectors of the subwords constituting the corresponding new word comprises: based on BERT, performing a whitening operation on the new word and the subwords constituting the corresponding new word to generate corresponding whitened semantic vectors; calculating the similarity between the semantic vector of the new word and the semantic vectors of the subwords constituting the corresponding new word; and if the similarity between the semantic vector of the new word and the semantic vector of a certain subword is lower than a threshold, deleting the new word from the set of new words.

[0122] A13. The method of any one of A1-12, wherein the step of processing the text data to obtain a plurality of sequences containing a plurality of words as word strings comprises: based on an N-Gram model, dividing the text data into sequences of length N.

[0123] By way of example, and not limitation, readable media can include volatile and non-volatile, removable and non-removable media implemented in a method or technology for storage and / or transmission of information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of the any of the above are also included within the scope of readable media.

[0124] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to not obscure the understanding of this description.

[0125] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to not obscure the understanding of this description.

[0126] Similarly, it is to be understood that the narrative provided above with respect to the exemplary embodiments of the present disclosure sometimes collectively sets forth various features of the present disclosure in groups rather than one at a time. This manner of description is not to be interpreted that the present disclosure contemplates a claim to more features than those expressly recited in each claim. Rather, as reflected in the claims below, a disclosed aspect is in terms of fewer than all of the features of a single embodiment disclosed earlier. Accordingly, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim is a separate embodiment of the present disclosure by itself.

[0127] Those skilled in the art will understand that the modules, or units, or components of the devices in the examples disclosed herein can be arranged in the devices as described in the examples, or alternatively can be located in one or more devices different from the devices in the examples. The modules in the foregoing examples can be combined into one module or further divided into multiple sub-modules.

[0128] Those skilled in the art will appreciate that the modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and further can be split into multiple sub-modules or sub-units or sub-components. Any combination of all the features disclosed in the specification (including the accompanying claims, abstract and drawings), and any method or process or steps of any such methods or processes disclosed in the specification (including the accompanying claims, abstract and drawings) can be made, except that at least some of such features and / or processes or units are mutually exclusive, unless explicitly stated otherwise. Each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features serving the same, equivalent or similar purpose, unless explicitly stated otherwise.

[0129] Furthermore, those skilled in the art will appreciate that the features of the various embodiments described herein are not mutually exclusive and can be combined in different embodiments. For example, in the following claims, any of the embodiments claimed can be used in any combination.

[0130] Furthermore, some of the embodiments described herein are described as a method or combination of elements of a method implementable by a processor of a computer system or by other means of carrying out the function. Accordingly, a processor with the necessary instructions for carrying out such a method or element of a method forms a means for carrying out the method or element of a method. Furthermore, an element of a method described herein that is implemented as a means for carrying out a particular function performs that function whether or not recited as a means for carrying out that function.

[0131] As used herein, unless otherwise indicated, the use of the ordinal adjectives "first", "second", "third", etc., merely to distinguish different instances of a same object, and are not intended to imply that a sequence or order to which the objects are described in the specification herein restricts the disclosure to a group or sequence of objects constructed and used in an order corresponding to that in which they are described herein.

[0132] While the present disclosure has been described in some embodiments, further modifications and variations from the teachings disclosed herein will occur to those skilled in the art once briefed on the above descriptions. Accordingly, it is to be understood that the application is not to be limited by the foregoing description, but is only limited by the scope of the appended claims. Moreover, the language used in this specification has been principally selected for readability and instructional purposes and can not have been selected to delineate or circumscribe the patent rights to which this disclosure relates.

Claims

1. A method for discovering new words, comprising: processing text data to obtain a plurality of sequences of words as word strings; determining candidate word strings by counting word information and a plurality of word features of each word string; generating a feature set based on feature vectors of the candidate word strings, wherein the feature vectors include the word features and topic vectors; iteratively processing the feature set using machine learning to generate a new word set including a plurality of new words; selecting new words from the new word set based on semantic vectors of each new word and semantic vectors of sub-words constituting the corresponding new word to determine final new words; wherein the step of determining candidate word strings by counting word information and a plurality of word features of each word string comprises: counting word information of each word string to generate a first candidate word string set; calculating word features of each candidate word string in the first candidate word string set and generating a second candidate word string set based on the word features, wherein the candidate word strings in the second candidate word string set are the determined candidate word strings; wherein the step of counting word information of each word string to generate a first candidate word string set comprises: counting word frequency and inverse text frequency index of each word string; judging based on at least values of the word frequency and the inverse text frequency index to filter out word strings less than corresponding median values to generate the first candidate word string set; wherein the word features include solidity, weighted left and right information entropy, and appearance probability, and the step of calculating word features of each candidate word string in the first candidate word string set and generating a second candidate word string set based on the word features comprises: calculating solidity of each candidate word string; calculating left information entropy and right information entropy of each candidate word string respectively and determining weighted left and right information entropy based on the left information entropy and the right information entropy; determining appearance probability of each candidate word string based on a language model; determining scores of each candidate word string based on the solidity, the weighted left and right information entropy, and the appearance probability, and selecting the candidate word strings based on the scores to generate the second candidate word string set; wherein the step of calculating solidity of each candidate word string comprises: counting minimum values of point mutual information of each sub-set constituting the candidate word string; processing the minimum values using length of the candidate word string to determine the solidity of the candidate word string; wherein the solidity is determined by the following formula: wherein, PLMI represents solidity, a, b, c represent sub-sets of the candidate word string, p(x) represents probability of appearance of x, word.length represents length of the candidate word string, and α1 and α2 are parameters.

2. The method of claim 1, wherein, the step of iteratively processing the feature set using machine learning to generate a new word set including a plurality of new words comprises: extracting labeled word samples from the text data using an AC automaton, wherein the AC automaton is constructed based on a user dictionary; selecting part of the feature vectors as a first feature set from the feature set, labeling the first feature set using the labeled word samples, and training to generate a regression model suitable for predicting probability that a candidate word string indicated by an input feature vector is a new word; selecting another part of the feature vectors from the feature set as a second feature set, inputting the second feature set into the trained regression model, and confirming a candidate string indicated by the feature vector as a new word when a predicted probability output by the regression model is greater than a preset value; adding the confirmed new word into a user dictionary and a new word set respectively to update the user dictionary and the new word set correspondingly; repeating the steps of iteratively extracting the annotated word samples, training the regression model, confirming the new word, and updating the user dictionary and the new word set until no new new word is confirmed for a plurality of times in succession, and ending the iteration.

3. The method of claim 1, wherein, The weighted left and right information entropy is calculated by the following formula: where LRE(W i ), LE(W i ), RE(W i ) represent the weighted left and right information entropy, left information entropy, right information entropy of the candidate word string W i , respectively, BE(W i ) represents the difference between the left information entropy and the right information entropy, and β1 is a parameter.

4. The method of claim 1, wherein, The step of screening the candidate strings based on the scores to generate a second candidate string set comprises: selecting candidate strings with scores not less than an average value of the scores to form the second candidate string set; The score of the candidate string is calculated by the following formula: Score(W i ) = λ1PLMI(W i ) + λ2LRE(W i ) + λ3LM(W i ), wherein PLMI(W i ), LRE(W i ), and LM(W i ) respectively represent the solidity, the weighted left and right information entropy, and the appearance probability of the candidate word string W i , Score(W i ) represents the score of the candidate word string W i , and λ1, λ2, and λ3 are weight parameters.

5. The method of any one of claims 1-4, wherein, The step of generating the feature set based on the feature vectors of the candidate strings further comprises: determining a topic vector of the candidate string.

6. The method of claim 2, wherein, The step of annotating the first feature set with the annotated word samples comprises: matching the candidate string indicated by the feature vector in the first feature set with the annotated word samples, and if the matching is consistent, annotating the feature vector as a positive sample, otherwise, annotating the feature vector as a negative sample.

7. The method of any one of claims 1-4, wherein, The step of screening the new words based on the semantic vectors of the new words and the semantic vectors of the sub-words constituting the corresponding new words comprises: based on BERT, performing a whitening operation on the new words and the sub-words constituting the corresponding new words to generate corresponding whitened semantic vectors; calculating the similarity between the semantic vector of the new word and the semantic vector of the sub-word; if the similarity between the semantic vector of the new word and the semantic vector of the sub-word is lower than a threshold value, deleting the new word from the new word set.

8. The method of any one of claims 1-4, wherein, The step of processing the text data to obtain a plurality of sequences containing a plurality of characters as word strings comprises: based on an N-Gram model, dividing the text data into sequences with a length of N.

9. A computing device comprising: one or more processors; memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the method of any one of claims 1-8.

10. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform the method of any one of claims 1-8.

Images