New word mining method, new word mining device and electronic equipment

Abstract

The embodiment of the invention discloses a new word mining method, a new word mining device and electronic equipment. A specific embodiment of the method comprises the following steps: standardizing an obtained original text to obtain a standard text; performing word segmentation on statements in the standard text by adopting an algorithm based on a statistical language model to obtain a word set of the standard text; according to the total number of the words in the word set, the words with the total number not smaller than a first threshold value are determined as candidate words, and a candidate word set is obtained; the candidate words in the candidate word set are screened according to preset index parameters, the candidate words obtained through screening are determined as the mined new words, and the preset index parameters are used for representing the probability that character combinations can serve as words. The embodiment is related to a text processing technology, a large amount of annotation data and model training do not need to be prepared, the method does not depend on empirical rules, and automatic mining of unsupervised potential new words can be achieved. The resource demand can be reduced while the data mining effect is ensured.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of text processing, and in particular to a new word mining method, a new word mining device, and an electronic device. Background Art

[0002] Natural language processing (NLP for short) is a key branch in the field of artificial intelligence and computer science, focusing on various theories and methods for achieving effective communication between humans and machines using natural language. NLP can automatically convert text in one language into another language to achieve cross-language communication. It can also help automatically classify text data into different categories. However, different languages ​​have different requirements for NLP technology. For example, the processing of Chinese is much more complicated than English. The Chinese writing system does not have clear word boundary markers, such as spaces. Therefore, when processing Chinese text, computers cannot directly identify vocabulary units through spaces. It needs to be solved by word segmentation operations based on and based on the word library. Therefore, a high-quality word library can reduce word segmentation errors, improve the accuracy of word segmentation results, and provide better input for subsequent natural language processing tasks.

[0003] However, the inventors have found that the word mining methods in the related art all have some problems. For example, the rule-based method depends largely on the quality and quantity of the rules. If the rules are not comprehensive or accurate enough, it may lead to omissions or false positives in new word mining. The new word discovery method based on deep learning requires a large amount of annotated data for training, and the performance and effect of the model are affected by the quality and quantity of the training data.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a new word mining method, a new word mining device, an electronic device, a computer-readable medium, and a computer program product to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a new word mining method, comprising: standardizing the acquired original text to obtain a standard text; using an algorithm based on a statistical language model to segment sentences in the standard text to obtain a word set of the standard text; determining words whose total number is not less than a first threshold as candidate words based on the total number of words in the word set to obtain a candidate word set; screening the candidate words in the candidate word set according to preset indicator parameters, and determining the screened candidate words as mined new words, wherein the preset indicator parameters are used to characterize the probability that a combination of characters can be used as a word.

[0008] In some embodiments, the acquired original text is standardized to obtain a standard text, including: removing characters in the original text that match preset characters, and using the original text without the preset characters as the standard text, wherein the preset characters include at least one of the following: modal particles, punctuation marks, and special symbols.

[0009] In some embodiments, candidate words in a candidate word set are screened according to preset index parameters, including: for each candidate word in the candidate word set, determining a solidification index value of the candidate word according to the number of times the candidate word appears in a standard text, wherein the solidification index is used to characterize the internal solidification degree of a word; and determining a candidate word having a solidification index value not less than a second threshold as a candidate word obtained by solidification screening, thereby obtaining a first candidate word set.

[0010] In some embodiments, the coagulation index value of the candidate word is determined based on the number of times the candidate word appears in the standard text, including: for candidate words with a character length greater than two, splitting the candidate word according to different splitting rules to obtain different character combinations of the candidate word, wherein during the splitting process, the position order of each character in the candidate word remains unchanged; for each character combination, the coagulation index value of the character combination is determined based on the number of times each character in the character combination appears in the standard text, and the number of times the candidate word appears in the standard text; the minimum value of the coagulation index values ​​of each character combination is used as the coagulation index value of the candidate word.

[0011] In some embodiments, the candidate words in the candidate word set are screened according to preset indicator parameters, and also include: for each candidate word in the candidate word set, determining the left and right information entropy indicator values ​​of the candidate word according to the characters that appear on the left and right sides of the candidate word in the standard text, respectively, wherein the left and right information entropy indicators are used to characterize the independence of the words; the candidate words whose left and right information entropy indicator values ​​are both not less than a third threshold are determined as candidate words obtained by left and right information entropy screening, to obtain a second candidate word set.

[0012] In some embodiments, determining the candidate words obtained by screening as the mined new words includes: determining the candidate words included in the first candidate word set and the second candidate word set as the mined new words.

[0013] In some embodiments, the method further includes: in response to determining that the original text contains multiple sentences, using a distributed cluster to process the multiple sentences separately, wherein the distributed cluster includes multiple server nodes, and each server node is provided with at least one processor; according to the number of server nodes in the distributed cluster and the number of sentences in the original text, the sentences in the original text are partitioned for processing, wherein statements in different partitions are processed by different server nodes.

[0014] In some embodiments, before screening the candidate words in the candidate word set according to preset indicator parameters, the method also includes: generating a word frequency dictionary based on the characters included in the candidate words in the candidate word set and the number of times the words indicated by the included characters appear in the standard text; and broadcasting the word frequency dictionary in a distributed cluster.

[0015] In a second aspect, some embodiments of the present disclosure provide a new word mining device, comprising: a text processing unit, configured to perform standardization processing on the acquired original text to obtain a standard text; a word segmentation unit, configured to use an algorithm based on a statistical language model to perform word segmentation on sentences in the standard text to obtain a word set of the standard text; a candidate word determination unit, configured to determine words whose total number is not less than a first threshold as candidate words based on the total number of words in the word set to obtain a candidate word set; a word screening unit, configured to screen candidate words in the candidate word set according to preset indicator parameters, and determine the screened candidate words as mined new words, wherein the preset indicator parameters are used to characterize the probability that a combination of characters can be used as a word.

[0016] In some embodiments, the text processing unit is further configured to remove characters in the original text that match preset characters, and use the original text without the preset characters as the standard text, wherein the preset characters include at least one of the following: modal particles, punctuation marks, and special symbols.

[0017] In some embodiments, the word screening unit is further configured to determine, for each candidate word in the candidate word set, a coagulation index value of the candidate word according to the number of times the candidate word appears in the standard text, wherein the coagulation index is used to characterize the internal coagulation degree of the word; and determine the candidate words whose coagulation index values ​​are not less than the second threshold as the candidate words obtained by the coagulation screening, thereby obtaining a first candidate word set.

[0018] In some embodiments, the word screening unit is further configured to split a candidate word having a character length greater than two into different character combinations of the candidate word according to different splitting rules, wherein during the splitting process, the position order of each character in the candidate word remains unchanged; for each character combination, the coagulation index value of the character combination is determined according to the number of times each character in the character combination appears in the standard text and the number of times the candidate word appears in the standard text; and the minimum value of the coagulation index values ​​of each character combination is used as the coagulation index value of the candidate word.

[0019] In some embodiments, the word screening unit is further configured to determine, for each candidate word in the candidate word set, the left and right information entropy index values ​​of the candidate word according to the characters appearing on the left and right sides of the candidate word in the standard text, respectively, wherein the left and right information entropy indexes are used to characterize the independence of the words; the candidate words whose left and right information entropy index values ​​are both not less than a third threshold are determined as the candidate words obtained by the left and right information entropy screening, thereby obtaining a second candidate word set.

[0020] In some embodiments, the word screening unit is further configured to determine candidate words included in both the first candidate word set and the second candidate word set as mined new words.

[0021] In some embodiments, the new word mining device also includes a distributed processing unit, which is configured to, in response to determining that the original text contains multiple sentences, use a distributed cluster to process the multiple sentences separately, wherein the distributed cluster includes multiple server nodes, and each server node is equipped with at least one processor; according to the number of server nodes in the distributed cluster and the number of sentences in the original text, the sentences in the original text are partitioned for processing, wherein sentences in different partitions are processed by different server nodes.

[0022] In some embodiments, before screening the candidate words in the candidate word set according to preset indicator parameters, the distributed processing unit is further configured to generate a word frequency dictionary based on the characters included in the candidate words in the candidate word set and the number of times the words indicated by the included characters appear in the standard text; and broadcast the word frequency dictionary in the distributed cluster.

[0023] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the new word mining method described in any implementation method in the above-mentioned first aspect.

[0024] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the new word mining method described in any implementation manner in the above-mentioned first aspect is implemented.

[0025] In a fifth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which, when executed by a processor, implements the new word mining method described in any implementation manner in the first aspect above.

[0026] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the new word mining method of some embodiments of the present disclosure proposes an unsupervised new word automatic mining method. Specifically, after obtaining the original text, it can first be standardized to obtain a standard text. In this way, the influence of interference factors (such as irrelevant characters such as punctuation marks) can be removed, and the accuracy of subsequent word segmentation can be improved. Then, the sentence in the standard text can be word segmented based on the algorithm of the statistical language model. That is, the sentence is converted into words of different lengths for subsequent analysis. After that, the candidate words are screened out by the total number of words, and the words with low probability are removed, thereby improving the efficiency and accuracy of subsequent new word mining. Finally, the combination of character representation characters can be used as the preset indicator parameters of the probability of the word to further screen the candidate words, so as to obtain the mined new words. It can be understood that the preset indicator parameters here are often commonly used and effective word analysis methods in the industry. The method disclosed in the present disclosure does not rely on too much prior knowledge, that is, it is not necessary to formulate a large number of cumbersome rules. At the same time, it is also not necessary to perform a large amount of sample data annotation and model training. Therefore, unsupervised new word automatic mining can be achieved, and the mining process does not require human intervention, and no manual labeling is required in advance. While ensuring the effectiveness of data mining, resource requirements can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0028] Figure 1 is a flow chart of some embodiments of the new word mining method disclosed in the present invention;

[0029] Figure 2 are flow charts of other embodiments of the new word mining method disclosed in the present invention;

[0030] Figure 3A yes Figure 2 Schematic diagram of some embodiments of the partitioning process;

[0031] Figure 3B yes Figure 2 Schematic diagram of some embodiments of stop word removal;

[0032] Figure 3C yes Figure 2 Schematic diagram of some embodiments of n-gram segmentation;

[0033] Figure 3D yes Figure 2 Schematic diagram of some embodiments of word frequency screening;

[0034] Figure 3E yes Figure 2 Flow chart of some embodiments of solidification degree screening;

[0035] Figure 3F yes Figure 2 Flow chart of some embodiments of left and right information entropy screening;

[0036] Figure 4 is a schematic diagram of the structure of some embodiments of the new word mining device disclosed in the present invention;

[0037] Figure 5 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0039] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0040] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0041] In addition, the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0042] It should be noted that word segmentation is usually the process of recombining continuous character sequences into word sequences according to certain specifications. In languages ​​such as English, words are generally separated by spaces as natural delimiters. In Chinese, characters, sentences, and paragraphs can be simply demarcated by obvious delimiters, but words do not have a formal delimiter. Although English also has the problem of dividing phrases, Chinese is much more complicated and difficult than English at the word level. Therefore, when segmenting Chinese words, it is often necessary to use a word library to achieve it.

[0043] The vocabulary is usually the basis and foundation of word segmentation. The word segmentation algorithm usually needs to identify the lexical boundaries in the text based on the entries in the vocabulary. A complete, accurate and rich vocabulary can often provide sufficient entry information for the word segmentation algorithm, enabling it to segment the text more accurately. The accuracy and completeness of the vocabulary directly affect the accuracy of word segmentation. A high-quality vocabulary can reduce word segmentation errors, improve the accuracy of word segmentation results, and provide better input for subsequent natural language processing tasks. Therefore, the vocabulary needs to be constantly updated to adapt to rapidly changing requests. And an outdated vocabulary may even cause erroneous word segmentation results. Based on this, the present application proposes an unsupervised method for automatically mining new words in text to achieve continuous updating of the vocabulary.

[0044] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0045] Figure 1 The process 100 of some embodiments of the new word mining method according to the present disclosure is shown. The method may include the following steps:

[0046] Step 101, standardize the acquired original text to obtain a standard text.

[0047] In some embodiments, the execution subject (e.g., server) of the new word mining method of some embodiments of the present disclosure can be connected to other electronic devices (e.g., databases or distributed clusters) through wired or wireless connections. Here, in order to realize new word mining in text data, users can obtain the required text data from public resources, or collect text data by themselves as the original text to be mined for new words. It should be noted that the original text here is usually a text containing a large number of Chinese characters, and the text may also contain at least one character such as numbers, letters, symbols, etc.

[0048] Here, the execution subject can perform standardization processing on the acquired original text to obtain a standard text. The standardization processing method and processing content here can be set according to actual needs. As an example, the execution subject can divide the sentences in the original text into paragraphs according to specific punctuation marks, such as commas, periods, semicolons, exclamation marks, etc. A sentence is generally located in the same paragraph, and different sentences are located in different paragraphs.

[0049] It is understandable that there are often some modal particles, punctuation marks, special symbols, etc. in Chinese natural language text. These characters usually affect the mining of new words, so they need to be removed. Optionally, the execution entity can remove the characters that match the preset characters in the original text, and use the original text without the preset characters as the standard text. Among them, the preset characters can include at least one of the following: modal particles, punctuation marks, special symbols, formulas, etc. Or Arabic numerals can also be converted to uppercase characters, etc. That is to say, the embodiments of the present disclosure can use preset regular matching rules to process each sentence to remove interfering characters.

[0050] Step 102, using an algorithm based on a statistical language model to segment sentences in the standard text into words, to obtain a word set of the standard text.

[0051] In some embodiments, the execution subject can use an algorithm based on a statistical language model (such as n-gram) to segment the sentences in the standard text into words, thereby obtaining a word set of the standard text. It can be understood that n-gram generally refers to the segmentation of a sentence into words with a length from 1 to n. Among them, n represents the length of the longest word and can be set according to actual needs. For example, in the sentence "I love computers", assuming that n is 3, the segmentation results of 1-gram are: "I", "love", "calculation", "calculation", "machine"; the segmentation results of 2-gram are: "I love", "love calculation", "calculation", "calculation machine"; the segmentation results of 3-gram are: "I love calculation", "love calculation", "computer". Although many of these words are obviously meaningless, word segmentation is an important prerequisite for discovering new words, and subsequent analysis and screening are based on these words.

[0052] Here, for each sentence in the standard text, the execution subject can summarize the words obtained by segmenting each sentence to obtain a word set of the standard text.

[0053] Step 103 , based on the total number of words in the word set, determine the words whose total number is not less than a first threshold as candidate words to obtain a candidate word set.

[0054] In some embodiments, the execution entity may count each word in the word set obtained in step 102 to determine the total quantity (i.e., number) of each word. The statistical method is also not limited. As an example, the execution entity may match each word in the word set, and treat the words that match the same as the same word, thereby obtaining the total quantity of the word, and recording the matching result. Then, the word may be removed from the word set to perform matching statistics for the next word until all word statistics are completed. For another example, the execution entity may cluster and group the words in the word set, thereby grouping the same word into one group, and then obtaining the quantity of each word by statistics.

[0055] It should be noted that, generally, the higher the number of times a word appears, the greater the possibility that it can be used as an independent word. Therefore, in order to improve the accuracy of new word mining, the execution subject can screen the words in the word set according to the first threshold. That is, the words whose total number is not less than the first threshold can be determined as candidate words, thereby obtaining a candidate word set. The first threshold can be set according to actual needs. This can avoid the subsequent processing of invalid data, reduce the amount of data to be processed, and help improve data processing efficiency.

[0056] Step 104 , screening the candidate words in the candidate word set according to preset indicator parameters, and determining the screened candidate words as mined new words.

[0057] In some embodiments, for the candidate word set obtained in step 103, the execution subject can further screen the candidate words therein according to preset indicator parameters. Thereby, the screened candidate words can be determined as new words mined. Among them, the preset indicator parameters can be used to characterize the probability that a combination of characters can be used as a word. The preset indicator parameters are also not limited, such as mutual information algorithm, chi-square test algorithm, etc. Here, mutual information is usually used to measure the degree of mutual dependence between two random variables. The chi-square test is a very widely used hypothesis testing method, which is usually the degree of deviation between the actual observed value and the theoretical inference value of the statistical sample. The degree of deviation between the actual observed value and the theoretical inference value determines the size of the chi-square value.

[0058] In some embodiments, the execution subject may use a solidification index for screening. The solidification index is used to characterize the internal solidification degree of a word. In other words, are the characters that make up the word solid enough? In simple terms, do these characters often make up the word? Specifically, for each candidate word in the candidate word set, the solidification index value of the candidate word can be determined based on the number of times the candidate word appears in the standard text. Then, the candidate word whose solidification index value is not less than the second threshold value can be determined as the candidate word obtained by solidification screening, and the first candidate word set is obtained.

[0059] As an example, take the two characters "足 (zú)" and "球 (qiú)". Suppose in a text of 50 million words, "足 (zú)" appears 40,000 times and "球 (qiú)" appears 1.5 million times. Then the probability of "足 (zú)" appearing is 0.0008, and the probability of "球 (qiú)" appearing is 0.03. If the appearances of the two characters are independent events, the expected probability of "足 (zú)" and "球 (qiú)" appearing together is 0.0008 * 0.03 = 2.4e-05. If in fact, "足球 (zúqiú)" appears 30,000 times, then the actual probability of "足 (zú)" and "球 (qiú)" appearing together is 6e-03. This probability is 250 times the expected probability, so the coagulation degree is 250. That is to say, the greater the coagulation degree, the more relevant the two characters are, and the greater the probability of forming a word.

[0060] Furthermore, for candidate words with a character length greater than two, first, according to different splitting rules, different character combinations of the candidate word can be split respectively. Among them, during the splitting process, the position order of each character in the candidate word remains unchanged. Then, for each character combination, based on the number of times each character in the character combination appears in the standard text and the number of times the candidate word appears in the standard text, the coagulation degree index value of the character combination is determined. After that, the minimum value among the coagulation degree index values of each character combination can be used as the coagulation degree index value of the candidate word.

[0061] That is to say, when the length of the word is greater than 2, the coagulation degrees of different combinations need to be calculated separately, and the minimum value is taken. For example, the word "守门员 (shǒuményuán)" can be split into two combinations: [守 (shǒu), 门员 (ményuán)] and [守门 (shǒumén), 员 (yuán)]. At this time, the coagulation degree of the word "守门员 (shǒuményuán)" can be determined according to the following formula:

[0062]

[0063] Among them, a, b, and c respectively represent the characters in the word; P() represents the number of times the word composed of characters appears in the text.

[0064] It can be understood that if the coagulation degree of the word is less than the second threshold, it means that the word is not stable enough and needs to be screened out. Optionally, the execution entity can determine the candidate words obtained by coagulation degree screening as the newly mined words.

[0065] In some application scenarios, the execution entity can also perform screening according to the left and right information entropy indicators. Among them, the left and right information entropy indicators can be used to characterize the independence of the word. Specifically, for each candidate word in the candidate word set, based on the characters that appear on the left and right sides of the candidate word in the standard text, the left and right information entropy index values of the candidate word can be determined respectively. After that, the candidate words with left and right information entropy index values not less than the third threshold can be determined as the candidate words obtained by left and right information entropy screening, and the second candidate word set is obtained.

[0066] It should be noted that through the screening process in step 103, it can be ensured that the candidate words obtained are usually relatively common combinations. However, considering the word "goalkeeper" again, the sub-word "keeper" inside is also very stable, and its appearance probability is much greater than the expectation of the simultaneous appearance of "door" and "member". However, every time "keeper" appears, it basically has to appear simultaneously with the "guard" on the left, that is, the characters on its left are very limited. "Goalkeeper" can obviously be placed in a very large number of contexts, so the characters on its left are very rich. The same is true for the right side. Therefore, if an index is used to represent the richness of the characters on the left / right side of a word, words with stronger independence can be screened out. Entropy represents the complexity and chaos of a thing, and information entropy can measure the richness of information. In this example, it is how random the adjacent characters on both sides of a word are, and the calculation formula is as follows:

[0067]

[0068] where p i represents the probability of the i-th adjacent character on the left or right; n represents the number of adjacent characters on the left or right.

[0069] Taking "Eating grapes but not spitting out grape skins, not eating grapes but spitting out grape skins" as an example, the word "grapes" appears four times, and the left adjacent characters are [eat, spit, eat, spit] respectively, and the right adjacent characters are [not, skin, pour, skin] respectively. According to the formula, the left information entropy of the word "grapes" is –(1 / 2)·log(1 / 2)–(1 / 2)·log(1 / 2)≈0.693, and its right information entropy is –(1 / 2)·log(1 / 2)–(1 / 4)·log(1 / 4)–(1 / 4)·log(1 / 4)≈1.04.

[0070] It can be understood that if the left information entropy or the right information entropy of a word is less than the third threshold, it needs to be screened out. Optionally, the execution entity can determine the candidate words obtained by screening the left and right information entropies as the newly mined words.

[0071] Optionally, in order to further improve the accuracy of new word mining, the execution entity can determine the candidate words jointly included in the first candidate word set and the second candidate word set as the newly mined words. That is, the first candidate word set and the second candidate word set can be finally merged and screened.

[0072] It should be noted that in the related art, the rule-based mining method usually relies on linguistic knowledge, domain-specific knowledge, and expert experience to formulate rules. These rules can be based on grammar, semantics, morphology, etc., to identify and extract specific information or patterns in the text, so as to discover new words. However, this method largely depends on the quality and quantity of the rules. If the rules are not formulated comprehensively or accurately, it may lead to missed reports or false reports in new word mining.

[0073] In addition, the mining methods based on deep learning in related technologies often require a large amount of labeled data for training, and the performance and effect of the model are affected by the quality and quantity of the training data. In addition, the selection of the model and parameter adjustment will also have a significant impact on the results. This method not only consumes manpower, but also has high requirements for hardware resources (such as graphics processors GPU), and the cost is very expensive.

[0074] Based on this, the new word mining method of some embodiments of the present disclosure proposes an unsupervised new word automatic mining method. Specifically, after obtaining the original text, it can first be standardized to obtain a standard text. In this way, the influence of interference factors (such as irrelevant characters such as punctuation marks) can be removed, and the accuracy of subsequent word segmentation can be improved. Then, the sentence in the standard text can be word segmented based on the algorithm of the statistical language model. That is, the sentence is converted into words of different lengths for subsequent analysis. After that, the candidate words are screened out by the total number of words, and the words with low probability are removed, thereby improving the efficiency and accuracy of subsequent new word mining. Finally, the combination of character representation characters can be used as the preset indicator parameters of the probability of the word to further screen the candidate words, so as to obtain the mined new words. It can be understood that the preset indicator parameters here are often commonly used and effective word analysis methods in the industry. The method disclosed in the present disclosure does not rely on too much prior knowledge, that is, it is not necessary to formulate a large number of cumbersome rules. At the same time, it is also not necessary to perform a large amount of sample data annotation and model training. Therefore, unsupervised new word automatic mining can be achieved, and the mining process does not require human intervention, and no manual labeling is required in advance. While ensuring the effectiveness of data mining, resource requirements can be reduced.

[0075] In some application scenarios, if it is determined that the original text contains multiple sentences, or there are multiple original texts that need to be mined for new words, a distributed cluster can be used to process these original texts or sentences separately. Among them, the distributed cluster can include multiple server nodes, and each server node can be provided with at least one processor. This can solve the problem of the limited amount of data that can be processed in a single-machine environment, thereby realizing a solution for mining new words in massive texts based on a distributed cluster. This can be achieved with only an ordinary CPU (central processing unit) computer, which can reduce hardware costs while increasing the amount of data processed.

[0076] Specific as Figure 2 As shown in the figure, the original text (i.e. the text to be mined) is first preprocessed, including data repartitioning, removal of stop words, word segmentation, and word frequency screening. After preprocessing, the sentences will become words of different lengths, and the second and third processes will be carried out, which are solidification screening and left and right information entropy screening. After obtaining two candidate sets, a Join operation is performed, and the intersection of the results of 2 and 3 is the mined new word.

[0077] It is understandable that the purpose of preprocessing is to segment sentences in the text into "candidate words" that meet the conditions for subsequent analysis. The so-called candidate words refer to a combination of single characters, which are just put together, and whether they can eventually be used as a word requires further analysis.

[0078] Here, in order to process massive amounts of data, the algorithm needs to be run in a distributed cluster. Figure 3A As shown in the figure, the distributed cluster consists of multiple servers (such as Node1, 2, and 3 in the figure), which have multiple processors inside. Such a structure can ensure parallel processing of data. In order to adapt to such a cluster environment, the algorithm needs to split the existing data into multiple parts, each of which can be processed by the cluster as a subtask. In other words, first of all, it is necessary to partition the sentences in the original text according to the number of server nodes in the distributed cluster and the number of sentences in the original text. Among them, the sentences in different partitions are processed by different server nodes.

[0079] from Figure 3A As can be seen from the figure, the Hash Partitioner algorithm can be used here to make the data in each partition as similar as possible. As shown in the figure, the seven address texts S1 to S7 are processed by the partitioner and split into six tasks corresponding to multiple processors. These tasks can be processed independently in parallel to improve overall efficiency. There is a dynamically configurable parameter here, namely the number of partitions P. The larger the number of partitions, the greater the parallelism of the task. In actual application, it is necessary to select a suitable value based on the number of servers and partitions.

[0080] Then as Figure 3B As shown in the figure, since the statement has been partitioned before, the elimination operation can be completed in parallel within each partition. For details, see Figure 1 The relevant description in the embodiments will not be repeated here.

[0081] Continue as Figure 3C For the n-gram segmentation process, in addition to segmenting the sentence, a distributed operation method flatMap is also needed. This is usually a method of splitting distributed elements into sub-elements. Each element we processed earlier is a sentence, that is, Figure 3BFrom S1 to S7, the target elements will be transformed into words after segmentation. In the figure, W1-1 represents the 1-gram set of sentence S1, and W2-3 represents the 3-gram set of sentence S2. It can be seen that the words after segmentation of each sentence are still located in the original partition, and the subsequent processing flow is still based on the parallel operation of the original partition. The dotted box on the far right is a distributed collection object, that is, the above word set. If the three boxes in the middle are the physical structure distribution of the data, the dotted box can be considered as a logical structure, and each element in it is a word, which is also a basic operation unit.

[0082] Then as Figure 3D When filtering the word frequency, you need to convert the word into a binary, that is, a word and a number. For example, the word "I love" is converted to (I love, 1). "I love" is the key and 1 is the value, where 1 represents 1 word. Then use the distributed operation reduceByKey to accumulate the value of each identical key, and finally get each key (word) and its total number. With each word and its total number, you need to filter out words whose total number is less than the threshold, such as the words marked with × in the figure. This is implemented here using the distributed filter operation.

[0083] After that, the distributed execution process of solidification screening is as follows Figure 3E As shown. First, based on the word frequency set obtained by the word frequency screening in the previous step, use the collect operation to localize all words and their word frequencies into a dictionary. Then broadcast it to the cluster, which is essentially to save a cache on each machine. The third step is to use the map operation based on the word frequency set to calculate the coagulation degree of each word. The calculation process is to split each word first, and then find the corresponding word frequency in the cached dictionary based on the word. Then use the formula to calculate the coagulation degree of the current word. The fourth step is to filter out words that are greater than the threshold, and the result is a coagulation degree candidate set, that is, the first candidate word set.

[0084] Continuing, the process of information entropy is as follows Figure 3F As shown. First, the neighborhood of the original sentence needs to be extracted. Here, the flatMap operation is used to extract the neighborhood of each word in each sentence, and a 3-tuple is obtained: left neighbor character, word, right neighbor character. Step 2, the 3-tuple is converted into key, value form through the map operation, with the word as the key and the 2-tuple consisting of the left and right neighbor characters as the value. Step 3, the neighbor character 2-tuples with the same key are grouped into a set through the reduceByKey operation. Step 4, calculate the left and right information entropy (i.e., neighborhood entropy) of each word. Finally, the filter operation is used to filter all words whose left and right information entropy is greater than the threshold. The result is the left and right information entropy candidate set, i.e., the second candidate word set.

[0085] Finally, the distributed Join operation can be used to perform the final merge screening, that is, to retain the words contained in both sets as the final result.

[0086] Further references Figure 4 , as a response to the above Figures 1 to 3F The present disclosure provides some embodiments of a new word mining device. These device embodiments are similar to Figures 1 to 3F The device can be specifically applied to various electronic devices.

[0087] like Figure 4 As shown, some embodiments of the new word mining device 400 may include: a text processing unit 401, configured to perform standardization processing on the acquired original text to obtain a standard text; a word segmentation unit 402, configured to use an algorithm based on a statistical language model to perform word segmentation on sentences in the standard text to obtain a word set of the standard text; a candidate word determination unit 403, configured to determine words whose total number is not less than a first threshold as candidate words based on the total number of words in the word set, to obtain a candidate word set; a word screening unit 404, configured to screen the candidate words in the candidate word set according to preset indicator parameters, and determine the screened candidate words as mined new words, wherein the preset indicator parameters are used to characterize the probability that a combination of characters can be used as a word.

[0088] In some embodiments, the text processing unit 401 can be further configured to remove characters in the original text that match preset characters, and use the original text without the preset characters as the standard text, wherein the preset characters include at least one of the following: modal particles, punctuation marks, and special symbols.

[0089] In some embodiments, the word screening unit 404 can be further configured to determine, for each candidate word in the candidate word set, a coagulation index value of the candidate word according to the number of times the candidate word appears in the standard text, wherein the coagulation index is used to characterize the internal coagulation degree of the word; and determine the candidate words whose coagulation index values ​​are not less than the second threshold as the candidate words obtained by the coagulation screening, thereby obtaining a first candidate word set.

[0090] In some embodiments, the word screening unit 404 can be further configured to split a candidate word having a character length greater than two into different character combinations of the candidate word according to different splitting rules, wherein during the splitting process, the position order of each character in the candidate word remains unchanged; for each character combination, the coagulation index value of the character combination is determined according to the number of times each character in the character combination appears in the standard text and the number of times the candidate word appears in the standard text; and the minimum value of the coagulation index values ​​of each character combination is used as the coagulation index value of the candidate word.

[0091] In some embodiments, the word screening unit 404 can be further configured to determine, for each candidate word in the candidate word set, the left and right information entropy index values ​​of the candidate word according to the characters that appear on the left and right sides of the candidate word in the standard text, respectively, wherein the left and right information entropy indexes are used to characterize the independence of the words; the candidate words whose left and right information entropy index values ​​are both not less than the third threshold are determined as the candidate words obtained by the left and right information entropy screening, thereby obtaining the second candidate word set.

[0092] In some embodiments, the word screening unit 404 may be further configured to determine the candidate words included in both the first candidate word set and the second candidate word set as mined new words.

[0093] In some embodiments, the new word mining device 400 may also include a distributed processing unit (not shown in the figure), which is configured to, in response to determining that the original text contains multiple sentences, use a distributed cluster to process the multiple sentences separately, wherein the distributed cluster includes multiple server nodes, and each server node is provided with at least one processor; according to the number of server nodes in the distributed cluster and the number of sentences in the original text, the sentences in the original text are partitioned for processing, wherein sentences in different partitions are processed by different server nodes.

[0094] In some embodiments, before screening the candidate words in the candidate word set according to preset indicator parameters, the distributed processing unit can be further configured to generate a word frequency dictionary based on the characters included in the candidate words in the candidate word set and the number of times the words indicated by the included characters appear in the standard text; and broadcast the word frequency dictionary in the distributed cluster.

[0095] It can be understood that the units recorded in the new word mining device 400 and the reference Figures 1 to 3F Therefore, the operations, features and beneficial effects described above for the method are also applicable to the new word mining device 400 and the units included therein, and will not be described in detail here.

[0096] Reference below Figure 5 , which shows a structural schematic diagram of an electronic device 500 suitable for implementing some embodiments of the present disclosure. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0097] like Figure 5As shown, the electronic device 500 may include a processing device 501 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0098] Typically, the following devices may be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a speaker, a vibrator, etc.; storage devices 508 including, for example, a disk, a hard disk, a memory card, etc.; and communication devices 509. The communication devices 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0099] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0100] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0101] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0102] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: performs standardization processing on the acquired original text to obtain a standard text; uses an algorithm based on a statistical language model to perform word segmentation on the sentences in the standard text to obtain a word set of the standard text; determines the words whose total number is not less than a first threshold as candidate words according to the total number of words in the word set to obtain a candidate word set; screens the candidate words in the candidate word set according to preset indicator parameters, and determines the screened candidate words as mined new words, wherein the preset indicator parameters are used to characterize the probability that a combination of characters can be used as a word.

[0103] In addition, computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including a text processing unit, a word segmentation unit, a candidate word determination unit, and a word screening unit. The names of these units do not constitute limitations on the units themselves in certain circumstances, for example, the text processing unit may also be described as a "unit that performs standardization processing on the acquired original text to obtain a standard text".

[0106] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0107] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which implements any of the above-mentioned new word mining methods when executed by a processor.

[0108] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A new word mining method, comprising: Standardize the acquired original text to obtain a standard text; Using an algorithm based on a statistical language model to segment sentences in the standard text into words, thereby obtaining a word set of the standard text; According to the total number of words in the word set, words whose total number is not less than a first threshold are determined as candidate words to obtain a candidate word set; The candidate words in the candidate word set are screened according to preset indicator parameters, and the screened candidate words are determined as mined new words, wherein the preset indicator parameters are used to characterize the probability that a combination of characters can be used as a word.

2. The new word mining method according to claim 1, wherein: The step of standardizing the acquired original text to obtain a standard text includes: The characters matching the preset characters in the original text are removed, and the original text without the preset characters is used as the standard text, wherein the preset characters include at least one of the following: modal particles, punctuation marks, and special symbols.

3. The new word mining method according to claim 1, wherein: The screening of the candidate words in the candidate word set according to the preset indicator parameters includes: For each candidate word in the candidate word set, determining a solidification index value of the candidate word according to the number of times the candidate word appears in the standard text, wherein the solidification index is used to characterize the internal solidification degree of the word; The candidate words whose solidification index values ​​are not less than the second threshold are determined as candidate words obtained by solidification screening, and a first candidate word set is obtained.

4. The new word mining method according to claim 3, wherein: Determining the solidification index value of the candidate word according to the number of times the candidate word appears in the standard text includes: For candidate words with a character length greater than two, different character combinations of the candidate words are obtained by splitting them according to different splitting rules, wherein the position order of each character in the candidate word remains unchanged during the splitting process; For each character combination, according to the number of times each character in the character combination appears in the standard text and the number of times the candidate word appears in the standard text, determining the solidification index value of the character combination; The minimum value of the coagulation index values ​​of each character combination is used as the coagulation index value of the candidate word.

5. The new word mining method according to claim 3, wherein: The screening of the candidate words in the candidate word set according to the preset indicator parameters also includes: For each candidate word in the candidate word set, according to the characters appearing on the left and right sides of the candidate word in the standard text, respectively determine the left and right information entropy index values ​​of the candidate word, wherein the left and right information entropy indexes are used to characterize the independence of the words; The candidate words whose left and right information entropy index values ​​are both not less than the third threshold are determined as the candidate words obtained by left and right information entropy screening, and the second candidate word set is obtained.

6. The new word mining method according to claim 5, wherein: The step of determining the screened candidate words as mined new words includes: The candidate words included in both the first candidate word set and the second candidate word set are determined as mined new words.

7. The new word mining method according to any one of claims 1 to 6, wherein: The method further comprises: In response to determining that the original text includes a plurality of sentences, using a distributed cluster to process the plurality of sentences respectively, wherein the distributed cluster includes a plurality of server nodes, and each server node is provided with at least one processor; According to the number of server nodes in the distributed cluster and the number of sentences in the original text, the sentences in the original text are partitioned and processed, wherein the sentences in different partitions are processed by different server nodes.

8. The new word mining method according to claim 7, wherein: Before screening the candidate words in the candidate word set according to the preset indicator parameters, the method further includes: Generate a word frequency dictionary according to the characters included in the candidate words in the candidate word set and the number of times the words indicated by the included characters appear in the standard text; The word frequency dictionary is broadcasted in the distributed cluster.

9. A new word mining device, comprising: A text processing unit is configured to perform standardization processing on the acquired original text to obtain a standard text; A word segmentation unit is configured to use an algorithm based on a statistical language model to perform word segmentation on the sentences in the standard text to obtain a word set of the standard text; a candidate word determination unit configured to determine, according to the total number of words in the word set, words whose total number is not less than a first threshold as candidate words, to obtain a candidate word set; The word screening unit is configured to screen the candidate words in the candidate word set according to preset indicator parameters, and determine the screened candidate words as the mined new words, wherein the preset indicator parameters are used to represent the probability that a combination of characters can be used as a word.

10. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the new word mining method as described in any one of claims 1-8.

11. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the new word mining method as described in any one of claims 1-8 is implemented.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the new word mining method according to any one of claims 1 to 8.

Images