New word discovery method, device, equipment, and computer-readable storage medium

Abstract

The present invention relates to the field of natural language processing technology, and discloses a new word discovery method, apparatus, device, and computer-readable storage medium. The new word discovery method comprises: obtaining a corpus to be processed; processing the corpus to be processed to obtain a set of segmented words to be processed; calculating a first parameter value for each first segmented word in the set of segmented words to be processed, the first parameter value including a first PMI value and a first left-right information entropy; inputting each first segmented word and its first parameter value into a pre-trained new word discovery model to obtain a first classification result; and determining whether each first segmented word is a new word based on the first classification result to obtain a new word set. The present invention can improve the accuracy of new word discovery results.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a new word discovery method, device, equipment and computer-readable storage medium. Background Art

[0002] New word discovery technology has high use value in the fields of text segmentation, named entity recognition, knowledge graph construction, text topic mining, etc. At present, new word mining is mainly based on statistical methods and methods based on open source tools. Among them, the statistical method usually looks for statistics that describe the characteristics of new words, such as PMI (Pointwise Mutual Information) value, left and right information entropy, etc., and then filters through the thresholds corresponding to the preset statistics to determine the new words. The method based on open source tools also often filters on the basis of statistics. For example, the Smoothnlp algorithm calculates the score of each candidate word after generating candidate words for the text, where the score is the sum of the absolute value of the left and right information entropy difference and the average value of the PMI, and then filters the candidate words based on the score to obtain new words.

[0003] The above-mentioned new word statistical method can only solve the problem of discovering a small amount of new words. It is highly dependent on the corpus. In mixed corpora from different fields, many new words with low frequency of occurrence are easily filtered out, resulting in the omission of new words, which leads to poor accuracy of new word discovery results. Summary of the Invention

[0004] The main purpose of the present invention is to provide a new word discovery method, device, equipment and computer-readable storage medium, aiming to solve the problem of poor accuracy of existing new word discovery methods.

[0005] To achieve the above object, the present invention provides a new word discovery method, which comprises:

[0006] Get the corpus to be processed;

[0007] Processing the corpus to be processed to obtain a word segmentation set to be processed;

[0008] Calculating a first parameter value of each first segmentation in the segmentation set to be processed, where the first parameter value includes a first PMI value and a first left-right information entropy;

[0009] Inputting each first participle and its first parameter value into a pre-trained new word discovery model to obtain a first classification result;

[0010] Determine whether each first participle is a new word according to the first classification result to obtain a new word set.

[0011] Preferably, the new word discovery method further comprises:

[0012] Obtain training corpus;

[0013] Processing the training corpus to obtain a training word set;

[0014] Calculate a second parameter value of each second segmentation in the training segmentation set, where the second parameter value includes a second PMI value and a second left-right information entropy;

[0015] The preset new word discovery model is trained using the training word segmentation set and the second parameter value to obtain the pre-trained new word discovery model.

[0016] Preferably, the step of training a preset new word discovery model using the training word set and the second parameter value to obtain the pre-trained new word discovery model includes:

[0017] Preliminarily screening the second participle in the training participle set according to the second PMI value, the second left-right information entropy and a preset threshold range to obtain a third participle;

[0018] Mark each third participle, and construct a training sample based on the marked third participle and its corresponding third PMI value and third left and right information entropy;

[0019] The preset new word discovery model is trained using the training samples to obtain the pre-trained new word discovery model.

[0020] Preferably, after the step of training the preset new word discovery model using the training samples to obtain the pre-trained new word discovery model, the new word discovery method further comprises:

[0021] Inputting each third participle and its corresponding third PMI value and third left-right information entropy into the pre-trained new word discovery model to obtain a second classification result;

[0022] The second classification result is compared with the marking result of the third word segmentation, and a new word screening rule is determined according to the comparison result.

[0023] Preferably, after the step of determining whether each first participle is a new word according to the first classification result to obtain a new word set, the new word discovery method further includes:

[0024] The new word set is screened again using the new word screening rule to obtain a final new word set.

[0025] Preferably, the step of processing the corpus to be processed to obtain a word segment set to be processed includes:

[0026] Removing punctuation marks from the corpus to be processed;

[0027] The corpus to be processed with punctuation removed is segmented to obtain a segmented word set to be processed.

[0028] Preferably, the step of performing word segmentation on the corpus to be processed from which punctuation marks have been removed to obtain a word segmentation set to be processed comprises:

[0029] The N-gram algorithm is used to perform word segmentation on the corpus to be processed from which punctuation marks have been removed, and a word segmentation set to be processed is obtained.

[0030] In addition, to achieve the above-mentioned purpose, the present invention further provides a new word discovery device, the new word discovery device comprising:

[0031] The first acquisition module is used to acquire the corpus to be processed;

[0032] A first processing module is used to process the corpus to be processed to obtain a word segmentation set to be processed;

[0033] A first calculation module is used to calculate a first parameter value of each first segmentation in the segmentation set to be processed, where the first parameter value includes a first PMI value and a first left-right information entropy;

[0034] A first classification module is used to input each first segmented word and its first parameter value into a pre-trained new word discovery model to obtain a first classification result;

[0035] The new word determination module is used to determine whether each first participle is a new word according to the first classification result, so as to obtain a new word set.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a new word discovery device, which includes: a memory, a processor, and a new word discovery program stored on the memory and runnable on the processor. When the new word discovery program is executed by the processor, the steps of the new word discovery method described above are implemented.

[0037] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium, on which a new word discovery program is stored. When the new word discovery program is executed by a processor, the steps of the new word discovery method described above are implemented.

[0038] The present invention provides a new word discovery method, device, equipment and computer-readable storage medium. The method first obtains a corpus to be processed, processes the corpus to be processed, and obtains a set of word segments to be processed; then, calculates a first parameter value of each first word in the set of word segments to be processed, the first parameter value including a first PMI value and a first left-right information entropy; finally, inputs each first word and its first parameter value into a pre-trained new word discovery model to obtain a first classification result, and then determines whether each first word is a new word based on the first classification result to obtain a new word set. The present invention adopts a statistical method as a basis, and then combines the new word discovery model to identify new words, which can greatly improve the accuracy of the new word discovery results. In addition, the new word discovery model also improves the recall rate of new word recognition, and has stronger applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the process of the first embodiment of the new word discovery method of the present invention;

[0041] Figure 3 Schematic diagram of the flow chart of the second embodiment of the new word discovery method of the present invention;

[0042] Figure 4 Schematic diagram of the functional modules of the first embodiment of the new word discovery device of the present invention.

[0043] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0046] The new word discovery device in the embodiment of the present invention may be a server, or a terminal device such as a PC (Personal Computer), a smart phone, a tablet computer, or a portable computer.

[0047] like Figure 1As shown, the new word discovery device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally also be a storage device independent of the aforementioned processor 1001.

[0048] Those skilled in the art will understand that Figure 1 The structure of the new word discovery device shown in the figure does not constitute a limitation to the new word discovery device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0049] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a new word discovery program.

[0050] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; and the processor 1001 can be used to call the new word discovery program stored in the memory 1005 and perform the following operations:

[0051] Get the corpus to be processed;

[0052] Processing the corpus to be processed to obtain a word segmentation set to be processed;

[0053] Calculating a first parameter value of each first segmentation in the segmentation set to be processed, where the first parameter value includes a first PMI value and a first left-right information entropy;

[0054] Inputting each first participle and its first parameter value into a pre-trained new word discovery model to obtain a first classification result;

[0055] Determine whether each first participle is a new word according to the first classification result to obtain a new word set.

[0056] Furthermore, the processor 1001 may call the new word discovery program stored in the memory 1005 and perform the following operations:

[0057] Obtain training corpus;

[0058] Processing the training corpus to obtain a training word set;

[0059] Calculate a second parameter value of each second segmentation in the training segmentation set, where the second parameter value includes a second PMI value and a second left-right information entropy;

[0060] The preset new word discovery model is trained using the training word segmentation set and the second parameter value to obtain the pre-trained new word discovery model.

[0061] Furthermore, the processor 1001 may call the new word discovery program stored in the memory 1005 and perform the following operations:

[0062] Preliminarily screening the second participle in the training participle set according to the second PMI value, the second left-right information entropy and a preset threshold range to obtain a third participle;

[0063] Mark each third participle, and construct a training sample based on the marked third participle and its corresponding third PMI value and third left and right information entropy;

[0064] The preset new word discovery model is trained using the training samples to obtain the pre-trained new word discovery model.

[0065] Furthermore, the processor 1001 may call the new word discovery program stored in the memory 1005 and perform the following operations:

[0066] Inputting each third participle and its corresponding third PMI value and third left-right information entropy into the pre-trained new word discovery model to obtain a second classification result;

[0067] The second classification result is compared with the marking result of the third word segmentation, and a new word screening rule is determined according to the comparison result.

[0068] Furthermore, the processor 1001 may call the new word discovery program stored in the memory 1005 and perform the following operations:

[0069] The new word set is screened again using the new word screening rule to obtain a final new word set.

[0070] Furthermore, the processor 1001 may call the new word discovery program stored in the memory 1005 and perform the following operations:

[0071] Removing punctuation marks from the corpus to be processed;

[0072] The corpus to be processed with punctuation removed is segmented to obtain a segmented word set to be processed.

[0073] Furthermore, the processor 1001 may call the new word discovery program stored in the memory 1005 and perform the following operations:

[0074] The N-gram algorithm is used to perform word segmentation on the corpus to be processed from which punctuation marks have been removed, and a word segmentation set to be processed is obtained.

[0075] Based on the above hardware structure, various embodiments of the new word discovery method of the present invention are proposed.

[0076] The present invention provides a new word discovery method.

[0077] Reference Figure 2 , Figure 2 Schematic diagram of the flow chart of the first embodiment of the new word discovery method of the present invention.

[0078] In this embodiment, the new word discovery method includes:

[0079] Step S10, obtaining the corpus to be processed;

[0080] The new word discovery method of this embodiment is implemented by a new word discovery device, which can be a server or a terminal device such as a PC, a smart phone, a tablet computer, a portable computer, etc. In this embodiment, a server is used as an example for description.

[0081] In this embodiment, the corpus to be processed is first obtained, and the obtaining methods include but are not limited to: 1) user upload, 2) crawling from web pages; 3) obtaining from a preset database.

[0082] Step S20, processing the corpus to be processed to obtain a word segmentation set to be processed;

[0083] Then, the corpus to be processed is processed to obtain the word set to be processed.

[0084] Specifically, step S20 includes:

[0085] Step a21, removing punctuation marks from the corpus to be processed;

[0086] Step a22: performing word segmentation on the corpus to be processed from which punctuation marks have been removed, to obtain a word segmentation set to be processed.

[0087] The specific processing process is: remove the punctuation marks of the corpus to be processed so that only the text part remains in the corpus to be processed, so as to facilitate subsequent word segmentation. Then, the corpus to be processed with the punctuation marks removed is segmented to obtain a set of segmented words to be processed. Among them, the word segmentation processing can be implemented through word segmentation tools, such as Jieba word segmentation, Chinese lexical analysis system ICTCLAS, Tsinghua University Chinese lexical analysis program THULAC, language technology platform LTP, etc.; word segmentation can also be performed through algorithm models, such as N-gram algorithm, HMM (Hidden Markov Model, Hidden Markov Model), etc. Word segmentation is mainly based on the characteristics of the Chinese language, cutting the corpus to be processed into individual words (recorded as segmentations).

[0088] Furthermore, step a22 includes:

[0089] The N-gram algorithm is used to perform word segmentation on the corpus to be processed from which punctuation marks have been removed, and a word segmentation set to be processed is obtained.

[0090] As a preferred embodiment, the N-gram algorithm can be used for word segmentation, wherein, for the numerical range of N, the numerical range of N in the N-gram can be preliminarily estimated based on the previous understanding of the knowledge in the domain corpus, and then the character lengths within the N range can be segmented. For example, when N is in the range of 2 to 4, all segmentations with a character length of 2, a character length of 3, and a character length of 4 can be obtained. This constitutes a set of segmented words to be processed. By using the N-gram algorithm for word segmentation, a set of segmented words with various character lengths can be obtained by taking different N values. However, the word segmentation tool generally uses an existing dictionary for word segmentation, which has some restrictions on character length and will result in the omission of some new words. Therefore, compared with other word segmentation methods, the present invention can reduce the probability of missing new words, thereby improving the accuracy of new word discovery results.

[0091] Step S30, calculating a first parameter value of each first segmentation in the segmentation set to be processed, wherein the first parameter value includes a first PMI value and a first left-right information entropy;

[0092] After obtaining the set of word segments to be processed, the parameter value of each word in the set of word segments to be processed (to distinguish it from the word segments of subsequent other word sets, it is recorded as the first word segment) is calculated (to distinguish it from the parameter values ​​of the word segments of subsequent other word sets, it is recorded as the first parameter value). The first parameter value includes the PMI value of each first word segment (recorded as the first PMI value) and the left and right information entropy of each first word segment (recorded as the first left and right information entropy).

[0093] The PMI (Pointwise Mutual Information) value is used to measure the degree of clustering within a word. Specifically, the occurrence probability of each word in the word segmentation can be calculated separately, and the joint probability of the word segmentation can be calculated. Then, the joint probability of the word segmentation is divided by the product of the occurrence probabilities of each word to obtain the PMI value. The corresponding calculation formula is as follows:

[0094] PMI=P(x,y) / (P(x)*P(y));

[0095] Among them, P(x) represents the probability of character x appearing in the word segmentation, P(y) represents the probability of character y appearing in the word segmentation, and P(x,y) represents the probability of word segmentation xy appearing in the corpus to be processed.

[0096] Information entropy is a measure of the amount of information. Higher entropy indicates richer information and greater uncertainty. For a character combination to form a word, it should appear in a rich context—that is, have a rich set of left and right neighboring characters. The first left and right neighboring entropy includes the first left neighboring character entropy and the first right neighboring character entropy. The first left neighboring character entropy is the entropy of the left neighboring character of the first segmented word, and the first right neighboring character entropy is the entropy of the right neighboring character of the first segmented word. The word frequencies of the characters on the left and right sides of the segmented word can be calculated as probabilities. The first left and right neighboring character entropies of each segmented word can then be calculated using the following formula.

[0097]

[0098] Among them, H(x) represents the information entropy of the left / right neighboring characters of the word segmentation x, X represents the set of the left / right neighboring characters of the word segmentation, and p(x) represents the word frequency of the characters on the left / right side of the word segmentation.

[0099] Step S40, inputting each first participle and its first parameter value into a pre-trained new word discovery model to obtain a first classification result;

[0100] Each first word segment and its first parameter value are input into a pre-trained new word discovery model to obtain a first classification result. The training process of the new word discovery model can refer to the second embodiment below and will not be described in detail here.

[0101] Step S50: Determine whether each first segmented word is a new word according to the first classification result to obtain a new word set.

[0102] After obtaining the first classification result, it can be determined whether each first segmented word is a new word, so that a new word set can be obtained by screening from the segmented word set to be processed.

[0103] An embodiment of the present invention provides a new word discovery method, which first obtains a corpus to be processed, processes the corpus to be processed, and obtains a set of word segments to be processed; then, calculates a first parameter value of each first word segment in the set of word segments to be processed, the first parameter value including a first PMI value and a first left and right information entropy; finally, inputs each first word segment and its first parameter value into a pre-trained new word discovery model to obtain a first classification result, and then determines whether each first word segment is a new word based on the first classification result to obtain a new word set. The present invention uses a statistical method as a basis, and then combines the new word discovery model to identify new words, which can greatly improve the accuracy of the new word discovery results. In addition, the new word discovery model also improves the recall rate of new word recognition, and has stronger applicability.

[0104] Further, refer to Figure 3 , Figure 3 2 is a flow chart of the second embodiment of the new word discovery method of the present invention.

[0105] based on Figure 2 In the first embodiment shown, the new word discovery method further includes:

[0106] Step S60, obtaining training corpus;

[0107] In this embodiment, a training corpus is first obtained to train a new word discovery model.

[0108] Step S70, processing the training corpus to obtain a training word set;

[0109] Then, the training corpus is processed to obtain a training word segmentation set. Specifically, the punctuation marks of the training corpus are removed so that only the text part of the training corpus remains, which is convenient for subsequent word segmentation. Then, the training corpus with punctuation marks removed is subjected to word segmentation processing to obtain a word segmentation set to be processed. When performing word segmentation processing, an N-gram algorithm can be used for word segmentation, wherein, for the numerical range of N, the numerical range of N in the N-gram can be preliminarily estimated based on the previous understanding of the knowledge in the domain corpus, and then the character length within the N range is segmented.

[0110] Step S80, calculating a second parameter value of each second segmentation in the training segmentation set, where the second parameter value includes a second PMI value and a second left-right information entropy;

[0111] After obtaining the training word segmentation set, the parameter value of each word in the training word segmentation set (to distinguish it from the word segmentations of other word sets, it is recorded as the second word segmentation) is calculated (to distinguish it from the parameter value of the word segmentations of other word sets, it is recorded as the second parameter value), the second parameter value includes the PMI value of each second word segmentation (recorded as the second PMI value) and the left and right information entropy of each second word segmentation (recorded as the second left and right information entropy), the second left and right information entropy includes the second left neighboring word information entropy and the second right neighboring word information entropy, the second left neighboring word information entropy is the information entropy of the left neighboring word of the second word segmentation, and the second right neighboring word information entropy is the information entropy of the right neighboring word of the second word segmentation. The calculation method of the second parameter value is similar to the calculation method of the first parameter value, and reference can be made to the above-mentioned first embodiment, which will not be repeated here.

[0112] Step S90: Use the training word segmentation set and the second parameter value to train the preset new word discovery model to obtain the pre-trained new word discovery model.

[0113] Finally, the preset new word discovery model is trained using the training word set and the second parameter value to obtain a pre-trained new word discovery model. Preferably, the preset new word discovery model can adopt a binary classification Bert model.

[0114] Specifically, step S90 includes:

[0115] Step a91, preliminarily screening the second segmented words in the training segmented word set according to the second PMI value, the second left-right information entropy, and a preset threshold range to obtain a third segmented word;

[0116] Step a92: annotate each third participle, and construct a training sample based on the annotated third participle and its corresponding third PMI value and third left-right information entropy;

[0117] Step a93: training the preset new word discovery model using the training samples to obtain the pre-trained new word discovery model.

[0118] In this embodiment, the specific training process of the new word discovery model is as follows:

[0119] First, the second participles in the training word segmentation set are preliminarily screened according to the second PMI value, the second left and right information entropy, and the preset threshold range to obtain the third participles, so as to remove the participles in the training word segmentation set that are more likely not to be new words. The preset threshold range includes a preset PMI threshold range, a preset left neighbor information entropy range, and a preset right neighbor information entropy range. Specifically, the participles in the training word segmentation set whose second PMI value is within the preset PMI threshold range, whose second left neighbor information entropy is within the preset left neighbor information entropy range, and whose second right neighbor information entropy is within the preset right neighbor information entropy range can be screened out and recorded as the third participles.

[0120] Then, each third participle is marked, which can be marked manually. Specifically, the third participle can be sent to the work end for the staff to mark and then receive the marking results. The purpose of marking is to divide the positive samples and negative samples, that is, to mark whether the third participle is a new word or a non-new word. Then, based on the marked third participle and its corresponding PMI value (to distinguish it from the PMI value of the participles in other word sets, it is recorded as the third PMI value) and the third left and right information entropy (to distinguish it from the left and right information entropy of the participles in other word sets, it is recorded as the third left and right information entropy) to construct a training sample.

[0121] Finally, the preset new word discovery model is trained through the training samples to obtain a pre-trained new word discovery model. The specific training process can refer to the existing technology and will not be described here.

[0122] In this embodiment, a new word discovery model can be trained through the above method, so that new words can be mined based on the PMI value and left and right adjacent entropy obtained by statistical methods, thereby improving the accuracy of the new word discovery results.

[0123] Furthermore, based on the above second embodiment, a third embodiment of the new word discovery method of the present invention is proposed.

[0124] In this embodiment, after step S90, the new word discovery method further includes:

[0125] Step A: inputting each third participle and its corresponding third PMI value and third left-right information entropy into the pre-trained new word discovery model to obtain a second classification result;

[0126] Step B: comparing the second classification result with the third word segmentation marking result, and determining a new word screening rule based on the comparison result.

[0127] In this embodiment, each third participle and its corresponding third PMI value and the third left and right information entropy are input into a pre-trained new word discovery model to obtain a second classification result, that is, the third participle in the training sample is classified by the trained new word discovery model. The second classification result is then compared with the annotation result of the third participle, and the participles (recorded as the fourth participle) whose model classification result is different from the manual annotation result can be obtained by comparison, and then the new word screening rules are determined based on this part of the participle. Specifically, the fourth participle can be sent to the work end for the staff to conduct a qualitative analysis of the fourth participle, summarize the new word screening rules, and then return them to the server.

[0128] Furthermore, after the above step S50, the new word discovery method further includes:

[0129] Step C: screening the new word set again using the new word screening rule to obtain a final new word set.

[0130] After the new words in the corpus to be processed are initially mined through the new word discovery model, the new word set can be further screened again through the new word screening rules to obtain the final new word set, so as to further improve the accuracy of the new word discovery results.

[0131] The present invention also provides a new word discovery device.

[0132] Reference Figure 4 , Figure 4 Schematic diagram of the functional modules of the first embodiment of the new word discovery device of the present invention.

[0133] like Figure 4 As shown, the new word discovery device includes:

[0134] A first acquisition module 10 is used to acquire the corpus to be processed;

[0135] A first processing module 20 is used to process the corpus to be processed to obtain a word segmentation set to be processed;

[0136] A first calculation module 30 is used to calculate a first parameter value of each first segmentation in the segmentation set to be processed, wherein the first parameter value includes a first PMI value and a first left-right information entropy;

[0137] A first classification module 40 is configured to input each first segmented word and its first parameter value into a pre-trained new word discovery model to obtain a first classification result;

[0138] The new word determination module 50 is configured to determine whether each first segmented word is a new word according to the first classification result, so as to obtain a new word set.

[0139] Furthermore, the new word discovery device further includes:

[0140] The second acquisition module is used to obtain training corpus;

[0141] A second processing module is used to process the training corpus to obtain a training word set;

[0142] A second calculation module is used to calculate a second parameter value of each second segmentation in the training segmentation set, where the second parameter value includes a second PMI value and a second left-right information entropy;

[0143] The model training module is used to train the preset new word discovery model using the training word set and the second parameter value to obtain the pre-trained new word discovery model.

[0144] Furthermore, the model training module includes:

[0145] A word segmentation preliminary screening unit, configured to perform preliminary screening on the second word segmentation in the training word segmentation set according to the second PMI value, the second left-right information entropy, and a preset threshold range to obtain a third word segmentation;

[0146] A sample construction unit is used to mark each third participle and construct a training sample based on the marked third participle and its corresponding third PMI value and third left and right information entropy;

[0147] The model training unit is used to train the preset new word discovery model through the training samples to obtain the pre-trained new word discovery model.

[0148] Furthermore, the new word discovery device further includes:

[0149] A second classification module is used to input each third participle and its corresponding third PMI value and third left-right information entropy into the pre-trained new word discovery model to obtain a second classification result;

[0150] A rule determination module is used to compare the second classification result with the marking result of the third word segmentation, and determine a new word screening rule according to the comparison result.

[0151] Furthermore, the new word discovery device further includes:

[0152] The new word re-screening module is used to re-screen the new word set according to the new word screening rules to obtain a final new word set.

[0153] Furthermore, the first processing module 20 includes:

[0154] a removal unit, configured to remove punctuation marks from the corpus to be processed;

[0155] The word segmentation unit is used to perform word segmentation on the corpus to be processed from which punctuation marks have been removed, to obtain a word segmentation set to be processed.

[0156] Furthermore, the word segmentation unit is specifically used for:

[0157] The N-gram algorithm is used to perform word segmentation on the corpus to be processed from which punctuation marks have been removed, and a word segmentation set to be processed is obtained.

[0158] Among them, the functional implementation of each module in the above-mentioned new word discovery device corresponds to the various steps in the above-mentioned new word discovery method embodiment, and their functions and implementation processes will not be repeated here one by one.

[0159] The present invention also provides a computer-readable storage medium having a new word discovery program stored thereon. When the new word discovery program is executed by a processor, the steps of the new word discovery method as described in any one of the above embodiments are implemented.

[0160] The specific embodiments of the computer-readable storage medium of the present invention are basically the same as the embodiments of the above-mentioned new word discovery method, and will not be described in detail here.

[0161] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0162] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0163] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0164] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A new word discovery method, characterized in that: The new word discovery method comprises: Get the corpus to be processed; Processing the corpus to be processed to obtain a word segmentation set to be processed; Calculating a first parameter value of each first segmentation in the segmentation set to be processed, where the first parameter value includes a first PMI value and a first left-right information entropy; Inputting each first participle and its first parameter value into a pre-trained new word discovery model to obtain a first classification result; determining whether each first participle is a new word according to the first classification result to obtain a new word set; The new word discovery method further includes: Obtain training corpus; Processing the training corpus to obtain a training word set; Calculate a second parameter value of each second segmentation in the training segmentation set, where the second parameter value includes a second PMI value and a second left-right information entropy; Training a preset new word discovery model using the training word segmentation set and the second parameter value to obtain the pre-trained new word discovery model; The step of training a preset new word discovery model using the training word set and the second parameter value to obtain the pre-trained new word discovery model includes: Preliminarily screening the second participle in the training participle set according to the second PMI value, the second left-right information entropy and a preset threshold range to obtain a third participle; Mark each third participle, and construct a training sample based on the marked third participle and its corresponding third PMI value and third left and right information entropy; The preset new word discovery model is trained using the training samples to obtain the pre-trained new word discovery model.

2. The new word discovery method according to claim 1, wherein: After the step of training the preset new word discovery model using the training samples to obtain the pre-trained new word discovery model, the new word discovery method further comprises: Inputting each third participle and its corresponding third PMI value and third left-right information entropy into the pre-trained new word discovery model to obtain a second classification result; The second classification result is compared with the marking result of the third word segmentation, and a new word screening rule is determined according to the comparison result.

3. The new word discovery method according to claim 2, wherein: After the step of determining whether each first participle is a new word according to the first classification result to obtain a new word set, the new word discovery method further includes: The new word set is screened again using the new word screening rule to obtain a final new word set.

4. The new word discovery method according to any one of claims 1 to 3, characterized in that: The step of processing the corpus to be processed to obtain a word segment set to be processed comprises: Removing punctuation marks from the corpus to be processed; The corpus to be processed with punctuation removed is segmented to obtain a segmented word set to be processed.

5. The new word discovery method according to claim 4, wherein: The step of performing word segmentation on the corpus to be processed from which punctuation marks have been removed to obtain a word segmentation set to be processed comprises: The N-gram algorithm is used to perform word segmentation on the corpus to be processed from which punctuation marks have been removed, and a word segmentation set to be processed is obtained.

6. A new word discovery device, characterized in that: The new word discovery device comprises: The first acquisition module is used to acquire the corpus to be processed; A first processing module is used to process the corpus to be processed to obtain a word segmentation set to be processed; A first calculation module is used to calculate a first parameter value of each first segmentation in the segmentation set to be processed, where the first parameter value includes a first PMI value and a first left-right information entropy; A first classification module is used to input each first segmented word and its first parameter value into a pre-trained new word discovery model to obtain a first classification result; a new word determination module, configured to determine whether each first participle is a new word according to the first classification result, so as to obtain a new word set; The new word discovery device further comprises: The second acquisition module is used to obtain training corpus; A second processing module is used to process the training corpus to obtain a training word set; A second calculation module is used to calculate a second parameter value of each second segmentation in the training segmentation set, where the second parameter value includes a second PMI value and a second left-right information entropy; A model training module, configured to train a preset new word discovery model using the training word set and the second parameter value to obtain the pre-trained new word discovery model; The model training module includes A word segmentation preliminary screening unit, configured to perform preliminary screening on the second word segmentation in the training word segmentation set according to the second PMI value, the second left-right information entropy, and a preset threshold range to obtain a third word segmentation; A sample construction unit is used to mark each third participle and construct a training sample based on the marked third participle and its corresponding third PMI value and third left and right information entropy; The model training unit is used to train the preset new word discovery model through the training samples to obtain the pre-trained new word discovery model.

7. A new word discovery device, characterized in that: The new word discovery device includes: a memory, a processor, and a new word discovery program stored in the memory and executable on the processor. When the new word discovery program is executed by the processor, the steps of the new word discovery method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a new word discovery program, which, when executed by a processor, implements the steps of the new word discovery method according to any one of claims 1 to 5.

Images