Rapid unsupervised lexicon construction method, system, equipment and medium in government affair service field

Abstract

The invention discloses a rapid unsupervised lexicon construction method, system and device in the government affair service field and a medium, and the method comprises the steps: cleaning corpora in the government affair service field to obtain a corpus; counting word segments in the corpus; calculating mutual information entropies of all word segments, and adding the word segments with the mutual information entropies higher than a threshold value into the initial word bank; performing word segmentation on the corpus in the government affair service field in the corpus based on the initial lexicon, and storing segmented words in a dictionary; the dictionary is traversed, non-compliant words are deleted, and words with semantic value are reserved to form an unsupervised word bank. According to the method, the accuracy and efficiency of unsupervised word bank construction are improved, and the method has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method, system, device and medium for constructing a fast unsupervised vocabulary library in the field of government services. Background Art

[0002] In many fields of Chinese information processing, corresponding functions need to be completed based on dictionaries. For example, in intelligent retrieval systems or intelligent dialogue systems, through word segmentation, question retrieval, similarity matching, determining retrieval results or answers to intelligent dialogues, etc., each process is calculated based on words as the smallest unit, and the calculation is based on word dictionaries, so word dictionaries have a great impact on the performance of intelligent systems.

[0003] The field of government services needs to process a large amount of text data, which contains a lot of useful information. However, most of the collected original text data is colloquial and has a strong regional color, which may contain misunderstandings and distortions. Therefore, whether a special domain dictionary can be built and updated in a timely manner has a decisive impact on the system efficiency of the intelligent dialogue system in the field of government services.

[0004] At present, there are two main methods for building a vocabulary: Manual screening and extraction methods It is only applicable to fields with small data volumes and extremely high professional requirements, such as cutting-edge scientific research.

[0005] 2. Statistical-based methods According to whether there is labeled data, it can be divided into supervised methods and unsupervised methods.

[0006] (1) Supervised methods The key to building a vocabulary is the segmentation of words, so it can be regarded as a classification or sequence labeling problem. The specific approach is to use certain statistics of the annotated corpus based on the text fragment as features to train a binary classification model or to directly obtain words based on sequence information for sequence labeling, or to make judgments on the obtained words. Such methods can usually be implemented using machine learning algorithms such as HMM, CRF, and SVM. However, in practical applications, it is very difficult to obtain a large amount of annotated corpus data, which requires a lot of manpower, material resources and time. Since there is already manually annotated data, why bother to solve it by modeling? (2) Unsupervised methods It does not rely on any existing word library, word segmentation tools, or annotated corpus. It uses statistical strategies to extract all possible text fragments from a large-scale corpus based on the common characteristics of words, and then uses language knowledge to exclude "useless fragments" that are not candidate words or calculate relevance to find the combination of words with the highest relevance. Finally, the word combinations that meet the conditions are collected to obtain the required word library. However, this method may lose some important semantic information, resulting in generally low quality of the obtained word library. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a method, system, device and medium for constructing a fast unsupervised word library in the field of government services, which can save manpower and material resources to the maximum extent and shorten the construction cycle of the word library in the field of government services while ensuring reliable quality.

[0008] To achieve the above object, the present invention adopts the following technical solutions: A fast unsupervised word library construction method in the field of government services, comprising the following steps: Clean the corpus in the field of government services to obtain a corpus; Count word fragments in a corpus; Calculate the mutual information entropy of all word fragments, and add the word fragments with mutual information entropy higher than the threshold to the initial vocabulary; Based on the initial word library, the government service field corpus in the corpus is segmented, and the segmented words are saved in the dictionary; Traverse the dictionary, delete non-compliant words, and retain words with semantic value to form an unsupervised vocabulary.

[0009] To optimize the above technical solutions, the specific measures taken also include: Furthermore, the cleaning of the corpus in the field of government services includes: Remove irrelevant symbols, numbers and special characters, separate the cleaned corpus by spaces, and save it line by line in plain text format.

[0010] Furthermore, the mutual information entropy is calculated as follows: If the length of the word fragment is 2 characters, the mutual information entropy calculation method of the word fragment is: The word fragment is divided into two parts based on the character, and the first part s1 and the second part s2 are obtained. The mutual information entropy is calculated according to the following formula: MI(s1, s2) = P(s1,s2)·ln(P(s1,s2) / (P(s1)·P(s2))) Where MI(s1, s2) represents the mutual information entropy between the first part s1 and the second part s2, P(s1, s2) represents the frequency of the word fragment in all the texts in the corpus, P(s1) represents the frequency of the first part s1 in all the texts in the corpus, and P(s2) represents the frequency of the second part s2 in all the texts in the corpus; If the length of the word fragment is greater than 2 characters, the mutual information entropy of the word fragment is calculated as follows: The word fragment is divided into two parts based on characters. There are multiple ways of division. Each division method obtains a combination pair of the first part s1 and the second part s2. The mutual information entropy is calculated once for each combination pair, and the minimum value of the mutual information entropy among these combination pairs is taken as the mutual information entropy of the entire word fragment.

[0011] Furthermore, the step of adding the word fragments whose mutual information entropy is higher than the threshold to the initial word library is specifically as follows: Different thresholds are set for word fragments of different lengths and stored in the form of a dictionary, min_proba = {2: R2, 3: R3, ..., n: Rn}, where min_proba represents the mutual information entropy threshold dictionary, R2 represents the mutual information entropy threshold of a word fragment with a length of 2 characters, R3 represents the mutual information entropy threshold of a word fragment with a length of 3 characters, and Rn represents the mutual information entropy threshold of a word fragment with a length of n characters. According to the length of the word fragment, the corresponding threshold is found in the mutual information entropy threshold dictionary and compared, and the word fragments with mutual information entropy higher than the threshold are added to the initial vocabulary.

[0012] Furthermore, the word segmentation of the government service field corpus in the corpus based on the initial lexicon is specifically as follows: The initial vocabulary G is constructed into a Trie tree, and the Trie tree is used to segment the government service field corpus in the corpus. The rule of word segmentation is: as long as any word fragment appears in the initial vocabulary G, this fragment will not be segmented; the segmentation result is saved in the dictionary words, the key value of words is the word, and the value value is the word frequency.

[0013] Furthermore, the specific rules for deleting non-compliant words are: Check whether the length of each word in the dictionary words is less than or equal to the set number of characters in the longest word n. If so, check whether the word has the same word fragment in the initial vocabulary G. If so, keep it, otherwise delete it. If the length of the word is greater than the set number of characters in the longest word n, check whether each n-word fragment of the word is in the initial vocabulary G. If any fragment is not there, delete the word.

[0014] The present invention also proposes a fast unsupervised vocabulary building system in the field of government services, comprising: The data cleaning module is used to clean the corpus in the field of government services to obtain a corpus; Statistics module, used to count word fragments in the corpus; An initial vocabulary building module is used to calculate the mutual information entropy of all word fragments and add the word fragments whose mutual information entropy is higher than a threshold to the initial vocabulary; The word segmentation module is used to segment the government service field corpus in the corpus based on the initial word library, and the segmented words are stored in the dictionary; The filtering module is used to traverse the dictionary, delete non-compliant words, and retain words with semantic value to form an unsupervised vocabulary.

[0015] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for constructing a fast unsupervised vocabulary library in the field of government services as described above is implemented.

[0016] The present invention also proposes a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the method for constructing a fast unsupervised vocabulary in the field of government services as described above.

[0017] The beneficial effects of the present invention are: This paper achieves a balance between the speed and effect of unsupervised word library construction by introducing C++ library and backtracking operation. In the past, the word library constructed unsupervised by statistical methods was not very satisfactory. At first glance, the generated word library can scan many unreasonable places, and the words that really need to be used need to go through more manual screening. However, the word library generated at one time by the present invention only needs simple manual screening or even no manual screening at all to meet the use requirements.

[0018] This paper provides a new word discovery algorithm based on information entropy, which can better reflect the distribution and semantic value of words or phrases in the corpus. This algorithm can improve the accuracy and efficiency of unsupervised word library construction and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the unsupervised vocabulary construction method proposed in the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] Embodiment 1

[0022] The present invention proposes a method for constructing a fast unsupervised word library in the field of government services. The flowchart is as follows: Figure 1 As shown, the following steps are included: Cleaning of corpus in the field of government services, including: Remove irrelevant symbols, numbers and special characters, separate the cleaned corpus by spaces, and save them line by line in plain text format to obtain a corpus.

[0023] Count the word fragments in the corpus; call the kenlm library to count the frequency information of each pre-segmented word fragment and save it. The files generated at this time are all in binary format and need to be converted later. Load the word fragment (ngram) information counted by the kenlm library, convert the binary file format and load it into memory.

[0024] Calculate the mutual information entropy of all word fragments. Mutual information entropy is an important concept in information theory. It is a dimensionless statistic used to measure the information that a random variable X can provide about the change of another random variable Y. In this algorithm, it is used to characterize the closeness between words in a word combination fragment (that is, the previous ngram). The calculation method of mutual information entropy is: If the length of the word fragment is 2 characters, the mutual information entropy calculation method of the word fragment is: The word fragment is divided into two parts based on the character, and the first part s1 and the second part s2 are obtained. The mutual information entropy is calculated according to the following formula: MI(s1, s2) = P(s1,s2)·ln(P(s1,s2) / (P(s1)·P(s2))) Where MI(s1, s2) represents the mutual information entropy between the first part s1 and the second part s2, P(s1, s2) represents the frequency of the word fragment in all the texts in the corpus, P(s1) represents the frequency of the first part s1 in all the texts in the corpus, and P(s2) represents the frequency of the second part s2 in all the texts in the corpus; If the length of the word fragment is greater than 2 characters, the mutual information entropy of the word fragment is calculated as follows: Split the word fragment into two parts based on the character. There are many ways to split it. Each way of splitting will get a combination of the first part s1 and the second part s2. The mutual information entropy is calculated once for each combination. The minimum value of the mutual information entropy among these combinations is taken as the mutual information entropy of the entire word fragment. For example: the character combination 'abcd' can be split into ('a', 'bcd'), ('ab', 'cd'), ('abc', 'd'), then MI('abcd') = min{MI('a', 'bcd'), MI('ab', 'cd'), MI('abc', 'd')}.

[0025] The word fragments whose mutual information entropy is higher than the threshold are added to the initial vocabulary; specifically: Different thresholds are set for word fragments of different lengths and stored in the form of a dictionary, min_proba = {2: R2, 3: R3, ..., n: Rn}, where min_proba represents the mutual information entropy threshold dictionary, R2 represents the mutual information entropy threshold of a word fragment with a length of 2 characters, R3 represents the mutual information entropy threshold of a word fragment with a length of 3 characters, and Rn represents the mutual information entropy threshold of a word fragment with a length of n characters. According to the length of the word fragment, the corresponding threshold is found in the mutual information entropy threshold dictionary and compared, and the word fragments with mutual information entropy higher than the threshold are added to the initial vocabulary.

[0026] Based on the initial word library, the corpus in the government service field is segmented, and the segmented words are saved in the dictionary; specifically: The initial vocabulary G is constructed into a Trie tree, and the Trie tree is used to segment the government service field corpus in the corpus. The rule of word segmentation is: as long as any word fragment appears in the initial vocabulary G, this fragment will not be segmented; for example, G = ['we', 'often', 'intentional', 'opinion', 'difference'], the sentence to be segmented is sentence = 'We often have differences of opinion', and the result after segmentation is result = ['we', 'often', 'have opinions', 'differences']. It can be seen that although the word 'have opinions' does not appear in G, since both the words 'intentional' and 'opinion' appear, the segmentation will not be separated from the two words 'intentional' or 'opinion'.

[0027] The segmented results are saved in the dictionary words, where the key value of words is the word and the value is the word frequency.

[0028] Traverse the dictionary and delete the non-compliant words. The specific rules for deleting non-compliant words are: Check whether the length of each word in the dictionary words is less than or equal to the number of characters n of the longest word set. If so, check whether there is the same word segment in the initial word bank G. If there is, keep it; if not, delete it. If the length of the word is greater than the number of characters n of the longest word set, check whether each n-character segment of the word is in the initial word bank G. As long as one segment is not in it, delete the word.

[0029] According to some semantic rules or domain knowledge, filter out common words and words with obvious "non-word" features, and finally screen out candidate words with certain semantic value. Keep the words with semantic value to form an unsupervised word bank.

[0030] Embodiment 2

[0031] A method for constructing a fast unsupervised word bank in the field of government affairs services includes the following steps: S11: Preprocess the original corpus. For example, if the corpus is corpus = "There is construction going on inside the Yujiatou Campus of Wuhan University of Technology on Youyi Avenue, Wuchang District, and the noise disturbs the residents. Please ask the functional department to verify and handle it.", since the subsequent kenlm library processing requires a space-segmented and plain-text format corpus as input, the result after text cleaning is text = "Wuchang District Youyi Avenue Wuhan University of Technology Yujiatou Campus There is construction going on Noise disturbs the residents Please ask the functional department to verify and handle it" S21: Call the count_ngrams program of kenlm to count ngrams and put its count_ngrams in a directory where the project files are located. Before use, the count_ngrams needs to be given executable permission S22: Select the ngram length to be 4 and load the statistically ngram information into memory. The data format of the ngram segment is as follows: {'Wu': 10, 'Wuhan': 6, 'Wuhan University of Technology': 1, 'Wuhan University of Technology': 4, 'Han': 13, 'Han University of Technology': 1, 'Han University of Technology': 2, 'University of Technology': 20, 'University of Technology': 8, 'University': 25} S31: Calculate mutual information entropy. In the present invention, the internal mutual information entropy of multiple words (ngrams) is considered at the same time. This definition requires enumerating all possible cutting methods, because a word should be very "tight" everywhere. The mutual information entropy of strings of 4 or more words is also defined similarly. Take the ngram fragment in the previous step, such as a certain fragment ngram0='Wuhan University of Technology', split it into n1=('Wuhan', 'Han University of Technology'), n2= ('Wuhan', 'Ligong'), n3=('Wuhan University of Technology', 'Ligong'), take min{MI(n1), MI(n2), MI(n3)} as the mutual information entropy of the fragment 'Wuhan University of Technology' S32: Filter ngrams based on mutual information entropy. Different ngram mutual information thresholds are different. [0, 2, 4, 6] is the mutual information threshold in this example. The first 0 is meaningless and is only used for padding. 2, 4, and 6 are the mutual information entropy thresholds for 2grams, 3grams, and 4grams, respectively. Add ngrams with mutual information entropy higher than the threshold to the set G. S41: Use the ngrams in the set G to build a Trie tree, and then use this Trie tree to do a basic "pre-segmentation" and count the frequency. G = ['each item', 'project', 'at the same time', 'forward', 'advance'] (the list is used for convenience of explanation, and the data structure of G in the experiment is a Trie tree) Split the text s = 'each item and advance forward at the same time', and the result is: each item / at the same time / forward / advance, and save the segmented result in the dictionary words, the key value of words is the word, and the value value is the word frequency, Words = {'each item': 2, 'at the same time': 2, 'forward': 5, 'advance'} S51, the purpose of backtracking is to exclude possible loose words by comparing with ngrams. After S41, text fragments such as 'each project' will be cut out. Similar examples include 'support', 'ball players', 'Zhuhai Port' and many other examples. However, these cases have a very low degree of solidification in 3grams, so we need a "backtracking" process. After obtaining the word list wotds in the above steps, filter the word list again. The filtering rule is that if the n-word inside is not in the original high-solidity ngrams, then it must be deleted.

[0032] S61. Iterate the ngrams in G until all ngrams have been backtracked. The ngrams that are finally retained are the candidate words for building the unsupervised vocabulary. S71. Based on some semantic rules or domain knowledge, common words and words with obvious "non-word" features are filtered out, and finally candidate words with certain semantic value are selected.

[0033] Examples of filtered candidate words: 'looks like', 'name Li', '12 years'.

[0034] Final vocabulary example: 2gram:|judgment|clouds|factors|occupy the road|primary|ism|for the people|initial|come again|party committee|overtime|fairy|ride|said again|medical science 3gram: Do nucleic acid test|District government|Property certificate|Especially|Department of Human Resources and Social Security|Secondary close contact|Family|Management|Inaction|Vaccination site 4gram:|Fever clinic|Personal account|Construction unit|Outlet Mansion|Negative nucleic acid|Test report|Face recognition|Related units|Express delivery company|Duplicate work order

[0035] 2gram represents a word with 2 characters, 3gram represents a word with 3 characters, and 4gram represents a word with 4 characters.

[0036] Embodiment 3

[0037] The present invention proposes a fast unsupervised word library construction system in the field of government services corresponding to the method of embodiment 1, comprising: The data cleaning module is used to clean the corpus in the field of government services to obtain a corpus; Statistics module, used to count word fragments in the corpus; An initial vocabulary building module is used to calculate the mutual information entropy of all word fragments and add the word fragments whose mutual information entropy is higher than a threshold to the initial vocabulary; The word segmentation module is used to segment the government service field corpus in the corpus based on the initial word library, and the segmented words are stored in the dictionary; The filtering module is used to traverse the dictionary, delete non-compliant words, and retain words with semantic value to form an unsupervised vocabulary.

[0038] The implementation methods of each module and module function in the system are completely consistent with the steps of the method in Example 1, so they will not be repeated here.

[0039] Embodiment 4

[0040] The present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for constructing a fast unsupervised vocabulary in the field of government services as described in Example 1 is implemented.

[0041] Embodiment 5

[0042] The present invention proposes a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the method for constructing a fast unsupervised vocabulary in the field of government services as described in the first embodiment.

[0043] In the embodiments disclosed in the present application, the computer storage medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of computer storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0044] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0045] The above are only preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A fast unsupervised word library construction method in the field of government services, characterized in that: The following steps are involved: Clean the corpus in the field of government services to obtain a corpus; Count word fragments in a corpus; Calculate the mutual information entropy of all word fragments, and add the word fragments with mutual information entropy higher than the threshold to the initial vocabulary; Based on the initial word library, the government service field corpus in the corpus is segmented, and the segmented words are saved in the dictionary; Traverse the dictionary, delete non-compliant words, and retain words with semantic value to form an unsupervised vocabulary.

2. The method for constructing a fast unsupervised word library in the field of government services as claimed in claim 1, characterized in that: The cleaning of the corpus in the field of government services includes: Remove irrelevant symbols, numbers and special characters, separate the cleaned corpus by spaces, and save it line by line in plain text format.

3. The method for constructing a fast unsupervised word library in the field of government services as claimed in claim 1, characterized in that: The mutual information entropy is calculated as follows: If the length of the word fragment is 2 characters, the mutual information entropy calculation method of the word fragment is: The word fragment is divided into two parts based on the character, and the first part s1 and the second part s2 are obtained. The mutual information entropy is calculated according to the following formula: MI(s1, s2) = P(s1,s2)·ln(P(s1,s2) / (P(s1)·P(s2))) Where MI(s1, s2) represents the mutual information entropy between the first part s1 and the second part s2, P(s1, s2) represents the frequency of the word fragment in all the texts in the corpus, P(s1) represents the frequency of the first part s1 in all the texts in the corpus, and P(s2) represents the frequency of the second part s2 in all the texts in the corpus; If the length of the word fragment is greater than 2 characters, the mutual information entropy of the word fragment is calculated as follows: The word fragment is divided into two parts based on characters. There are multiple ways of division. Each division method obtains a combination pair of the first part s1 and the second part s2. The mutual information entropy is calculated once for each combination pair, and the minimum value of the mutual information entropy among these combination pairs is taken as the mutual information entropy of the entire word fragment.

4. The method for constructing a fast unsupervised word library in the field of government services as claimed in claim 1, characterized in that: The step of adding word fragments whose mutual information entropy is higher than a threshold to the initial word library is specifically as follows: Different thresholds are set for word fragments of different lengths and stored in the form of a dictionary, min_proba = {2: R2, 3: R3, ..., n: Rn}, where min_proba represents the mutual information entropy threshold dictionary, R2 represents the mutual information entropy threshold of a word fragment with a length of 2 characters, R3 represents the mutual information entropy threshold of a word fragment with a length of 3 characters, and Rn represents the mutual information entropy threshold of a word fragment with a length of n characters. According to the length of the word fragment, the corresponding threshold is found in the mutual information entropy threshold dictionary and compared, and the word fragments with mutual information entropy higher than the threshold are added to the initial vocabulary.

5. The method for constructing a fast unsupervised word library in the field of government services as claimed in claim 1, characterized in that: The word segmentation of the government service field corpus in the corpus based on the initial lexicon is specifically as follows: The initial vocabulary G is constructed into a Trie tree, and the Trie tree is used to segment the government service field corpus in the corpus. The rule of word segmentation is: as long as any word fragment appears in the initial vocabulary G, this fragment will not be segmented; the segmentation result is saved in the dictionary words, the key value of words is the word, and the value value is the word frequency.

6. The method for constructing a fast unsupervised word library in the field of government services as claimed in claim 1, characterized in that: The specific rules for deleting non-compliant words are: Check whether the length of each word in the dictionary words is less than or equal to the set number of characters in the longest word n. If so, check whether the word has the same word fragment in the initial vocabulary G. If so, keep it, otherwise delete it. If the length of the word is greater than the set number of characters in the longest word n, check whether each n-word fragment of the word is in the initial vocabulary G. If any fragment is not there, delete the word.

7. A fast unsupervised vocabulary building system in the field of government services, characterized in that: include: The data cleaning module is used to clean the corpus in the field of government services to obtain a corpus; Statistics module, used to count word fragments in the corpus; An initial vocabulary building module is used to calculate the mutual information entropy of all word fragments and add the word fragments whose mutual information entropy is higher than a threshold to the initial vocabulary; The word segmentation module is used to segment the government service field corpus in the corpus based on the initial word library, and the segmented words are stored in the dictionary; The filtering module is used to traverse the dictionary, delete non-compliant words, and retain words with semantic value to form an unsupervised vocabulary.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for constructing a fast unsupervised vocabulary in the field of government services as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to execute the method for constructing a fast unsupervised vocabulary in the field of government services as described in any one of claims 1 to 6.

Images