Distribution network maintenance text information extraction method based on improved Jieba word segmentation

By using the TF-IDF algorithm to identify proprietary vocabulary in the power field and improve the lexicon in Jieba word segmentation tool, the problem of Jieba's word segmentation accuracy in the power field is solved, and effective word segmentation and higher number of texts for distribution network patrol recording Word participle accuracy.

CN113988066BActive Publication Date: 2024-11-05ELECTRIC POWER SCI & RES INST OF STATE GRID TIANJIN ELECTRIC POWER CO +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111108711.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-22
Publication Date
2024-11-05
Estimated Expiration
2041-09-22

AI Technical Summary

Technical Problem

The existing Chinese word segmentation tool Jieba is difficult to accurately segment word segmentation in the application of the power field, resulting in the split or merge of keywords, affecting the accuracy of text preprocessing and cannot be effectively used for intelligent analysis.

Method used

By analyzing the text characteristics of the power grid inspection, using the TF-IDF algorithm to identify the proprietary vocabulary in the power field, improving the Jieba vocabulary library, combining Jieba word segmentation tools to form a keyword list of the power inspection text, and finding the maximum probability through dynamic planning path to improve word segmentation accuracy.

Benefits of technology

It realizes effective word segmentation for the text recorded by the distribution network patrol inspection, improves word segmentation accuracy and text characteristics retention rate, enhances the ability to recognize power vocabulary, and ensures higher word segmentation accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113988066B_ABST
    Figure CN113988066B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for extracting information from distribution network maintenance text based on improved Jieba word segmentation, comprising the following steps: step 1, analyzing distribution network inspection text features; step 2, cleaning distribution network inspection text; step 3, forming a power inspection text keyword table; step 4, using the TF-IDF algorithm to improve the power inspection text keyword table formed in step 3; step 5, loading the improved power inspection text keyword table obtained by using the TF-IDF algorithm in step 4 into the Jieba dictionary, completing the update of the keyword table and the stop word table in the Jieba dictionary. The present invention can use the TF-IDF algorithm to identify proprietary vocabulary in the power field according to the characteristics of the distribution network maintenance record text data, improve the Jieba word library, and then realize the effective word segmentation of the distribution network inspection record text.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of application of natural language processing in power system, and relates to a method for extracting distribution network operation and maintenance text information, in particular to a method for extracting distribution network maintenance text information based on improved Jieba word segmentation. Background Art

[0002] With the rapid development of informatization, the amount of data accumulated by various businesses of power grid enterprises is increasing. In the maintenance link of the distribution network, a large amount of raw data is accumulated and saved in the form of text, including maintenance records, test records, inspection records and other information, which contains rich power information and is of great significance for guiding operation and maintenance work. Text processing is the basis of text analysis and understanding, and the processing results can directly affect the accuracy of text analysis. Words are the basic units that constitute sentences. Before analyzing sentences, it is necessary to segment words and segment the words in the text as feature values. For English, word segmentation is relatively simple because there are obvious separators between words in English sentences. Chinese sentences have complex structures, there are no separators between words, and the same word has different meanings in different sentences and different texts. The sentence structure of the distribution network inspection text is complex, and there are no obvious separation marks between words. Word segmentation is required to understand the sentence. Word segmentation is the basis of text processing, and the accuracy of word segmentation affects the text processing results. However, due to the complexity of semi-structured data in Chinese text, a large amount of power text cannot be directly used in the learning process of intelligent analysis.

[0003] Jieba Chinese word segmentation is a widely used word segmentation tool, a third-party word segmentation library based on Python. Due to the specificity of the power field, the results obtained by directly using Jieba's power word segmentation cannot meet the needs of power text segmentation. It is very easy to split keywords or merge some words, which affects the accuracy of text preprocessing.

[0004] Therefore, it is very important and urgent to improve the Jieba word segmentation tool based on the characteristics of power text and realize the research on information mining technology of distribution network inspection record text. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a distribution network maintenance text information extraction method based on improved Jieba word segmentation. According to the characteristics of the distribution network maintenance record text data, the TF-IDF algorithm is used to identify proprietary vocabulary in the power field, improve the Jieba word library, and then realize the effective word segmentation of the distribution network inspection record text.

[0006] The present invention solves the practical problem by adopting the following technical solutions:

[0007] A method for extracting text information from distribution network maintenance based on improved Jieba word segmentation includes the following steps:

[0008] Step 1: Analyze the characteristics of distribution network inspection text;

[0009] Step 2: Clean the distribution network inspection text according to the distribution network inspection text characteristics of step 1;

[0010] Step 3: Based on the distribution network inspection text after data cleaning in step 2, use the Jieba word segmentation tool to perform word segmentation, count the word frequencies and form a keyword table for the power inspection text.

[0011] Step 4: For the distribution network inspection text, the TF-IDF algorithm is used to improve the power inspection text keyword table formed in step 3;

[0012] Step 5: Load the improved power inspection text keyword table obtained by using the TF-IDF algorithm in step 4 into the Jieba dictionary to complete the update of the keyword table and the stop word table in the Jieba dictionary.

[0013] Moreover, the distribution network inspection text features of step 1 include:

[0014] (1) The inspection text structure is inconsistent;

[0015] (2) The length of inspection texts varies;

[0016] (3) The inspection text contains characters with special meanings;

[0017] (4) Some of the inspection texts contain a high degree of vocabulary repetition.

[0018] Moreover, the specific steps of step 2 include:

[0019] (1) Delete duplicate records to reduce the differences caused by different writing habits of different personnel;

[0020] (2) Delete blank records to reduce the invalid length of some texts;

[0021] (3) Delete punctuation marks contained in the text;

[0022] (4) Delete meaningless words;

[0023] Moreover, the specific steps of step 3 include:

[0024] (1) Through Jieba's one-time word segmentation, sample part of the original text and its word segmentation data, and observe the recognition and word segmentation results;

[0025] (2) Compare the original text, extract the misclassified words, and construct a keyword list;

[0026] (3) Perform numerical statistics on the segmented text;

[0027] (4) Finally, the high-frequency words obtained by statistics are added to the keyword table, and the keyword table of power inspection text is established through Jieba word segmentation.

[0028] Moreover, the specific steps of step 4 include:

[0029] (1) Calculate TF

[0030] TF represents the probability of a certain power word appearing in the entire article, and its calculation formula is:

[0031]

[0032] Among them, n ij is the number of times the distribution network inspection feature words appear in the text, ∑ k n kj is the number of all feature words in the text; TF ij The result of the calculation is the frequency of a certain power characteristic word;

[0033] (2) Calculate IDF

[0034] IDF stands for inverse text frequency, and its calculation formula is:

[0035]

[0036] Where N represents the total number of articles in the corpus, 1+N power Indicates the number of words with obvious characteristics in the article. In order to prevent the word from not existing in the corpus, the denominator is 0, and 1+N is used. power As the denominator, IDF ij Represents the inverse text frequency.

[0037] (3) Calculate TF-IDF

[0038] The calculation formula is:

[0039] TF-IDF ij =TF ij IDF ij (3)

[0040] The value of TF-IDF is composed of the word frequency TF ij and inverse text word frequency IDF ij The larger the TF-IDF value is, the more important the power feature word is to the text.

[0041] (4) After using the TF-IDF algorithm to extract keywords, a complete keyword table for power inspection text is formed, and the Jieba vocabulary is improved to enhance the word segmentation ability of the Jieba dictionary in the power field.

[0042] Moreover, the specific steps of step 5 include:

[0043] (1) Jieba word segmentation implements efficient word graph scanning based on prefix dictionaries, generating a directed acyclic graph consisting of all possible word formation situations of Chinese characters in a sentence;

[0044] (2) Dynamic programming is used to find the maximum probability path between the corpus and the keyword table, and to find the maximum segmentation combination based on word frequency.

[0045] (3) Jieba is used again to perform word segmentation through the updated vocabulary to ensure that the words in the vocabulary can be correctly classified, thereby improving the recognition and segmentation accuracy of power vocabulary.

[0046] Moreover, after step 5, the method further includes the following steps:

[0047] After completing the update of the keyword table and stop word list in step 5, use Jieba to perform word segmentation again, add the keyword table to ensure that the words in the word list can be correctly classified, and continue to iterate and update the keyword table and stop word list 2 to 3 times.

[0048] Advantages and beneficial effects of the present invention:

[0049] Based on the Jieba word segmentation tool, the present invention studies the text preprocessing technology of distribution network maintenance records, proposes to use the TF-IDF algorithm to identify words in the specific field of electricity, and adds the identified power words into the Jieba word library, improves the Jieba word segmentation tool, uses the stop word list and keyword list of the field, and more important words are retained. When constructing text features, more features are retained compared with direct word segmentation, and effective word segmentation of distribution network inspection records is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a process flow chart of the present invention. DETAILED DESCRIPTION

[0051] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0052] A distribution network maintenance text information extraction method based on improved Jieba segmentation, such as Figure 1 As shown, the following steps are included:

[0053] Step 1: Analyze the characteristics of distribution network inspection text;

[0054] The distribution network inspection text features of step 1 include:

[0055] (1) Inspection text structure is inconsistent

[0056] The contents of the distribution network inspection record text generally include the information of the writer and the power company, relevant contents of the station house, maintenance personnel information and suggested countermeasures, etc., and due to the different habits of the writers, the title, paragraph format and text structure of the text are inconsistent.

[0057] (2) Inspection text length varies

[0058] Due to the different complexities of various hidden dangers and maintenance methods, as well as the differences in the degree of detail of the records kept by the writers, the lengths of individual fault texts vary greatly.

[0059] (3) The inspection text contains characters with special meanings.

[0060] Most inspection text sentences are interspersed with characters, quantitative units and numbers with special meanings, such as "SO2 mass fraction is 1.41×10-6, H2S mass fraction is 0.5×10-6". These fields containing quantitative information and type information are very meaningful, but the information they contain is easily lost during text processing.

[0061] (4) Some of the inspection texts contain a high degree of vocabulary repetition.

[0062] The inspection text is basically described by recording information such as time, location, equipment object, and equipment status.

[0063] In this embodiment, compared with general Chinese text, the distribution network inspection record text is oriented to power equipment and environment, recording equipment information and environmental conditions during the inspection process, and has characteristics different from general text, so the record text needs to be analyzed.

[0064] Step 2: Clean the distribution network inspection text according to the distribution network inspection text characteristics of step 1;

[0065] The specific steps of step 2 include:

[0066] (1) Delete duplicate records to reduce the differences caused by different writing habits of different personnel;

[0067] (2) Delete blank records to reduce the invalid length of some texts;

[0068] (3) Delete punctuation marks contained in the text;

[0069] (4) Delete meaningless words, such as prepositions, modal particles, etc. These words have no practical significance for understanding the text and should be removed.

[0070] In this embodiment, before processing the distribution network inspection record text, some preprocessing is required for the corpus text, including deleting meaningless records such as duplicate records and blank records. The text contains punctuation marks, prepositions, modal particles, etc. These words have no practical significance for understanding the text and should be removed from the word segmentation results. These words are called stop words. This improves the efficiency and accuracy of understanding the sentence and processing the text.

[0071] Step 3: Based on the distribution network inspection text after data cleaning in step 2, use the Jieba word segmentation tool to perform word segmentation, count the word frequencies and form a keyword table for the power inspection text.

[0072] The specific steps of step 3 include:

[0073] (1) Through Jieba's one-time word segmentation, sample part of the original text and its word segmentation data, and observe the recognition and word segmentation results;

[0074] Jieba word segmentation tool is based on prefix dictionary dict.txt. For new words that cannot be segmented in the application, you can modify or add dictionaries. The format of custom dictionary is the same as dict.txt, one word occupies one line; each line is divided into three parts: word, word frequency (optional), part of speech (optional), separated by spaces. When word frequency is omitted, automatic calculation can ensure the word frequency of the word. Use Jieba to specify the Wikipedia general field dictionary to segment words.

[0075] (2) Compare the original text, extract the misclassified words, and construct a keyword list;

[0076] Although Jieba has the ability to recognize new words and misclassified words, extracting misclassified words by comparison can ensure a higher accuracy rate.

[0077] (3) Perform numerical statistics on the segmented text;

[0078] Since power data has domain characteristics, extra attention needs to be paid to high-frequency words.

[0079] (4) Finally, the high-frequency words obtained by statistics are added to the keyword table, and the keyword table of power inspection text is established through Jieba word segmentation.

[0080] In this embodiment, the Jieba word segmentation tool is based on the prefix dictionary dict.txt. New words that cannot be segmented in the application need to be modified or added to the dictionary. At the same time, users can also customize the dictionary. Developers can specify a custom dictionary to segment the unregistered words in the Jieba dictionary. Although Jieba has the ability to recognize new words, adding new words by yourself can ensure a higher accuracy rate. The format of the custom dictionary is the same as dict.txt, with one word occupying one line; each line is divided into three parts: words, word frequency (optional), and part of speech (optional), separated by spaces. When the word frequency is omitted, automatic calculation can be used to ensure the word frequency of the word.

[0081] After a word segmentation, Jieba samples part of the original text and its word segmentation data, observes and recognizes the word segmentation results, compares them with the original text, extracts the wrongly segmented words, and constructs a keyword table. Then, the word segmentation text is numerically counted. Since power data has domain characteristics, high-frequency words need to be identified and added to the keyword table.

[0082] After Jieba performs word segmentation once, the present invention samples part of the original text and its word segmentation data, and observes and recognizes the word segmentation results. The original text is compared, and the wrongly segmented words are extracted to construct a keyword table. Then, numerical statistics are performed on the word segmentation text. Since power data has domain characteristics, high-frequency words require extra attention. Finally, high-frequency words are added to the keyword table.

[0083] Step 4: For the distribution network inspection text, the TF-IDF algorithm is used to improve the power inspection text keyword table formed in step 3;

[0084] The specific steps of step 4 include:

[0085] (1) Calculate TF

[0086] TF represents the probability of a certain power word appearing in the entire article, and its calculation formula is:

[0087]

[0088] Among them, n ij is the number of times the distribution network inspection feature words appear in the text, ∑ k n kj is the number of all feature words in the text. TF ij The result of the calculation is the word frequency of a certain power characteristic word.

[0089] (2) Calculate IDF

[0090] IDF stands for Inverse Text Frequency, which is used to measure the frequency of a certain power word in the entire corpus. It is used to remove some frequently encountered but unimportant words in the document while retaining some words with obvious characteristics. The calculation formula is:

[0091]

[0092] Where N represents the total number of articles in the corpus, 1+N power Indicates the number of words with obvious characteristics in the article. In order to prevent the word from not existing in the corpus, the denominator is 0, and 1+N is used. power As the denominator, IDF ij Represents the inverse text frequency.

[0093] (3) Calculate TF-IDF

[0094] The main purpose of the TF-IDF algorithm is that if a word or phrase appears very frequently in the inspection text, but very rarely in other texts, it means that the word or phrase has obvious characteristics, and the word or phrase can be used as a power inspection keyword. The calculation formula is:

[0095] TF-IDF ij =TF ij IDF ij (3)

[0096] The value of TF-IDF is composed of the word frequency TF ij and inverse text word frequency IDF ij The larger the TF-IDF value is, the more important the power feature word is to the text.

[0097] (4) After using the TF-IDF algorithm to extract keywords, a complete keyword table for power inspection text is formed, and the Jieba vocabulary is improved to enhance the word segmentation ability of the Jieba dictionary in the power field.

[0098] In this embodiment, the TF-IDF algorithm is used to improve the vocabulary established in step 3. TF-IDF is a commonly used weighting technology for information retrieval and text mining. TF-IDF is a statistical method used to evaluate the importance of a word to a document set or one of the documents in a corpus. The importance of a word increases in direct proportion to the number of times it appears in a document, but at the same time it decreases in inverse proportion to the frequency of its appearance in the corpus. TF-IDF can be used to extract keywords, improve the Jieba vocabulary, and improve the accuracy of the system's word segmentation.

[0099] Step 5: Load the improved power inspection text keyword table obtained by using the TF-IDF algorithm in step 4 into the Jieba dictionary to complete the update of the keyword table and the stop word table in the Jieba dictionary;

[0100] The specific steps of step 5 include:

[0101] (1) Jieba word segmentation implements efficient word graph scanning based on prefix dictionaries, generating a directed acyclic graph consisting of all possible word formation situations of Chinese characters in a sentence;

[0102] (2) Dynamic programming is used to find the maximum probability path between the corpus and the keyword table, and to find the maximum segmentation combination based on word frequency.

[0103] (3) Jieba is used again to perform word segmentation through the updated vocabulary to ensure that the words in the vocabulary can be correctly classified, thereby improving the recognition and segmentation accuracy of power vocabulary.

[0104] In this embodiment, after extracting keywords from the power inspection text using the TF-IDF algorithm, the keyword table is loaded into the Jieba dictionary to improve the word segmentation ability of the Jieba dictionary in the power field. Jieba word segmentation based on the improved dictionary has a better word segmentation effect. Jieba word segmentation first implements efficient word graph scanning based on the prefix dictionary, and then generates a directed acyclic graph consisting of all possible word formation situations of Chinese characters in the sentence, and uses dynamic programming to find the maximum probability path to find the maximum segmentation combination based on word frequency. Through the updated word list, the recognition and segmentation accuracy of power vocabulary is improved.

[0105] Step 6: After completing the update of the keyword table and stop word list in step 5, use Jieba to perform word segmentation again, add the keyword table to ensure that the words in the word list can be correctly classified, and continuously iterate to update the keyword table and stop word list 2 to 3 times.

[0106] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation manner. Any other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the scope of protection of the present invention.

Claims

1. A method for extracting text information from distribution network maintenance based on improved Jieba word segmentation, characterized in that: The following steps are involved: Step 1: Analyze the characteristics of distribution network inspection text; Step 2: Clean the distribution network inspection text according to the distribution network inspection text characteristics of step 1; Step 3: Based on the distribution network inspection text after data cleaning in step 2, use Jieba word segmentation tool to perform word segmentation, count the word frequency and form a power inspection text keyword table; Step 4: For the distribution network inspection text, the TF-IDF algorithm is used to improve the power inspection text keyword table formed in step 3; Step 5: Load the improved power inspection text keyword table obtained by using the TF-IDF algorithm in step 4 into the Jieba dictionary to complete the update of the keyword table and the stop word table in the Jieba dictionary; The distribution network inspection text features of step 1 include: (1) The inspection text structure is inconsistent; (2) The length of inspection texts varies; (3) The inspection text contains characters with special meanings; (4) Some of the inspection texts contain a high degree of vocabulary repetition. The specific steps of step 2 include: (1) Delete duplicate records to reduce the differences caused by different writing habits of different personnel; (2) Delete blank records to reduce the invalid length of some texts; (3) Delete punctuation marks contained in the text; (4) Delete meaningless words; The specific steps of step 3 include: (1) Through Jieba's one-time word segmentation, sample part of the original text and its word segmentation data, and observe the recognition and word segmentation results; (2) Compare the original text, extract the misclassified words, and construct a keyword list; (3) Perform numerical statistics on the segmented text; (4) Finally, the high-frequency words obtained by statistics are added to the keyword table, and the keyword table of power inspection text is established through Jieba word segmentation; The specific steps of step 4 include: (1) Calculate TF TF represents the probability of a certain power word appearing in the entire article, and its calculation formula is: Among them, n ij is the number of times the distribution network inspection feature words appear in the text, ∑ k n kj is the number of all feature words in the text; TF ij The result of the calculation is the frequency of a certain power characteristic word; (2) Calculate IDF IDF stands for inverse text frequency, and its calculation formula is: Where N represents the total number of articles in the corpus, 1+N power Indicates the number of words with obvious characteristics in the article. In order to prevent the word from not existing in the corpus, the denominator is 0, and 1+N is used. power As the denominator, IDF ij represents the inverse text frequency; (3) Calculate TF-IDF The calculation formula is: TF-IDF ij =TF ij ·IDF ij (3) The value of TF-IDF is composed of the word frequency TF ij and inverse text word frequency IDF ij The larger the TF-IDF value is, the more important the power feature word is to the text. (4) After using the TF-IDF algorithm to extract keywords, a complete keyword table for power inspection texts is formed, and the Jieba dictionary is improved to enhance the word segmentation capability of the Jieba dictionary in the power field; The specific steps of step 5 include: (1) Jieba word segmentation implements efficient word graph scanning based on prefix dictionaries, generating a directed acyclic graph consisting of all possible word formation situations of Chinese characters in a sentence; (2) Dynamic programming is used to find the maximum probability path between the corpus and the keyword table, and to find the maximum segmentation combination based on word frequency; (3) Jieba is used again to perform word segmentation through the updated vocabulary to ensure that the words in the vocabulary can be correctly classified, thereby improving the recognition and segmentation accuracy of power vocabulary.

2. The method for extracting text information of distribution network maintenance based on improved Jieba word segmentation according to claim 1 is characterized in that: After step 5, the following steps are also included: After completing the update of the keyword table and stop word list in step 5, use Jieba to perform word segmentation again, add the keyword table to ensure that the words in the word list can be correctly classified, and continue to iterate and update the keyword table and stop word list 2 to 3 times.