A method for constructing a professional vocabulary for NLP word segmentation in the coal industry

Abstract

The present invention proposes a method for constructing a professional lexicon for NLP word segmentation in the coal industry, which relates to the technical field of lexicon construction. The method includes: extracting coal industry keywords from coal journal data, performing a first frequency ranking on the extracted multiple coal industry word segments, and constructing a first lexicon and a second lexicon based on the first frequency ranking; identifying coal industry standard terms using text recognition technology to construct a third lexicon; expanding the second and third lexicons based on a deep learning model to obtain an initial lexicon; performing a second frequency ranking on the initial lexicon, and constructing a fourth lexicon based on the second frequency ranking; and performing repeated segmentation screening on the fourth lexicon and the first lexicon to form a fifth lexicon to obtain a coal industry word segmentation lexicon. This improves the quality of coal industry NLP word segmentation results and enhances the accuracy of large-scale data processing using large models in the coal industry.

Claims

1. A method for constructing a professional vocabulary for NLP word segmentation in the coal industry, characterized in that: The method comprises: Extract coal industry keywords from coal journal data to obtain multiple coal industry segmentations; Performing a first frequency sorting on the multiple coal industry segmentations to select a first vocabulary constructed by the target coal industry segmentations that are ranked first by frequency, and a second vocabulary constructed by the remaining coal industry segmentations except the target coal industry segmentations; Use text recognition technology to identify coal industry standard terms to obtain coal mining industry standard word segmentation to build a third word library; Based on the deep learning model, learning data features of the second vocabulary and the third vocabulary, and expanding the second vocabulary and the third vocabulary according to the data features to obtain a new initial vocabulary; Perform a second word frequency sorting in the initial word library, and select the backup coal industry segmentation words with the highest frequency sorting to construct the fourth word library; The fourth word library and the first word library are combined into a fifth word library, and repeated word segmentation screening is performed on the fifth word library to obtain a coal industry word segmentation word library; The coal journal data includes structured data, semi-structured data and unstructured data. Coal industry keywords are extracted from the coal journal data to obtain multiple coal industry segmentations, including: We use the label index in the two-dimensional table data structure DataFrame to access the keyword columns corresponding to the coal journal data, complete keyword extraction from the structured data, and obtain structured first-category industry segmentation. The semi-structured data corresponding to the coal journal data is parsed into structured data through a parser to extract keywords from the structured data and obtain the second category of industry segmentation corresponding to the semi-structured data; Using string matching algorithms, we locate keyword rows for the directly editable and non-directly editable categories in the unstructured data to extract keywords from the unstructured data and obtain the third and fourth industry segmentation words. The first category industry segmentation words, the second category industry segmentation words, the third category industry segmentation words, and the fourth category industry segmentation words are collectively regarded as multiple coal industry segmentation words; The string matching algorithm is used to locate the keyword rows of the directly editable and non-directly editable categories in the unstructured data respectively to complete the keyword extraction of the unstructured data, and obtain the third category industry segmentation and the fourth category industry segmentation, including: Use string matching algorithms to locate keywords in the directly editable categories in unstructured data to extract keywords from the directly editable categories and obtain the third category of industry segmentation. Performing data preprocessing on the non-directly editable class in the unstructured data to obtain an initialized non-directly editable class, wherein the data preprocessing includes noise reduction, binarization processing, and tilt correction; Performing text detection, recognition, correction and editing on the initialized non-directly editable classes in order to obtain usable industry texts; The available industry texts are respectively located in keyword rows by a string matching algorithm to complete the extraction of keywords that cannot be directly edited, and obtain the fourth category of industry segmentation.

2. The method according to claim 1, characterized in that The first frequency sorting of the plurality of coal industry segmentations is performed to select the first vocabulary constructed by the target coal industry segmentations ranked first by frequency, and the second vocabulary constructed by the remaining coal industry segmentations except the target coal industry segmentations in the coal industry segmentations, including: Based on the hash table, the tool class is used to perform a first frequency sorting on the multiple coal industry segmentations, and the frequency is sorted from high to low, so as to select the first vocabulary constructed by the target coal industry segmentations with the top x frequencies, and the second vocabulary constructed by the remaining coal industry segmentations in the coal industry segmentations except the target coal industry segmentations; Here, x is determined by the number of coal industry word segmentation extractions and the application feedback results of the first and second word libraries in the coal industry NLP word segmentation.

3. The method according to claim 1, characterized in that The second frequency sorting is performed in the initial vocabulary, and the backup coal industry segmentation words with the highest frequency sorting y are selected to construct the fourth vocabulary, including: The TF-IDF algorithm is used to perform a second word frequency sorting on the initial word library to obtain the TF-IDF value of each coal industry word segmentation; Based on the descending order of TF-IDF values, select the top y backup coal industry segmentations in frequency order to construct the fourth vocabulary; Among them, y is determined by the application feedback results of the backup coal industry word segmentation in the coal industry NLP word segmentation.

4. A device for constructing a professional vocabulary for NLP word segmentation in the coal industry, used to execute the method for constructing a professional vocabulary for NLP word segmentation in the coal industry according to claim 1, characterized in that: The device comprises: The extraction module is used to extract coal industry keywords from coal journal data to obtain multiple coal industry segmentations; A first sorting module is configured to perform a first frequency sorting on the plurality of coal industry segmentations to select a first vocabulary constructed by the target coal industry segmentations that are ranked first by frequency, and a second vocabulary constructed by the remaining coal industry segmentations except the target coal industry segmentations; The recognition module is used to identify coal industry standard terms through text recognition technology to obtain coal industry standard category segmentation to build a third vocabulary; an expansion module, configured to learn data features of the second and third lexicons based on a deep learning model, and to expand the second and third lexicons according to the data features to obtain a new initial lexicon; The second sorting module is used to perform a second word frequency sorting in the initial word library, and select the backup coal industry segmentation words with the highest frequency sorting to construct a fourth word library; The screening module is used to combine the fourth vocabulary and the first vocabulary into a fifth vocabulary, and perform repeated word segmentation screening on the fifth vocabulary to obtain a coal industry word segmentation vocabulary.

5. The device according to claim 4, characterized in that The extraction module comprises: The first extraction unit is used to access the keyword column corresponding to the coal journal data using the label index through the two-dimensional table data structure DataFrame, complete the keyword extraction of the structured data, and obtain the structured first-category industry segmentation; The second extraction unit is used to parse the semi-structured data corresponding to the coal journal data into structured data through a parser, so as to extract keywords from the structured data and obtain the second type of industry segmentation words corresponding to the semi-structured data; A positioning unit is used to locate keyword rows for directly editable and non-directly editable categories in the unstructured data using a string matching algorithm, so as to complete keyword extraction from the unstructured data and obtain third-category industry segmentation words and fourth-category industry segmentation words; The generating unit is used to combine the first-category industry segmentation words, the second-category industry segmentation words, the third-category industry segmentation words, and the fourth-category industry segmentation words as a plurality of coal industry segmentation words.

6. The device according to claim 5, characterized in that The positioning unit is specifically used to: Use string matching algorithms to locate keywords in the directly editable categories in unstructured data to extract keywords from the directly editable categories and obtain the third category of industry segmentation. Performing data preprocessing on the non-directly editable class in the unstructured data to obtain an initialized non-directly editable class, wherein the data preprocessing includes noise reduction, binarization processing, and tilt correction; Performing text detection, recognition, correction and editing on the initialized non-directly editable classes in order to obtain usable industry texts; The available industry texts are respectively located in keyword rows by a string matching algorithm to complete the extraction of keywords that cannot be directly edited, and obtain the fourth category of industry segmentation.

7. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-3.

Images