Professional lexicon construction method for NLP word segmentation in coal industry

Abstract

The invention provides a professional lexicon construction method for coal industry NLP word segmentation, and relates to the technical field of lexicon construction, and the method comprises the steps: carrying out coal industry keyword extraction on coal periodical data, carrying out first word frequency sorting on a plurality of extracted coal industry segmented words, and carrying out second word frequency sorting on a plurality of extracted coal industry segmented words; constructing a first word bank and a second word bank according to the first word frequency sorting; coal industry standard terms are recognized through a character recognition technology to construct a third word library; based on a deep learning model, expanding the second lexicon and the third lexicon to obtain an initial lexicon; performing second word frequency sorting in the initial word bank, and constructing a fourth word bank according to the second word frequency sorting; and performing repeated word segmentation screening on a fifth word bank formed by the fourth word bank and the first word bank to obtain the coal industry word segmentation word bank. Therefore, the quality of the NLP word segmentation result in the coal industry is improved, and the accuracy of processing large-scale data by a large model in the coal industry is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of word library construction, and in particular to a method, device, electronic equipment and storage medium for constructing a professional word library for NLP word segmentation in the coal industry. Background Art

[0002] The coal industry is a field with a long history and complex technology. Over a long period of time, coal mining technology has been continuously developing, accumulating a large amount of professional corpus in the coal industry and precipitating rich professional knowledge. In related technologies, the use of these professional corpus and professional knowledge mainly stays at the expert experience stage, which is difficult to promote on a large scale, and the knowledge value is limited. It is urgent to extract, integrate and utilize the knowledge of the coal industry, so a professional vocabulary construction method for NLP word segmentation in the coal industry is urgently needed. Summary of the invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0004] To this end, the first purpose of the present invention is to propose a professional vocabulary construction method for NLP word segmentation in the coal industry, which improves the quality of NLP word segmentation results in the coal industry and enhances the accuracy of large-scale data processing by large models in the coal industry.

[0005] The second object of the present invention is to propose a professional vocabulary building device for NLP word segmentation in the coal industry.

[0006] A third objective of the present invention is to provide an electronic device.

[0007] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium storing computer instructions.

[0008] To achieve the above purpose, the first embodiment of the present invention proposes a method for constructing a professional vocabulary for NLP word segmentation in the coal industry, the method comprising:

[0009] Extract coal industry keywords from coal journal data to obtain multiple coal industry segmentations;

[0010] Performing a first frequency sorting on the plurality of coal industry segmentations to select a first vocabulary constructed by the target coal industry segmentations with the top x frequencies, and a second vocabulary constructed by the remaining coal industry segmentations except the target coal industry segmentations;

[0011] The coal industry standard terms are identified through text recognition technology to obtain coal mining industry standard word segmentation to construct the third word library;

[0012] Based on the deep learning model, learning data features of the second vocabulary and the third vocabulary, so as to expand the second vocabulary and the third vocabulary according to the data features to obtain a new initial vocabulary;

[0013] Perform a second frequency sorting in the initial word library, and select the spare coal industry segmentation words with the highest frequency sorting to construct the fourth word library;

[0014] The fourth word library and the first word library are combined into a fifth word library, and the fifth word library is repeatedly segmented and screened to obtain a coal industry word segmentation word library.

[0015] To achieve the above purpose, the second embodiment of the present invention proposes a professional vocabulary building device for NLP word segmentation in the coal industry, the device comprising:

[0016] The extraction module is used to extract coal industry keywords from coal journal data to obtain multiple coal industry segmentation words;

[0017] A first sorting module is used to perform a first word frequency sorting on the plurality of coal industry segmentations, so as to select a first word library constructed by the target coal industry segmentations with the top x frequencies sorted, and a second word library constructed by the remaining coal industry segmentations except the target coal industry segmentations in the coal industry segmentations;

[0018] The recognition module is used to recognize coal industry standard terms through text recognition technology to obtain coal industry standard word segmentation to construct a third word library;

[0019] An expansion module, used for learning data features of the second vocabulary and the third vocabulary based on a deep learning model, so as to expand the second vocabulary and the third vocabulary according to the data features to obtain a new initial vocabulary;

[0020] The second sorting module is used to perform a second word frequency sorting in the initial word library, and select the spare coal industry segmentation words before y in frequency sorting to construct a fourth word library;

[0021] The screening module is used to combine the fourth word library and the first word library into a fifth word library, and perform repeated word segmentation screening on the fifth word library to obtain a coal industry word segmentation word library.

[0022] To achieve the above-mentioned purpose, the third aspect of the present invention proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0023] In order to achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0024] The method, device, electronic device and storage medium for constructing a professional word library for NLP word segmentation in the coal industry of the embodiment of the present invention extract coal industry keywords from coal journal data to perform a first word frequency sorting on the extracted multiple coal industry word segments, and construct a first word library and a second word library based on the first word frequency sorting; identify coal industry standard terms through text recognition technology to construct a third word library; expand the second word library and the third word library based on a deep learning model to obtain an initial word library; perform a second word frequency sorting in the initial word library, and construct a fourth word library based on the second word frequency sorting; perform repeated word segmentation screening on the fifth word library composed of the fourth word library and the first word library to obtain a coal industry word segmentation word library. As a result, the quality of the coal industry NLP word segmentation results is improved, and the accuracy of the large-scale data processing of the coal industry large model is enhanced.

[0025] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A flow chart of a method for constructing a professional word library for NLP word segmentation in the coal industry provided by an embodiment of the present invention;

[0028] Figure 2 A flowchart of a method for constructing a professional word library for NLP word segmentation in the coal industry provided by an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of the structure of a professional vocabulary building device for NLP word segmentation in the coal industry provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0031] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of national laws and regulations.

[0032] The following describes a method and device for constructing a professional vocabulary for NLP word segmentation in the coal industry according to an embodiment of the present invention with reference to the accompanying drawings.

[0033] Figure 1 A flowchart of a method for constructing a professional vocabulary for NLP word segmentation in the coal industry provided by an embodiment of the present invention.

[0034] like Figure 1 As shown, the method comprises the following steps:

[0035] Step 101, extract coal industry keywords from coal journal data to obtain multiple coal industry segmentations.

[0036] In some embodiments, coal journal data includes structured data, semi-structured data and unstructured data. One implementation method of extracting coal industry keywords from coal journal data to obtain multiple coal industry segmentation words can be to use a two-dimensional table data structure DataFrame to use a label index to access the keyword column corresponding to the coal journal data to complete the keyword extraction of structured data to obtain structured first-class industry segmentation words; parse the semi-structured data corresponding to the coal journal data into structured data through a parser to extract keywords from the structured data to obtain the second-class industry segmentation words corresponding to the semi-structured data; locate the keyword rows of the directly editable and non-directly editable classes in the unstructured data through a string matching algorithm to complete the keyword extraction of the unstructured data to obtain the third-class industry segmentation words and the fourth-class industry segmentation words; and use the first-class industry segmentation words, the second-class industry segmentation words, the third-class industry segmentation words and the fourth-class industry segmentation words together as multiple coal industry segmentation words. Therefore, the data involved comes from coal journal data, which greatly improves the standardization and reliability of the word library.

[0037] Optionally, the Document Object Model (DOM), event-based XML parser (simple API for XML, SAX), lightweight data exchange format (JavaScript Object Notation, JSON), and comma-separated values ​​(CSV) can be used as parsers to parse the semi-structured data corresponding to the coal journal data into structured data, so as to extract keywords from the structured data and obtain the second type of industry segmentation corresponding to the semi-structured data.

[0038] Furthermore, a string matching algorithm is used to locate keyword rows for the directly editable class and the non-directly editable class in the unstructured data respectively to complete keyword extraction of the unstructured data, and obtain the third category of industry segmentation and the fourth category of industry segmentation. An implementation method can be to locate keyword rows for the directly editable class in the unstructured data by a string matching algorithm to complete keyword extraction of the directly editable class and obtain the third category of industry segmentation; perform data preprocessing on the non-directly editable class in the unstructured data to obtain an initialized non-directly editable class, wherein the data preprocessing includes noise reduction, binarization processing and tilt correction; perform text detection and recognition and correction editing on the initialized non-directly editable class in turn to obtain usable industry text; and locate keyword rows for the available industry text by a string matching algorithm to complete keyword extraction of the non-directly editable class and obtain the fourth category of industry segmentation.

[0039] Optionally, the unstructured data is mainly text type, and the non-directly editable type may be an image type.

[0040] Specifically, for the directly editable class, the string matching algorithm may be Rabin-Karp, and then the Rabin-Karp algorithm is used to locate the keyword row of the directly editable class to complete the keyword extraction of the directly editable class and obtain the third category industry segmentation.

[0041] For the non-directly editable class, denoising is performed by removing random noise from the image in the image class by using median filtering, Gaussian filtering, and wavelet transform methods respectively, and taking the denoised image with the best denoising effect; binarization processing is performed by using the histogram method to determine the binarization threshold, and binarizing the denoised image to obtain a binary image; tilt correction is performed by performing tilt correction on the binarized image through Hough transform to obtain the initialization non-directly editable class (initialization image) constructed by the target image.

[0042] Among them, text detection and recognition uses a text recognition algorithm (Convolutional Recurrent Neural Network, CRNN) to perform text detection, and completes text segmentation and character recognition through vertical projection and recurrent neural network (Recurrent Neural Network, RNN) to obtain text character text.

[0043] Corrective editing is to use a language model (N-gram) to correct and edit the extracted character text to obtain usable industry text, thereby improving the quality and usability of industry text.

[0044] Optionally, the Rabin-Karp algorithm can be used to quickly locate the keyword lines, and constraints can be added using regularization formulas to complete the extraction of coal industry keywords in the area between keywords and graph classification numbers for coal periodicals.

[0045] Step 102: Perform the first word frequency sorting on multiple coal industry word segments to select the first word library constructed from the target coal industry word segments ranked in the top x in terms of frequency, and the second word library constructed from the remaining coal industry word segments except the target coal industry word segments among the coal industry word segments.

[0046] In some embodiments, an implementation manner of performing the first word frequency sorting on multiple coal industry word segments to select the first word library constructed from the target coal industry word segments ranked in the top x in terms of frequency, and the second word library constructed from the remaining coal industry word segments except the target coal industry word segments among the coal industry word segments can be as follows: Based on a hash table, use a tool class to perform the first word frequency sorting on multiple coal industry word segments, sorting the word frequencies from high to low, to select the first word library (N1) constructed from the target coal industry word segments ranked in the top x in terms of frequency, and the second word library (N2) constructed from the remaining coal industry word segments except the target coal industry word segments among the coal industry word segments; where x is determined by the quantity of coal industry word segment extraction and the application feedback results of the first word library and the second word library in coal industry NLP word segmentation.

[0047] Optionally, 0 < x < 1, and the selection of x should make the coal industry NLP word segmentation have strong robustness and generalization.

[0048] Step 103: Identify the coal industry standard terms through text recognition technology to obtain the coal mine industry standard type word segments and construct the third word library.

[0049] Among them, the coal industry standard terms can be the term content in relevant coal mine science and technology term documents to establish an electronic document of coal industry standard terms.

[0050] Among them, the term content in relevant coal mine science and technology term documents can include relevant standard terms for roadway engineering, underground mining, open-pit mining, hoisting and transportation, mine surveying, coal mine safety, and coal mine electricity.

[0051] In some embodiments, optical character recognition (OCR) is used to identify relevant standard terms for shaft and tunnel engineering, underground mining, open-pit mining, hoisting and transportation, mine surveying, coal mine safety, and coal mine electricity, and obtain coal mine industry standard segmentation, and check and filter the incorrectly formatted segmentation, repeated segmentation, and incorrectly recognized segmentation of the coal mine industry standard segmentation in the electronic document one by one, so as to sort out a high-quality third word library (N3) and complete the extraction of coal mine industry standard segmentation. As a result, the data involved comes from the coal industry standard terminology, which greatly improves the standardization and reliability of the word library.

[0052] Step 104 , based on the deep learning model, learn the data features of the second vocabulary and the third vocabulary, so as to expand the second vocabulary and the third vocabulary according to the data features to obtain a new initial vocabulary.

[0053] Among them, the deep learning model can be a pre-trained language model (Generative Pre-trained Transformer, GPT) based on the Transformer structure in Transformer, but is not limited to this.

[0054] Therefore, based on the GPT model in Transformer, a new vocabulary (N2+N3) is expanded and generated in the process of learning the data features of the second and third vocabulary using self-supervised learning technology. The GPT model can optimize the generation effect by continuously adjusting its parameters, complete the vocabulary update iteration, and obtain a new initial vocabulary (N0).

[0055] Step 105, performing a second frequency sorting in the initial word library, selecting y spare coal industry segment words before the frequency sorting, to construct a fourth word library.

[0056] In some embodiments, a second frequency sorting is performed in the initial vocabulary, and y backup coal industry segmentations are selected before the frequency sorting to construct a fourth vocabulary. One implementation method can be to use the TF-IDF algorithm to perform a second frequency sorting on the initial vocabulary to obtain the TF-IDF value of each coal industry segmentation; based on the TF-IDF value sorting from large to small, select y backup coal industry segmentations before the frequency sorting to construct the fourth vocabulary; wherein y is determined by the application feedback result of the backup coal industry segmentation in the coal industry NLP segmentation. Therefore, the GPT model in the transformer is introduced to perform self-supervised learning of data features, so that the vocabulary has the necessary update and iteration capabilities, and can always meet the needs of subsequent NLP segmentation of new knowledge texts in the coal industry.

[0057] Optionally, before performing the second word frequency sorting on the initial word library using the TF-IDF algorithm, it is also necessary to select an open corpus with a relatively large scale, high diversity, and good balance for the second word frequency sorting. Among them, the open corpus can be a Chinese dataset (Microsoft Research Asia, MSRA), but is not limited to this, and this embodiment does not make specific limitations on this.

[0058] Furthermore, term frequency–inverse document frequency (TF-IDF) is a statistical method used for information retrieval weighting. When the TF-IDF algorithm performs the second word frequency sorting on the initial word library N0, in the first step, calculate the term frequency (TF). Suppose a certain word segmentation appears n times in the documents of the open corpus, and the documents of the open corpus contain a total of N words, then the calculation method of the TF value is as shown in Equation (1):

[0059]

[0060] Among them, t represents a specific word segmentation; d represents the documents of the open corpus.

[0061] In the second step, calculate the inverse document frequency (IDF). The calculation method of the IDF value is as shown in Equation (2):

[0062]

[0063] Among them, D is the total number of documents in the open corpus; df t is the number of documents containing the word segmentation t. By taking the logarithm, the problem of excessive numerical values can be avoided, and at the same time, the monotonically decreasing characteristic of the IDF is ensured.

[0064] In the third step, calculate the TF-IDF value. The calculation method of the TF-IDF value is as shown in Equation (3):

[0065] TF-IDF = TF * IDF

[0066] In the fourth step, sort the TF-IDF values of each word segmentation in the initial word library from largest to smallest, and select the top y spare coal industry word segmentations to construct the fourth word library (N4), where 0 < y ≤ 1, and y should be selected as the maximum value on the basis of ensuring that the coal industry NLP word segmentation has strong generalization ability, which is determined according to the application feedback results of the spare coal industry word segmentations in the coal industry NLP word segmentation.

[0067] Step 106: Combine the fourth word library and the first word library to form the fifth word library, and perform repeated word segmentation screening on the fifth word library to obtain the coal industry word segmentation library.

[0068] The method for constructing a professional word library for NLP word segmentation in the coal industry of the embodiment of the present invention extracts coal industry keywords from coal journal data to perform a first word frequency sorting on the extracted multiple coal industry word segments, and constructs a first word library and a second word library based on the first word frequency sorting; recognizes coal industry standard terms through text recognition technology to construct a third word library; expands the second word library and the third word library based on a deep learning model to obtain an initial word library; performs a second word frequency sorting in the initial word library, and constructs a fourth word library based on the second word frequency sorting; and performs repeated word segmentation screening on the fifth word library composed of the fourth word library and the first word library to obtain a coal industry word segmentation word library. Thus, the quality of the coal industry NLP word segmentation results is improved, and the accuracy of the large-scale data processing of the coal industry large model is enhanced.

[0069] In addition, in some embodiments, the present invention also provides an implementation flow chart of a method for constructing a professional word library for NLP word segmentation in the coal industry, such as Figure 2 As shown, coal industry keywords are extracted from coal journal data to obtain multiple coal industry segmentations; multiple coal industry segmentations are sorted by frequency for the first time; the first vocabulary (N1) is constructed by selecting the target coal industry segmentations with the frequency ranking before x, and the second vocabulary (N2) is constructed by the remaining coal industry segmentations with the frequency ranking (1-x); the coal industry standard terms are identified to obtain coal mine industry standard segmentations to construct the third vocabulary (N3); the second vocabulary (N2) and the third vocabulary (N3) are expanded, and updated and iterated to obtain a new initial vocabulary (N0); based on the open corpus, the second frequency ranking is performed in the initial vocabulary, and the backup coal industry segmentations with the frequency ranking before y are selected to construct the fourth vocabulary (N4); the fourth vocabulary and the first vocabulary are combined to form the fifth vocabulary (N5), and the fifth vocabulary (N5) is screened for repeated segmentations to obtain the coal industry segmentation vocabulary (N6). Therefore, the professional vocabulary construction method for NLP word segmentation in the coal industry constructed by the present invention has wide applicability, and provides support for the promotion of large-model technology and digital transformation in the coal industry and even other vertical industries.

[0070] In order to implement the above embodiment, the present invention also proposes a professional vocabulary building device for NLP word segmentation in the coal industry.

[0071] Figure 3 A schematic diagram of the structure of a professional vocabulary building device for NLP word segmentation in the coal industry provided by an embodiment of the present invention.

[0072] like Figure 3 As shown, the professional vocabulary building device 30 for NLP word segmentation in the coal industry includes: an extraction module 31, a first sorting module 32, a recognition module 33, an expansion module 34, a second sorting module 35 and a screening module 36.

[0073] An extraction module 31 is used to extract coal industry keywords from coal journal data to obtain multiple coal industry segmentation words;

[0074] A first sorting module 32 is used to perform a first frequency sorting on the plurality of coal industry segmentations, so as to select a first vocabulary constructed by the target coal industry segmentations with the top x frequencies sorted, and a second vocabulary constructed by the remaining coal industry segmentations except the target coal industry segmentations in the coal industry segmentations;

[0075] The recognition module 33 is used to recognize coal industry standard terms through text recognition technology to obtain coal industry standard word segmentation to construct a third word library;

[0076] An expansion module 34, configured to learn data features of the second vocabulary and the third vocabulary based on a deep learning model, so as to expand the second vocabulary and the third vocabulary according to the data features to obtain a new initial vocabulary;

[0077] The second sorting module 35 is used to perform a second word frequency sorting in the initial word library, and select the spare coal industry segmentation words before the frequency sorting y to construct a fourth word library;

[0078] The screening module 36 is used to combine the fourth word library and the first word library into a fifth word library, and perform repeated word segmentation screening on the fifth word library to obtain a coal industry word segmentation word library.

[0079] Furthermore, in a possible implementation of the embodiment of the present invention, the extraction module 31 includes:

[0080] The first extraction unit is used to access the keyword column corresponding to the coal journal data by using the label index through the two-dimensional table data structure DataFrame, complete the keyword extraction of the structured data, and obtain the structured first category industry segmentation;

[0081] The second extraction unit is used to parse the semi-structured data corresponding to the coal journal data into structured data through a parser, so as to extract keywords from the structured data and obtain the second type of industry segmentation words corresponding to the semi-structured data;

[0082] A positioning unit is used to locate the keyword rows of the directly editable category and the non-directly editable category in the unstructured data by using a string matching algorithm, so as to complete the keyword extraction of the unstructured data and obtain the third category industry segmentation and the fourth category industry segmentation;

[0083] The generating unit is used to take the first-category industry segmentation words, the second-category industry segmentation words, the third-category industry segmentation words, and the fourth-category industry segmentation words together as a plurality of coal industry segmentation words.

[0084] Furthermore, in a possible implementation of the embodiment of the present invention, the positioning unit is specifically configured to:

[0085] The keyword rows of the directly editable category in the unstructured data are located by using a string matching algorithm to complete the keyword extraction of the directly editable category and obtain the third category of industry segmentation;

[0086] Performing data preprocessing on the non-directly editable class in the unstructured data to obtain an initialized non-directly editable class, wherein the data preprocessing includes noise reduction, binarization processing and tilt correction;

[0087] Performing text detection, recognition and correction editing on the initialized non-directly editable categories in turn to obtain usable industry texts;

[0088] The available industry texts are respectively located for keyword rows through a string matching algorithm to complete the extraction of keywords that cannot be directly edited, and obtain the fourth category of industry segmentation.

[0089] Further, in a possible implementation of the embodiment of the present invention, the first sorting module 32 is specifically configured to:

[0090] Based on the hash table, the tool class is used to perform a first frequency sorting of the multiple coal industry segmentations, and the frequency is sorted from high to low, so as to select the first word library constructed by the target coal industry segmentations with the top x frequencies in the frequency sorting, and the second word library constructed by the remaining coal industry segmentations except the target coal industry segmentations in the coal industry segmentations;

[0091] Among them, x is determined by the number of coal industry word segmentation extractions and the application feedback results of the first vocabulary and the second vocabulary in the coal industry NLP word segmentation.

[0092] Further, in a possible implementation of the embodiment of the present invention, the second sorting module 35 is specifically configured to:

[0093] The TF-IDF algorithm is used to sort the initial word library for the second time to obtain the TF-IDF value of each coal industry word segment;

[0094] Based on the descending order of TF-IDF values, select the first y backup coal industry segmentations in frequency order to construct the fourth word library;

[0095] Among them, y is determined by the application feedback results of the backup coal industry word segmentation in the coal industry NLP word segmentation.

[0096] The professional word library construction device for NLP word segmentation of the coal industry of the embodiment of the present invention extracts coal industry keywords from coal journal data to perform a first word frequency sorting on the extracted multiple coal industry word segments, and constructs a first word library and a second word library according to the first word frequency sorting; recognizes coal industry standard terms through text recognition technology to construct a third word library; expands the second word library and the third word library based on a deep learning model to obtain an initial word library; performs a second word frequency sorting in the initial word library, and constructs a fourth word library according to the second word frequency sorting; and performs repeated word segmentation screening on the fifth word library composed of the fourth word library and the first word library to obtain a coal industry word segmentation word library. Thus, the quality of the coal industry NLP word segmentation results is improved, and the accuracy of the large-scale data processing of the coal industry large model is enhanced.

[0097] In order to implement the above embodiment, the present invention further provides an electronic device, including:

[0098] at least one processor; and

[0099] a memory communicatively connected to the at least one processor; wherein,

[0100] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the aforementioned method.

[0101] In order to implement the above embodiment, the present invention further proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the above method.

[0102] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0103] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0104] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0105] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0106] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0107] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0108] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0109] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for constructing a professional word library for NLP word segmentation in the coal industry, characterized in that: The method comprises: Extract coal industry keywords from coal journal data to obtain multiple coal industry segmentations; Performing a first frequency sorting on the plurality of coal industry segmentations to select a first vocabulary constructed by the target coal industry segmentations with the top x frequencies, and a second vocabulary constructed by the remaining coal industry segmentations except the target coal industry segmentations; The coal industry standard terms are identified through text recognition technology to obtain coal mining industry standard word segmentation to construct the third word library; Based on the deep learning model, learning data features of the second vocabulary and the third vocabulary, so as to expand the second vocabulary and the third vocabulary according to the data features to obtain a new initial vocabulary; Perform a second frequency sorting in the initial word library, and select the spare coal industry segmentation words with the highest frequency sorting to construct the fourth word library; The fourth word library and the first word library are combined into a fifth word library, and the fifth word library is repeatedly segmented and screened to obtain a coal industry word segmentation word library.

2. The method according to claim 1, characterized in that: The coal journal data includes structured data, semi-structured data and unstructured data. Coal industry keywords are extracted from the coal journal data to obtain multiple coal industry segmentations, including: The two-dimensional table data structure DataFrame is used to access the keyword column corresponding to the coal journal data using the label index to complete the keyword extraction of structured data to obtain the structured first-category industry segmentation; The semi-structured data corresponding to the coal journal data is parsed into structured data by a parser to extract keywords from the structured data and obtain the second category of industry segmentation corresponding to the semi-structured data; The keyword rows of the directly editable and non-directly editable categories in the unstructured data are located by using a string matching algorithm to complete keyword extraction of the unstructured data and obtain the third-category industry segmentation and the fourth-category industry segmentation. The first-category industry segmentation words, the second-category industry segmentation words, the third-category industry segmentation words, and the fourth-category industry segmentation words are collectively regarded as multiple coal industry segmentation words.

3. The method according to claim 2, characterized in that The keyword rows of the directly editable and non-directly editable categories in the unstructured data are respectively located by the string matching algorithm to complete the keyword extraction of the unstructured data, and obtain the third category industry segmentation and the fourth category industry segmentation, including: The keyword rows of the directly editable category in the unstructured data are located by using a string matching algorithm to complete the keyword extraction of the directly editable category and obtain the third category of industry segmentation; Performing data preprocessing on the non-directly editable class in the unstructured data to obtain an initialized non-directly editable class, wherein the data preprocessing includes noise reduction, binarization processing and tilt correction; Performing text detection, recognition and correction editing on the initialized non-directly editable categories in turn to obtain usable industry texts; The available industry texts are respectively located for keyword rows through a string matching algorithm to complete the extraction of keywords that cannot be directly edited, and obtain the fourth category of industry segmentation.

4. The method according to claim 1, characterized in that The first frequency sorting of the plurality of coal industry segmentations is performed to select the first word library constructed by the target coal industry segmentations with the top x frequency sorting, and the second word library constructed by the remaining coal industry segmentations except the target coal industry segmentations in the coal industry segmentations, including: Based on the hash table, the tool class is used to perform a first frequency sorting of the multiple coal industry segmentations, and the frequency is sorted from high to low, so as to select the first word library constructed by the target coal industry segmentations with the top x frequencies in the frequency sorting, and the second word library constructed by the remaining coal industry segmentations except the target coal industry segmentations in the coal industry segmentations; Among them, x is determined by the number of coal industry word segmentation extractions and the application feedback results of the first vocabulary and the second vocabulary in the coal industry NLP word segmentation.

5. The method according to claim 1, characterized in that The second frequency sorting is performed in the initial word library, and the spare coal industry segmentation words before the frequency sorting are selected to construct the fourth word library, including: The TF-IDF algorithm is used to sort the initial word library for the second time to obtain the TF-IDF value of each coal industry word segment; Based on the descending order of TF-IDF values, select the first y backup coal industry segmentations in frequency order to construct the fourth word library; Among them, y is determined by the application feedback results of the backup coal industry word segmentation in the coal industry NLP word segmentation.

6. A professional word library construction device for NLP word segmentation in the coal industry, characterized in that: The device comprises: The extraction module is used to extract coal industry keywords from coal journal data to obtain multiple coal industry segmentation words; A first sorting module is used to perform a first word frequency sorting on the plurality of coal industry segmentations, so as to select a first word library constructed by the target coal industry segmentations with the top x frequencies sorted, and a second word library constructed by the remaining coal industry segmentations except the target coal industry segmentations in the coal industry segmentations; The recognition module is used to recognize coal industry standard terms through text recognition technology to obtain coal industry standard word segmentation to construct a third word library; An expansion module, used for learning data features of the second vocabulary and the third vocabulary based on a deep learning model, so as to expand the second vocabulary and the third vocabulary according to the data features to obtain a new initial vocabulary; The second sorting module is used to perform a second word frequency sorting in the initial word library, and select the spare coal industry segmentation words before y in frequency sorting to construct a fourth word library; The screening module is used to combine the fourth word library and the first word library into a fifth word library, and perform repeated word segmentation screening on the fifth word library to obtain a coal industry word segmentation word library.

7. The device according to claim 6, characterized in that The extraction module comprises: The first extraction unit is used to access the keyword column corresponding to the coal journal data by using the label index through the two-dimensional table data structure DataFrame, complete the keyword extraction of the structured data, and obtain the structured first category industry segmentation; The second extraction unit is used to parse the semi-structured data corresponding to the coal journal data into structured data through a parser, so as to extract keywords from the structured data and obtain the second type of industry segmentation words corresponding to the semi-structured data; A positioning unit is used to locate the keyword rows of the directly editable category and the non-directly editable category in the unstructured data by using a string matching algorithm, so as to complete the keyword extraction of the unstructured data and obtain the third category industry segmentation and the fourth category industry segmentation; The generating unit is used to take the first-category industry segmentation words, the second-category industry segmentation words, the third-category industry segmentation words, and the fourth-category industry segmentation words together as a plurality of coal industry segmentation words.

8. The device according to claim 7, characterized in that The positioning unit is specifically used for: The keyword rows of the directly editable category in the unstructured data are located by using a string matching algorithm to complete the keyword extraction of the directly editable category and obtain the third category of industry segmentation; Performing data preprocessing on the non-directly editable class in the unstructured data to obtain an initialized non-directly editable class, wherein the data preprocessing includes noise reduction, binarization processing and tilt correction; Performing text detection, recognition and correction editing on the initialized non-directly editable categories in turn to obtain usable industry texts; The available industry texts are respectively located for keyword rows through a string matching algorithm to complete the extraction of keywords that cannot be directly edited, and obtain the fourth category of industry segmentation.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-5.

Images