Knowledge hierarchical extraction method for coal industry scientific literature

Abstract

The application provides a knowledge hierarchical extraction method for coal industry scientific literature, relates to the technical field of natural language processing, and comprises the following steps: carrying out pure text MD format conversion on a PDF format coal industry scientific literature, deleting non-text identifiers at the beginning of lines, and obtaining a target coal industry document; defining language identifiers of titles at all levels and an identifier rule library formed by level identifiers; training a title hierarchical model; the title hierarchical model identifies multiple target level titles in the target coal industry document and respective corresponding texts; the multiple target level titles are added with identifiers through the identifier rule library and combined with the texts to generate MD text files; and regular matching identifiers are used for directional knowledge hierarchical extraction in the MD text files to obtain extracted texts.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a knowledge hierarchical extraction method for scientific and technological literature in the coal industry. Background Art

[0002] Due to the specialized nature and complexity of the coal industry, scientific and technological literature contains a wealth of industry knowledge and practical experience, which is a crucial foundation for building intelligent coal mines. However, traditional scientific and technological literature information extraction methods struggle to efficiently process this massive, unstructured volume of data. This can lead to biases or anomalies in the responses generated by intelligent coal mine models, resulting in information that is inconsistent with reality or lacks accuracy. Therefore, a more reliable, hierarchical knowledge extraction method for coal industry scientific and technological literature is urgently needed. Summary of the Invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0004] To this end, the first purpose of the present invention is to propose a knowledge hierarchical extraction method for scientific and technological literature in the coal industry, which improves the accuracy and efficiency of knowledge hierarchical extraction for scientific and technological literature in the coal industry through PDF document processing, title classification model, and title-oriented identifier rule library.

[0005] The second purpose of the present invention is to provide a knowledge classification extraction device for coal industry scientific and technological literature.

[0006] A third object of the present invention is to provide an electronic device.

[0007] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium storing computer instructions.

[0008] To achieve the above objectives, the first embodiment of the present invention proposes a knowledge hierarchical extraction method for scientific and technological literature in the coal industry, the method comprising:

[0009] Convert the coal industry scientific and technological literature in PDF format into a coal industry document in plain text MD format, and delete the non-text identifiers at the beginning of each line in the coal industry document to obtain the target coal industry document;

[0010] Define a title-oriented identifier rule base, the identifier rule base includes a language identifier defined according to the language type of each level of title, and a level identifier corresponding to each level of title;

[0011] The large model is used to synthesize multiple training titles of different levels and the training texts of each level of training titles to form a title classification dataset. Then, combined with the pre-trained language model, the semantic features of the training titles and training texts are extracted, and the decision tree classification algorithm is trained to obtain a title classification model.

[0012] Identify multiple target-level titles and the text of each target-level title in the target coal industry document through the title classification model;

[0013] According to the identifier rule library, the target language identifier and the corresponding target level identifier are added to the beginning of each target level title line respectively, and the target level titles and texts after adding the target language identifier and target level identifier are combined to form a standard MD text file;

[0014] According to the user's question request information, a corresponding regularized matching identifier is generated to match the target level title in the MD text file, and a targeted knowledge hierarchical extraction of the text under the target level title is performed to obtain the extracted text of the question request information.

[0015] To achieve the above-mentioned purpose, the second embodiment of the present invention proposes a knowledge classification extraction device for scientific and technological literature in the coal industry, the device comprising:

[0016] A conversion module is used to convert the coal industry scientific and technological literature in PDF format into a coal industry document in plain text MD format, and delete the non-text identifier contained at the beginning of each line in the coal industry document to obtain the target coal industry document;

[0017] A definition module is used to define a title-oriented identifier rule base, wherein the identifier rule base includes a language identifier defined according to the language type of each level of title, and a level identifier corresponding to each level of title;

[0018] The training module is used to use the large model to synthesize multiple training titles of different levels and the training text of each level of training titles to form a title classification dataset. It is then combined with the pre-trained language model to extract the semantic features of the training titles and training texts, and train the decision tree classification algorithm to obtain a title classification model;

[0019] An identification module, configured to identify multiple target-level titles and the text of each target-level title in a target coal industry document through a title classification model;

[0020] A building module is used to add the target language identifier and the corresponding target level identifier at the beginning of each target level title according to the identifier rule library, and combine the target level titles and texts after adding the target language identifier and target level identifier to form a standard MD text file;

[0021] The extraction module is used to generate a corresponding regularized matching identifier based on the user's question request information to match the target level title in the MD text file, and perform targeted knowledge hierarchical extraction of the text under the target level title to obtain the extracted text of the question request information.

[0022] To achieve the above-mentioned purpose, the third aspect embodiment of the present invention proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0023] In order to achieve the above-mentioned objectives, an embodiment of the fourth aspect of the present invention proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0024] The embodiment of the present invention provides a knowledge hierarchical extraction method, device, electronic device, and storage medium for coal industry scientific and technological documents. The method converts coal industry scientific and technological documents in PDF format into plain text MD format and then deletes non-text identifiers at the beginning of the line to obtain a target coal industry document; defines language identifiers for titles of each level and an identifier rule library composed of level identifiers; trains a title hierarchical model; the title hierarchical model identifies multiple target level titles and their corresponding texts in the target coal industry document; multiple target level titles are added with identifiers through the identifier rule library and combined with the text to generate an MD text file; regularized matching identifiers are used to perform directional knowledge hierarchical extraction in the MD text file to obtain extracted text. Thus, through PDF document processing, title hierarchical model, and title-oriented identifier rule library, the accuracy and efficiency of knowledge hierarchical extraction for coal industry scientific and technological documents are improved.

[0025] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A schematic diagram of a process for extracting knowledge from coal industry scientific and technological literature according to an embodiment of the present invention;

[0028] Figure 2A schematic flow chart of another method for extracting knowledge from coal industry scientific and technological literature according to an embodiment of the present invention;

[0029] Figure 3 A schematic structural diagram of a knowledge hierarchical extraction device for coal industry scientific and technological literature provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0031] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of the present invention comply with the relevant provisions of relevant laws and regulations.

[0032] The following describes, with reference to the accompanying drawings, a method, device, electronic device, and storage medium for extracting knowledge from coal industry scientific and technological literature according to an embodiment of the present invention.

[0033] Figure 1 A schematic flow chart of a method for extracting knowledge from coal industry scientific and technological literature at a hierarchical level provided by an embodiment of the present invention.

[0034] like Figure 1 As shown, the method includes the following steps:

[0035] Step 101: convert the coal industry scientific and technological literature in PDF format into a coal industry document in plain text MD format, and delete the non-text identifier contained at the beginning of each line in the coal industry document to obtain the target coal industry document.

[0036] In some possible implementations, coal industry scientific and technological documents in PDF format are converted into coal industry documents in plain text MD format, and non-text identifiers at the beginning of each line in the coal industry document are deleted to obtain a target coal industry document, including: using an optical character recognition (OCR) method to convert coal industry scientific and technological documents in PDF format into coal industry documents in plain text MD format (Markdown format); using a matching algorithm to search each line in the coal industry document, and when each line is found to contain a non-text identifier at the beginning of the line, using a deletion method to delete the non-text identifier to obtain the target coal industry document.

[0037] Specifically, coal industry documents can be:

[0038] 4.7 Car barriers and buffers

[0039] 4.7.1 Wire rope

[0040] When using wire rope as a barrier and buffer element, round stranded, cross-twisted wire rope should be used. The wire rope should be free of broken wires and rust. When the structure or diameter of the wire rope changes, the resistance value of the buffer should be recalibrated.

[0041] 4.7.2 Resistance value of buffer

[0042] The resistance value of the buffer should be calibrated, and within the designed resistance value range, the resistance value difference between two buffers should not exceed 20%. After calibration, the parts should have no permanent deformation or damage.

[0043] 4.7.3 Fluorescent signs on vehicle barriers

[0044] The vehicle barriers should have red and white fluorescent signs.

[0045] Step 102: define a title-oriented identifier rule base, which includes language identifiers defined according to the language types of titles at each level, and level identifiers corresponding to titles at each level.

[0046] In some possible implementations, where each level of headings includes four levels of headings, the level identifier of the first-level heading is one preset identifier, the level identifier of the second-level heading is two preset identifiers, the level identifier of the third-level heading is three preset identifiers, and the level identifier of the fourth-level heading is four preset identifiers. This facilitates rapid and accurate differentiation and segmentation of target coal industry documents. To achieve accurate classification and extraction of target coal industry documents, specific identifiers are added before the titles to facilitate rapid search and extraction.

[0047] Specifically, you can add "*" as a language identifier at the beginning of the Chinese title line, and add "&" as a language identifier at the beginning of the English title line; the preset identifier of the first-level title can be set to "#", the level identifier of the second-level title can be set to "##", the level identifier of the third-level title can be set to "###", and the level identifier of the fourth-level title can be set to "####".

[0048] Among them, Chinese titles generally appear in the first line of the target coal industry document. Add "*" at the beginning of the first line as a language identifier to identify the language type of the title; search for the first line of English in the target coal industry document to confirm that it is an English title. Add "&" before the English title. The results after adding are as follows:

[0049] * Safety technical requirements for protective devices for sports cars used in coal mines

[0050] &Safety technical requirements for coal mine-used car catcher

[0051] In step 103, a large model is used to synthesize a plurality of training titles of different levels and training texts of training titles of each level to form a title classification data set. Then, the pre-trained language model is combined to extract the semantic features of the training titles and training texts, and a decision tree classification algorithm is trained to obtain a title grading model.

[0052] In some possible implementations, a large model is used to synthesize a plurality of training titles of different levels and training texts of the training titles of each level to form a title classification dataset, which is then combined with a pre-trained language model to extract semantic features of the training titles and training texts, and a decision tree classification algorithm is trained to obtain a title hierarchy model, including: using the large model to synthesize a plurality of training titles of different levels and training texts of the titles of each level to form a title classification dataset; based on the title classification dataset, combined with a pre-trained language model (Bidirectional Encoder Representations from Transformers, BERT), extracting semantic features of the training titles and training texts, training a decision tree classification algorithm (GBDT), and generating an initial title hierarchy model; using a regularized matching method to check whether there are numbers and "dots" at the beginning of each line in the title classification dataset, and calculating the number of "dots" and the number of numbers between the "dots"; if the number of "dots" is greater than the number of numbers by n, it is used as a training title line, where n is a preset threshold; and optimizing the initial title hierarchy model through the training title lines to obtain a title hierarchy model (BERT-GBDT), thereby improving the accuracy of title hierarchy recognition.

[0053] Among them, each level of training titles can be set to no less than 5,000, totaling no less than 25,000; the training text contains data similar to training titles of different levels, such as "The wind duct length of 2.5 meters needs to be arranged in the tunneling head". This type of data is similar to the second-level title, but it is not a second-level title.

[0054] Step 104: Identify multiple target-level titles and the text of each target-level title in the target coal industry document through the title classification model.

[0055] In some possible implementations, specifically, if the Bert-GBDT model determines that the text is body text, no level identifier is added; if it is a level one title, a single "#" is added to the beginning of the line; if it is a level two title, two "#"s are added to the beginning of the line, resulting in "##"; if it is a level three title, three "#"s are added to the beginning of the line, resulting in "###". If it is a level four title, four "#"s are added to the beginning of the line, resulting in "###". The target level titles with added level identifiers and the text of each target level title are as follows:

[0056] #1 Achievements in the Development of Coal Mining Theory

[0057] ## 1.1 Achievements in Theoretical Development

[0058] "Practical Mine Pressure Control Theory", from the systematicity, depth, integrity of the research, and the achievements in guiding mining practice

[0059] ### 1.1.1 Clarify the guiding ideology and system

[0060] (1) Strictly distinguish and define two basic concepts

[0061] "Mine pressure" and "mine pressure manifestation" use "existing mining" to indicate the absolute existence of the former, and control "mine pressure manifestation" within a safe, reliable, technically possible and economically reasonable range as mine pressure control.

[0062] Step 105: Add the target language identifier and the corresponding target level identifier at the beginning of each target level title according to the identifier rule library, and combine the target level titles and texts after adding the target language identifier and target level identifier to form a standard MD text file.

[0063] In some possible implementations, after forming a standard MD text file, it also includes: searching each line of the MD text file, recording multiple target lines with target level identifiers, and recording the line number and target line content of each target line; extracting the target line content in order of line number size, saving it in sequence, and removing all target level identifiers to generate a directory of scientific and technological literature in the coal industry.

[0064] Specifically, search each line of the MD text file. If it is found that the beginning of the line contains "####", add the same number of spaces as the number of "#" signs at the beginning of the line, and then extract all the contents of the line (target line content) and record the line number of the line (target line).

[0065] Search each line of the MD text file. If the line begins with "###", add the same number of spaces as the number of "#"s at the beginning of the line, and then extract all the contents of the line. Record the line number.

[0066] Search each line of the MD text file. If it is found that the line begins with "##", add the same number of spaces as the number of "#" signs at the beginning of the line, then extract all the contents of the line and record the line number.

[0067] Search each line of the MD text file. If it is found that the line begins with "#", add the same number of spaces as the number of "#" signs at the beginning of the line, then extract all the contents of the line and record the line number.

[0068] Extract the above contents strictly in the order of line numbers, save them in order, and remove all special identifiers "#", then a directory of scientific and technological literature in the coal industry can be generated.

[0069] For the successfully identified target level titles, level identifiers are added according to the level-based identifier rule library. Subsequent level title recognition only requires regularized matching and classification of titles with level identifiers, which is an order of magnitude faster than the response speed of reusing classification algorithms for hierarchical classification matching.

[0070] Step 106, based on the user's question request information, generates a corresponding regularized matching identifier to match the target level title in the MD text file, and performs directional knowledge hierarchical extraction of the text under the target level title to obtain the extracted text of the question request information.

[0071] In some possible implementations, a corresponding regularized matching identifier is generated based on the user's question request information to match the target level title in the MD text file, and a targeted knowledge hierarchical extraction of the text under the target level title is performed to obtain the extracted text of the question request information, including: when the user's question request information is to extract the text under each first-level title, a corresponding regularized matching identifier is generated as a preset identifier to match multiple first-level title lines in the MD text file, each of which contains only one preset identifier at the beginning of the line, and a program (python) script is used to extract the text between two first-level title lines from the multiple first-level title lines as the extracted text under each first-level title; when the user's question request information is to extract the text under any target first-level title, In this case, a corresponding regularized matching identifier is generated as a preset identifier to match multiple first-level title rows in the MD text file, each of which contains only one preset identifier at the beginning of each row, and a program script is used to extract the text between the target first-level title row and the next first-level title row from the multiple first-level title rows as the extracted text under the target first-level title; in the case where the user's question request information is to extract the text under any target second-level title, a corresponding regularized matching identifier is generated as two preset identifiers to match multiple second-level title rows in the MD text file, each of which contains only two preset identifiers at the beginning of each row, and a program script is used to extract the text between the target second-level title row and the next second-level title row from the multiple second-level title rows as the extracted text under the target second-level title.

[0072] Optionally, based on the written MD text file, traverse each line and search for all first-level headings. Search each line and check whether it contains only a single "#" sign at the beginning of the line. If so, it can be determined that this line is the title of a first-level heading. Then record the line number and continue searching the full text, recording each first-level heading line that contains only a single "#" sign at the beginning of the line. Use a Python script to copy and save the text between two first-level heading lines in sequence, thereby achieving title-based segmentation of the entire MD text file. For example, assuming that the MD text file has a total of seven first-level headings, the seven line numbers will be located. Let the line numbers be na, nb, nc, nd, ne, nf, and ng respectively. Use a Python script to extract and process the text between na and nb, the text between nb and nc, the text between nc and nd, the text between nd and ne, the text between ne and nf, and the text between nf and ng. Then, integrate the text under the same first-level heading into a single paragraph to achieve semantic coherence, completing the hierarchical segmentation of the entire MD text file.

[0073] Alternatively, to extract only the text below a specific first-level heading, for example, "2 Coal Mine Fire Prevention and Extinguishing Specifications," a Python script can be used with regularization. Regularization not only matches the level identifier "#" at the beginning of a line but also the target first-level heading. Then, each line is searched, the line containing the target first-level heading is located, and the line number is obtained, set to "a." Using the regularized matching principle, the search continues downward until the nearest first-level heading line containing a # is found. This line is then located and its line number is obtained, set to "b." The Python script then copies the text between the two line numbers (a, b) to extract the text.

[0074] Alternatively, if you need to extract the text under a specific sub-heading, such as content under '2.6 Coal Mining Specifications,' you can use a Python script with regularization. Regularization matches the line with the level identifier "##" at the beginning of the line and the target sub-heading. Locate the line containing the target sub-heading and obtain the line number, setting it to c. Using the regularized matching principle, continue matching downward until you find the nearest sub-heading line containing "##." Locate that line and obtain its line number, setting it to d. Use the Python script to copy the text between the two line numbers (c and d) to extract the text.

[0075] The embodiment of the present invention provides a knowledge hierarchical extraction method for coal industry scientific and technological documents. The method converts the coal industry scientific and technological documents in PDF format into plain text MD format and then deletes the non-text identifiers at the beginning of the line to obtain the target coal industry document; defines the language identifiers of each level of titles and an identifier rule library composed of level identifiers; trains a title hierarchical model; the title hierarchical model identifies multiple target level titles and their corresponding texts in the target coal industry document; multiple target level titles are added with identifiers through the identifier rule library and combined with the text to generate an MD text file; regularized matching identifiers are used to perform directional knowledge hierarchical extraction in the MD text file to obtain extracted text. Thus, through PDF document processing, title hierarchical model, and title-oriented identifier rule library, the accuracy and efficiency of knowledge hierarchical extraction for coal industry scientific and technological documents are improved.

[0076] By adding title-level identifiers, the structured parsing and modular classification of coal science and technology literature are accurately achieved. Combined with regular expression matching and line number positioning methods, core knowledge such as first-level titles and text, second-level titles and text, and third-level titles and text can be accurately extracted. It also supports the merging and splitting of content at the same level, solving problems such as low extraction efficiency and high error rates caused by the mixing of unstructured text information, and laying the foundation for building a high-precision industry knowledge vector database. At the same time, by extracting textual knowledge from coal science and technology literature, it can support the rapid and efficient construction of a highly reliable industry knowledge base, helping to enhance retrieval generation for large models.

[0077] In order to clearly illustrate the above embodiment, Figure 2 A flowchart of another method for knowledge hierarchical extraction of scientific and technological literature in the coal industry, provided by an embodiment of the present invention, includes: converting coal industry scientific and technological literature in PDF format into a target coal industry document in plain text (MD) format; simultaneously, using a large model to synthesize multiple training titles of different levels and training text for each training title to form a title classification dataset; then, combining the pre-trained language model, extracting semantic features of the training titles and training text, and training a decision tree classification algorithm to obtain an initial title classification model; then, using a regularized matching method to match and check the titles in the title classification dataset to optimize the initial title classification model to obtain a title classification model; defining a title-oriented identifier rule base; adding target-level identifiers corresponding to each target-level title in the target coal industry document through the title classification model and the identifier rule base; and performing targeted knowledge hierarchical extraction of the text under the target-level title by matching the identifiers with the target-level identifiers through regularized matching to obtain extracted text. Thus, by modularly extracting key information from coal industry scientific and technological literature, an industry knowledge vector database is constructed. The industry knowledge vector database provides a rich and structured industry knowledge resource for the generative large language model, thereby effectively improving the accuracy and reliability of the generative large language model in coal industry applications.

[0078] In order to implement the above embodiment, the present invention also proposes a knowledge classification extraction device for coal industry scientific and technological literature.

[0079] Figure 3 A schematic structural diagram of a knowledge hierarchical extraction device for coal industry scientific and technological literature provided by an embodiment of the present invention.

[0080] like Figure 3 As shown, the knowledge hierarchical extraction device 30 for coal industry scientific and technological literature includes: a conversion module 31, a definition module 32, a training module 33, an identification module 34, a construction module 35, and an extraction module 36.

[0081] The conversion module 31 is used to convert the coal industry scientific and technological literature in PDF format into a coal industry document in plain text MD format, and delete the non-text identifier contained in the beginning of each line in the coal industry document to obtain the target coal industry document;

[0082] A definition module 32 is used to define a title-oriented identifier rule base, the identifier rule base including language identifiers defined according to the language type of each level of title, and level identifiers corresponding to each level of title;

[0083] The training module 33 is used to use the large model to synthesize multiple training titles of different levels and the training text of each level of training titles to form a title classification dataset. Then, combined with the pre-trained language model, the semantic features of the training titles and training text are extracted, and a decision tree classification algorithm is trained to obtain a title classification model.

[0084] An identification module 34 is configured to identify multiple target-level titles and the text of each target-level title in the target coal industry document through a title classification model;

[0085] A building module 35 is configured to add a target language identifier and a corresponding target level identifier to the beginning of each target level title according to an identifier rule base, and to combine each target level title and the text after adding the target language identifier and the target level identifier to form a standard MD text file;

[0086] The extraction module 36 is used to generate a corresponding regularized matching identifier based on the user's question request information to match the target level title in the MD text file, and perform targeted knowledge hierarchical extraction of the text under the target level title to obtain the extracted text of the question request information.

[0087] Furthermore, in a possible implementation of the embodiment of the present invention, the conversion module 31 is specifically configured to:

[0088] Use optical character recognition to convert coal industry scientific and technological documents in PDF format into coal industry documents in plain text MD format;

[0089] A matching algorithm is used to search each line in the coal industry document. When a non-text identifier is found at the beginning of each line, the non-text identifier is deleted using a deletion method to obtain the target coal industry document.

[0090] Furthermore, in a possible implementation of an embodiment of the present invention, where each level of titles includes four level titles, the level identifier of the first-level title is one preset identifier, the level identifier of the second-level title is two preset identifiers, the level identifier of the third-level title is three preset identifiers, and the level identifier of the fourth-level title is four preset identifiers.

[0091] Furthermore, in a possible implementation of the embodiment of the present invention, the training module 33 is specifically configured to:

[0092] Use the large model to synthesize multiple titles of different levels and the training text of titles of each level to form a title classification dataset;

[0093] Based on the title classification dataset, combined with the pre-trained language model, the semantic features of the training titles and training texts are extracted, a decision tree classification algorithm is trained, and an initial title classification model is generated;

[0094] Using a regularized matching method, we check whether there are numbers and dots at the beginning of each line in the title classification dataset, and calculate the number of dots and the number of numbers between the dots.

[0095] If the number of symbol "dots" is greater than the number of numbers by n, it is used as a training title row, where n is the preset threshold;

[0096] By training the title lines, the initial title grading model is optimized to obtain a title grading model.

[0097] Furthermore, in a possible implementation of the embodiment of the present invention, the apparatus further includes:

[0098] A search module is used to search each line of the MD text file, record multiple target lines with target level identifiers, and record the line number and content of each target line;

[0099] The generation module is used to extract the target line content in order of line number size, save it in order, and remove all target level identifiers to generate a directory of coal industry scientific and technological literature.

[0100] Furthermore, in a possible implementation of the embodiment of the present invention, the extraction module 36 is specifically configured to:

[0101] When the user's question request information is to extract the text under each first-level heading, a corresponding regularized matching identifier is generated as a preset identifier to match multiple first-level heading lines in the MD text file, each of which contains only one preset identifier at the beginning of each line, and a program script is used to extract the text between two first-level heading lines from the multiple first-level heading lines as the extracted text under each first-level heading;

[0102] When the user's question request information is to extract the text under any target first-level title, a corresponding regularized matching identifier is generated as a preset identifier to match multiple first-level title lines in the MD text file, each of which contains only one preset identifier at the beginning of each line, and a program script is used to extract the text between the target first-level title line and the next first-level title line from the multiple first-level title lines as the extracted text under the target first-level title;

[0103] When the user's question request information is to extract the text under any target secondary title, the corresponding regularized matching identifier is generated as two preset identifiers to match multiple secondary title lines in the MD text file, each of which contains only two preset identifiers at the beginning of each line, and use the program script to extract the text between the target secondary title line and its next secondary title line from the multiple secondary title lines as the extracted text under the target secondary title.

[0104] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment and will not be repeated here.

[0105] The knowledge hierarchical extraction device for coal industry scientific and technological documents of the embodiment of the present invention converts the coal industry scientific and technological documents in PDF format into plain text MD format and then deletes the non-text identifiers at the beginning of the line to obtain the target coal industry document; defines the language identifiers of each level of titles and the identifier rule library composed of level identifiers; trains a title hierarchical model; the title hierarchical model identifies multiple target level titles and their corresponding texts in the target coal industry document; multiple target level titles are added with identifiers through the identifier rule library and combined with the text to generate an MD text file; regularized matching identifiers are used to perform directional knowledge hierarchical extraction in the MD text file to obtain extracted text. Therefore, through PDF document processing, title hierarchical model, and title-oriented identifier rule library, the accuracy and efficiency of knowledge hierarchical extraction for coal industry scientific and technological documents are improved.

[0106] In order to implement the above embodiment, the present invention further provides an electronic device, including:

[0107] at least one processor; and

[0108] a memory communicatively connected to the at least one processor; wherein,

[0109] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the aforementioned method.

[0110] In order to implement the above embodiment, the present invention further proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the above method.

[0111] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0113] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0114] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" is any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (not exhaustive) of computer-readable media include: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0115] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any of the following technologies known in the art, or a combination thereof, may be used: a discrete logic circuit having logic gates for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gates, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0116] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0117] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0118] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A knowledge hierarchical extraction method for scientific and technological literature in the coal industry, characterized by: The method comprises: Convert the coal industry scientific and technological literature in PDF format into a coal industry document in plain text MD format, and delete the non-text identifiers at the beginning of each line in the coal industry document to obtain the target coal industry document; Define a title-oriented identifier rule base, the identifier rule base includes a language identifier defined according to the language type of each level of title, and a level identifier corresponding to each level of title; The large model is used to synthesize multiple training titles of different levels and the training texts of each level of training titles to form a title classification dataset. Then, combined with the pre-trained language model, the semantic features of the training titles and training texts are extracted, and the decision tree classification algorithm is trained to obtain a title classification model. Identify multiple target-level titles and the text of each target-level title in the target coal industry document through the title classification model; According to the identifier rule library, the target language identifier and the corresponding target level identifier are added to the beginning of each target level title line respectively, and the target level titles and texts after adding the target language identifier and target level identifier are combined to form a standard MD text file; Generate a corresponding regularized matching identifier based on the user's question request information to match the target level title in the MD text file, and perform a targeted knowledge hierarchical extraction of the text under the target level title to obtain the extracted text of the question request information; The method uses the large model to synthesize multiple training titles of different levels and the training texts of the training titles of each level to form a title classification dataset, and then combines it with the pre-trained language model to extract the semantic features of the training titles and training texts, and train the decision tree classification algorithm to obtain a title classification model, including: Use the large model to synthesize multiple titles of different levels and the training text of titles of each level to form a title classification dataset; Based on the title classification dataset, combined with the pre-trained language model, the semantic features of the training titles and training texts are extracted, a decision tree classification algorithm is trained, and an initial title classification model is generated; Using a regularized matching method, we check whether there are numbers and dots at the beginning of each line in the title classification dataset, and calculate the number of dots and the number of numbers between the dots. If the number of symbol "dots" is greater than the number of numbers by n, it is used as the training title row, where n is the preset threshold; By training the title lines, the initial title grading model is optimized to obtain a title grading model.

2. The method according to claim 1, characterized in that The process of converting the coal industry scientific and technological literature in PDF format into a coal industry document in plain text MD format and deleting the non-text identifier at the beginning of each line in the coal industry document to obtain the target coal industry document includes: Use optical character recognition to convert coal industry scientific and technological documents in PDF format into coal industry documents in plain text MD format; A matching algorithm is used to search each line in the coal industry document. When a non-text identifier is found at the beginning of each line, the non-text identifier is deleted using a deletion method to obtain the target coal industry document.

3. The method according to claim 1, characterized in that in, When each level of title includes four level titles, the level identifier of the first-level title is one preset identifier, the level identifier of the second-level title is two preset identifiers, the level identifier of the third-level title is three preset identifiers, and the level identifier of the fourth-level title is four preset identifiers.

4. The method according to claim 1, wherein After forming a standard MD text file, it also includes: Search each line of the MD text file, record multiple target lines with target level identifiers, and record the line number and content of each target line; The target row contents are extracted in order of row numbers, stored in order, and all target level identifiers are removed to generate a catalog of scientific and technological literature in the coal industry.

5. The method according to claim 3, characterized in that The method of generating a corresponding regularized matching identifier based on the user's question request information to match a target level title in the MD text file, and performing a targeted knowledge hierarchical extraction of the text under the target level title to obtain the extracted text of the question request information includes: When the user's question request information is to extract the text under each first-level heading, a corresponding regularized matching identifier is generated as a preset identifier to match multiple first-level heading lines in the MD text file, each of which contains only one preset identifier at the beginning of each line, and a program script is used to extract the text between two first-level heading lines from the multiple first-level heading lines as the extracted text under each first-level heading; When the user's question request information is to extract the text under any target first-level title, a corresponding regularized matching identifier is generated as a preset identifier to match multiple first-level title lines in the MD text file, each of which contains only one preset identifier at the beginning of each line, and a program script is used to extract the text between the target first-level title line and the next first-level title line from the multiple first-level title lines as the extracted text under the target first-level title; When the user's question request information is to extract the text under any target secondary title, the corresponding regularized matching identifier is generated as two preset identifiers to match multiple secondary title lines in the MD text file, each of which contains only two preset identifiers at the beginning of each line, and use the program script to extract the text between the target secondary title line and its next secondary title line from the multiple secondary title lines as the extracted text under the target secondary title.

6. A knowledge classification extraction device for coal industry scientific and technological literature, characterized in that: The device comprises: A conversion module is used to convert the coal industry scientific and technological literature in PDF format into a coal industry document in plain text MD format, and delete the non-text identifier contained at the beginning of each line in the coal industry document to obtain the target coal industry document; A definition module is used to define a title-oriented identifier rule base, wherein the identifier rule base includes a language identifier defined according to the language type of each level of title, and a level identifier corresponding to each level of title; The training module is used to use the large model to synthesize multiple training titles of different levels and the training text of each level of training titles to form a title classification dataset. It is then combined with the pre-trained language model to extract the semantic features of the training titles and training texts, and train the decision tree classification algorithm to obtain a title classification model; An identification module, configured to identify multiple target-level titles and the text of each target-level title in a target coal industry document through a title classification model; A building module is used to add the target language identifier and the corresponding target level identifier at the beginning of each target level title according to the identifier rule library, and combine the target level titles and texts after adding the target language identifier and target level identifier to form a standard MD text file; An extraction module is used to generate a corresponding regularized matching identifier based on the user's question request information, so as to match the target level title in the MD text file, and perform a targeted knowledge hierarchical extraction of the text under the target level title to obtain the extracted text of the question request information; The training module is specifically used to: Use the large model to synthesize multiple titles of different levels and the training text of titles of each level to form a title classification dataset; Based on the title classification dataset, combined with the pre-trained language model, the semantic features of the training titles and training texts are extracted, a decision tree classification algorithm is trained, and an initial title classification model is generated; Using a regularized matching method, we check whether there are numbers and dots at the beginning of each line in the title classification dataset, and calculate the number of dots and the number of numbers between the dots. If the number of symbol "dots" is greater than the number of numbers by n, it is used as the training title row, where n is the preset threshold; By training the title lines, the initial title grading model is optimized to obtain a title grading model.

7. The device according to claim 6, characterized in that The conversion module is specifically used to: Use optical character recognition to convert coal industry scientific and technological documents in PDF format into coal industry documents in plain text MD format; A matching algorithm is used to search each line in the coal industry document. When a non-text identifier is found at the beginning of each line, the non-text identifier is deleted using a deletion method to obtain the target coal industry document.

8. The device according to claim 6, characterized in that in, When each level of title includes four level titles, the level identifier of the first-level title is one preset identifier, the level identifier of the second-level title is two preset identifiers, the level identifier of the third-level title is three preset identifiers, and the level identifier of the fourth-level title is four preset identifiers.

9. The device according to claim 6, characterized in that The device further comprises: A search module is used to search each line of the MD text file, record multiple target lines with target level identifiers, and record the line number and content of each target line; The generation module is used to extract the target line content in order of line number size, save it in order, and remove all target level identifiers to generate a directory of coal industry scientific and technological literature.

10. The device according to claim 8, characterized in that The extraction module is specifically used to: When the user's question request information is to extract the text under each first-level heading, a corresponding regularized matching identifier is generated as a preset identifier to match multiple first-level heading lines in the MD text file, each of which contains only one preset identifier at the beginning of each line, and a program script is used to extract the text between two first-level heading lines from the multiple first-level heading lines as the extracted text under each first-level heading; When the user's question request information is to extract the text under any target first-level title, a corresponding regularized matching identifier is generated as a preset identifier to match multiple first-level title lines in the MD text file, each of which contains only one preset identifier at the beginning of each line, and a program script is used to extract the text between the target first-level title line and the next first-level title line from the multiple first-level title lines as the extracted text under the target first-level title; When the user's question request information is to extract the text under any target secondary title, the corresponding regularized matching identifier is generated as two preset identifiers to match multiple secondary title lines in the MD text file, each of which contains only two preset identifiers at the beginning of each line, and use the program script to extract the text between the target secondary title line and its next secondary title line from the multiple secondary title lines as the extracted text under the target secondary title.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

Images