Coal mine unstructured data extraction method combining dynamic segmentation and quality evaluation
By using dynamic segmentation and quality assessment methods, combined with a large language model to process unstructured documents from coal mines, the problem of low information extraction efficiency in existing technologies is solved, generating high-quality structured data to support the intelligent development of coal mines.
Patent Information
- Application Number
- CN202610066290.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-19
- Publication Date
- 2026-03-13
AI Technical Summary
Existing technologies cannot efficiently and accurately extract structured information from unstructured documents in the coal mining industry, thus limiting the intelligent development of the coal mining industry.
By combining a dynamic structure-aware segmentation strategy with a large language model, documents are segmented through token length constraints, and a structured extraction and quality assessment system is adopted to generate a high-quality structured dataset.
It enables efficient and accurate processing of unstructured documents from coal mines, and the generated structured data conforms to the logical structure and semantic information of the original documents, providing a solid data foundation to support intelligent coal mining and safety risk early warning.
Smart Images
Figure CN121659958A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent information processing in coal mines, specifically to a method for extracting unstructured coal mine data that combines dynamic segmentation and quality assessment. Background Technology
[0002] The coal mining industry is rapidly entering a data-driven, intelligent phase. Massive amounts of documents, such as geological exploration reports, safety assessments, design drawings, and technical specifications, are increasingly valuable as key mediums carrying industry knowledge and decision-making elements. However, these crucial documents are currently mostly stored in unstructured or semi-structured formats such as PDFs, Word documents, or scanned copies, generally characterized by high information density, complex structures, and varied styles. How to efficiently and accurately extract structured information from such documents has become one of the key bottlenecks restricting the intelligent development of the coal mining industry. Existing technologies, whether using traditional rules or directly applying general large models, cannot perfectly solve the problem of extracting unstructured long documents from the coal mining industry. Summary of the Invention
[0003] To overcome at least one deficiency in the prior art, this application provides a method for extracting unstructured coal mine data that combines dynamic segmentation and quality assessment.
[0004] Firstly, a method for extracting unstructured coal mine data that combines dynamic segmentation and quality assessment is provided, including: The document to be processed is obtained from the coal mining industry. The document is then parsed to represent each paragraph of the document as a triple. The triple includes the paragraph content, the paragraph's token length, and the paragraph's structural tag. A dynamic structure-aware segmentation strategy is adopted to divide all paragraphs into multiple semantically coherent input blocks; A large language model is used to extract the structured data from each input block to obtain the extraction results. Each extraction result is evaluated using an extraction quality structure scoring system to obtain an extraction result score; extraction results with scores greater than the quality threshold are selected to form the final high-quality structured dataset.
[0005] In one embodiment, a dynamic structure-aware segmentation strategy is used to divide all paragraphs into multiple semantically coherent input blocks, including: Step 21: Divide the current remaining paragraphs into several paragraphs based on the Token length constraint to form the initial state of the current input block; the Token length constraint is constructed based on the Token length of the paragraph; if the current input block is the first input block, then the current remaining paragraphs are all paragraphs. Step 22: Generate several paragraph division schemes based on the initial state of the current input block; Step 23: Calculate the structure awareness score for each paragraph segmentation scheme; the structure awareness score includes logical integrity score, element integrity score, and structural mutation penalty item; Step 24: Select the paragraph segmentation scheme corresponding to the maximum value of the structure-aware score as the optimal paragraph segmentation scheme; the paragraphs in the optimal paragraph segmentation scheme constitute the final current input block. Step 25: Remove the paragraphs included in the current input block from the current remaining paragraphs, and use them as the remaining paragraphs for the next calculation. Return to step 21 until there are no remaining paragraphs.
[0006] In one embodiment, based on a token length constraint, several segments are divided from the current remaining segments to form the initial state of the current input block, including: The token length constraint is: the sum of the tokens of all paragraphs in the input block does not exceed the upper limit of the large model context window. ; Starting from the first paragraph of the remaining paragraphs, select several paragraphs that satisfy the Token length constraint, and select the maximum number of paragraphs to form the initial state of the current input block.
[0007] In one embodiment, several segmentation schemes are generated based on the initial state of the current input block, including: The first paragraph in the initial state of the current input block is selected as the first paragraph division scheme; the first paragraph and the second paragraph are selected as the second paragraph division scheme; the next paragraph division scheme is based on the previous paragraph division scheme with one more paragraph added.
[0008] In one embodiment, a structure-aware score is calculated for each paragraph segmentation scheme using the following formula:
[0009] in, Paragraph division scheme Structural perception score Paragraph division scheme The beginning paragraph of the text, Paragraph division scheme The ending paragraph in Paragraph division scheme Logical integrity score Paragraph division scheme The element integrity score, Paragraph division scheme Structural mutation penalty term, for The weight, for The weight, for The weight.
[0010] In one embodiment, the logical integrity score is calculated using the following formula:
[0011] in, Paragraph division scheme Logical integrity score Paragraph division scheme The Middle The structure tag for each paragraph, T={heading} Subtitle}, T is the first structural tag set, C={ }, where C is the second structure tag set. It is an indicator function; it returns 1 if the condition is true and 0 if the condition is false. To avoid constants that are divisible by zero, Represents AND operation.
[0012] In one embodiment, the element integrity score is calculated using the following formula:
[0013] in, Paragraph division scheme The element integrity score, Paragraph division scheme The Middle The structural tags of each paragraph, for The corresponding weight value.
[0014] In one embodiment, the structural mutation penalty term is expressed by the following formula:
[0015] in, Paragraph division scheme Structural mutation penalty term, Paragraph division scheme The Middle The structural tags of each paragraph, Paragraph division scheme The Middle The structural tags of each paragraph, For To begin the structure tag, The penalty value for ending the structure label.
[0016] In one embodiment, an extraction quality structure scoring system is used to evaluate each extraction result to obtain an extraction result score, using the following formula:
[0017] in, For the first Extraction results corresponding to each input block for The extraction results are scored. for Logical integrity score for The element integrity score, for Content alignment consistency score, for The weight, for The weight, for The weight.
[0018] In one embodiment, the content alignment consistency score is calculated using the following formula:
[0019] in, Indicates the first The input block starts from the paragraph. To the end paragraph The content of the paragraph, Indicates the first Extraction results corresponding to each input block and Levenshtein edit distance between them express The length of the token.
[0020] Secondly, a coal mine unstructured data extraction device combining dynamic segmentation and quality assessment is provided to realize the above-mentioned coal mine unstructured data extraction method combining dynamic segmentation and quality assessment.
[0021] Compared with the prior art, this application has the following beneficial effects: 1. This application leverages the powerful semantic understanding capabilities of large language models and introduces a token-constrained dynamic segmentation strategy to efficiently process long documents, avoiding information loss or semantic breaks caused by exceeding model input limits.
[0022] 2. This application establishes an automated assessment system for the quality of structured content, which can perform automated and multi-dimensional quantitative assessments of the extracted data in terms of structural integrity, entity coverage, and content consistency.
[0023] 3. This application can accurately and efficiently automate the processing of unstructured documents in coal mines. The generated results not only conform to the logical structure and semantic information of the original documents, but also pass the automated quality assessment, with high accuracy. This can better realize the structuring of mine knowledge and provide a solid data foundation for downstream tasks such as intelligent coal mining and safety risk early warning. Attached Figure Description
[0024] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings: Figure 1 A flowchart illustrating a method for extracting unstructured coal mine data that combines dynamic segmentation and quality assessment is shown. Detailed Implementation
[0025] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.
[0026] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution according to this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.
[0027] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.
[0028] This application provides a method for extracting unstructured coal mine data that combines dynamic segmentation and quality assessment. Figure 1 A flowchart illustrating a method for extracting unstructured coal mine data that combines dynamic segmentation and quality assessment is shown. (See attached image.) Figure 1 The method mainly includes the following steps: Step S1: Obtain the documents to be processed in the coal mining industry, and parse the documents to be processed to represent each paragraph of the documents as a triple. The triple includes the paragraph content, the token length of the paragraph, and the structural tag of the paragraph.
[0029] Here, the documents to be processed in the coal mining industry include geological exploration reports, safety assessment materials, academic papers, standards and specifications, etc. Document processing tools can be used to apply different parsing strategies to different types of data in the documents, representing each paragraph as a triple. ,in, Indicates the first Paragraph triples, These represent the paragraph content, the paragraph's token length, and the paragraph's structure tags, respectively. The paragraph's token length refers to the number of characters in the paragraph.
[0030] The data types in a document can include graphical data, tabular data, and formula data. The specific parsing strategy is as follows: For image data, OCR + visual large model can be used. For scanned documents / images, high-precision OCR technology can be used to recognize and extract text content. For complex multi-image files, visual large model can be used to directly extract content.
[0031] For tabular data, rule matching and open-source code libraries can be used. For simple tables, rule matching can be used to extract content and row and column position information. For complex tables, open-source code libraries (such as Marker) can be used to accurately extract the data into Markdown format.
[0032] For formula data, Mathpix is used for initial extraction based on Mathpix + Doc2X, and then Doc2X is used to convert the formulas into LaTeX format for secondary use.
[0033] Step S2: Use a dynamic structure-aware segmentation strategy to divide all paragraphs into multiple semantically coherent input blocks.
[0034] Step S3: Use a large language model to perform structured extraction on each input block to obtain the extraction results.
[0035] Specifically, a suggestion template is constructed, and this template is combined with the paragraph content and structural tags corresponding to each paragraph in the input block. This combination is then input into a large language model to generate structured Markdown text. Here, the large oracle model can be, for example, a Gemini series model. Then, a parser converts the structured Markdown text into a unified JSON data structure, i.e., extracts the results. For example, for the first... input blocks After structured extraction, the first... Extraction results corresponding to each input block .
[0036] Step S4: Each extraction result is evaluated using an extraction quality structure scoring system to obtain an extraction result score; extraction results with scores greater than a quality threshold are selected to constitute the final high-quality structured dataset. Here, the quality threshold is determined based on practical experience and is not specifically limited.
[0037] This embodiment leverages the powerful semantic understanding capabilities of large language models and introduces a token-constrained dynamic segmentation strategy to efficiently process lengthy documents, avoiding information loss or semantic breaks caused by exceeding model input limits. It also establishes an automated structured content quality assessment system that can automatically and multidimensionally quantify the extracted data in terms of structural integrity, entity coverage, and content consistency.
[0038] In one embodiment, step S2 involves using a dynamic structure-aware segmentation strategy to divide all paragraphs into multiple semantically coherent input blocks, including: Step 21: Divide the current remaining paragraphs into several paragraphs based on the Token length constraint to form the initial state of the current input block; the Token length constraint is constructed based on the Token length of the paragraphs; if the current input block is the first input block, then the current remaining paragraphs are all paragraphs.
[0039] Specifically, the token length constraint is: the sum of the tokens of all paragraphs in the input block does not exceed the upper limit of the large model context window. Here, the upper limit of the large model context window. Select according to the actual situation, without specific restrictions.
[0040] Starting from the first paragraph of the remaining paragraphs, select several paragraphs that satisfy the Token length constraint, and select the maximum number of paragraphs to form the initial state of the current input block.
[0041] Step 22: Generate several paragraph division schemes based on the initial state of the current input block.
[0042] Specifically, the first paragraph in the initial state of the current input block is selected as the first paragraph division scheme; the first paragraph and the second paragraph are selected as the second paragraph division scheme; and so on, the next paragraph division scheme adds one paragraph to the previous paragraph division scheme.
[0043] Step 23: Calculate the structure perception score for each paragraph segmentation scheme.
[0044] Here, the structure-aware scoring includes logical integrity score, element integrity score, and structural mutation penalty item.
[0045] Specifically, a structure awareness score is calculated for each paragraph segmentation scheme using the following formula:
[0046] in, Paragraph division scheme Structural perception score Paragraph division scheme The beginning paragraph of the text, Paragraph division scheme The ending paragraph in Paragraph division scheme Logical integrity score Paragraph division scheme The element integrity score, Paragraph division scheme Structural mutation penalty term, for The weight, for The weight, for The weight.
[0047] All weights are adjustable and can be dynamically adjusted based on the type of input data. For example, when using academic papers as input data, a greater emphasis is placed on logical integrity, so the weights can be increased. The value is used to ensure that more logical blocks are divided.
[0048] Specifically, the logical integrity score is mainly used to detect whether there are logical connection structures in the paragraph division scheme, such as the logical pairing of "heading → body text", using the following formula:
[0049] in, Paragraph division scheme Logical integrity score Paragraph division scheme The Middle The structure tag for each paragraph, T={heading} ,subtitle }, T is the first structural label set, C={ }, where C is the second structure tag set. It is an indicator function; it returns 1 if the condition is true and 0 if the condition is false. To avoid constants that are divisible by zero, Represents AND operation.
[0050] Specifically, the element integrity score is mainly used to detect the number of unstructured elements in the paragraph segmentation scheme, including tables, formulas, and images, using the following formula:
[0051] in, Paragraph division scheme The element integrity score, Paragraph division scheme The Middle The structural tags of each paragraph, for The corresponding weight value, for example, the weight can be set to: , , , A higher weight indicates that the element is more important in verifying the integrity of other elements.
[0052] Specifically, the structural mutation penalty term is mainly used to detect whether there is a jump in tags within the paragraph division scheme, such as whether there is a "header → footer" tag combination. The following formula is used:
[0053] in, Paragraph division scheme Structural mutation penalty term, Paragraph division scheme The Middle The structural tags of each paragraph, Paragraph division scheme The Middle The structural tags of each paragraph, For To begin the structure tag, The penalty value is for the end of the structure label. Table 1 shows the penalty values for different cases.
[0054] Table 1
[0055] Here, Header represents the header, Footer represents the footer, Title represents the heading, Paragraph represents the paragraph, and List represents the heading. .
[0056] Step 24: Select the paragraph segmentation scheme corresponding to the maximum value of the structure-aware score as the optimal paragraph segmentation scheme; the paragraphs in the optimal paragraph segmentation scheme constitute the final current input block.
[0057] Step 25: Remove the paragraphs included in the current input block from the current remaining paragraphs, and use them as the remaining paragraphs for the next calculation. Return to step 21 until there are no remaining paragraphs.
[0058] This embodiment introduces a token-constrained dynamic segmentation strategy to efficiently process long documents and avoid information loss or semantic breaks caused by exceeding model input limits.
[0059] In one embodiment, an extraction quality structure scoring system is used to evaluate each extraction result to obtain an extraction result score, using the following formula:
[0060] in, For the first Extraction results corresponding to each input block for The extraction results are scored. for Logical integrity score for The element integrity score, for Content alignment consistency score, for The weight, for The weight, for The weight.
[0061] It needs to be explained that, Logical integrity score and Element integrity score The specific calculation method is the same as in the aforementioned embodiments, and will not be repeated here.
[0062] Specifically, since some data is removed during the large model extraction process, the structural mutation penalty term is adjusted to be calculated based on content alignment consistency. The content alignment consistency score is calculated based on edit distance and penalizes content misalignment, omission, or redundancy using the following formula:
[0063] in, Indicates the first The input block starts from the paragraph. To the end paragraph The content of the paragraph, express and Levenshtein edit distance between them express The length of the token.
[0064] This embodiment establishes an automatic evaluation system for structured content quality, which can perform automated, multi-dimensional quantitative evaluation of the extracted data in terms of structural integrity, entity coverage, and content consistency.
[0065] To further verify the effectiveness of the coal mine unstructured data extraction method combining dynamic segmentation and quality assessment proposed in this application, 1959 real documents from the coal mining industry, including academic papers, professional books, and standards and specifications, were selected for the experiment. The specific data are shown in Table 2.
[0066] Table 2
[0067] Experimental results show that the proposed method exhibits good robustness and accuracy when processing large-scale, multi-type real-world documents. Statistical analysis shows that the proposed method achieved an average score of 0.92 in structural integrity, 0.95 in content alignment consistency, and a final comprehensive score of 0.902. These high scores demonstrate that the proposed method effectively preserves the original document structure and remains highly faithful to the original content.
[0068] Furthermore, when unstructured data extraction was performed on papers containing complex tables using the method described in this application, the boundaries of the original tables, cells spanning multiple rows, table header hierarchy, and all data information were accurately identified and restored into a standard Markdown format with a clear structure and complete content. This demonstrates that the method described in this application exhibits superior performance for complex tables that are common in documents and are difficult to extract.
[0069] This application also provides a coal mine unstructured data extraction device that combines dynamic segmentation and quality assessment, used to implement the coal mine unstructured data extraction method combining dynamic segmentation and quality assessment described in the foregoing embodiments.
[0070] In summary, this application has the following technical effects: 1. This application leverages the powerful semantic understanding capabilities of large language models and introduces a token-constrained dynamic segmentation strategy to efficiently process long documents, avoiding information loss or semantic breaks caused by exceeding model input limits.
[0071] 2. This application establishes an automated assessment system for the quality of structured content, which can perform automated and multi-dimensional quantitative assessments of the extracted data in terms of structural integrity, entity coverage, and content consistency.
[0072] 3. This application can accurately and efficiently automate the processing of unstructured documents in coal mines. The generated results not only conform to the logical structure and semantic information of the original documents, but also pass the automated quality assessment, with high accuracy. This can better realize the structuring of mine knowledge and provide a solid data foundation for downstream tasks such as intelligent coal mining and safety risk early warning.
[0073] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for extracting unstructured coal mine data by combining dynamic segmentation and quality assessment, characterized in that, include: Obtain documents to be processed from the coal mining industry, parse the documents to represent each paragraph of the documents as a triple, the triple including paragraph content, paragraph token length and paragraph structure tag; A dynamic structure-aware segmentation strategy is adopted to divide all paragraphs into multiple semantically coherent input blocks; A large language model is used to extract the structured data from each input block to obtain the extraction results. Each extraction result is evaluated using an extraction quality structure scoring system to obtain an extraction result score; Extraction results with scores greater than the quality threshold are selected to form the final high-quality structured dataset.
2. The method as described in claim 1, characterized in that, in, A dynamic structure-aware segmentation strategy is used to divide all paragraphs into multiple semantically coherent input blocks, including: Step 21: Divide the current remaining paragraphs into several paragraphs based on the token length constraint to form the initial state of the current input block; the token length constraint is constructed according to the token length of the paragraph; if the current input block is the first input block, then the current remaining paragraphs are all paragraphs. Step 22: Generate several paragraph division schemes based on the initial state of the current input block; Step 23: Calculate the structure awareness score for each paragraph segmentation scheme; the structure awareness score includes logical integrity score, element integrity score, and structural mutation penalty item; Step 24: Select the paragraph segmentation scheme corresponding to the maximum value of the structure-aware score as the optimal paragraph segmentation scheme; the paragraphs in the optimal paragraph segmentation scheme constitute the final current input block. Step 25: Remove the paragraphs included in the final current input block from the current remaining paragraphs as the remaining paragraphs for the next calculation, and return to step 21 until there are no remaining paragraphs.
3. The method as described in claim 2, characterized in that, in, Based on the token length constraint, several segments are divided from the current remaining segments to form the initial state of the current input block, including: The token length constraint is that the sum of the tokens of all paragraphs in the input block does not exceed the upper limit of the large model context window. ; Starting from the first paragraph of the remaining paragraphs, select several paragraphs that satisfy the Token length constraint, and select the maximum number of paragraphs to form the initial state of the current input block.
4. The method as described in claim 2, characterized in that, in, Based on the initial state of the current input block, several segmentation schemes are generated, including: The first paragraph in the initial state of the current input block is selected as the first paragraph division scheme; the first paragraph and the second paragraph are selected as the second paragraph division scheme; the next paragraph division scheme is based on the previous paragraph division scheme with one more paragraph added.
5. The method as described in claim 2, characterized in that, in, The structure awareness score is calculated for each paragraph segmentation scheme using the following formula: in, Paragraph division scheme Structural perception score Paragraph division scheme The beginning paragraph of the text, Paragraph division scheme The ending paragraph in Paragraph division scheme Logical integrity score Paragraph division scheme The element integrity score, Paragraph division scheme Structural mutation penalty term, for The weight, for The weight, for The weight.
6. The method as described in claim 5, characterized in that, The logical integrity score is calculated using the following formula: in, Paragraph division scheme Logical integrity score Paragraph division scheme The Middle The structure tag for each paragraph, T={heading} Subtitle}, T is the first structural tag set, C={ }, where C is the second structure tag set. It is an indicator function; it returns 1 if the condition is true and 0 if the condition is false. To avoid constants that are divisible by zero, Represents AND operation.
7. The method as described in claim 5, characterized in that, The element integrity score is calculated using the following formula: in, Paragraph division scheme The element integrity score, Paragraph division scheme The Middle The structural tags of each paragraph, for The corresponding weight value.
8. The method as described in claim 5, characterized in that, The structural mutation penalty term is formulated using the following formula: in, Paragraph division scheme Structural mutation penalty term, Paragraph division scheme The Middle The structural tags of each paragraph, Paragraph division scheme The Middle The structural tags of each paragraph, For To begin the structure tag, The penalty value for ending the structure label.
9. The method as described in claim 1, characterized in that, in, Each sampling result is evaluated using a sampling quality structure scoring system to obtain a sampling result score, using the following formula: in, For the first Extraction results corresponding to each input block for The extraction results are scored. for Logical integrity score for The element integrity score, for Content alignment consistency score, for The weight, for The weight, for The weight.
10. The method as described in claim 9, characterized in that, The content alignment consistency score is calculated using the following formula: in, Indicates the first The input block starts from the paragraph. To the end paragraph The content of the paragraph, Indicates the first Extraction results corresponding to each input block and Levenshtein edit distance between them express The length of the token.
11. A coal mine unstructured data extraction device combining dynamic segmentation and quality assessment, characterized in that, This method is used to implement the coal mine unstructured data extraction method combining dynamic segmentation and quality assessment as described in any one of claims 1-10.