A water conservancy knowledge structured extraction and verification method and device

Abstract

The application provides a water conservancy knowledge structured extraction and verification method and device, and belongs to the technical field of artificial intelligence. The method comprises the following steps: differentiating text processing to generate an intermediate file for different format files; classifying the intermediate file into a rule category or a non-rule category based on a preset rule library; identifying a hierarchical title to form an entry knowledge block for the rule category file, and converting table content into an HTML knowledge block; performing semantic segmentation on the non-rule category file to generate a knowledge block; implementing knowledge block verification, archiving, and marking abnormal alarm blocks; converting the table knowledge block into a natural language description by using a large model; positioning the context of the alarm knowledge block, and intelligently correcting the alarm knowledge block by using a large model. The application innovatively combines a traditional semantic analysis model and a large language model to form a closed loop process of preprocessing, extraction, verification, and correction, solves the problems of structured analysis and error correction of complex texts in the water conservancy field, and significantly improves the knowledge processing efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for extracting and verifying structured knowledge in water conservancy. Background Technology

[0002] In the field of water resources knowledge processing, traditional information processing technologies have revealed significant shortcomings when dealing with multi-source, heterogeneous, and highly unstructured water resources data. Their reliance on fully automated processing based on fixed rules makes it difficult to comprehensively cover the complexities of domain-specific expressions and nested engineering parameters in water resources texts, often leading to biases in knowledge element identification and severely impacting the efficiency of data resource transformation into knowledge assets.

[0003] While emerging large model (Large Language Model, LLM) technology has demonstrated powerful capabilities in general domains, it faces challenges in applications within the water conservancy industry. Due to the specialized and complex nature of water conservancy knowledge, relying solely on large models for processing not only demands extremely high computing power and is time-consuming, but also struggles to meet the stringent standards for data accuracy in the water conservancy sector. Summary of the Invention

[0004] This invention provides a method and apparatus for structured extraction and verification of water conservancy knowledge, which addresses the shortcomings of low processing efficiency and poor accuracy of complex water conservancy texts in existing technologies.

[0005] In a first aspect, the present invention provides a method for structured extraction and verification of water conservancy knowledge, comprising: extracting text from files of different formats using different text processing methods to form intermediate files; matching the text content in the intermediate files according to text matching rules in a preset matching rule library to classify the intermediate files into regulatory documents or non-regulatory documents; for regulatory documents, identifying the text content according to hierarchical heading regular expressions and forming a knowledge block from the content of the lowest-level heading; generating knowledge blocks from the HTML format text corresponding to the table content in the intermediate files; for non-regulatory documents, segmenting the text content using semantic segmentation to form knowledge blocks; verifying and archiving the formed knowledge blocks, and marking knowledge blocks that fail the verification as alarm knowledge blocks; converting the knowledge blocks of tables into knowledge blocks described in natural language using a large model; locating the original text and related context in the intermediate files based on the alarm knowledge blocks, and correcting the alarm knowledge blocks based on the large model.

[0006] According to the water conservancy knowledge structure extraction and verification method provided by the present invention, when the file to be extracted includes a PDF file, the method for extracting text from the PDF file includes: a reading step: reading the PDF file; an identification step: locating and dividing page elements; the types of page elements include formulas, text, and tables; an extraction step: selecting an extraction method according to the element type to extract the text content of the elements; and a backfilling step: reconstructing and backfilling the extracted text content to form an intermediate file of the PDF file; wherein the reading, identification, extraction, and backfilling steps form a parallel four-level pipeline architecture.

[0007] According to the water conservancy knowledge structure extraction and verification method provided by the present invention, when the file to be extracted also includes image files, the method for extracting text from the image files includes: identifying the image files in the same directory in the original database; sorting the image files according to preset rules based on the parameters of the image files, and generating a PDF file; and extracting the image files in the manner of processing PDF files.

[0008] According to the water conservancy knowledge structure extraction and verification method provided by the present invention, the text content of intermediate files is matched according to the text matching rules in the preset matching rule library to classify the intermediate files into rule-based files or non-rule-based files, including: classifying the successfully matched intermediate files into rule-based files; inputting the unmatched intermediate files into the large model for further judgment; if the judgment result of the large model is that it belongs to the rule-based file, generating a new text matching rule corresponding to the intermediate file and updating the original matching rule library.

[0009] According to the water conservancy knowledge structure extraction and verification method provided by the present invention, for non-regulatory documents, the text content is segmented using semantic segmentation to form knowledge blocks. The method includes: initializing a sliding window of a preset size and loading the text; scanning from the end of the sliding window forward; locating the segmentation point that meets preset conditions through a semantic analysis model; wherein the preset conditions include: the text before and after the segmentation point is a complete sentence; and the length of the text before the segmentation point is maximized within the window range. After determining the segmentation point, the text before the segmentation point is extracted to form a complete knowledge block, and the sliding window is moved to the current segmentation point for the next segmentation.

[0010] According to the water conservancy knowledge structure extraction and verification method provided by the present invention, the formed knowledge blocks are verified, and knowledge blocks that fail the verification are marked as alarm knowledge blocks. This includes: for regulatory documents, analyzing and verifying the content of the text titles to identify knowledge blocks that fail the verification and marking them as alarm knowledge blocks; and performing closure verification on HTML tags to mark knowledge blocks of tables that cannot close HTML tags as alarm knowledge blocks.

[0011] According to the water conservancy knowledge structure extraction and verification method provided by the present invention, the text matching rule is a regular expression.

[0012] The present invention provides a method and apparatus for structured extraction and verification of water conservancy knowledge, which innovatively combines traditional semantic analysis models and large-scale models. The traditional model is used to perform preliminary analysis, classification, and structured extraction and verification of water conservancy texts, and the large-scale model is used for further natural language description and verification. While reducing development costs, this method achieves rapid and accurate processing of complex water conservancy texts, effectively making up for the shortcomings of existing methods and significantly improving the efficiency and quality of water conservancy knowledge processing. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating the structured extraction and verification method for water conservancy knowledge provided by the present invention.

[0015] Figure 2 This is an overall architecture diagram of the water conservancy knowledge structure extraction and verification method provided by the present invention;

[0016] Figure 3 This is a pipeline timing diagram provided by the present invention;

[0017] Figure 4 This is a flowchart of the self-learning mechanism provided by the present invention;

[0018] Figure 5 This is a flowchart of semantic segmentation provided by the present invention;

[0019] Figure 6 This is a flowchart of the alarm verification process provided by the present invention;

[0020] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0023] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more.

[0024] The following is combined Figures 1-7 This invention describes the method and apparatus for extracting and verifying structured water conservancy knowledge provided in embodiments of the present invention.

[0025] Figure 1 This is a flowchart illustrating the structured extraction and verification method for water conservancy knowledge provided by the present invention. Figure 2 This is an overall architecture diagram of the water conservancy knowledge structure extraction and verification method provided by the present invention. See below for reference. Figure 1 and Figure 2 The process / modules of the present invention will be described.

[0026] This invention proposes an extraction technique based on model recognition and rule processing. It can extract each clause from the original document individually, extract tables as a whole and convert them into natural language, and perform semantic segmentation of long paragraphs to generate several small knowledge blocks. Finally, it outputs these knowledge blocks in a structured format as a knowledge JSON file. This invention forms an intelligent extraction tool that combines domain knowledge and an automated verification mechanism suitable for this application scenario. The extraction tool includes three modules in its processing stage: a text extraction module, a structuring module, and an intelligent processing and verification module. The modules work sequentially, starting from the input original document and ultimately outputting a structured knowledge JSON file divided according to knowledge blocks.

[0027] Part 1: Text Extraction Process / Module

[0028] This section on text extraction can target data from a database of original water resources knowledge documents. The database is manually maintained, with a data administrator periodically adding new water resources knowledge documents. The administrator places the documents into different paths based on their subject matter. This section can be implemented using steps 101 (coarse processing unit) and 102 (cleaning and classification unit).

[0029] Step 101: Use different text processing methods to extract text from files of different formats to form intermediate files.

[0030] Step 101 processes files in the database, including PDFs, Word documents, Excel spreadsheets, and images, achieving full coverage of mainstream document formats and supporting automatic normalization of mainstream encodings such as GBK and UTF-8. Therefore, in the initial processing, this invention employs different processing methods for different file types, and finally stores the extracted plain text in a text file format (such as a TXT file). Detailed processing methods are as follows:

[0031] (1) PDF file

[0032] When the file to be extracted includes a PDF file, the method for extracting text from the PDF file includes: a reading step: reading the PDF file; a recognition step: locating and dividing page elements; the types of page elements include formulas, text, and tables; an extraction step: selecting an extraction method based on the element type to extract the text content of the elements; and a backfilling step: reconstructing and backfilling the extracted text content to form an intermediate PDF file; wherein the reading, recognition, extraction, and backfilling steps form a parallel four-level pipeline architecture.

[0033] As an optional embodiment, specifically, the PDF intelligent parsing engine of the present invention adopts a four-stage pipeline architecture of "file reading → layout recognition → formula / table / text extraction → backfilling and reconstruction". The final output of the PDF is a plain text file that closely resembles the original PDF layout, with tables stored in HTML format and formulas in LaTeX format, referred to as the intermediate file.

[0034] During the file reading phase, the binary stream parsing function of the PyMuPDF library, combined with multi-threaded block loading technology, enables memory-mapped reading of large files. In the layout recognition phase, a segmentation strategy based on the DocLayoutYOLO model is used to locate and divide page elements. Formula extraction utilizes the visual-speech and language alignment mechanism of the UniMERNet model to generate LaTeX, table extraction employs the Struct_EQTABLE model to recognize complex structures and output HTML-formatted text, and text extraction deploys the PaddleOCR enhanced network. All of these utilize mature open-source solutions. In the backfilling and reconstruction phase, a spatial-semantic joint encoder is used to inject the outputs of each modality into the document structure reconstruction module, achieving coordinate alignment and logical flow reconstruction based on the initial layout topology tree.

[0035] The unique feature of this invention lies in its creative and efficient integration of multiple open-source models into a four-stage pipeline, as illustrated in the timing diagram. Figure 3 As shown, Figure 3 This is a pipeline timing diagram provided by the present invention, which significantly improves efficiency compared to traditional serial processing. Through process connection and module concurrency, it realizes intelligent and efficient decomposition of PDF files.

[0036] (2) Use a document processor to extract text. Its paragraph segmentation function provides convenience for subsequent text structure recognition and reconstruction, and outputs a txt file.

[0037] (3) xls spreadsheet file

[0038] This program uses pandas to automatically read plain table files and automatically identifies table header content and main / subtitles using preset rules. For each row of the table, it extracts the cell content of each column corresponding to the header and matches it with the header. Based on a descriptive text template, which is `In the table {title}, for {first field of a row}, its {header column 1} is {the content of the corresponding cell in that row}, ...`, it performs formatted output.

[0039] The results of parsing this type of file can skip the structured extraction and verification steps and be directly output as structured text.

[0040] (4) Image files

[0041] When the file to be extracted from also includes image files, the method for extracting text from the image files includes: identifying the image files in the same directory as the original database; sorting the image files according to preset rules based on the parameters of the image files, and then generating a PDF file; and extracting the image files in the same way as processing PDF files.

[0042] As an optional implementation, specifically, by searching for image files in the same search directory, the images are precisely sorted based on file metadata (such as shooting time, creation time, modification time, etc.) or feature information in the filename (such as serial number, timestamp, etc.) to ensure that the image order conforms to the content logic. With the help of Pillow, the sorted image files are automatically combined to generate a PDF file according to the predetermined order, and the subsequent processing steps are the same as those for the PDF file in (1).

[0043] Step 102: Match the text content in the intermediate file according to the text matching rules in the preset matching rule library (self-learning rule library) to classify the intermediate file into rule-based files or non-rule-based files.

[0044] This step categorizes the input files into "rule-based files" and "non-rule-based files" based on their text content, corresponding to different parsing and storage methods in the subsequent structured processing module. This step innovatively combines the characteristics and unique structure of water conservancy documents with rule filtering and semantic analysis techniques to achieve accurate and efficient text classification.

[0045] Specifically, the organizational characteristics of rule-based documents are that they have regular title formats, such as “Article 1”, “Article (1)”, “3.1.1”, “I.”, etc., but the specific format of different document titles is unknown. Therefore, this invention proposes to use a large model-assisted rule base to dynamically learn the classification process of rule-based documents. Figure 4 This is a flowchart of the self-learning mechanism provided by the present invention. The specific self-learning process is as follows: Figure 4 As shown:

[0046] First, maintain a local rule base (i.e., a preset matching rule base / self-learning rule base) containing text matching rules for mainstream title formats (i.e., regular expressions required to match different formats), such as matching the formats "item 5", "3.1.1", and "item 16".

[0047] Subsequently, for the currently processed file, file samples are randomly extracted and matched using expressions from the local rule base. The following rules are used to determine whether a file is a rule-based file: (Rule base example follows).

[0048] if'r\d{1,2}\s*\.\s*\d{1,2}\s*' matches

[0049] elifr'7\s*\d{1,2}(?!\.\d)' matches

[0050] elifr'Article [1,2,3,4,5,6,7,8,9,10]{1,2}' matches

[0051] ……

[0052] return True

[0053] If none of the text matching rules in the local rule library can determine the file as a regulatory document, then further use the large model for judgment. Specifically, input the sample content to the large model with the following prompt:

[0054] "Please strictly process the input text content according to the following rules:

[0055] 1. **Input analysis**: Scan the text `{text}` to detect whether there are hierarchical structured headings (e.g., numeric numbering "1.1", Chinese serial number "Section 1", alphabetic numbering "A.2", etc.).

[0056] 2. **Matching rules**:

[0057] - If there is a tree structure, extract the lowest-level heading pattern (e.g., "2.3.1" corresponds to "\d+\.\d+\.\d+"), and generate its regular expression template (covering all variants of the same-level headings).

[0058] - If there is no clear hierarchical structure, directly return "No".

[0059] 3. **Output format**:

[0060] - When there is a structure, return in JSON format:

[0061] ```json

[0062] {"result": "Yes", "regex": "Lowest-level regular template"}

[0063] ```

[0064] - When there is no structure, return:

[0065] ```json

[0066] {"result": "No"}

[0067] ```

[0068] 4. **Examples**:

[0069] - Input: "1.1 Overview\n1.2 Methods\n2.1 Experiments" → Output: `{"result": "Yes", "regex": "\\d+\\.\\d+"}`

[0070] Input: "This is a paragraph text" → Output: `{"result": "No"}`

[0071] If the result of the large model's judgment is a rule-type file, the output of the large model will include a regular expression that matches the corresponding format, and this regular expression will be added to the local rule library.

[0072] After the above steps are completed, the file with a typical structured entry format is identified as a rule / regulation file.

[0073] For example, the text sample of "Specification for Manufacturing, Installation and Acceptance of Steel Gates for Hydropower Projects (GBT14173-2008)" states: "3.1.1 Before manufacturing gates and embedded parts, the following materials shall be available: a) Design drawings, construction drawings and technical documents. The design drawings shall include the general drawing of the gate and embedded parts, and the construction drawings shall include the assembly drawing and parts drawings of the gate and embedded parts. 3.1.2..."

[0074] Part Two: Structured Processes / Modules

[0075] This section's function is to select different modules for processing based on the text extraction module's results and the file category, ultimately outputting structured knowledge blocks and combining them into a JSON file. This section can be achieved through steps 103 and 104.

[0076] Step 103: For rule-type documents, identify the text content based on the hierarchical heading regular expression, and form a knowledge block from the content of the items under the lowest level heading; for the HTML format text corresponding to the table content in the intermediate documents, generate knowledge blocks; for non-rule-type documents, use semantic segmentation to segment the text content to form knowledge blocks.

[0077] Step 104: Verify and archive the formed knowledge blocks, and mark any knowledge blocks that fail the verification as alarm knowledge blocks.

[0078] The specific implementation methods for steps 103 and 104 are as follows:

[0079] (1) Smart extraction steps for regulations (short and standard intelligent extraction unit)

[0080] For files identified as regulations by the previous module, the module will output a regular expression for parsing the regulation headings. In this step, this regular expression is used to identify the entire text and organize the content of the items under the lowest-level headings into a knowledge block.

[0081] Example: Detected 4.1, 4.1.1, 4.1.2, 4.2, 4.2.1, 4.2.2

[0082] The content under headings 4.1.1, 4.1.2, 4.2.1, and 4.2.2 will be organized into four knowledge blocks.

[0083] For rule-based documents, by analyzing and verifying the content of the text titles, this invention can identify knowledge blocks with abnormalities and mark them as alarm knowledge blocks:

[0084] Specifically, during the execution of the extraction module, the extracted title sequence number can be verified, and such anomalies will be captured. The following situations will be considered as abnormal extraction results and will be output as alarm logs.

[0085] The subsequent intelligent processing and verification modules will then perform verification and correction.

[0086] 1) Incorrect clause numbers, such as “3.1.>”, “.2.3”, etc.

[0087] 2) Adjacent numbers within the same chapter or section do not strictly increase by +1. For example, "3.1.2" is followed directly by "3.2.3".

[0088] (2) Table extraction steps (table extraction unit)

[0089] In the first part, the tables in the file will be identified and extracted into standard HTML format. This invention stores each table as a separate JSON knowledge block.

[0090] Therefore, at this stage, the HTML tags of the tables that appear can be closed and matched, and the results can be temporarily stored as intermediate results. An alarm can be given for tables whose HTML tags cannot be closed (marked as an alarm knowledge block), and the tables can be verified and corrected in the subsequent intelligent processing and verification modules.

[0091] (3) Text natural segmentation steps (text natural segmentation unit)

[0092] For non-rule-based documents, semantic segmentation is used to divide the text content into knowledge blocks. This includes: initializing a sliding window of a preset size and loading the text; scanning from the end of the sliding window forward; and locating segmentation points that meet preset conditions using a semantic analysis model. The preset conditions include: the text before and after the segmentation point is a complete sentence, and the length of the text before the segmentation point is maximized within the window range. After determining the segmentation point, the text before the segmentation point is extracted to form a complete knowledge block, and the sliding window is moved to the current segmentation point for the next segmentation.

[0093] As an optional embodiment, Figure 5 This is a flowchart of semantic segmentation provided by the present invention, such as... Figure 5As shown, a sliding window of initial size k (in practice, k is set to 500) is first used. Initially, text of length k is input into the sliding window. Starting from the end of the window, the `sentence_split` method of the LTP model set is called to find the text segmentation point. This method guarantees:

[0094] 1. The sentences before and after the split point are complete statements.

[0095] 2. The length of the text before the split point should be as large as possible within the range of k values.

[0096] After each segmentation is completed, the preceding text within the sliding window is output as a complete knowledge block, and the sliding window is moved to the current segmentation point to proceed with the next segmentation.

[0097] Finally, each knowledge block is output as a structured JSON string, for example:

[0098] {

[0099] "id": "Notice on Clarifying the List and Scope of Rivers (River Sections) and Lakes for Which the Yangtze River Water Resources Commission is Responsible for Reviewing and Signing Water Project Construction Planning Approval Letters (Trial Implementation)_Water Planning and Design No. 2010175_3",

[0100] "type": "Natural Partition",

[0101] "file_title": "Notice on Clarifying the List and Scope of Rivers (River Sections) and Lakes for Which the Yangtze River Water Resources Commission is Responsible for Reviewing and Signing Water Project Construction Planning Approval Letters (Trial Implementation)_Water Planning and Design No. 2010175",

[0102] "name": "Notice on Clarifying the List and Scope of Rivers (River Sections) and Lakes for Which the Yangtze River Water Resources Commission is Responsible for Reviewing and Signing Water Project Construction Planning Approval Letters (Trial Implementation)_Water Planning and Design No. 2010175_3",

[0103] IV. The Yangtze River Water Resources Commission and the relevant provincial, autonomous region, and municipal water administrative departments shall, in accordance with the requirements of the Ministry of Water Resources Order No. 31 on the "Trial Implementation of the Management Measures for the Approval System of Water Project Construction Planning" and the General Office of the Ministry of Water Resources Notice on Conscientiously Implementing the "Trial Implementation of the Management Measures for the Approval System of Water Project Construction Planning" (Office Regulation No. 20087), implement the water project construction planning approval system in accordance with the law, and organize the formulation of detailed implementation rules for the "Trial Implementation of the Management Measures for the Approval System of Water Project Construction Planning." The Yangtze River Water Resources Commission shall closely cooperate and coordinate with the relevant provincial, autonomous region, and municipal water administrative departments to comprehensively strengthen the effective management of the implementation of the water project construction planning approval system.

[0104] }

[0105] Part Three: Intelligent Processing and Verification Flow / Module

[0106] This section is mainly implemented by step 105 (table intelligent processing unit) and step 106 (alarm content verification unit).

[0107] Step 105: Use a large model to transform the knowledge blocks in the table into knowledge blocks described in natural language.

[0108] In the structured module, the table content is extracted into standard HTML text, containing numerous HTML formatting symbols, and the text is organized by rows and columns, resulting in unclear natural semantic representation. Therefore, this invention uses a large model to provide a natural language description of the table content. Specifically, the following prompt words are used to input the large model:

[0109] Problem Description: Based on the content and logic of the input HTML table, use accurate and specific language to summarize the table into a complete descriptive text without omitting any information.

[0110] Input text: `{text}`

[0111] Requirements: The language should be concise and no more than `{length}` words; paragraphs are not allowed; sentences unrelated to the table content are not permitted in the reply; please try your best to correct typos.

[0112] The output of the large model is used as the modified knowledge block and then re-archived.

[0113] Step 106: Locate the original text and related context in the intermediate file based on the alarm knowledge block, and revise the alarm knowledge block based on the large model.

[0114] In practice, the structured module will output alarm knowledge blocks up to 5%. This invention processes alarms using a large model, as shown in Figure 6. Figure 6 Here is the alarm verification flowchart provided by this invention:

[0115] First, based on the alarm file name and entry information, locate the corresponding original text and relevant context in the corresponding intermediate file, and input it into the large model according to the following prompts:

[0116] Please strictly follow these rules when processing the input text content:

[0117] 1. **Input Analysis:** Scan the text `{text}`, locate and extract the corresponding item content under the structured title `{item}`.

[0118] 2. **Output Requirements**:

[0119] Because the text content may be corrupted (e.g., the title is corrupted and becomes "1.2.<", "1.2.", "1.3"), please use your reasoning to correctly reconstruct the structured title based on the context (e.g., if 1.2.1 and 1.2.3 appear, the text between the two titles likely belongs to 1.2.2), and accurately locate the relevant text to output the correctly reconstructed title and its corresponding content. If it is determined that there is no content corresponding to the title, return "No".

[0120] 3. **Output Format**:

[0121] - Returns results in JSON format:

[0122] json

[0123] {"result": "Yes", "item": "Correctly reconstructed article title", "text": "Content corresponding to the title"}

[0124] ```

[0125] - Returns the following if no result is found:

[0126] json

[0127] {"result": "No"}

[0128] ```

[0129] 4. **Example**:

[0130] - Input: text="1.1this is a item 1.22this is a secondary item 1.3 endof item", item='1.22' → Output: {"result": "is", "item": "1.2","text": "this is a secondary item"}

[0131] Input: text="This is a paragraph text" item="1.22' → Output: `{"result": "No"}`

[0132] The output of the large model is used as the modified knowledge block and then re-archived.

[0133] Experiments revealed that this invention processes PDF and image files at an average speed of approximately 5 seconds per page, and DOC and XLS files at the millisecond level. In current practice, approximately 350 files were processed, resulting in about 30,000 knowledge blocks, with a total processing time of approximately 5 hours. Based on current practical experience, it is estimated that the same workload would require approximately 5 months for 5 experienced and skilled personnel to complete manually, line by line.

[0134] In summary, the water conservancy knowledge structure extraction and verification method and apparatus provided by this invention innovatively combines traditional semantic analysis models and large-scale models. The traditional model performs preliminary parsing, classification, and structure extraction and verification of water conservancy texts, while the large-scale model is used for further natural language description and verification. This reduces development costs while achieving rapid and accurate processing of complex water conservancy texts, effectively making up for the shortcomings of existing methods and significantly improving the efficiency and quality of water conservancy knowledge processing.

[0135] The water conservancy knowledge structure extraction and verification device provided by the present invention includes: a text extraction module, a structure module, and an intelligent processing and verification module; the text extraction module includes a coarse processing unit and a cleaning and classification unit; the structure module includes a rule intelligent extraction unit, a table extraction unit, and a text natural segmentation unit; the intelligent processing and verification module includes a table intelligent processing unit and an alarm content verification unit.

[0136] The coarse processing unit is used to extract text from files of different formats using different text processing methods to form intermediate files;

[0137] The cleaning and classification unit is used to match the text content in the intermediate file according to the text matching rules in the preset matching rule library, so as to classify the intermediate file into a rule-based file or a non-rule-based file.

[0138] The intelligent extraction unit for rule-based documents identifies text content based on hierarchical heading regular expressions and forms a knowledge block from the content of items under the lowest-level heading; the table extraction unit for table files generates knowledge blocks based on intermediate files of the table files; the natural text segmentation unit for non-rule-based documents segments text content using semantic segmentation to form knowledge blocks.

[0139] The text extraction module is also used to verify and archive the formed knowledge blocks, and to mark the knowledge blocks that fail the verification as alarm knowledge blocks;

[0140] The table intelligent processing unit is used to transform the knowledge blocks of the table into knowledge blocks described in natural language using a large model;

[0141] The alarm content verification unit is used to locate the original text and related context in the intermediate file based on the alarm knowledge block, and to correct the alarm knowledge block based on the large model.

[0142] It should be noted that the water conservancy knowledge structure extraction and verification device provided in this embodiment of the invention can execute the water conservancy knowledge structure extraction and verification method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0143] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions from the memory 730 to execute a method for extracting and verifying structured water conservancy knowledge.

[0144] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the water conservancy knowledge structure extraction and verification method provided in the above embodiments.

[0146] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the water conservancy knowledge structure extraction and verification method provided in the above embodiments.

[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for structured extraction and verification of hydrological knowledge, characterized in that, The method comprises the following steps: Different text processing methods are used to extract text from files of different formats to form intermediate files; According to the text matching rules in the preset matching rule library, the text content of the intermediate file is matched to classify the intermediate file into a table rule type file or a non-table rule type file; For table rule type files, the text content is identified according to the hierarchical title regular expression, and the entry content under the lowest level title forms a knowledge block; for the html format text corresponding to the table content in the intermediate file, a knowledge block is generated; for non-table rule type files, the text content is segmented by using semantic segmentation to form a knowledge block; The formed knowledge blocks are tested and archived, and the knowledge blocks that fail the test are marked as alarm knowledge blocks; The knowledge blocks of the table are converted into natural language description knowledge blocks by using a large model; The original text and related context of the alarm knowledge block in the intermediate file are located, and the alarm knowledge block is corrected based on the large model; According to the text matching rules in the preset matching rule library, the text content of the intermediate file is matched to classify the intermediate file into a table rule type file or a non-table rule type file, which comprises: Divide the intermediate file that matches successfully into a table rule type file; Input the intermediate file that fails to match into a large model for further judgment: In the case that the judgment result of the large model is a table rule type file, generate a new text matching rule corresponding to the intermediate file, and update the original matching rule library.

2. The method for hydrologic knowledge structuring extraction and verification according to claim 1, characterized in that, In the case that the file to be subjected to text extraction includes a PDF file, the text extraction method for the PDF file comprises: Reading step: reading the PDF file; Identification step: positioning and dividing the page elements; the types of the page elements include formulas, texts and tables; Extraction step: selecting an extraction method according to the element type to extract the text content of the element; Backfilling step: reconstructing and backfilling the extracted text content to form an intermediate file of the PDF file; The reading, identification, extraction and backfilling steps form a four-stage pipeline architecture in parallel.

3. The method for hydrologic knowledge structuring extraction and verification of claim 2, wherein, In the case that the file to be subjected to text extraction also includes a picture file, the text extraction method for the picture file comprises: Determine the picture files under the same directory in the original database; According to the parameters of the picture files, sort the picture files according to the preset rules to generate a PDF file; Extract the picture file according to the method of processing the PDF file.

4. The method for hydrologic knowledge structuring extraction and verification of claim 1, wherein, For non-table rule type files, the text content is segmented by using semantic segmentation to form a knowledge block, which comprises: Initialize a sliding window of a preset size and load the text, scan from the tail of the sliding window to the front, and locate the segmentation point that meets the preset condition by using a semantic analysis model; wherein the preset condition includes: the text before and after the segmentation point is a complete sentence, and the length of the text before the segmentation point is maximized within the window range; After determining the segmentation point, the text before the segmentation point is extracted to form a complete knowledge block, and the sliding window is moved to the current segmentation point for the next segmentation.

5. The method for hydrologic knowledge structuring extraction and verification of claim 1, wherein, The formed knowledge blocks are tested, and the knowledge blocks that fail the test are marked as alarm knowledge blocks, which comprises: For the rule type file, the content of the text title is analyzed and tested to determine the knowledge block that is tested abnormally, and the knowledge block is marked as an alarm knowledge block; and The html tag is closed and tested, and the knowledge block of the table that cannot be closed is marked as an alarm knowledge block.

6. The method for hydrologic knowledge structuring extraction and verification of claim 1, wherein, The text matching rule is a regular expression.

7. A water knowledge structured extraction and verification apparatus, characterized by, The method for realizing the water conservancy knowledge structured extraction and verification method as claimed in any one of claims 1 to 6 comprises a text extraction module, a structured module and an intelligent processing and verification module; the text extraction module comprises a rough processing unit and a cleaning and classification unit, the structured module comprises a rule intelligent extraction unit, a table extraction unit and a text natural segmentation unit, and the intelligent processing and verification module comprises a table intelligent processing unit and an alarm content verification unit; The rough processing unit is configured to extract texts from files in different formats by using different text processing methods to form intermediate files. The cleaning and classification unit is configured to match the text content of the intermediate files according to the text matching rules in the preset matching rule library to classify the intermediate files into rule type files or non-rule type files. The rule intelligent extraction unit is configured to, for the rule type file, identify the text content according to a hierarchical title regular expression, and form a knowledge block from the content of the entries under the lowest level title. The text extraction module is further configured to test and archive the formed knowledge blocks into a database, and mark the knowledge blocks that are tested abnormally as alarm knowledge blocks. The table intelligent processing unit is configured to convert the knowledge blocks of the table into natural language description knowledge blocks by using a large model. The alarm content verification unit is configured to locate the original text and related context of the alarm knowledge block in the intermediate file, and correct the alarm knowledge block based on a large model.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the water conservancy knowledge structured extraction and verification method as claimed in any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the water conservancy knowledge structured extraction and verification method as claimed in any one of claims 1 to 6.

Images