A method for standardization processing and quality pretreatment control of a water conservancy field corpus

Abstract

This invention discloses a method for standardizing and controlling the quality preprocessing of corpora in the field of water conservancy, comprising the following steps: determining the scope of water conservancy disciplines and the coverage of the corpus; selecting authoritative journal articles as the source of the corpus; statistically analyzing the annual distribution, length, chart density, and high-frequency vocabulary features of the corpus to obtain original PDF documents in the field of water conservancy; parsing and extracting structured data from the original PDF documents to generate structured data; standardizing the structured data; performing automated quality detection on the cleaned corpus, calculating a comprehensive quality score, and filtering the corpus according to a preset threshold; and dividing the corpus into mutually exclusive training, validation, and test sets based on a time segmentation strategy according to the publication year, and outputting a reproducible training dataset in JSONL format. This invention can achieve fully automated conversion from original PDF water conservancy documents to high-quality, structured, standardized, and reproducible training corpora, providing reliable data support for large-scale vertical models in water conservancy.

Description

Technical Field

[0001] This invention relates to the field of intelligent water conservancy knowledge service technology, and in particular to a method for standardization processing and quality preprocessing control of corpora in the field of water conservancy. Background Technology

[0002] In large-scale model research targeting vertical domains, data quality is always the fundamental factor determining the model's professional capabilities and output reliability. Unlike general texts, water conservancy literature typically contains a large number of technical terms, unit symbols, formulas, charts, and complex chapter structures. The original corpus documents are mostly saved in PDF format, which is difficult to process. If such documents are directly used for model training, problems such as text misordering, OCR (Optical Character Recognition) errors, duplicate content, and inconsistent expression will enter the corpus, thereby affecting the model's learning effect on domain knowledge.

[0003] Currently, existing general document parsing methods face many challenges when processing literature in the field of water conservancy: First, the page layout is complex, with mixed single / double column layouts and serious interference from headers and footers; second, it is difficult to accurately extract formulas and figures; and third, the expression of professional terms and units is inconsistent. As large-scale model research gradually shifts from "model-driven" to "data-model collaborative driving," the standardization, traceability, and reproducibility of the corpus construction process have become important standards for measuring research quality. However, defects such as reference parsing errors, inability to quantify quality assessment, non-reproducibility of the processing process, and lack of data traceability cannot meet the data requirements for continued pre-training and multi-task supervised fine-tuning of large-scale vertical water conservancy models. Summary of the Invention

[0004] The purpose of this invention is to provide a method for the standardized processing and quality preprocessing control of corpora in the field of water conservancy. This method can realize the fully automated conversion from raw PDF water conservancy documents to high-quality, structured, standardized, and reproducible training corpora, providing reliable data support for large-scale vertical models of water conservancy.

[0005] The above-mentioned technical objective of the present invention is achieved through the following technical solution: A method for standardization and quality preprocessing control of corpora in the field of water conservancy includes the following steps: S1. Selection and Feature Analysis of Corpus Sources in the Water Conservancy Field: Determine the scope of water conservancy disciplines and the coverage of corpus, select authoritative journal articles as corpus sources, and statistically analyze the annual distribution, length, chart density and high-frequency vocabulary characteristics of the corpus to obtain original PDF documents in the water conservancy field. S2. Complex document parsing and extraction: Parse and extract structured data from the original PDF document to generate structured data. The parsing process adopts a hybrid strategy, namely: first, use a general PDF parsing tool for preliminary extraction, and then use a post-processing correction strategy based on rules and learners for domain-specific correction. S3. Corpus standardization processing: Perform corpus standardization processing on the structured data, including structure verification, text denoising, duplicate content filtering, unit standardization and citation repair, to generate cleaned corpus; S4. Quality Inspection: Perform automated quality inspection on the cleaned corpus, calculate the comprehensive quality score, and filter the corpus according to a preset threshold. The overall quality score Q is calculated based on a weighted average of six indicators: parsing availability, format compliance rate, unit consistency score, duplicate rate penalty, semantic integrity, and reference resolution rate. S5. Dataset Partitioning: Based on the time segmentation strategy, the corpus is divided into mutually exclusive training, validation and test sets according to the publication year, and a reproducible training dataset is output in JSONL format.

[0006] Furthermore, in S1, the corpus in the field of water conservancy includes at least academic literature on hydrology and water resources, water conservancy and hydropower engineering, farmland water conservancy and irrigation and drainage, soil and water conservation, water environment and water ecology management, water disaster prevention, water conservancy project operation and management, and digital twin water conservancy and water conservancy informatization.

[0007] Furthermore, in S2, the structured extraction is based on a predefined JSON Schema-like organization, whose fields include: document number, title, author, author affiliation, abstract, keywords, chapter tree, references, illustrations, and tables.

[0008] Furthermore: In S2, the document structure recognition includes: Chapter recognition: By combining heuristic methods based on typesetting rules with classification methods based on feature learning, the title hierarchy and content boundaries are restored by recognizing the font size, bolding style, numbering prefix and contextual semantics of the title, thus forming a chapter tree; Paragraph segmentation: Combining syntactic markers and layout features, it identifies sentence-end punctuation, line breaks, and paragraph indentation patterns. For text breaks caused by two-column layout or page breaks, it uses a context similarity judgment strategy to merge them.

[0009] Furthermore, in S2, the processing methods for charts, formulas, and references include: Formula processing: Identifying formula areas through layout analysis, prioritizing the use of mathematical formula recognition tools to convert them into LaTeX expressions, and performing syntax standardization; Table processing: Use table recognition tools to convert tables into structured data format and perform semi-automatic correction of tables spanning multiple pages or multi-level headers; Reference processing: A two-stage processing strategy is adopted. First, the original reference string is preserved in its entirety. Then, the structured fields are parsed out. Finally, the domain dictionary is used for a posteriori correction.

[0010] Furthermore: In S2, the success rate of parsing the original PDF document is defined as: ; in, Number of articles published in a year; OCR quality is defined as: in, For character error rate, , , ,and These represent the number of characters replaced, deleted, and inserted, respectively. This represents the total number of actual text characters.

[0011] Furthermore: In S3, the corpus normalization process includes: performing JSON structure verification and completion on the parsing results according to a preset schema; The rule-based matching method is used to filter out garbled characters and non-text information such as headers, footers, and page numbers generated by OCR. A paragraph-level similarity detection method is used to identify, merge, or delete duplicate paragraphs; Construct a unit mapping dictionary to uniformly convert engineering units into a standard representation; By combining rule matching and template recognition methods, reference entries are completed and standardized.

[0012] Furthermore: In S4, the quality inspection includes: Header and footer and duplicate paragraph detection: Determine and remove headers and footers based on text strings with fixed positions; detect and delete duplicate paragraphs based on normalized hash and Jaccard similarity. Garbled text and out-of-order detection: Garbled text is determined based on the proportion of non-CJK characters and the perplexity of the language model; out-of-order detection is based on the cosine similarity mutation of paragraph embedding; Paragraph quality initial value calculation: Define character cleanliness With semantic coherence Initial paragraph quality value: .

[0013] Furthermore: In S4, the overall quality score is: ; in, P parse To resolve availability, P formatPformat To improve the format compliance rate, U For consistency scores, S For semantic completeness, R For reference resolution rate, D For the repetition rate penalty term, the weight coefficients are as follows: Non-negative and summing to 1, initial weights are set as follows: ; The preset threshold is: if If it is judged to be of high quality, it can be directly put into the warehouse; if If it is placed in the low-quality pool, manual review or targeted automatic correction is recommended; if Then temporarily remove or retain only the metadata for manual recovery.

[0014] In summary, the present invention has the following beneficial effects: Firstly, this invention can achieve fully automated conversion from raw PDF water conservancy documents to high-quality, structured, standardized, and reproducible training corpora, providing reliable data support for large-scale vertical water conservancy models. Secondly, this invention adopts a method that combines "preliminary analysis using general tools with post-processing correction using rules" to structurally restore the content such as titles, abstracts, main text, figures, formulas, and references, thereby improving the completeness and usability of academic literature analysis. Thirdly, this invention combines noise filtering, field normalization, and quality scoring mechanisms to effectively eliminate low-quality samples, ultimately forming a large-scale, high-quality, and traceable corpus asset in the water conservancy field. This provides reliable corpus support for subsequent pre-training, supervised fine-tuning, and prompting optimization experiments in the field. Attached Figure Description

[0015] Figure 1 This is a flowchart of the operation of the present invention; Figure 2 This is a flowchart of the document parsing process of the present invention; Figure 3 This is a flowchart of the corpus standardization process of the present invention; Figure 4 This is a distribution chart of the annual publication volume of journals A and B in this invention. Figure 5 This is a graph showing the annual paper length and chart density of journal AB, which is related to the present invention. Figure 6 This is a graph showing the percentage of high-frequency words in the journal AB of this invention. Figure 7This is the corpus quality multidimensional graph of the present invention; Figure 8 This is a distribution chart of the overall quality score Q of this invention; Figure 9 This is a quality score distribution chart for each journal in this invention; Figure 10 This is a graph showing the relationship between the number of pages and the quality score of the journal of this invention; Figure 11 This invention compares key indicators of journal data before and after cleaning. Figure 12 This invention relates to the distribution of the improvement in the quality score of journal documents. Detailed Implementation

[0016] The present invention will be further described in detail below with reference to the accompanying drawings.

[0017] In the description of this invention, it should be understood that the terms "upper", "lower", "left", "right", "front", "rear", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0018] Example, refer to Figure 1 A method for standardization and quality preprocessing control of corpora in the field of water conservancy includes the following steps: S1. Selection and Feature Analysis of Corpus Sources in the Water Resources Field: Determine the scope of water resources disciplines and the coverage of corpus, select authoritative journal articles as corpus sources, including at least academic literature on hydrology and water resources, water conservancy and hydropower engineering, farmland water conservancy and irrigation and drainage, soil and water conservation, water environment and water ecology management, water disaster prevention, water conservancy project operation and management, and digital twin water conservancy and water conservancy informatization. Statistically analyze the annual distribution, length, chart density and high-frequency vocabulary characteristics of the corpus, and obtain original PDF documents in the water resources field. S2. Complex document parsing and extraction: Parse and extract structured data from the original PDF document to generate structured data. The parsing process adopts a hybrid strategy, namely: first, use a general PDF parsing tool for preliminary extraction, and then use a post-processing correction strategy based on rules and learners for domain-specific correction. Reference Figure 2 The document parsing process is divided into five stages: 1. Perform page-level layout analysis to detect and locate text blocks, image blocks, and table blocks on the page; 2. Semantically categorize various text blocks, such as bibliographic information, abstracts, main text, figure captions, table captions, and headers and footers; 3. Perform block-level splicing and column merging on the detection results to restore a linear text flow that conforms to the natural reading order; 4. Identify chapter titles and reconstruct the document's hierarchical structure; 5. Special content such as images, tables, formulas, and references are extracted and stored in a structured format. After PDF parsing, the proportion of documents that can be successfully parsed into machine-readable text and OCR are used as parsing quality evaluation indicators. The former represents whether the document can be successfully converted into machine text, and the latter represents the quality of the converted text. The success rate of parsing the original PDF document is defined as: ; in, Number of articles published in a year; OCR converts text in scanned PDFs or page images into machine-readable text, using character error rate as the basic evaluation metric. Let the number of characters replaced, deleted, and inserted between the actual text and the OCR output text be... , and , The total number of actual text characters. Character error rate is defined as: ; Based on this, OCR quality is defined as: ; The closer the value is to 1, the closer the recognition result is to the real text, and the higher the OCR quality. Structured extraction is based on a predefined JSON Schema-like organization, whose fields include: document number, title, author, author affiliation, abstract, keywords, chapter tree, references, illustrations and tables. This allows multiple types of information that are discretely distributed in the original document to be uniformly incorporated into a computable data framework, thereby enhancing the consistency and operability of subsequent processing. Document structure recognition includes: Chapter recognition: By combining heuristic methods based on typesetting rules with classification methods based on feature learning, the title hierarchy and content boundaries are restored by recognizing the font size, bolding style, numbering prefix and contextual semantics of the title, thus forming a chapter tree; Paragraph segmentation: Combining syntactic tags and page layout features, it identifies sentence-end punctuation, line breaks, and paragraph indentation patterns. For text breaks caused by double-column layout or page breaks, it uses a context similarity judgment strategy to merge them. The processing methods for charts, formulas, and references include: Formula processing: Identifying formula areas through layout analysis, prioritizing the use of mathematical formula recognition tools to convert them into LaTeX expressions, and performing syntax standardization; Table processing: Using table recognition tools to convert tables into structured data formats, and performing semi-automatic correction for cross-page tables or multi-level table headers; For images and captions: a strategy of "image resource preservation + caption association extraction" is adopted. First, the original image resources in the paper are saved. Then, the corresponding caption text is matched according to the page adjacency relationship to establish the correspondence between images and descriptions. For images containing coordinate axis marks, curve descriptions, or other key text information, OCR recognition is further performed on the image area, and the obtained text is stored as additional descriptive information of the image to improve the usability of image content in subsequent indexing and semantic understanding. For reference processing: a two-stage processing strategy is adopted. First, the original reference string is completely preserved. Then, the structured fields such as author, title, source, year, page number, and DOI are parsed out. Finally, a post-hoc correction is performed in combination with a domain dictionary. S3. Corpus Normalization Processing: Perform corpus normalization processing on the structured data (refer to...). Figure 3 The process includes structure verification, text denoising, duplicate content filtering, unit standardization, and citation repair, generating a cleaned corpus. The parsing results are all stored in JSONL format, including fields such as title, abstract, chapter content, and references. Structure verification includes: checking the field integrity of the parsed results according to the preset schema, focusing on verifying the existence and reasonable format of key fields such as title, abstract, sections and references; and using rule-based methods to complete missing fields that can be repaired. The rule-based matching method is used to filter out garbled characters, meaningless symbol sequences, and abnormal character combinations generated by OCR, while deleting non-text information such as headers, footers, and page numbers to improve the purity of the main text content. A unit mapping dictionary is constructed for common unit expressions (m³ / s, mm / d, kg / m³) in engineering literature, which unifies the engineering units into standard representations and improves the consistency and computability of the corpus; To address issues such as inconsistent reference formats and incomplete field recognition, a method combining rule matching and template recognition is used to complete and standardize reference entries. S4. Quality Inspection: Perform automated quality inspection on the cleaned corpus, calculate the comprehensive quality score, and filter the corpus according to preset thresholds; Parsing availability measures the degree to which key structural fields of a document are successfully parsed. Let the set of key fields in the document that need to be parsed be... This includes core fields such as title, abstract, and sections. The parsing availability is defined as follows: ; in This represents the set of fields that were successfully parsed. The value range is [0,1]. A higher value indicates a more complete parsing of the document structure.

[0019] To ensure the structural consistency of the corpus, this paper uses JSON schema to perform structural validation on the parsed results. Let the number of fields that pass the schema validation be... The total number of fields is The format compliance rate is defined as follows: ; Used to evaluate whether the parsing results conform to a predefined data structure.

[0020] Unit Consistency score is used to standardize units to ensure the computability of numerical information. Let the number of successfully identifiable and standardized numerical units in the document be . The total number of units is The unit consistency score is defined as follows: ; The higher this indicator is, the more standardized the expression of engineering parameters in the document.

[0021] The duplicate penalty term is introduced to reduce the impact of noise caused by issues such as paragraph repetition, header / footer interference, or text splicing errors during the parsing process on the quality of the corpus. ; in For the length of the repeated text, The length of the full text is used as a reference. The higher the repetition rate, the greater the penalty for the quality score.

[0022] Semantic completeness measures the degree of semantic coherence between paragraphs in a document. Assuming the document has a total of [number missing] paragraphs... There are 1 paragraph, and the semantic similarity between adjacent paragraphs is 1. Then semantic completeness is defined as: ; This metric can be obtained by calculating paragraph semantic similarity using a sentence vector model.

[0023] Reference parsing rate is an important component of academic literature, and its degree of structuring directly affects the effectiveness of knowledge network construction. Let the number of reference entries in a document that are successfully parsed into structured citations be... The total number of references is The reference resolution rate is then defined as: ; Based on the above indicators, the overall quality score is: ; in, P parse To resolve availability, P formatPformat To improve the format compliance rate, U For consistency scores, S For semantic completeness, R For reference resolution rate, D For the repetition rate penalty term, the weight coefficients are as follows: Non-negative and summing to 1, initial weights are set as follows: The preset threshold is: if If it is judged to be of high quality, it can be directly put into the warehouse; if If it is placed in the low-quality pool, manual review or targeted automatic correction is recommended; if Then temporarily remove or retain only the metadata for manual recovery.

[0024] Quality inspection includes: Header, footer, and repeated paragraph detection: If a text appears more than a threshold in the document and its position is fixed, it is marked as a header or footer and removed from the whole document or saved separately. Paragraph breaks caused by pagination can be judged by the semantic continuity between the end of the block boundary and the beginning of the next page block and merged into a single paragraph. Based on normalized hashing and Jaccard similarity or cosine similarity detection, duplicate text segments should be retained in one copy and other copies deleted. At the same time, the source page number should be recorded for manual review. Garbled text and out-of-order detection: On the one hand, the proportion of non-CJK characters and the frequency of rare characters in the paragraph are calculated. On the other hand, the average token-log-probability or perplexity of the sentence is evaluated using a word- or sub-word-based language model. When the perplexity of the sentence is much higher than the document mean and also contains abnormal characters, it is marked as garbled text or OCR error. Paragraph quality initial value calculation: Define character cleanliness With semantic coherence Initial paragraph quality value: ; Desirable -0.6, which is then included in the overall quality score along with other indicators; S5. Dataset partitioning: Based on the time segmentation strategy, the corpus is divided into mutually exclusive training, validation and test sets according to the publication year, and a reproducible training dataset is output in JSONL format. Time-based split: Dividing the corpus into three parts—training, validation, and testing—based on the year of publication; Source / Paper ID de-leakage: Ensure complete isolation based on paper ID, that is, all derived data (tables, figure captions, appendices) of the same paper appear in only one split. For multiple papers by the same author, especially in the case of "continuing research" or "the same project", stratified sampling by project or author can be considered to avoid information leakage. Topic / Domain Balancing: When allocating training / test sets, ensure that the topics are distributed across the splits with approximate coverage, or explicitly specify cross-domain test sets; In set theory terms, if If the set contains all documents, it should be divided into mutually exclusive sets. and ( ); The reproducible pipeline should be deterministic and documented in version control, including the following steps: data collection → parsing → cleaning → normalization → partitioning → exporting JSONL. Each step should generate checksums and logs to facilitate tracking which version produced which type of training data.

[0025] Setting up the experiment: 1. Select well-known and authoritative journals A and B as core corpus sources in the field of water conservancy, and conduct statistical analysis on their data scale and year distribution from 2006 to 2025. Calculate the annual publication volume, average word count per article, and average number of formulas / figures per article for each source.

[0026] Reference Figure 4 The two journals formed a relatively continuous and stable annual sample coverage during the research period, indicating that the constructed corpus can well support cross-year domain language modeling research.

[0027] Reference Figure 5Through corpus feature analysis, the average page count of both journals showed an overall upward trend. Journal A had a larger overall page count, indicating that it was more comprehensive in terms of content development and research expression. Although the latter had a shorter overall page count, it also showed a significant increase in recent years. In contrast, the graph density of Journal A fluctuated less overall, indicating that its graph usage was relatively stable. The graph density of Journal B showed more obvious phased changes, indicating that there are differences in the research expression methods of different journals.

[0028] Reference Figure 6 The analysis of high-frequency word proportions reveals that these words cover areas such as water conservancy projects, hydrology and water resources, and agricultural water management. Among these, "water resources" has the highest proportion at 3.2%, followed by "water conservancy projects" at 3%, and "irrigation" at 2.8%. This distribution of high-frequency words reflects both the engineering nature of knowledge in the water conservancy field and the important role of themes such as resource management, ecological protection, and risk prevention in the corpus.

[0029] 2. Analysis of Corpus Results Reference Figure 7 It can be seen that the overall parsability of the corpus was good during the study period, and the parsing success rate remained at a high level in most years, gradually approaching or exceeding the preset qualified and excellent lines. At the same time, the OCR quality also showed an overall improving trend, but there were still some fluctuations between different years. In other words, the year of publication not only affects the size of the corpus, but also indirectly affects the text parsing effect through the evolution of document format.

[0030] 3. Overall quality score distribution Reference Figure 8 As can be seen, the corpus quality scores are mainly concentrated in the 80-90 range, indicating that most documents have complete parsing structures and high text recognition quality, suggesting that the overall data quality is good and the distribution is biased towards the high-quality range.

[0031] 4. Differences in the quality of corpora from different journals Reference Figure 9 As can be seen, the average quality score of journal A is about 85.1, and the overall distribution is relatively concentrated, indicating that the journal has a standardized document structure and high parsing stability. The average score of journal B is about 75.8, which is slightly lower than that of journal A, and the distribution range is wider. The main reasons for this difference include differences in journal layout format, different scanning quality of historical documents, and the complexity of chart structure.

[0032] 5. Relationship between page count and quality score Reference Figure 10 It can be seen that there is a moderate positive correlation between the number of pages in a paper and the quality score. Overall, papers with more pages usually have a more complete chapter structure, more contextual information, and a more standardized reference structure.

[0033] 6. Analysis of Corpus Cleaning Results 6.1 Comparison of key indicators before and after cleaning Reference Figure 11 As can be seen, after data cleaning, all quality indicators showed significant improvement. The JSON parsing success rate increased from approximately 85% to nearly 100%; the unit standardization rate increased from 78% to 95%; the term frequency increased by approximately 26%; the abstract completeness rate increased by approximately 10%; and the reference completeness rate increased by approximately 22%. In addition, the average comprehensive quality score also increased from approximately 72 points to 82 points, an overall improvement of approximately 13.9%, further demonstrating that data cleaning can significantly improve the quality of the corpus.

[0034] 6.2 Analysis of the extent of improvement in corpus quality Reference Figure 12 As can be seen, the quality score improvement of most documents is concentrated in the 0-15 point range, with an average improvement of about 7.8 points. This indicates that for the vast majority of documents, the cleaning process can effectively repair parsing errors and improve the usability of the corpus. At the same time, the quality score improvement of some documents exceeds 20 points, mainly due to the successful repair of scanned PDFs with severe OCR noise; the removal of duplicate content in paragraphs; and a significant improvement in the success rate of structured parsing of references.

[0035] Therefore, this invention constructs a corpus processing workflow oriented towards model training, realizing the transformation from raw documents to high-quality structured corpus, ensuring the reliability of the corpus in terms of temporal continuity, disciplinary representativeness, and professional standardization, effectively eliminating low-quality samples, and ultimately forming a large-scale, high-quality corpus asset in the water conservancy field with good traceability.

[0036] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make inventive modifications to this embodiment as needed, but as long as they are within the scope of the claims of the present invention, they are protected by patent law.

Claims

1. A method for standardization processing and quality preprocessing control of corpora in the field of water conservancy, characterized in that, Includes the following steps: S1. Selection and Feature Analysis of Corpus Sources in the Water Conservancy Field: Determine the scope of water conservancy disciplines and the coverage of corpus, select authoritative journal articles as corpus sources, and statistically analyze the annual distribution, length, chart density and high-frequency vocabulary characteristics of the corpus to obtain original PDF documents in the water conservancy field. S2. Complex document parsing and extraction: Parse and extract structured data from the original PDF document to generate structured data. The parsing process adopts a hybrid strategy, namely: first, use a general PDF parsing tool for preliminary extraction, and then use a post-processing correction strategy based on rules and learners for domain-specific correction. S3. Corpus standardization processing: Perform corpus standardization processing on the structured data, including structure verification, text denoising, duplicate content filtering, unit standardization and citation repair, to generate cleaned corpus; S4. Quality Inspection: Perform automated quality inspection on the cleaned corpus, calculate the comprehensive quality score, and filter the corpus according to a preset threshold. The overall quality score Q is calculated based on a weighted average of six indicators: parsing availability, format compliance rate, unit consistency score, duplicate rate penalty, semantic integrity, and reference resolution rate. S5. Dataset Partitioning: Based on the time segmentation strategy, the corpus is divided into mutually exclusive training, validation and test sets according to the publication year, and a reproducible training dataset is output in JSONL format.

2. The method for standardization and quality preprocessing control of corpora in the field of water conservancy according to claim 1, characterized in that: In S1, the corpus in the field of water conservancy includes at least academic literature on hydrology and water resources, water conservancy and hydropower engineering, farmland water conservancy and irrigation and drainage, soil and water conservation, water environment and water ecology management, water disaster prevention, water conservancy project operation and management, and digital twin water conservancy and water conservancy informatization.

3. The method for standardization and quality preprocessing control of corpora in the field of water conservancy according to claim 1, characterized in that: In S2, the structured extraction is based on a predefined JSON Schema-like organization, whose fields include: document number, title, author, author affiliation, abstract, keywords, chapter tree, references, illustrations, and tables.

4. The method for standardization and quality preprocessing control of corpora in the field of water conservancy according to claim 1, characterized in that: In S2, the document structure recognition includes: Chapter recognition: By combining heuristic methods based on typesetting rules with classification methods based on feature learning, the title hierarchy and content boundaries are restored by recognizing the font size, bolding style, numbering prefix and contextual semantics of the title, thus forming a chapter tree; Paragraph segmentation: Combining syntactic markers and layout features, it identifies sentence-end punctuation, line breaks, and paragraph indentation patterns. For text breaks caused by two-column layout or page breaks, it uses a context similarity judgment strategy to merge them.

5. The method for standardization and quality preprocessing control of corpora in the field of water conservancy according to claim 1, characterized in that: In S2, the processing methods for charts, formulas, and references include: Formula processing: Identifying formula areas through layout analysis, prioritizing the use of mathematical formula recognition tools to convert them into LaTeX expressions, and performing syntax standardization; Table processing: Use table recognition tools to convert tables into structured data format and perform semi-automatic correction of tables spanning multiple pages or multi-level headers; Reference processing: A two-stage processing strategy is adopted. First, the original reference string is preserved in its entirety. Then, the structured fields are parsed out. Finally, the domain dictionary is used for a posteriori correction.

6. The method for standardization and quality preprocessing control of corpora in the field of water conservancy according to claim 1, characterized in that: In S2, the success rate of parsing the original PDF document is defined as: ； in, Number of articles published in a year; OCR quality is defined as: in, For character error rate, , , ,and These represent the number of characters replaced, deleted, and inserted, respectively. This represents the total number of actual text characters.

7. The method for standardization and quality preprocessing control of corpora in the field of water conservancy according to claim 1, characterized in that: In S3, the corpus normalization process includes: performing JSON structure verification and completion on the parsing results according to a preset schema; The rule-based matching method is used to filter out garbled characters and non-text information such as headers, footers, and page numbers generated by OCR. A paragraph-level similarity detection method is used to identify, merge, or delete duplicate paragraphs; Construct a unit mapping dictionary to uniformly convert engineering units into a standard representation; By combining rule matching and template recognition methods, reference entries are completed and standardized.

8. The method for standardization and quality preprocessing control of corpora in the field of water conservancy according to claim 1, characterized in that: In S4, the quality inspection includes: Header and footer and duplicate paragraph detection: Determine and remove headers and footers based on text strings with fixed positions; detect and delete duplicate paragraphs based on normalized hash and Jaccard similarity. Garbled text and out-of-order detection: Garbled text is determined based on the proportion of non-CJK characters and the perplexity of the language model; out-of-order detection is based on the cosine similarity mutation of paragraph embedding; Paragraph quality initial value calculation: Define character cleanliness With semantic coherence Initial paragraph quality value: 。 9. The method for standardization and quality preprocessing control of corpora in the field of water conservancy according to claim 1, characterized in that: In S4, the overall quality score is: in, P parse To resolve availability, P formatPformat To improve the format compliance rate, U For consistency scores, S For semantic completeness, R For reference resolution rate, D For the repetition rate penalty term, the weight coefficients are as follows: Non-negative and summing to 1, initial weights are set as follows: ； The preset threshold is: if If it is judged to be of high quality, it can be directly put into storage; if If it is placed in the low-quality pool, manual review or targeted automatic correction is recommended; if Then temporarily remove or retain only the metadata for manual recovery.

Images