A Batch Structured Extraction Method for Plant Protection Literature Information Based on AI Large Models

Abstract

This invention discloses a method for batch structured extraction of literature information in the field of plant protection based on a large AI model, belonging to the field of computer data processing and artificial intelligence application technology. The method converts PDF documents into Markdown text using an adaptive preprocessing module and a Markdown conversion module. Then, a dual-strategy adaptive AI extraction module processes text using different methods based on a text length threshold, and combines this with a JSON data repair strategy to extract JSON data. Next, a two-stage parallel scheduling module processes CPU-intensive conversion tasks and I / O-intensive AI analysis tasks in parallel. Finally, a multi-dimensional aggregation and export module aggregates and generates structured database files according to predefined mapping rules. This invention achieves end-to-end automation from literature acquisition to structured data output, solving the problem of low efficiency in extracting information from manually read literature.

Description

Technical Field

[0001] This invention belongs to the field of computer data processing and artificial intelligence application technology, specifically relating to a method for batch structured extraction of literature information in the field of plant protection based on a large AI model. Background Technology

[0002] In plant protection and related agricultural sciences, a large number of research papers are published and widely disseminated in PDF format. The information contained in these papers, such as crop types, pest and disease names, environmental factors, experimental designs, and quantitative results, is a crucial data source for meta-analysis, knowledge graph construction, and decision support. However, PDFs are designed for visual presentation rather than semantic annotation. Chapters, tables, and figure captions often lose clear structural labels after being converted to page coordinates and text blocks, making it impossible to automatically and reliably restore the seemingly clear "chapter hierarchy." Diverse journal formats, two-column or multi-column layouts, and cross-page tables mixed with text and figures further weaken the effectiveness of coordinate-based or simple rule-based text reconstruction, resulting in uncontrollable quality of the raw input received by downstream extraction modules, directly restricting the accuracy and completeness of information extraction.

[0003] To improve the efficiency of automated processing, shallow methods such as OCR or regular expressions are often used in engineering to perform text recovery and field matching on PDFs. However, these methods are inherently deficient in semantic understanding and cross-paragraph association. The recognition accuracy of OCR is significantly affected by the scanning quality, embedded fonts, and the complexity of symbols. Rule matching is prone to misjudgment or missed detection when dealing with scientific texts with diverse terminology, synonym substitution, and contextual dependence. It cannot reliably associate discrete numbers and units with their semantic subjects, and it is also difficult to handle domain entities and quantitative indicators with contextual constraints. Therefore, shallow tools alone cannot meet the requirements of data quality and semantic accuracy in the field of plant protection.

[0004] Recent large language models have demonstrated strong capabilities in semantic understanding, but they still face engineering limitations when dealing with long academic texts and multi-field extraction tasks. The model's context window and the length of a single output constitute physical boundaries. Extremely long texts or texts with numerous fields often suffer from information omissions, field mismatches, or non-standard output formats. Furthermore, when models undertake complex structured tasks simultaneously, they are more prone to uncontrollable generation (illusions) or inconsistent formatting, leading to additional burdens on parsing and validation. While splitting long texts or querying them in stages can alleviate contextual pressure, it introduces the complexity of merging, deduplication, and consistency maintenance, increasing system engineering costs. In large-scale batch processing scenarios, these problems are amplified into challenges to system reliability and cost. Concurrency and scheduling, API rate limiting and retries, fault recovery and idempotency, and automatic correction of non-standard outputs are all issues that must be addressed to ensure stable batch processing operation. Compared to general scientific literature, plant protection papers often simultaneously include ecological observations, experimental designs, statistical analyses, and geographical / temporal information. Terminology exhibits regional and linguistic variations, and quantitative indicators frequently include units and confidence intervals. Therefore, this places higher demands on the spatial order and semantic hierarchy of text reconstruction, the embedding of domain knowledge, and the traceability of numerical data. In summary, there is an urgent need for a comprehensive solution that can both restore the original semantic structure of PDF documents and collaborate with large models, providing concurrency and fault tolerance guarantees at the batch processing level, to meet the agricultural research field's demand for high-quality structured data. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention aims to propose a batch structured extraction method for plant protection literature information based on a large AI model. The method first uses an adaptive preprocessing module and a Markdown conversion module to accurately convert PDF documents into Markdown text that preserves semantic structure. Then, a dual-strategy adaptive AI extraction module adaptively switches between a phased extraction strategy and an intelligent block extraction strategy based on text length, combined with a JSON data repair strategy, to extract structured JSON data. Finally, a two-stage parallel scheduling module and a multi-dimensional aggregation and export module efficiently integrate the scattered results and output them as a multi-dimensional structured database file. This invention achieves an end-to-end automated extraction process from unstructured PDF documents to a readily usable structured database, providing an efficient and accurate macro-data analysis foundation for research in the field of plant protection.

[0006] The technical solution adopted in this invention:

[0007] A method for batch structured extraction of literature information in the field of plant protection based on AI large-scale models is proposed. Based on a Python IDE platform, it achieves end-to-end automation from PDF documents to structured data output through a multi-stage, parallel processing flow. The processing flow includes the following:

[0008] S1 constructs an adaptive preprocessing module for PDF documents and a Markdown conversion module. PDF documents obtained from the document library are processed through intelligent PDF text extraction, deep text cleaning, combined with semantic structure recognition and paragraph reconstruction, and idempotent caching strategy, and are batch converted into Markdown format text that is readable by computers and retains the semantic hierarchy. This achieves accurate conversion from visual unstructured format to semantic structured format.

[0009] S2, a dual-strategy adaptive AI extraction module is constructed. Markdown-formatted text is input, and the module adaptively switches between a phased extraction strategy and an intelligent block extraction strategy based on a text length threshold. The phased extraction strategy uses several rounds of focused professional Prompt instructions to call the AI ​​model to extract basic metadata, domain entities, and methods / results, returning JSON formatted data. The intelligent block extraction strategy segments multiple text blocks using chapter boundaries in the Markdown-formatted text as semantic segmentation units, and dynamically matches focused professional Prompt instructions for each text block to extract JSON formatted data. Furthermore, a highly robust JSON formatted data repair strategy is incorporated to obtain parsable structured JSON formatted data, overcoming the technical limitations of AI models in handling long texts and complex multi-field extraction tasks.

[0010] S3. Construct a two-stage parallel scheduling module to concurrently process CPU-intensive conversion tasks and I / O-intensive AI analysis tasks. Construct a multi-dimensional aggregation and export module to aggregate the generated structured JSON format data according to predefined mapping rules, generate a multi-dimensional structured database file, and output an adaptive and localized format, realizing efficient integration from scattered extraction results to macro-analysis datasets.

[0011] Preferably, in step S1, the specific processing procedure for intelligent PDF text extraction is as follows:

[0012] Intelligent PDF Text Extraction Based on Physical Layout Analysis and Coordinate Space Clustering: Addressing the lack of logical structure in PDF documents, a "coordinate space inverse reconstruction" strategy is employed; details are as follows:

[0013] The "coordinate space reverse reconstruction" strategy includes metadata vectorization extraction, multi-column detection and topology sorting mechanism, and noise filtering based on spatial and frequency features;

[0014] a. Metadata Vectorization Extraction: Using a parsing package (such as the PyMuPDF library), traverse the PDF document to obtain all text blocks on the document page and construct feature vectors for each text block. To obtain a set of two-dimensional text blocks, where ( () represents the bounding box coordinates of the text block in the two-dimensional page space. Indicates font style, Indicates font size, Represents text content;

[0015] b. Multi-column detection and topological sorting mechanism: To solve the problem of disordered reading order caused by two-column or multi-column layouts on the page, a one-dimensional coordinate clustering algorithm is performed on the two-dimensional text block set; the distribution of blank gaps on the page in the horizontal direction is detected by projection analysis, thereby dynamically dividing the page into... Each column range is denoted as... Based on this, a hierarchical topological sorting rule is applied, prioritizing sorting by column index, with sorting within each column following a "top-down" approach. (Axis increments), from left to right () The geometric logic of "axis increasing" maps a discrete set of two-dimensional text blocks into a linear one-dimensional logical text stream;

[0016] c. Noise filtering based on spatial and frequency characteristics: Establish a dual verification mechanism of location and content; set a threshold area at the page edge. (Header / Footer area), and perform cross-page frequency analysis on text blocks within the page edge threshold area; for text blocks that appear repeatedly in fixed coordinate areas across multiple pages and whose content similarity is higher than the threshold. The text blocks were identified as layout noise and removed.

[0017] Preferably, in step S1, deep text cleaning includes, but is not limited to:

[0018] Deep text stream cleaning based on heuristic rules and conservative strategies: To preserve the original semantics to the greatest extent and repair artifacts in the extraction process, a conservative heuristic cleaning algorithm is adopted, which triggers modifications only when strict preset conditions are met. The cleaning algorithm includes: character-level normalization and encoding repair, line break repair, and residual noise removal, as detailed below:

[0019] a. Character-level normalization and encoding repair: Construct a pre-defined character mapping table and perform standardized mapping; In response to the encoding confusion problem unique to PDF documents, hyphens, non-standard control characters, and diverse quotation marks / dashes are uniformly mapped to standard ASCII or Unicode characters to eliminate the heterogeneity of the underlying character encoding;

[0020] b. Line break repair based on context lookahead: Design context-aware regular expressions to process pseudo line breaks. Instead of simply deleting all line breaks, it makes judgments based on the "lookahead" and "lookbehind" logic of the context, merging hyphens across lines, separating numbers and units, and abnormal spaces or line breaks between Chinese characters, avoiding Chinese word adhesion and English line breaks, and eliminating the representation differences of equivalent characters;

[0021] c. Removal of residual noise in regular pattern matching: Deploy high-precision anchor regular expressions and position rules to accurately match and identify residual noise such as headers, footers, and page numbers with specific format features and delete the residual noise to avoid accidental deletion of the main text content.

[0022] Preferably, in the S1, the specific process of combining semantic structure recognition and paragraph reconstruction is as follows:

[0023] (1) Semantic structure recognition based on a domain pattern dictionary: Establish a mapping mechanism from unstructured text lines to structured Markdown tags; first, construct a preset domain keyword pattern dictionary, which is a high-dimensional pattern dictionary containing the characteristics of the plant protection field , where is an anchor-based regular expression (such as ^(Introduction|引言) for matching chapter titles), is the corresponding Markdown syntax tag (such as ## (second-level heading), ** (bold)); then, define a regular mapping function for matching; for any text line in the input text stream, traverse the pattern dictionary ; the calculation formula for performing regular matching is:

[0024]

[0025] If the text line hits a certain regular pattern in the dictionary , it is converted into a structured line with a semantic prefix , that is, a structured line in Markdown format, otherwise it remains as a normal text line; this mechanism realizes the explicitization of the document logical structure;

[0026] (2) Paragraph reconstruction based on a dynamic semantic buffer mechanism: To solve the problem of semantic fragmentation caused by hard carriage returns in PDF layout, construct a two-state semantic buffer algorithm for text reorganization; first, initialize a volatile semantic buffer , and set its space state to ; then, define a boundary judgment function. For the current text line traversed, define a boolean judgment function :

[0027] ,in This represents the set of structured header lines identified by the semantic structure mapping. Indicates a blank line. The regular expression pattern representing the list item;

[0028] Next, state transitions and execution logic are performed, and the judgment condition is checked. When a condition is met (i.e., the current line is not a chapter title or blank line, etc.), it is determined that the current text line belongs to a continuous part of the previous paragraph that was forcibly cut off by the layout; at this time, a "push and concatenate" operation is performed, and the current text line is concatenated. Appended to semantic buffer The text ends with spaces, connecting the visually fragmented short lines into a logically complete long sentence; when the condition is judged... When established (i.e., the current line is a clearly defined structured boundary), "semantic truncation and commit" is triggered; the following operations are performed in strict order, buffering... The accumulated complete paragraphs are written to the output; then the buffer is reset to empty; finally, the current structured line is... (As in the title) are output as independent nodes to reconstruct the complete paragraph;

[0029] By combining semantic structure recognition and paragraph reconstruction, the structured markup in Markdown format and the complete paragraphs formed by reconstructing paragraphs based on structure awareness ultimately output Markdown format text with a clear hierarchical structure and semantic coherence.

[0030] Preferably, in step S1, the idempotent caching strategy based on timestamps is as follows:

[0031] To ensure the reentrancy and computational efficiency of batch processing tasks after crashes or interruptions, a condition-triggered gating mechanism based on timestamp metadata is designed; assuming the PDF document set is... The Markdown format text collection is For any task Define the modification timestamp of the source file (PDF format document) as... The target file (Markdown format text) was modified with the timestamp. (If the file does not exist, the value is 0); then, the gating function is executed, defining the Boolean value for the execution decision of the conversion task. The calculation formula is:

[0032]

[0033] During the initialization phase, traverse the file list information (including...) All PDF documents and The mapping relationship between the Chinese and Markdown formatted text), only when The corresponding task will only be assigned when the result is true. PDF documents are added to the computation queue; this mechanism ensures the idempotency of the system, meaning that multiple executions of the same batch of tasks produce consistent results, and the computational complexity is reduced from... Reduce to (Only new or modified files are processed), significantly improving the engineering efficiency of large-scale document database processing;

[0034] Preferably, in step S2, the dual-strategy adaptive AI extraction module adaptively switches between a staged extraction strategy and an intelligent block extraction strategy based on a text length threshold. The specific process is as follows:

[0035] The principle of the intelligent decision-making routing mechanism based on text length: To address the problem of limited input length in the context window of AI models, an automatic routing mechanism is designed; the input Markdown formatted text is treated as an object to be processed. First, calculate its complexity, which is proportional to the text length. Set a safety threshold for the measurement standard. (For example, 100,000 characters represent the limit that the AI ​​model can process at one time), then the decision logic calculation formula is as follows:

[0036]

[0037] when (Standard length text) indicates that the length of the Markdown text is moderate, and the AI ​​model can process it in one go. It executes a phased extraction strategy, focusing on close reading. The same Markdown text is input into the AI ​​model three times (i.e., the AI ​​model reads it three times repeatedly), and each time it extracts information at a different depth.

[0038] when (Extremely long text) indicates that the length of the Markdown text is too long for the AI ​​model to process in one go. The intelligent block extraction strategy is executed, which focuses on "block reading" and divides the Markdown text into multiple text blocks, which are then input into the AI ​​model for processing.

[0039] (1) Staged extraction strategy: Based on the "orthogonal task decomposition strategy" for focused extraction, for standard length text, in order to prevent the AI ​​model from "attention distraction" when extracting a large number of fields, a "divide and conquer" strategy is adopted to simplify the originally complex overall task. The task is broken down into three independent (orthogonal) subtasks. The AI ​​model is asked three times consecutively, each time using a different focus-specific Prompt instruction, ensuring a completely different focus each time. The calculation formula for task synthesis is as follows:

[0040]

[0041] (Basic Macro Level): In the first round of questions, the AI ​​model was instructed to focus only on metadata such as "title, author, publication time, and chapter structure," ignoring details;

[0042] (Specialty Entity Layer): The second round of questions instructs the AI ​​model to focus only on entities in specialized fields such as "crop name, type of pests and diseases, and factors causing outbreaks";

[0043] (Quantitative Data Layer): The third round of questions instructed the AI ​​model to focus only on numbers, statistical methods, and results such as "temperature, humidity, wind speed, variance, and average value";

[0044] Finally, the three answers containing some focused information are merged, deduplicated, and formed a complete parsable structured JSON data;

[0045] (2) Intelligent block extraction strategy: Based on the "semantic alignment block extraction strategy", the intelligent block extraction uses the ## chapter title (i.e. chapter boundary) in the Markdown format text as the semantic segmentation unit to divide the ultra-long text into multiple logically coherent text blocks; to ensure that the text blocks match the most accurate extraction Prompt instruction, vector similarity matching is then used; specifically, the content of the segmented text blocks and each dedicated professional Prompt instruction in the Prompt template library are converted into high-dimensional semantic vectors, and the text block vectors are defined and calculated. With instruction vector The cosine similarity is calculated using the following formula:

[0046]

[0047] The Prompt instruction with the highest similarity is selected for extraction; for example, text blocks containing the semantics of "analysis of variance" will automatically be matched with the "statistical method extraction" instruction with high confidence, rather than the general summary instruction, and finally the JSON data format output is obtained.

[0048] Preferably, the JSON data format repair strategy in step S2 is as follows:

[0049] When the standard json.loads() parsing fails, a backup process for JSON data format repair is automatically triggered. Through a series of string and regular expression operations, non-standard JSON data is repaired, and the repaired and standardized JSON data is then sent back to the parser.

[0050] Preferably, in step S3, the specific process of the two-stage parallel scheduling module is as follows:

[0051] In the first stage, a CPU thread pool focused on CPU-intensive tasks is created and started. All PDF documents to be processed are evenly distributed to the threads in the CPU thread pool. Each thread independently executes the process of step S1 and generates Markdown text.

[0052] In the second stage, an IO thread pool focused on I / O-intensive tasks is created and started. All Markdown text to be processed is evenly distributed to the threads in the IO thread pool. Each thread independently executes the process in step S2 and generates structured JSON data.

[0053] Finally, the asyncio.run() wrapper is used to efficiently execute asynchronous API calls in a synchronous thread, thus achieving optimal resource utilization.

[0054] After all PDF documents have been processed and the resulting structured JSON data has been generated, an aggregation script is started using the scheduler. The aggregation script iterates through the structured JSON data and integrates the generated JSON data file.

[0055] Preferably, in step S3, based on the multi-dimensional aggregation export module, the generated structured JSON format data text is aggregated according to predefined mapping rules to generate a multi-dimensional structured database file, and an adaptive and localized format is output, as follows:

[0056] (1) Generate by aggregation using predefined mapping rules: Use the pandas library to read the JSON format data file into memory; according to the predefined mapping rules, append the JSON fields in different JSON format data to different DataFrame multidimensional data tables; finally, use pandas.ExcelWriter to write the DataFrame multidimensional data table as different worksheets to the same Excel file, where each worksheet represents a specific data dimension, forming a multidimensional structured database file (i.e., an Excel structured database file);

[0057] (2) Output adaptive and localized format: Set a field name Chinese-English dictionary; before writing the DataFrame to Excel, replace the column names of the DataFrame with Chinese table headers using the field name Chinese-English dictionary; after writing the DataFrame to Excel, use the openpyxl or xlsxwriter engine to traverse each column of each worksheet, calculate the maximum display width of all cell contents in the column, and set the optimal column width with the best pixel value accordingly, so that the generated Excel file can be read clearly without any manual adjustment, improving the usability and human readability of the final output.

[0058] Compared with existing technologies, this invention proposes a method for batch structured extraction of literature information in the field of plant protection based on AI large-scale models. The advantages of this method are:

[0059] 1. A dual-strategy adaptive AI extraction mechanism significantly improves the completeness and accuracy of structured information extraction.

[0060] This invention introduces a "text complexity-driven dual-strategy adaptive mechanism" for the first time in the structured processing of plant protection professional literature. This mechanism automatically determines the text complexity based on the length, number of chapters, and semantic density of the Markdown text, and switches between a phased extraction strategy and an intelligent block extraction strategy. When the Markdown text is of moderate size, the system breaks down the complex extraction task into three sub-tasks: basic information, professional factors, and quantitative data, which are submitted to the AI ​​model separately. This results in higher focus for each call, reducing information omissions and formatting issues. When encountering extremely long or structurally complex documents, the system performs semantic block segmentation based on chapter titles and automatically matches the most suitable Prompt template for different chapters, thus maintaining the integrity of field extraction even when processing long texts. Because this dual strategy can adapt to scientific literature of different sizes and structures, it reduces the contextual pressure and output instability of the model. This makes the field extraction accuracy of this invention in different types of plant protection literature about 25% to 40% higher than that of conventional single-strategy methods. Moreover, the average API call cost per document can be reduced to about 0.056 yuan, realizing a large-scale automated knowledge extraction capability that combines high accuracy and low cost.

[0061] 2. The two-stage parallelization architecture, which separates heterogeneous tasks, significantly improves batch processing speed.

[0062] The system significantly improves the efficiency of batch document processing through a cleverly designed two-stage asynchronous parallel processing pipeline. The first stage utilizes a thread pool to execute all PDF-to-Markdown conversions in parallel (CPU-intensive tasks), while the second stage uses another thread pool to process all API communications with the AI ​​model in parallel (I / O-intensive tasks). This architecture, which separates and concurrently processes different types of tasks, effectively overlaps operation waiting times, fully utilizes the potential of hardware resources, and enables the system to process multiple documents simultaneously. The overall throughput far exceeds that of traditional serial processing methods. Under the same hardware environment, the overall processing speed of this invention can be 2 to 3 times faster than traditional serial processing methods, thereby significantly shortening the processing cycle of large document databases and significantly improving the overall efficiency of the data preparation stage.

[0063] 3. The multi-layered fault-tolerance strategy significantly improved the system's robustness and availability in real-world data environments.

[0064] The system incorporates a multi-layered fault-tolerance mechanism to ensure stability and data integrity when processing large-scale, uncontrollable data sources. A robust JSON data repair strategy intelligently identifies and corrects non-standard data formats returned by AI models (e.g., removing markers, repairing unescaped characters), significantly improving the success rate of data parsing. A timestamp-based idempotent caching mechanism avoids redundant work and supports safe interruption and resumption of tasks. Comprehensive anomaly detection and handling logic ensures that the failure of processing a single document will not affect the entire batch task. Due to the introduction of the above fault-tolerance system, this invention maintains high stability when dealing with plant protection PDF documents from different journals, in different languages, and with different layouts. The success rate and consistency of data processing are significantly improved, enabling it to operate stably for a long time in real-world research environments.

[0065] 4. The use and application value of scientific research data are improved through structured outputs that are deeply customized for specific domains.

[0066] The entire system, from its underlying data structure (such as the customized PaperContent class) and AI instruction (Prompt) templates to its final output format, is deeply customized to meet the specific needs of literature analysis in the field of plant protection. The resulting multi-worksheet Excel file contains approximately 50 specialized fields, including title, abstract, crop type, pest and disease names, environmental factors, experimental methods, and statistical results, forming a readily usable thematic research database. Users can directly use this data for subsequent data analysis, trend studies, or attribution analysis without complex data cleaning and integration, significantly improving the efficiency of scientific research data preparation. Attached Figure Description

[0067] Figure 1This is an overall flowchart of the method for batch structured extraction of literature information in the field of plant protection based on AI large model, as proposed in this invention.

[0068] Figure 2 This is a schematic diagram of the overall architecture of the method for batch structured extraction of literature information in the field of plant protection based on AI large model according to the present invention;

[0069] Figure 3 This is a schematic diagram of the architecture of the PDF document adaptive preprocessing module and the Markdown conversion module of the present invention;

[0070] Figure 4 This is a schematic diagram of the architecture of the dual-strategy adaptive AI extraction module of the present invention;

[0071] Figure 5 The figure shows the performance index results of the batch structured extraction method of plant protection literature information based on AI large model of the present invention. Detailed Implementation

[0072] The technical solutions of the embodiments of this application will be further described clearly and completely below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0073] To make the inventive objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings: In order to better understand the above-mentioned objectives, features, and advantages of this invention, the advantages of this invention will be further illustrated below by comparing the embodiments with the accompanying drawings and specific implementation methods.

[0074] This invention proposes a method for batch structured extraction of literature information in the field of plant protection based on a large AI model, such as... Figure 1 and Figure 2 As shown, the Python-based IDE platform (PyCharm) achieves end-to-end automation from PDF documents to structured data output through a multi-stage, parallel processing flow. The processing flow is described in detail below:

[0075] S1 constructs an adaptive preprocessing module for PDF documents and a Markdown conversion module. PDF documents obtained from the document database undergo intelligent PDF text extraction, deep text cleaning, semantic structure recognition and paragraph reconstruction, and an idempotent caching strategy to be batch-converted into Markdown format text with a semantic hierarchy suitable for computer processing. This achieves accurate conversion from visual unstructured format to semantic structured format, such as... Figure 3 As shown;

[0076] Specifically, in step S1, the specific processing procedure for intelligent PDF text extraction is as follows:

[0077] Intelligent PDF Text Extraction Based on Physical Layout Analysis and Coordinate Space Clustering: Traditional PDF text extraction methods (such as get_text()) treat the page as a flat text stream, ignoring the visually structured layout features of the document page, which can disrupt the reading order of complex layouts such as multi-column layouts and mixed text and images. Therefore, considering the lack of logical structure in PDF documents, the PDF text extraction method adopts a "coordinate space reverse reconstruction" strategy; as follows:

[0078] The "coordinate space reverse reconstruction" strategy includes metadata vectorization extraction, multi-column detection and topology sorting mechanism, and noise filtering based on spatial and frequency features;

[0079] a. Metadata Vectorization Extraction: By calling the get_text("blocks") function in the PyMuPDF library (parsing package), all text blocks and their metadata information on the pages of PDF documents are extracted. Each text block and its metadata information includes bounding box coordinates (used to record the physical position of the text block on the page, such as bounding box: x0, y0, x1, y1), font, and font size; a feature vector of the text block is constructed. To obtain a set of two-dimensional text blocks, where ( () represents the bounding box coordinates of the text block in the two-dimensional page space. Indicates font style, Indicates font size, Represents text content;

[0080] b. Multi-column detection and topological sorting mechanism: To solve the problem of disordered reading order caused by two-column or multi-column layouts on the page, a one-dimensional coordinate clustering algorithm is performed on the two-dimensional text block set; the distribution of blank gaps on the page in the horizontal direction is detected by projection analysis, thereby dynamically dividing the page into... Each column range is denoted as... Based on this, a hierarchical topological sorting rule is applied, prioritizing sorting by column index, with sorting within each column following a "top-down" approach. (Axis increments), from left to right () The geometric logic of "axis increment" maps a discrete set of two-dimensional text blocks into a linear one-dimensional logical text stream, ensuring that the extracted PDF text is read in the correct order, and thus perfectly reconstructs the correct reading text stream.

[0081] c. Noise filtering based on spatial and frequency characteristics: Establish a dual verification mechanism of location and content; set a threshold area at the page edge. (Header / Footer area), and perform cross-page frequency analysis on text blocks within the page edge threshold area; identify text blocks that repeatedly appear in different or multiple pages within a fixed coordinate area (or fluctuate within a very small range), and have the same or content similarity higher than the threshold. Text blocks (such as page numbers, DOI links, etc.) are identified and filtered, and identified as layout noise and removed, so as to remove layout noise such as headers, footers, and page numbers, significantly improving the accuracy and usability of PDF text extraction;

[0082] Specifically, in S1, deep text cleaning includes, but is not limited to:

[0083] Deep Text Stream Cleaning Based on Heuristic Rules and Conservative Strategies: To maximize the preservation of original semantics and repair artifacts during the extraction process, a conservative heuristic cleaning algorithm is employed to clean the text stream extracted from PDF documents, triggering modifications only when strict preset conditions are met. The cleaning algorithm includes character-level normalization and encoding repair, line break repair, and residual noise removal, as detailed below:

[0084] a. Character-level normalization and encoding repair: A pre-built character mapping table is constructed and used to perform normalized mapping for encoding repair. Addressing the issue of numerous encoding errors and typesetting noise introduced during PDF text extraction (get_text()), the pre-built character mapping table only repairs explicit and identifiable errors, avoiding guesswork about ambiguous typesetting, and preserving the original information to the maximum extent. It replaces encoding issues unique to the extracted PDF documents, such as English hyphens (e.g., fi, fl), non-standard punctuation, and soft hyphens (e.g., soft hyphens, \u00ad). It also maps various non-standard dashes (e.g., long dashes, short dashes) and quotation marks (e.g., curved quotation marks, angled quotation marks) to standard ASCII or Chinese punctuation to ensure text standardization at the character level.

[0085] b. Context-Aware Line Break Repair: This approach repairs automatic line breaks in PDF documents without relying on a dictionary (which is prone to misinterpretation). It uses context-aware regular expressions to handle pseudo-line breaks. Specifically, it employs the following types of regular expression logic for repair: 1. Repairing hyphens across lines using regular expressions. This expression matches the hyphen at the end of a line and the word fragment at the beginning of the next line by capturing groups, such as " "Seamless merge into" 2. Remove pseudo-newline characters using regular expressions. (For English) or For Chinese texts, instead of simply deleting all line breaks, it is determined based on the "forward-looking" and "backward-looking" logic of the context to avoid Chinese word adhesion and English line breaks. That is, all PDF-format documents are normalized using NFC (Normalization Form C) to eliminate the representational differences of equivalent characters.

[0086] c. Removal of residual noise by regular pattern matching: Deploy high-precision anchor regular expressions and position rules (such as matching ^Page\s*\d+$ and ^DOI:), accurately match and identify residual noise such as headers, footers, and page numbers with specific format features, and delete the residual noise, rather than making fuzzy judgments based on position or length to avoid accidentally deleting the content of the main text.

[0087] Specifically, in S1, the specific process of combining semantic structure recognition and paragraph reconstruction is as follows:

[0088] (1) Semantic structure recognition based on a domain pattern dictionary: To reconstruct the hierarchical structure of the document, the algorithm does not rely on fragile layout information such as fonts and font sizes in PDFs. A mapping mechanism from unstructured text lines to structured Markdown tags is established. Taking the cleaned text stream as input, first construct a preset domain keyword pattern dictionary, where the pattern dictionary includes a high-dimensional pattern dictionary with characteristics of the plant protection field , where is an anchor-based regular expression (such as ^(Introduction|引言) for matching chapter titles), is the corresponding Markdown syntax tag (such as ## (second-level title), ** (bold)); then, define a regular mapping function to match the cleaned text stream with the pattern dictionary. For example, match the chapter title expression with (Introduction|引\s*言) and the caption expression of figures with (Fig\.\s*\d+), and convert the recognized content into structured tags in Markdown format (#, ##, ). For any text line in the input text stream, traverse the pattern dictionary ; the calculation formula for performing regular matching is:

[0089]

[0090] If a text line hits a certain regular pattern in the dictionary , it is converted into a structured line with a semantic prefix , that is, a structured line in Markdown format, otherwise it remains as a normal text line; this mechanism realizes the explicitization of the logical structure of the document.

[0091] (2) Paragraph reconstruction based on dynamic semantic buffering mechanism: In order to solve the semantic fragmentation problem caused by hard returns in PDF typesetting, and to distinguish between structured line breaks (such as headings, lists, and table rows) and typesetting line breaks (hard returns inserted within the same paragraph due to page width limitations), a dual-state semantic buffering algorithm is constructed for text reconstruction. First, a volatile semantic buffer is initialized. Its spatial state is set as This is used to accumulate text lines; then, all text lines are traversed, and a boundary judgment function is defined, setting the current text line as... Define the Boolean decision function as follows: :

[0092] ,

[0093] in This represents the set of structured header lines identified by the semantic structure mapping. Indicates a blank line. The regular expression pattern representing the list item;

[0094] Next, state transitions and execution logic are performed, and the judgment condition is checked. When a condition is met (i.e., the current line is not a chapter title or blank line, etc.), it is determined that the current text line belongs to a continuous part of the previous paragraph that was forcibly cut off by the layout; at this time, a "push and concatenate" operation is performed, and the current text line is concatenated. Appended to semantic buffer The text ends with a space, and is not immediately output, thus connecting visually fragmented short lines into logically complete long sentences; when the condition is judged... When a line is established (i.e., the current line is a clearly defined structured boundary, such as starting with ## or enclosed by **), "semantic truncation and commit" is triggered; the following operations are performed in strict order, buffering... The accumulated complete paragraphs are written to the output; then the buffer is reset to empty; finally, the current structured line is... (As shown in the title) as independent node outputs to reconstruct the complete paragraph; this algorithm reconstructs visually scattered paragraphs in PDF documents into semantically continuous single text paragraphs that can be understood by LLM (Large Language Model), thus ensuring the core of semantic continuity;

[0095] By combining semantic structure recognition and paragraph reconstruction, the structured markup in Markdown format and the complete paragraphs formed by reconstructing paragraphs based on structure awareness ultimately output Markdown format text with a clear hierarchical structure and semantic coherence.

[0096] Specifically, in S1, the idempotent caching strategy based on timestamps is as follows:

[0097] For time-consuming batch tasks, to ensure reentrancy and computational efficiency after crashes or interruptions, fault tolerance mechanisms must be designed to handle unexpected interruptions (such as program crashes or network problems). Idempotency is a key characteristic to ensure that task reruns do not produce side effects (such as duplicate processing or data corruption). PDF format conversion is a CPU-intensive operation; cache checks should be implemented before conversion to avoid repetitive work. Specifically, before performing the PDF to Markdown conversion, a condition-triggered gating mechanism based on timestamp metadata should be designed. Let the set of PDF format documents be... The Markdown format text collection is For any task Define the modification timestamp of the source file (PDF format document) as... The target file (Markdown format text) was modified with the timestamp. (If the file does not exist, the value is 0); then, check and compare the last modification timestamps of the PDF document with those of the existing Markdown text, execute the gating function, and define the Boolean value for the execution decision of the conversion task. The calculation formula is:

[0098]

[0099] During the initialization phase, traverse the file list information (including...) All PDF documents and The mapping relationship between the Chinese and Markdown formatted text), only when The corresponding task will only be assigned when the result is true. PDF documents are added to the computation queue; that is, the PDF to Markdown conversion operation is triggered only when the modification time of the PDF document is later than that of an existing Markdown file (i.e., the update time of the PDF document); this mechanism guarantees the idempotency of batch tasks, that is, multiple executions of the same batch task produce consistent results, and the computational complexity is reduced from... Reduce to (Only new or modified files are processed), significantly improving the engineering efficiency of large-scale document database processing;

[0100] S2, a dual-strategy adaptive AI extraction module is constructed. The Markdown text is input into the AI ​​module, and a phased extraction strategy or an intelligent block extraction strategy is adaptively switched based on the text length threshold. The phased extraction strategy uses several rounds of focused professional Prompt instructions to call the AI ​​model to extract basic metadata, domain entities, and methods / results respectively, returning JSON format data. The intelligent block extraction strategy segments multiple text blocks using chapter boundaries in the Markdown text as semantic segmentation units, and dynamically matches focused professional Prompt instructions for each text block to extract JSON format data. Furthermore, a highly robust JSON data repair strategy is combined to obtain parsable structured JSON format data text, overcoming the technical limitations of AI models in handling long texts and complex multi-field extraction tasks. Figure 4 As shown;

[0101] Specifically, in step S2, the dual-strategy adaptive AI extraction module adaptively switches between a staged extraction strategy and an intelligent block extraction strategy based on a text length threshold. The specific process is as follows:

[0102] The dual-strategy adaptive AI (DeepSeek-V3.2-Exp model) extraction module adopts a "divide and conquer" and "adaptive strategy switching" architecture to overcome the context window limitations and output length limitations of AI models when processing long texts. It automatically extracts unstructured text information (Markdown format text content) and transforms it into structured data (such as JSON format key-value pairs), and solves the problem that it is difficult to complete complex multi-field extraction tasks in a single request.

[0103] The switching principle of the dual-strategy adaptive AI extraction module: To address the issue of limited input length in the AI ​​model's context window, an automatic splitter is designed (i.e., based on text length thresholds and complexity as decision points); the input Markdown text is treated as an object to be processed. First, calculate its complexity, which is proportional to the text length. Set a safety threshold for the measurement standard. (For example, 100,000 characters represent the limit that the AI ​​model can process at one time), then the decision logic calculation formula is as follows:

[0104]

[0105] Check the length of Markdown formatted text: when (Text length < threshold) is judged as standard length text, indicating that the length of the Markdown format text is moderate and the AI ​​model can process it in one go. The execution focuses on close reading. The same Markdown format text is input into the AI ​​model three times (that is, the AI ​​model reads it three times repeatedly) and each time it extracts information at a different depth.

[0106] when (Text length > threshold) is judged as excessively long text, indicating that the length of the Markdown format text is too long for the AI ​​model to process in one go. The intelligent block extraction strategy is executed, which focuses on "block reading", dividing the Markdown format text into multiple text blocks and inputting them into the AI ​​model for processing respectively.

[0107] (1) Staged extraction strategy: Focused extraction based on the "orthogonal task decomposition strategy" is used for standard length text (e.g., 60,000 characters), whose length is within the capacity of the AI ​​model (e.g., 128k context window capacity). However, extracting the required information (50 fields) at once is too complex and may exceed the output limit of the AI ​​model (e.g., 8k). To prevent the AI ​​model from "distracting attention" when extracting a large number of fields, a "divide and conquer" strategy is adopted. This strategy reduces the original complex total task The task is broken down into three independent (orthogonal) subtasks, which are sent to the AI ​​model for processing three times consecutively. However, each time a different focusing-specific Prompt instruction is used, ensuring that the focus of attention is completely different each time. The calculation formula for task synthesis is as follows:

[0108]

[0109] (Basic Macro Layer): In the first round of questions, the AI ​​model is instructed to focus only on metadata such as "title, author, publication time, and chapter structure" to extract "basic information and chapters" (such as title, abstract, introduction, and conclusion). At this stage, the instruction identifies the metadata and macro structure of the text, ignoring details.

[0110] (Specialty Entity Layer): In the second round of questioning, the AI ​​model is instructed to focus only on specialty entities such as "crop name, pest and disease type, outbreak factors" and extract "specialty factors" (such as crop type, pest and disease name, temperature, humidity data). At this stage, the instructions focus on specialty entities in the text.

[0111] (Quantitative Data Layer): In the third round of questioning, the AI ​​model is instructed to focus only on numbers, statistical methods and results such as "temperature, humidity, wind speed, variance, and average value" to extract "quantitative data and methods" (such as statistical correlation, prediction models, and geographical scope). At this stage, the instructions focus on research methods and quantitative results.

[0112] More specifically, to ensure the standardization of extraction, all the focused professional Prompt instructions at the above levels are constructed using structured prompt word templates; with ' Taking the 'Professional Entity Layer' as an example, its Prompt template structure is as follows:

[0113] You are a professional expert in analyzing academic papers on agricultural pests and diseases. Please extract technical terms and environmental management factors from the following papers, adhering to the core extraction principles (which must be strictly followed):

[0114] 1. Completeness Principle: Once relevant keywords are identified, the complete sentence or paragraph containing the keyword must be extracted;

[0115] 2. Complete context: Don't just extract the keywords themselves; extract the complete description and related data in which the keywords appear.

[0116] 3. Data integrity: All specific numbers, dates, times, percentages, statistical values, and other numerical information must be retained;

[0117] 4. Avoid omissions: Do not use "..." or "etc." to omit content; include all relevant information.

[0118] 5. Strictly distinguish between chapters: Materials and Methods, Results, Discussion, and Conclusion; it is strictly forbidden to extract data information from the introduction, literature review, or previous research review chapters;

[0119] 6. Language fidelity: Strictly extract from the original text; no rewriting, summarizing, or fabrication is permitted.

[0120] Key points for chapter identification:

[0121] Introduction / Foreword typically includes: previous research, research background, and literature review.

[0122] Materials and methods typically include: experimental design, survey methods, and data collection.

[0123] Results typically include: observational data, experimental results, and statistical analysis.

[0124] Discussions typically include: outcome analysis, causal explanation, and mechanism exploration.

[0125] Conclusions typically include: key findings and summaries.

[0126] Field Extraction Guidelines (Example): Research Subjects (can be extracted from the abstract and individual research sections)

[0127] crop_type: Crop name (e.g., rice, wheat)

[0128] pest_disease_name: The complete name of the pest or disease (e.g., brown planthopper, white-backed planthopper, rice blast).

[0129] pest_disease_category: Category (e.g., pests, diseases)

[0130] pathogen_type: The type of pathogen (e.g., fungus, virus, bacteria).

[0131] Please return in JSON format:

[0132] {{

[0133] "crop_type": "crop type",

[0134] "pest_disease_name": "Name of pest or disease",

[0135] "pest_disease_category": "Pest and disease category",

[0136] "pathogen_type": "pathogen type",

[0137] }}

[0138] Final reminder:

[0139] 1. Once relevant keywords are identified in the Materials and Methods, Results, Discussion, and Conclusion sections, the complete sentence or paragraph containing the keyword must be extracted;

[0140] 2. Don't just extract keywords; extract complete causal relationships, data relationships, and time relationships.

[0141] 3. It is strictly forbidden to extract content from the introduction or literature review;

[0142] 4. Only return an empty string if a field does not actually exist in the research section;

[0143] Finally, the three results containing partial focus information are returned as JSON data and stored in local memory. These three JSON data are then merged, duplicates are removed, and the format is standardized to form a complete, parsable, structured JSON data containing all target fields. This strategy avoids the output length limitation of AI models and improves extraction accuracy through task focusing. It is like having three experts review a manuscript: one for the format, one for the content, and one for the data, and then summarizing their opinions.

[0144] (2) Intelligent block extraction strategy: Intelligent block extraction based on the "semantic alignment block strategy" is used for ultra-long text (such as 150,000 characters), whose length exceeds the context window required by the AI ​​model. Forced input will result in information loss or API error. The semantic block strategy is adopted to divide the ultra-long text into multiple logically coherent text blocks (such as document header information 'header', introduction 'introduction', methods 'methods', etc.). Instead of using the general Prompt instruction for all text blocks, the most matching special Prompt instruction is dynamically selected from the Prompt template library according to the name of each text block (such as 'methods') (such as a Prompt instruction specifically used to extract "research design" and "data source"). Finally, these text blocks are traversed and vector similarity matching is used. Specifically, the content of the segmented text blocks and each special focused professional Prompt instruction in the Prompt template library are converted into high-dimensional semantic vectors, and the text block vectors are defined and calculated. With instruction vector The cosine similarity is calculated using the following formula:

[0145]

[0146] The Prompt command with the highest similarity is selected for extraction. For example, text blocks containing the semantics of "analysis of variance" will be automatically matched with the "statistical method extraction" command with high confidence, rather than the general summary command. Then, the AI ​​model is called independently for each text block to extract information, and all the scattered JSON data returned from different blocks are integrated and merged to form a complete extraction result. This strategy ensures that the contextual information is completely preserved when processing text of any length.

[0147] Specifically, the JSON data repair strategy in S2 follows as follows:

[0148] The JSON data output of AI models (even when a JSON data output mode is specified) may contain syntactic flaws (such as containing ```json, trailing commas, or unescaped characters). These flaws can cause strict parsers (such as Pydantic) to fail. To ensure a high success rate, an automatically triggered JSON data repair backup process is designed. Its workflow is as follows: When the standard json.loads() parsing fails, the JSON data repair backup process is automatically triggered. Through a series of string and regular expression operations (such as removing code block markers, removing redundant commas, fixing unescaped \n, and automatically completing missing characters), the non-standard JSON data text is repaired, and then parsing is attempted again. The repaired, compliant JSON data is then sent back to the parser to improve the system's fault tolerance.

[0149] S3. Construct a two-stage parallel scheduling module, which is used to execute the CPU-intensive conversion task of step S1 and the I / O-intensive AI analysis task of step S2, respectively; construct a multi-dimensional aggregation and export module, which aggregates the generated structured JSON format data according to predefined mapping rules to generate a multi-dimensional structured database file, realizing efficient integration from scattered extraction results to macro-analysis dataset.

[0150] Specifically, the process of the two-stage parallel scheduling module in S3 is as follows:

[0151] The two-stage parallel scheduling module and the multi-dimensional aggregation and export module act as the "brain" and "factory" of the system, responsible for efficiently scheduling steps S1 and S2, concurrently managing the entire data processing flow, and aggregating the fragmented, single-article JSON format data extraction results into a unified, multi-dimensional, and readily available dataset. The document processing flow contains a large number of I / O-intensive (file reading and writing, network API requests) and CPU-intensive (text processing) operations. Concurrent processing greatly overlaps the waiting time of these operations, improving system throughput. Instead of using a single thread pool for mixed processing, a two-stage parallel pipeline is constructed.

[0152] In the first stage, a CPU thread pool focused on CPU-intensive tasks is created and started. All PDF documents to be processed are evenly distributed to the threads in the CPU thread pool. Each thread independently executes the process of step S1 and generates Markdown text.

[0153] In the second stage, after all PDF conversion tasks in the first stage are completed, an IO thread pool focused on I / O-intensive tasks is started. All Markdown text to be processed is evenly distributed to threads in the IO thread pool. Each thread independently executes the process of step S2 and generates structured JSON data, which is used to handle all communication with the AI ​​model API and execute all AI analysis tasks (I / O intensive) in parallel.

[0154] Finally, the asyncio.run() wrapper is used to efficiently execute asynchronous API calls in a synchronous thread, thus achieving optimal resource utilization.

[0155] The JSON data results of a single document are micro-level. Only by aggregating them and reorganizing them according to logical dimensions can a macro-level, analyzable dataset be formed. After all PDF documents have been processed to generate structured JSON data, an aggregation script is started using a scheduler. The aggregation script traverses the structured JSON data and integrates the generated JSON data files.

[0156] Specifically, in step S3, based on the multi-dimensional aggregation export module, the generated structured JSON format data is aggregated according to predefined mapping rules to generate a multi-dimensional structured database file, and an adaptive and localized format is output, as follows:

[0157] (1) Predefined mapping rules are used to aggregate and generate: the data is read into memory using the pandas library; according to the predefined data pattern mapping rules, the JSON fields (such as basic_info.title, env_factors.temperature) in different JSON format data are appended to different DataFrames (used to store and process structured data) (such as "basic information" and "environmental factors"); finally, pandas.ExcelWriter (a tool used to write data to an Excel file) is used to write these DataFrames as different worksheets to the same Excel file, each worksheet representing a specific data dimension (i.e., an Excel structured database file), forming a multi-dimensional structured database;

[0158] (2) Output format adaptation and localization: improve the usability and human readability of the final output; the system has a built-in Chinese-English dictionary for field names ({'title': 'title', 'temperature_celsius': 'temperature(°C)'}); before writing the DataFrame to Excel, the column names (original English fields) of the DataFrame are replaced with Chinese table headers through this dictionary; after writing the DataFrame to Excel, the openpyxl or xlsxwriter engine is used to traverse each column of each worksheet, calculate the maximum display width of all cell contents in the column, and set the optimal column width with the best pixel value accordingly, ensuring that the generated Excel file can be read clearly without any manual adjustment.

[0159] (3) Algorithm performance monitor: Monitoring logic is injected through Python decorators (@timing_decorator and @performance_logger); this enables the system to automatically record the time, memory usage (requires psutil) and success status when performing key steps (such as PDF conversion, AI analysis, and export), providing data support for performance tuning and troubleshooting.

[0160] The specific experiments of this invention are described in detail below with reference to the technical solution and accompanying drawings:

[0161] This example is executed on a server or personal computer equipped with a multi-core CPU (4 cores or more), running Windows 10, and with Python 3.13 and related third-party libraries installed, including PyMuPDF (fitz), the OpenAI API client, Pandas, and openpyxl. The AI ​​model is the DeepSeek large language model, which supports long context processing (128k tokens) and JSON output mode. It batch processes 10 plant protection and pest / disease literature articles from a specified folder. The execution steps of this example are as follows:

[0162] Step 1: System Initialization and File Preparation

[0163] Ten PDF documents (named paper_01.pdf to paper_10.pdf for ease of explanation) on different crop pests and diseases were saved to the local computer's preset input folder directory . / pdf_papers / . These documents cover a variety of layouts, including single-column, double-column, and mixed text and image layouts, to simulate the complexity of real-world documents.

[0164] When the main() function is executed, the system proceeds sequentially as follows:

[0165] Log initialization: Call the setup_logging() method to configure the logging system, and output INFO and higher level log information to the console and the log file paper_extraction.log to facilitate process monitoring and problem tracing;

[0166] Processor instantiation: Create a PaperBatchProcessor object, whose initialization process includes:

[0167] Load the API key DEEPSEEK_API_KEY, API base address DEEPSEEK_BASE_URL, and maximum concurrent worker threads MAX_WORKERS (set to 3 in this example) from the configuration file config.py.

[0168] Automatically create the . / converted_papers / directory to store intermediate Markdown files and the . / analysis_results / directory to store the final analysis results;

[0169] Initialize three core functional modules: PDFToMarkdownConverter (format converter), DeepSeekAPI (AI intelligent analyzer), and DataExporter (result exporter).

[0170] Step 2: Parallel PDF to Markdown conversion (Stage 1)

[0171] The main program calls the processor.run_complete_workflow() method to start the complete workflow; the first stage focuses on efficiently and accurately converting PDF documents into structured Markdown text.

[0172] Task distribution: The system scans the . / pdf_papers / directory to obtain a list of all PDF files; then, it creates a ThreadPoolExecutor thread pool with a maximum of 3 worker threads, iterates through 10 PDF file paths, and submits the conversion task of 10 PDF documents, self._convert_single_pdf(pdf_file), to the thread pool;

[0173] Parallel Conversion and Caching Mechanism: The thread pool starts 3 worker threads. Taking paper_01.pdf as an example, when a worker thread performs conversion, it first checks whether the target Markdown format text. / converted_papers / paper_01.md exists and compares its last modification time with that of the PDF format document. If the Markdown format text exists, it calls pdf_path.stat().st_mtime (PDF modification time) and output_path.stat().st_mtime (md modification time) respectively. If pdf_mtime ≤ output_mtime (that is, the PDF format document has not been updated and the Markdown format text has been updated), the Markdown format text is considered to have been processed, and the conversion is directly skipped. The thread directly returns the Markdown format text, using the cached result, which significantly improves the efficiency of repeated processing.

[0174] Content Extraction and Deep Cleaning: For the format text that needs to be converted (cache invalid), the thread performs the following steps:

[0175] Text Extraction: Use the PyMuPDF library to open the PDF, traverse each page page, and call page.get_text() (not "blocks") to extract the original text stream page by page.

[0176] Text Preprocessing: Call the _advanced_preprocessing method for deep cleaning, including: fixing encoding problems (such as replacing the ligature fi with fi), merging words broken due to line breaks (such as Fix to ), removing headers and footers (such as lines matching the pattern ^Page\s*\d+), and punctuation marks.

[0177] Structure Identification and Marking: Call the _identify_paper_structure method to identify chapter titles based on predefined regular expressions (such as r'(Introduction|引\s*言)'). Subsequently, use the _intelligent_formatting method to mark the identified main titles as secondary Markdown headings (##), and bold the figure captions (such as 'Fig.1') (**).

[0178] Paragraph Rearrangement and Saving: Call the _reflow_markdown_paragraphs method, traverse all lines, merge consecutive text lines that are not headings / lists / tables (connected with spaces) into semantically complete paragraphs, and finally generate a clearly formatted and structured Markdown format text and save it.

[0179] Step 3: Parallel AI extraction of structured information from documents (Phase 2)

[0180] The system uses an AI model to extract information from Markdown formatted text in a parallel and structured manner.

[0181] Task distribution: The system obtains a list of 10 Markdown formatted text paths generated in the first stage, creates a second thread pool (max_workers=3), and submits the analysis task self._analyze_single_markdown_sync(md_file) to the second thread pool;

[0182] Dynamic dual-strategy extraction: Within the worker thread, the system reads Markdown formatted text content and intelligently selects the extraction strategy based on the text length;

[0183] Phased Extraction Strategy (Regular Length Text): For Markdown formatted text with a length of 100,000 characters or less (such as paper_01.md), the system executes the _extract_in_phases method; this method adopts a phased, progressive extraction approach.

[0184] Phase 1: Extracting basic metadata (title, author, abstract, chapter structure);

[0185] Phase 2: Extracting information from the relevant professional field (names of the crops and pests / diseases being studied).

[0186] Phase 3: Extracting quantitative data (statistical methods, models, environmental factors such as temperature and humidity);

[0187] The extraction results from each stage are merged into a unified PaperContent data object, ensuring the integrity and relevance of the information;

[0188] Intelligent chunking extraction strategy (ultra-long text): For Markdown formatted text exceeding 100,000 characters (such as paper_08.md), the system executes the _extract_ultra_long_text method; this method employs a chapter-based, refined extraction approach.

[0189] Smart Blocking: Uses Markdown formatted text ## headings to divide long articles into chapters;

[0190] Targeted extraction: For each chapter (e.g., "Methodology"), dynamically generate the most relevant Prompt and call the AI ​​model for extraction; for example, extract the statistical model and experimental design in the "Methodology" chapter, and extract the numerical results and graph conclusions in the "Results" chapter.

[0191] The extraction results from all chapters are also integrated into the same PaperContent object;

[0192] JSON parsing and fault tolerance: The AI ​​model's return results are set to JSON format data; the system will clean the returned text (such as removing the `json` tag) and try to parse it using json.loads; if the parsing fails, the system will log a warning and continue the process using an empty dictionary to ensure the robustness of the overall process, rather than being interrupted by a single failed request;

[0193] Step 4: Results Summary and Multidimensional Information Export (Phase 3)

[0194] This stage involves integrating and formatting all extracted structured information for output.

[0195] Data integration: Convert the 10 PaperContent objects into a list of dictionaries and load them into a PandasDataFrame;

[0196] Multidimensional Excel Export: Create an Excel file with a timestamp (e.g., extracted_papers_20251017_225205.xlsx); create a pd.ExcelWriter writer, and the system maps dictionaries to preset fields (e.g., basic_info_columns, research_object_columns, etc.); taking the "Basic Information" worksheet as an example, its writing process embodies the structured output characteristics of this invention:

[0197] Column selection and mapping: Execute `basic_df=df[list(basic_info_columns.keys())].copy()` to filter columns belonging to the "Basic Information" dimension from the overall data frame;

[0198] Rename columns: Execute `basic_df.columns = list( basic_info_columns. values())`; this step replaces internal field names (such as `file_name`) with Chinese names (such as "filename"), greatly improving the readability and usability of the final report;

[0199] Worksheet writing: Call basic_df.to_excel(writer, sheet_name='Basic Information', index=False) to write the processed subset of data into the specified worksheet of the Excel file;

[0200] The system cycles through this process, sequentially creating and populating seven worksheets: “Research Subjects,” “Environmental Factors,” “Agricultural Practices,” “Ecological Factors,” “Occurrence and Hazards,” and “Quantitative Data.” This design solves the problem of clutter caused by piling all information into a single worksheet, achieving effective data classification and management.

[0201] After all data has been written, the system calls the `self._adjust_column_width(writer, sheet_name)` method to optimize the format of each worksheet, specifically including:

[0202] Dynamic column width calculation: Traverse each column of the worksheet and retrieve the content of all cells in that column;

[0203] Calculate the optimal width: Take the maximum value of the content length of all cells in the column, max_length, and calculate the adjusted width based on this: adjusted_width = min(max(max_length + 2, 8), 50); This formula ensures that the column width can accommodate most of the content (max_length + 2) while being limited to a reasonable range (minimum 8 characters, maximum 50 characters).

[0204] Apply optimization settings: Apply the calculated width to the corresponding column;

[0205] This automated format optimization process overcomes the drawbacks of inconsistent column widths in the default export format, which require manual adjustments, and significantly improves the professionalism and direct usability of the output.

[0206] File persistence: When the context manager of pd.ExcelWriter ends, the system automatically writes the formatted data to disk and saves it to the preset . / analysis_results / directory. At this point, all extracted literature information has been transformed from unstructured text into a structured, multi-dimensional, and beautifully formatted spreadsheet.

[0207] Automatic format optimization: Iterates through all worksheets and automatically adjusts the width of each column to match the content length, improving the readability of the report;

[0208] Step 5: Process End

[0209] Once the workflow is complete, the system prints "Complete workflow execution completed!" and the final result file storage path to the console; at this point, all steps in this embodiment have been executed.

[0210] Step Six: Large-Sample Test Validation

[0211] To further verify the stability, efficiency, economy, and accuracy of the method provided by this invention in large-scale practical applications, an additional large-scale batch processing test was conducted in this embodiment of the invention.

[0212] 1. Test environment: Same as the environment described in Example 1;

[0213] 2. Test data: 930 academic papers in PDF format on the field of crop pest and disease research were downloaded from relevant databases such as Web of Science and CNKI, from different journal sources and with different layout formats (including single column, double column, multiple charts, English and Chinese). Among them, there were 266 English papers and 664 Chinese papers.

[0214] 3. Execution process: Execute the complete workflow of this invention (steps one to four), wherein the maximum concurrent worker thread parameter is modified to 32 to fully utilize the performance of multi-core hardware;

[0215] 4. Test Results: In the first phase, the system successfully converted 896 documents from PDF to Markdown format out of 930 documents. In the second phase, the system successfully extracted and exported the structured information from 893 documents, with an overall success rate of over 96.0%. The failed documents (accounting for 4.0%) were mainly due to the PDF documents being scanned images from which text could not be extracted or having serious physical damage after manual inspection and tracing.

[0216] 5. Processing efficiency: The total processing time for all 930 documents was approximately 2 hours.

[0217] 6. Deep Seek API Usage Consumption and Cost: During the processing of 893 successful documents, the system initiated 2761 API requests, consuming a total of 25,484,266 tokens. Based on the publicly available pricing for the Deep Seek model API, the total cost of API calls was approximately 50 RMB, making it extremely cost-effective.

[0218] 7. Accuracy Verification: To verify the accuracy of the extracted literature information and data, this invention employs two experts to randomly select 100 successfully extracted documents and manually verify the completeness and accuracy of the extracted content for key fields (including 18 fields such as file name, paper title, journal name, publication time, temperature data, humidity data, precipitation data, cultivation factors, fertilization effects, irrigation effects, host sensitivity, landscape factors, outbreak triggering factors, outbreak severity, quantitative results, geographical range, time scale, and threshold). Figure 5 As shown, after verification and calculation, among the 100 randomly sampled documents, the total precision of extraction of 18 key fields reached 95.64%, the recall rate reached 92.55%, and the overall F1-score reached 93.87%.

[0219] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0220] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for batch structuring and extracting information from literature in the field of plant protection based on an AI large model, characterized in that, The IDE running platform based on Python language realizes an end-to-end automatic process from PDF format literature to structured data output through a multi-stage and parallel processing process, and the processing process includes the following: S1, a PDF literature adaptive preprocessing module and a Markdown conversion module are constructed, PDF format literature is obtained, intelligent PDF text extraction and reconstruction reading text flow are based on, through text deep cleaning, combined with semantic structure recognition and paragraph reconstruction, and idempotent caching strategy, the literature is batch converted into Markdown format text with semantic hierarchical structure suitable for computer processing, realizing the conversion from visual unstructured format to semantic structured format; S2, a double-strategy adaptive AI extraction module is constructed, the Markdown format text is input into the AI module, the adaptive switching between the phased extraction strategy and the intelligent block extraction strategy is realized according to the text length threshold, and the extraction JSON format data strategy is executed through the focused professional Prompt instruction; The AI module also includes a high-robustness JSON format data repair strategy for JSON format data to obtain parseable structured JSON format data; In S2, through the double-strategy adaptive AI extraction module, adaptive switching between the phased extraction strategy and the intelligent block extraction strategy is realized according to the text length and structure complexity, and the specific process is as follows: Set a text length threshold as a decision point; After the Markdown format text is input, first check the length of the Markdown format text: if the text length < threshold, determine that it is a standard length text, and start the phased extraction strategy; If the text length > threshold, determine that it is an ultra-long length text, and start the intelligent block extraction strategy; (1) Phased extraction strategy: adopt task decomposition strategy, input the same Markdown format text into the AI model for three times, use different focused professional Prompt instructions each time, decompose one task into three subtasks with different focus points, as follows: Stage one: extract "basic information and chapters", this stage instruction identifies the metadata of the Markdown format text; Stage two: extract "professional factors", this stage instruction identifies professional domain entities in the Markdown format text; Stage three: extract "quantitative data and methods", this stage instruction identifies the methods and results of the Markdown format text; Finally, the results of the three stages are returned as JSON format data and stored in local memory, the three JSON format data are merged, duplicate items are removed and the format is unified, and finally a complete JSON format data containing all target fields is formed; (2) Intelligent block extraction strategy: adopt semantic alignment block strategy, use the chapter boundary in Markdown format text as semantic segmentation unit, divide the text of super long length into multiple logically coherent text blocks; according to the name of each text block, dynamically match the focused professional Prompt instruction and traverse the text block, then call the AI model independently for each text block to extract information, extract the scattered JSON format data, integrate and merge all the returned scattered JSON format data distributed in different blocks to form complete JSON format data; S3, a two-stage parallel scheduling module is constructed, which is respectively used for executing the CPU intensive conversion task of step S1 and the I / O intensive AI analysis task of step S2; a multi-dimensional aggregation and export module is constructed, the generated structured JSON format data is aggregated according to the pre-defined mapping rule to generate multi-dimensional structured database files, and the adaptive and localized format is output, realizing efficient integration from scattered extraction results to analyzable data sets.

2. The AI large model-based literature information batch structuring extraction method in the plant protection field according to claim 1, characterized in that, In the S1, the intelligent PDF text extraction includes: Call the parsing package to traverse the PDF format literature, obtain all text blocks and corresponding metadata in the literature page, the metadata includes the bounding box coordinates, font style, font size information of the text block; based on the x coordinate distribution of the text block, column structure recognition is implemented, and the reading order is reconstructed in each column according to the y coordinate from top to bottom; it also includes identifying and identifying the text blocks that repeatedly appear in different pages and filtering them as page headers or footers to reconstruct the reading text stream. 3.The AI large model-based bulk structured extraction method for plant protection field literature information according to claim 1, characterized in that, In the S1, the text depth cleaning includes the following: Text depth cleaning includes encoding repair, line break repair and noise removal; the encoding repair replaces the ligature and non-standard punctuation in the text stream based on the pre-set character mapping table; the line break repair combines the cross-line hyphen, separates the number and unit, and removes the Chinese abnormal line break based on the regular expression; The noise removal identifies and deletes the header, footer and page number noise based on the regular expression and position rule.

4. The AI large model-based literature information batch structuring extraction method in the plant protection field according to claim 1, characterized in that, In the S1, combined with semantic structure recognition and paragraph reconstruction, specifically including: (1) Semantic structure recognition: use the pre-set domain keyword pattern dictionary to perform regular matching with the text stream to identify chapter titles and table captions, and output the structured mark in Markdown format; (2) Paragraph reconstruction: traverse the text lines in the text stream to establish a buffer area; when a non-structured mark normal text line is identified, accumulate it in the buffer area and merge it into a semantically coherent paragraph, and output until a blank line or a new structured mark line is detected, eliminating the non-semantic line break in the PDF format literature; By combining semantic structure recognition and paragraph reconstruction, combining structured mark in Markdown format and semantically coherent paragraphs formed based on structure perception, Markdown format text with clear hierarchical structure and semantic coherence is output.

5. The AI large model-based literature information batch structuring extraction method in the plant protection field according to claim 1, characterized in that, In the S1, the idempotency caching strategy based on timestamp is as follows: Before the PDF format conversion work, the cache check is implemented; specifically, before the conversion from PDF to Markdown is performed, the last modification timestamps of the PDF format literature and the corresponding Markdown format text are compared, and the conversion is triggered only when the modification time of the PDF format literature is later than that of the corresponding Markdown format text or the hash values of the two are inconsistent, so as to ensure the idempotency of batch tasks and avoid repeated processing.

6. The AI large model-based literature information batch structuring extraction method in the plant protection field according to claim 4, characterized in that, In S2, the JSON format data repair strategy is as follows: When the standard json.loads() parsing fails, the JSON format data repair backup process is automatically triggered, a series of string and regular expression operations are performed to repair the non-standard JSON format data, and the repaired JSON format data meeting the standard is sent to the parser again to obtain the parseable structured JSON format data.

7. The AI large model-based literature information batch structuring extraction method in the plant protection field according to claim 1, characterized in that, In S3, the specific process of the two-stage parallel scheduling module is as follows: In the first stage, a CPU thread pool dedicated to CPU-intensive tasks is created and started, and all PDF format documents to be processed are evenly distributed to the threads in the CPU thread pool, each thread independently executes the process of step S1 to generate Markdown format text; In the second stage, an IO thread pool dedicated to I / O intensive tasks is created and started, and all Markdown format texts to be processed are evenly distributed to the threads in the IO thread pool, each thread independently executes the process of step S2 to generate structured JSON format data; Finally, the asyncio.run() wrapper is used to efficiently execute asynchronous API calls in a synchronous thread, so that resources are optimally utilized; After the structured JSON format data generated by processing all PDF format documents, an aggregation script is started by using the scheduler, and the aggregation script traverses the structured JSON format data and integrates the generated JSON format data files. 8.The method of claim 1, wherein the method is characterized by, In S3, based on the multi-dimensional aggregation export module, the structured database file is aggregated and generated based on the pre-defined mapping rules, and the adaptive and localized format is output, as follows: (1) Predefined mapping rule aggregation generation: use the pandas library to read the JSON format data file into memory; according to the pre-defined mapping rules, the JSON fields in different JSON format data are respectively appended to different DataFrame multi-dimensional tables; finally, use pandas.ExcelWriter to write the DataFrame multi-dimensional table as different worksheets into the same Excel file, each worksheet represents a specific data dimension, forming a multi-dimensional structured database file; (2) Output adaptive and localized format: Set a field name Chinese-English dictionary; Before writing the DataFrame into Excel, replace the column name of the DataFrame with the Chinese table header through the field name Chinese-English dictionary; After writing the DataFrame into Excel, use the openpyxl or xlsxwriter engine to traverse each column of each worksheet, calculate the maximum display width of all cell contents in the column, and set the best column width with the best pixel value to ensure that the generated Excel file can be clearly read without any manual adjustment, improving the usability and human readability of the final output results.

Images