Intelligent document batch processing system and method based on large language model

Abstract

This invention discloses an intelligent document batch processing system and method based on a large language model, belonging to the field of intelligent document processing technology. Addressing the problems of high local deployment costs, weak fault tolerance, and poor adaptability in existing technologies, it adopts cloud-based service deployment. Through collaboration between a local intelligent scheduling engine and a cloud-based large language model, it integrates core technologies such as multi-level JSON fault tolerance repair, multi-mode PDF text extraction, and cross-domain adaptive configuration. Users configure custom fields and prompt word templates through a graphical interface, and the system can automatically complete batch document extraction, API calls, anomaly repair, field validation, and structured output. This solution significantly lowers the technical threshold, requires no high-performance hardware, and is suitable for multiple fields such as scientific research, medicine, education, and finance. It significantly improves the reliability, efficiency, and flexibility of document processing, and is applicable to the intelligent organization and information extraction of massive amounts of literature.

Description

Technical Field

[0001] This invention relates to the field of intelligent document processing technology, and in particular to an intelligent document batch processing system and method based on a large language model. Background Technology

[0002] With the continuous development of artificial intelligence technology in the field of document processing, AI-based document structure extraction, classification, and information analysis technologies have been gradually applied to scenarios such as scientific research, academia, and enterprise office work. Currently, existing technologies in this field mainly include CN120336416A "A Method and System for Structured Document Extraction Based on Artificial Intelligence," CN120337937A "A Method and System for Extracting Academic Viewpoints from Academic Literature," CN119760143A "A Method for Intelligent Classification and Automatic Review Generation of Multiple Documents Based on a Large Language Model," and CN119293342A "An Intelligent Technology Dynamic Tracking System for the Scientific Field," etc. These technologies have achieved partial automation of document processing to a certain extent, but still have many prominent problems and significant shortcomings in practical applications, making it difficult to meet the needs of ordinary users, small and medium-sized teams, and industrialized batch processing.

[0003] First, existing technologies generally adopt the local pre-trained model deployment method, which has extremely high requirements for hardware devices. It requires high-performance computing hardware such as GPUs and TPUs. Not only is the initial hardware investment cost high, but the cost of subsequent model maintenance and updates also continues to increase, resulting in a high technical threshold that is difficult for ordinary individual users and small and medium-sized teams to afford, which greatly limits the popularization and application of the technology.

[0004] Secondly, the poor user experience is a significant issue. The existing system requires specialized technical personnel for model configuration, parameter tuning, and routine maintenance, lacking user-friendly support for non-professional users. Ordinary researchers and other target user groups cannot independently complete the entire literature processing workflow, severely impacting the actual efficiency of the technology.

[0005] Furthermore, existing technologies suffer from a significant lack of fault tolerance and repair mechanisms. Their design assumes that the output of the AI ​​model is always stable and standardized. However, when processing long texts or complex subject matter, the model output is highly susceptible to formatting errors (such as incomplete JSON structures or syntax errors), content truncation, or data corruption. Existing systems cannot handle these anomalies, directly causing interruptions and errors in the entire information extraction process. This necessitates manual intervention for debugging, resulting in extremely low reliability and success rates for batch processing, making it difficult to meet the demands of large-scale document processing.

[0006] Meanwhile, existing solutions suffer from severely insufficient adaptability and a rigid overall design, failing to intelligently adapt to different document structures, domain-specific terminology, personalized extraction needs of different users, and API characteristics of different service providers. This results in extremely poor generalization capabilities; each time a new application domain or document type is encountered, it often requires retraining the model or performing a large amount of tedious manual configuration, failing to achieve true "intelligence" and "adaptability."

[0007] Furthermore, batch processing suffers from poor reliability and lacks a stability guarantee mechanism for large-scale batch processing tasks. In scenarios involving the processing of hundreds or thousands of documents, any unexpected API call failure, network fluctuation, or content parsing error can cause the entire task queue to freeze or terminate, failing to meet the basic requirements of industrial applications for task completion rate and business continuity. Users need to monitor the processing process at all times, which is not only cumbersome but also seriously affects processing efficiency.

[0008] Finally, weak human-computer interaction design is also a significant shortcoming of existing technologies. Current systems lack intuitive graphical interfaces, making user operations such as custom configuration and result visualization cumbersome, with high learning costs and low interaction efficiency, further diminishing the user experience. Summary of the Invention

[0009] The purpose of this invention is to provide an intelligent document batch processing system and method based on a large language model. Through cloud service deployment and multiple fault-tolerant repair mechanisms, it realizes automated batch processing of documents throughout the entire process, while lowering the technical application threshold, allowing non-professional users to use professional-grade document analysis capabilities, and providing an intuitive and easy-to-use graphical interface to improve processing efficiency and user experience.

[0010] To achieve the above objectives, this invention provides an intelligent document batch processing system based on a large language model, comprising: 1. User interaction configuration layer It includes a graphical user interface module, a cross-domain adaptive configuration module, a domain-specific prompt word automatic generation engine, and a task queue management module. The graphical user interface (GUI) module provides an intuitive operating interface, supports bibliographic selection, processing mode configuration, and real-time progress monitoring; the interface supports theme switching to adapt to different usage scenarios.

[0011] The cross-domain adaptive configuration module is used for users to customize extracted fields and set domain-specific prompt word templates. It supports saving, importing and sharing of custom templates, enabling flexible adaptation of processing rules.

[0012] The task queue management module is used to handle the creation, priority scheduling, and lifecycle management of tasks, and supports task suspension, resume, cancellation, and breakpoint resumption functions.

[0013] 2. Local intelligent processing engine, including a multi-mode PDF text extraction module, a large language model (LLM) API interaction and fault tolerance module, a field integrity verification engine, and an intelligent file renaming and classification module; The multi-mode PDF text extraction module is used to extract text content from PDF documents, integrating four strategies: head_only, tail_only, head_and_tail, and full_text; The large language model API interaction and fault tolerance module is used to communicate with cloud APIs and correct and complete non-standard results returned by APIs. It integrates a multi-level JSON repair algorithm and adopts a progressive repair strategy at the syntax level, structure level, and semantic level; The field integrity verification engine is used to perform integrity verification on data (implementing intelligent matching of Chinese and English fields, such as automatically mapping "Author" to "作者"), performing secondary processing on the repaired data, executing dynamic field mapping, mandatory verification of required fields, and intelligently filling standardized default values (such as "unknown") for fields that cannot be obtained, ensuring the structuring and integrity of the output data; The intelligent file renaming and classification module is used for renaming and classified storage of PDF documents. According to the extracted metadata (title, author, year, field, etc.), it automatically renames and classifies and stores the original PDF files, achieving automated management of processing results; 3. Cloud service integration layer: including a multi-source large language model API proxy module and a real-time status monitoring and logging module. The multi-source large language model API proxy module is used to uniformly encapsulate the API interfaces of different service providers and built-in an API request rate limiting mechanism to avoid triggering the interface limits of service providers due to high-frequency calls.

[0014] The real-time status monitoring and logging module is used to record the API call status, processing progress, and system exceptions; specifically, it includes recording the API call status (success / failure), processing progress, error types (such as timeout errors, format errors), and repair records. The logs support filtering and querying by time range and error type, providing data support for operation and maintenance and optimization; supporting the export of logs as TXT format files to meet the needs of data traceability and auditing.

[0015] 4. Data output layer: including a structured result storage module, a multi-format report generation module, and a processing log and auditing module, The structured results storage module is used for persistent storage of processing results, persisting the processing results to a local database (SQLite) or file system, supporting incremental processing and breakpoint resumption, and can load unfinished tasks to continue processing on the next startup.

[0016] The multi-format report generation module is used to generate standardized reports, including standardized Excel reports, which can be exported to JSON, CSV and other formats. The Excel reports include data summary tables and abnormal file statistics tables, and support data filtering, sorting and conditional formatting highlighting (such as highlighting records with missing required fields in red).

[0017] The processing log and auditing module is used to record detailed operation logs, including user configuration parameters, task processing time, file processing results, API call details, etc. The log retention time is configurable (default 90 days) to meet data traceability and auditing needs.

[0018] Preferably, the domain-specific prompt word automatic generation engine has pre-set basic templates for scientific research, PICOS framework templates for the medical field, teaching case analysis modules, and financial analysis templates for the financial field, and supports the import and saving of user-defined templates; it associates user-defined fields with template placeholders through a field mapping mechanism, automatically adds field description context (e.g., the "experimental model" field is associated with "briefly introduce the experimental model used in the article, including cell lines, animal models, clinical samples, etc., including model characteristics and application scenarios; if no specific model is specified, it returns 'none'"); it uses string interpolation technology to inject fields into the template to generate complete prompt words, supports dynamic adjustment of prompt words based on API feedback, and supports users to manually modify or add constraints.

[0019] Preferably, the multi-mode PDF text extraction module integrates four strategies: beginning-first, ending-first, beginning-and-end combined, and full-text extraction. The built-in priority rule is: beginning-and-end combined > full-text extraction > beginning-first > ending-first. The rule engine parses the PDF text stream, identifies the position markers of "Abstract", "Introduction", "Conclusion" and "References", calculates the confidence level by combining the document length and structural complexity, and automatically selects the extraction strategy.

[0020] Preferably, the large language model API interaction and fault tolerance module integrates a multi-level JSON repair algorithm, adopting a progressive repair strategy at the syntax, structure, and semantic levels. Syntax-level repair includes fixing bracket matching issues, quotation mark mismatch issues, and escape character issues. Structure-level repair includes extracting key-value pairs using strict regular expressions and matching key-value pairs using loose regular expressions. Based on a user-defined list of expected fields, it intelligently infers missing fields in the API response and reconstructs a complete JSON structure. For missing fields, it can intelligently fill in default values ​​or trigger secondary queries. Semantic-level repair performs semantic verification on the repaired JSON by calling a lightweight large language model or rule engine.

[0021] Preferably, the multi-source large language model API proxy module uniformly encapsulates the API interfaces of AI service providers (such as Kimi, DeepSeek, etc.) containing different LLM models, and supports automatic switching of service nodes according to service load, thereby improving service availability and flexibility; the real-time status monitoring and log module records API call status, processing progress, error type and repair records, providing data support for system operation and maintenance, problem investigation and performance optimization.

[0022] A method for intelligent batch processing of documents based on a large language model, comprising the following steps: S1. System Initialization: Start the system, load the preset configuration (such as default API parameters, extraction mode, required fields) and historically saved user configuration information, and initialize each functional module; S2. Task Configuration and Queue Establishment: Users configure API parameters (service provider, API key, model parameters), select the domain, PDF extraction mode, processing options (such as whether to process encrypted files, whether to retain original files), customize analysis requirements (custom fields, prompt word templates), and output options (report format, save path) through a graphical user interface (GUI). Users select required fields (literature classification, corresponding author, author affiliation, publication year, journal, title, introduction, innovation, experimental model, journal impact factor, etc.), select the folder containing the PDF documents to be processed, and drag and drop import of local folders is supported. The system traverses the folder to search for all PDF documents and establishes a batch processing task queue. S3, PDF Text Extraction: Extract text content from PDF documents based on the configured extraction mode or the extraction mode intelligently recommended by the system; S4. Cloud API Call: Prepare API call parameters (such as model version, request timeout), construct system prompt words, send the extracted text to the cloud-based large language model API for analysis, and monitor the API call status in real time. S5. Response Preprocessing and Fault Tolerance Repair: Organize the API response. The organization process includes validating JSON, removing HTML tags, cleaning up error message prefixes, standardizing the format, removing non-JSON content, and determining whether the preprocessed response is valid JSON. If it is invalid JSON, a multi-level progressive repair strategy is initiated. S6. Field Integrity Validation: Perform field name mapping conversion, required field checking, custom column processing, and default value setting on the repaired data; S7. Document organization and storage: Create new file names, create category folders, handle duplicate files, copy PDF documents to the target directory, and rename PDF files according to document category; S8. Output Results: Organize the analysis results into structured data, add processing time and record file path, construct data rows and add them to the data frame, generate a standardized Excel report, and support exporting JSON and CSV format files.

[0023] Preferably, in step S3, the logic for the system's intelligent recommendation extraction mode includes: If the "Abstract" or "Introduction" is located in the first 10% of the text area of ​​the PDF document and accounts for 5%-15% of the length of the full text, the "First to Enter" mode is recommended. If the "Conclusion" or "Discussion" section is located in the last 10% of the text area of ​​the PDF document and is clearly separated from the preceding text, the "End First" mode is recommended. If the PDF document has a complex structure with multiple chapters and numerous appendices, enable the "head and tail combined" mode; If the above features are not significant or the user requires complete content extraction, the "full text extraction" mode will be used.

[0024] Preferably, in step S4, after the cloud API call is made, the call status is judged. If the call fails, tiered retries are performed according to the retry threshold and time interval configured by the user. After the retries are exhausted, the failure log is recorded and the current PDF document processing is terminated without affecting the processing flow of other PDF documents in the task queue. After all documents are processed, the system automatically summarizes and generates a batch processing report.

[0025] Preferably, in step S6, the field integrity verification includes: achieving intelligent matching of Chinese and English fields through a dynamic field mapping mechanism; required fields include document classification, author, author's affiliation, publication year, journal, title, and introduction; mandatory verification is performed on required fields, and standardized default values ​​are automatically filled in for missing fields.

[0026] Preferably, in step S7, the document metadata includes title, author, publication year, field classification, journal name, and impact factor; the classification folder is created based on field classification, processing result tags, or user-defined rules; duplicate files are distinguished by serial numbers.

[0027] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: (1) Cost and threshold reduction: The cloud service model avoids huge hardware investment and complex operation and maintenance; the graphical configuration enables non-professional users to easily use professional-grade AI capabilities, which greatly promotes the popularization of technology.

[0028] (2) Revolutionary improvement in processing reliability: The original multi-level intelligent fault-tolerant repair mechanism effectively solves the industry pain point of uncertainty in the output of large language model APIs, and increases the success rate of large-scale batch processing tasks from less than 70% in traditional solutions to more than 99%, reaching the industrial application standard.

[0029] (3) Strong cross-domain adaptability: Through the configurable field library and template engine, the system can quickly switch to completely different fields such as scientific research, medical care, education, and finance without modifying the core code, which solves the core problem of poor generalization ability of traditional solutions and has extremely strong scalability.

[0030] (4) Robust and efficient batch processing: Task queue management, exception isolation, automatic retry and breakpoint resume mechanism ensure the continuity and stability of large-scale operations. Users do not need to monitor at all times, and truly realize "unattended" automated processing.

[0031] (5) Significantly optimized user experience: The entire process is guided by a graphical interface. From file selection, configuration, monitoring to result export, the operation is intuitive and simple, which greatly reduces the learning cost and improves work efficiency.

[0032] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating an embodiment of an intelligent document batch processing method based on a large language model according to the present invention. Figure 2This is a structural block diagram of an embodiment of an intelligent document batch processing system based on a large language model according to the present invention; Figure 3 This is a flowchart illustrating the multi-level fault-tolerant repair mechanism of an embodiment of the present invention. Figure 4 These are screenshots of the relevant GUI interfaces in an embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0036] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Based on Figure 1 The method shown and Figure 2 The system shown is used in areas such as scientific literature review, medical clinical case analysis, and investment material research. The system interface is as follows: Figure 4 As shown.

[0037] Example 1: Intelligent organization and information extraction for scientific research literature It is suitable for researchers and academic institutions to automatically organize massive amounts of scientific literature, extract core information, and build structured knowledge bases, solving the problems of low efficiency and fragmented information in traditional literature processing.

[0038] I. Core Configuration 1. API Service Selection: The system integrates multi-source large language model APIs (such as Kimi, DeepSeek, etc.), allowing users to switch according to their needs, reflecting the system's service independence and scalability.

[0039] 2. Adaptive PDF extraction strategy selection: The system provides multiple text extraction modes (such as "first-of-its-kind", "first-and-last-of-its-kind", "first-and-last-of-its-kind", and "full-text extraction"), which can intelligently recommend based on the document's structural characteristics to maximize the acquisition of effective content and avoid interference from invalid information.

[0040] 3. Custom Analysis Configuration: Define structured extraction fields, covering literature classification, corresponding author, author affiliation, publication year, journal, title, abstract, novelty, experimental model, impact factor, etc.; the system calls the basic prompt word template in the scientific research field, maps the custom fields to the corresponding placeholders in the template, and automatically adds field description context (such as the "experimental model" field being associated with "briefly introduce the experimental model used in the article, including cell lines, animal models, clinical samples, etc., including model characteristics and application scenarios; if no specific model is specified, it returns 'none'"), generating complete prompt words, and supporting users to manually modify or add constraints based on API feedback.

[0041] II. Processing Flow 1. System initialization and configuration: Start the intelligent document processing system and load the preset configuration and historically saved user configuration information; users can complete the configuration of API parameters, extraction mode, custom fields and output options through the GUI.

[0042] 2. Establish a batch processing task queue: The user specifies the folder containing the research literature to be processed. The system automatically traverses the folder, searches for all PDF documents, generates a batch processing task queue, and clarifies the task processing order and priority.

[0043] 3. Document Text Extraction: Extract text content from documents based on user configuration or system-recommended extraction modes; the system parses the PDF text stream through a rule engine, identifies key chapter location markers such as "Abstract", "Introduction", "Conclusion" and "References", and prioritizes capturing core content areas.

[0044] 4. Cloud API Call and Anomaly Monitoring: Construct system prompt words and API call parameters, send the extracted text to the selected cloud-based large language model API for analysis, monitor the API call status in real time, and record the call results and response time.

[0045] 5. Intelligent fault tolerance and repair, such as Figure 3 As shown: If the API returns content with formatting errors, truncation, or non-standard JSON issues, a multi-level progressive repair strategy is initiated: First, syntax-level repair is performed to correct errors such as mismatched brackets, mismatched quotation marks, and missing closing braces; then, structural-level repair is performed by comparing the user-defined field list, inferring missing fields, and reconstructing the complete JSON structure; finally, semantic-level repair is performed by using a lightweight large language model or rule engine to verify the reasonableness of field content format (such as checking whether the author list is in the format of personal names), while supplementing valid information by using strict regular expressions to extract key-value pairs and loose regular expressions to match key-value pairs.

[0046] 6. Field Integrity Verification: Perform field name mapping conversion on the repaired data, check whether required fields such as corresponding author, author affiliation, and publication year are complete, automatically fill in standardized default values ​​such as "unknown" or "not yet extracted" for missing fields, and complete custom column processing and format standardization.

[0047] 7. Document organization and storage: The system automatically renames PDF documents based on the extracted metadata (title, author, publication year, journal name, etc.), creates folders according to document categories, distinguishes duplicate files by serial number, and copies the organized documents to the target directory to achieve standardized management of document assets.

[0048] 8. Structured Results Output: Integrates the extraction results from all literature to generate a standardized Excel report containing all custom fields, which can be exported to JSON, CSV and other formats to form a structured knowledge base that can be directly used for scientific research analysis.

[0049] III. Implementation Results

[0050] Breakthrough in reliability: Through an intelligent fault-tolerant repair mechanism, the success rate of batch processing tasks exceeds 99% when facing the inherent output uncertainty of large language model APIs, reaching the industrial-grade application standard.

[0051] Significant efficiency improvement: The fully automated process frees people from tedious document reading and organization, increasing processing efficiency by more than 50 times compared to manual methods.

[0052] High output quality: Thanks to field validation and intelligent data filling, the generated structured data can be directly imported into professional software for subsequent analysis, greatly improving the efficiency and quality of scientific research data preparation.

[0053] Strong adaptability: By configuring different field templates, the same system can be easily applied to different disciplines such as biomedicine, materials science, and computer science, proving the versatility and strong vitality of the technology of this invention.

[0054] Example 2: Intelligent Analysis and Evidence-Based Structured Analysis of Clinical Case Reports Aimed at medical institutions and clinical researchers, it extracts structured information from clinical case reports and standardizes medical terminology to provide data support for evidence-based medicine research and clinical decision support.

[0055] I. Core Configuration 1. Clinical processing logic configuration: Load a medical-specific configuration template, specify that case data must be associated and standardized based on standard medical terms such as SNOMEDCT and ICD-11, key information extraction follows the evidence-based medicine PICOS framework, and enforces the patient privacy protection mechanism to automatically block sensitive information such as names and ID numbers.

[0056] 2. Medical-specific field configuration: Define PICOS-related extraction fields, including patient (P) elements such as basic patient information, inclusion criteria, and exclusion criteria; intervention and comparison (I / C) elements such as treatment plan, intervention measures, control measures, and medication details; outcome (O) elements such as primary efficacy endpoints, secondary efficacy endpoints, and adverse events; study design (S) elements such as study type, blinding settings, and follow-up time; and clinical standard information such as chief complaint, present illness, past medical history, physical examination, auxiliary examinations, preliminary diagnosis, and prognostic assessment.

[0057] 3. Research parameter configuration: Users can select the clinical research mode (such as randomized controlled trial, cohort study, case control study, etc.) through the GUI, and the system will automatically adjust the extracted fields according to the selection; users can further refine the settings of specific efficacy indicator names (such as progression-free survival, PFS), intervention details and other parameters to ensure compliance with medical ethics and the Personal Information Protection Law.

[0058] 4. Prompt word configuration: Construct medical-specific prompt words, which must strictly follow the evidence-based medicine PICOS framework for information extraction, use standardized medical terminology, and return results in JSON format. The prompt words must cover the specific requirements of core extraction dimensions such as patient population, intervention measures, control measures, outcome indicators, and study design.

[0059] II. Processing Flow 1. System Initialization and Template Loading: After the system starts, it loads the medical-specific configuration template and PICOS framework-related configurations, switches to the medical professional view interface, and provides an adapted environment for clinical case processing.

[0060] 2. Establish a case processing queue: The user specifies the folder containing the clinical case reports to be processed. The system scans the case reports in PDF or structured text format within the folder, automatically creates a batch processing queue, and records the file path and basic information of each case.

[0061] 3. Case text extraction: Based on the structural characteristics of clinical case reports, a full-text extraction mode is adopted to ensure the complete acquisition of key information such as medical history, examination results, treatment plan, and prognosis, and to avoid the loss of clinical data due to partial extraction.

[0062] 4. Medical AI Analysis and Information Extraction: The extracted case text (with patient privacy information hidden) and medical-specific prompts are sent to the cloud-based large language model API. Medical-related models are called to analyze the data and extract the structured information corresponding to the PICOS framework, including basic patient characteristics, details of intervention and control measures, efficacy and safety results, and study design parameters.

[0063] 5. Data standardization processing: Activate the professional terminology standardization engine to automatically map the diagnostic names in the case to ICD-11 codes and the surgical procedures to ICD-9-CM-3 codes; unify drug brand names and common names to standard generic names, which can be associated with ATC codes; standardize the units and classify the numerical ranges of laboratory test results (such as serum creatinine 120 umol / L) to ensure that the data format is standardized and consistent.

[0064] 6. Fault Tolerance Repair and Integrity Verification: Perform multi-level repair on non-standard JSON data returned by the API to correct format errors and content truncation issues; verify the integrity of required fields (such as patient basic information, intervention measures, and efficacy indicators), and fill missing fields with default values ​​according to medical data specifications to ensure that no core clinical information is omitted.

[0065] 7. Clinical Knowledge Integration: The system automatically links the extracted PICOS elements to construct a clinical knowledge graph of "disease-treatment-outcome", visually displaying the evidence level and effect differences of different treatment options, providing an intuitive reference for clinical decision-making.

[0066] 8. Professional Report Generation: Generates a standard PICOS structured summary for each case to facilitate rapid assessment of research value; automatically labels the level of evidence (e.g., Level 1b, Level 2a) according to the research type; generates a comparative outcome table to summarize and compare the main outcomes of different interventions; and outputs a safety overview report to summarize the occurrence of adverse reactions in all cases, forming professional results that can be directly used for clinical research or management.

[0067] III. Implementation Results

[0068] The system uses a medical terminology standardization engine and the PICOS framework to accurately extract core clinical information, achieving an accuracy rate of over 99%. The ICD-11 diagnostic code mapping matching rate and the standardization rate of generic drug names both exceed 99%, and the unit uniformity rate of laboratory test results is 100%. The generated structured data fully meets the data specification requirements for evidence-based medicine research.

[0069] The constructed "disease-treatment-outcome" knowledge graph and its visualization comparison can intuitively present the evidence strength of different interventions, providing data support for clinicians to choose the optimal treatment plan and helping to implement evidence-based decision-making.

[0070] By configuring extraction fields for different research types (RCT, cohort studies, case-control studies), the system can seamlessly adapt to various clinical research scenarios, quickly switch processing logic without modifying the core code, and meet diverse clinical research needs.

[0071] Example 3: Investment Research Application Serving investment institutions and researchers, it performs batch analysis of industry research reports and company financial reports, extracting key information such as core financial indicators, industry trends, and investment recommendations to support investment decisions, portfolio analysis, and risk assessment.

[0072] I. Core Configuration 1. Financial processing logic configuration: Clearly define that investment research reports require comprehensive analysis, financial data units are interconnected according to standard accounting principles, and key investment indicators must be extracted accurately and in a timely manner; divide research reports into modules such as industry analysis, company finance, and investment recommendations, and set the key information extraction points for each module.

[0073] 2. Financial-Specific Field Configuration: Define structured extraction fields, including industry analysis, company overview, financial indicators, profitability, growth analysis, risk assessment, valuation level, investment advice, target price, catalyst events, risk warnings, etc., covering the core dimensions of investment decision-making.

[0074] 3. Analysis parameter configuration: Users can select industry categories (such as artificial intelligence and new energy), analysis time range (such as 2023-2024), and analysis depth (such as in-depth research report) through the GUI, and set the output format as portfolio analysis table, visual dashboard, etc., to meet the needs of different investment analysis scenarios.

[0075] 4. Prompt Keyword Configuration: Construct financial-specific prompt keywords, clearly requiring the extraction of core information such as industry trends, financial data, valuation indicators, and investment recommendations from investment research reports, ensuring that the extracted content complies with financial data standards and investment analysis needs.

[0076] II. Processing Flow 1. System Initialization and Configuration: After the system starts, a configuration template specifically for the financial field is loaded. Users complete the configuration of parameters such as industry classification, time range, analysis depth, and output format, and determine the financial-specific fields and prompt words.

[0077] 2. Establish a financial report processing queue: The user specifies the folder where the investment research reports are located (e.g., D: / Investment Research / AI Industry 2024). The system scans the PDF reports in the folder, automatically identifies the document type, creates a batch processing queue, and records the number of reports and storage path.

[0078] 3. Report Text Extraction: Based on the structural characteristics of investment research reports, a combined approach of beginning and end is used to extract core chapter content (such as industry overview, financial summary, and investment recommendations). If the user requests a complete analysis, a full-text extraction mode is used to ensure the acquisition of key investment information.

[0079] 4. Intelligent extraction of financial information: The extracted text and financial-specific prompts are sent to the cloud-based big language model API, which calls financial-related models for analysis to extract structured information such as industry analysis, company overview, financial indicators (such as revenue growth rate and gross profit margin), investment recommendations (such as buy or hold), risk assessment, and target price.

[0080] 5. Financial Data Verification and Standardization: Verify the completeness and accuracy of the extracted financial data, standardize the units of financial indicators (such as RMB 100 million, percentage), standardize valuation indicators (PE, PB, PS, etc.), and standardize investment recommendation levels (buy, overweight, neutral, etc.) to ensure that the data can be directly used for investment analysis.

[0081] 6. Portfolio Analysis and Risk Assessment: Based on standardized financial data, conduct industry distribution analysis (such as the proportion of AI chips, new energy, biomedicine, etc.), classify reports into high, medium and low risk, use a 1-10 rating system to score investment value, and build a portfolio analysis model and risk assessment system.

[0082] 7. Research Results Output: Generate investment analysis reports (e.g., AI Industry Investment Analysis.docx), export structured data to a standardized investment database (e.g., Investment Research Database.xlsx), and create investment visualization dashboards to intuitively display industry trends, investment value, and risk distribution.

[0083] 8. Processing Completion: Complete the intelligent processing of all investment research reports to form complete investment research results, supporting subsequent investment decision-making.

[0084] III. Implementation Results

[0085] Successfully processes investment research reports with an information extraction rate of 96%, accurately extracting core financial indicators and investment recommendations; generates a standardized investment database containing 11 professional fields, with unified data format and direct reusability; automatically categorizes research reports by industry, risk level, and investment value for easy filtering and review; constructs a comprehensive portfolio analysis model and risk assessment system to provide data support for investment decisions; and boasts high processing efficiency, with a total processing time of approximately 50 minutes for every 100 reports, averaging 30 seconds per report.

[0086] The remaining technical features in the above embodiments can be flexibly selected by those skilled in the art to meet different specific practical needs according to actual circumstances. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims. In the above description, numerous specific details have been set forth to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that these specific details are not necessary to implement the present invention. In other instances, to avoid obscuring the present invention, well-known techniques, such as specific construction details, operating conditions, and other technical conditions, have not been specifically described.

Claims

1. A smart document batch processing system based on a large language model, characterized in that, include: The user interaction configuration layer includes a graphical user interface module, a cross-domain adaptive configuration module, a domain prompt word automatic generation engine, and a task queue management module. The graphical user interface module is used for literature catalog selection, processing mode configuration, and real-time progress monitoring. The cross-domain adaptive configuration module is used for users to customize extraction fields and set domain-specific prompt word templates. The task queue management module is used for handling task creation, priority scheduling, and lifecycle management. The local intelligent processing engine includes a multi-mode PDF text extraction module, a large language model API interaction and fault tolerance module, a field integrity verification engine, and an intelligent file renaming and classification module. The multi-mode PDF text extraction module is used to extract text content from PDF documents. The large language model API interaction and fault tolerance module is used to communicate with cloud APIs and correct and complete non-standard results returned by APIs. The field integrity verification engine is used to verify the integrity of data. The intelligent file renaming and classification module is used to rename and classify PDF documents for storage. Cloud service integration layer: includes a multi-source large language model API proxy module and a real-time status monitoring and logging module. The multi-source large language model API proxy module is used to uniformly encapsulate the API interfaces of different service providers, and the real-time status monitoring and logging module is used to record API call status, processing progress and system anomalies. Data persistence and output layer: includes a structured result storage module, a multi-format report generation module, and a processing log and auditing module. The structured result storage module is used for persistent storage of processing results, the multi-format report generation module is used to generate standardized reports, and the processing log and auditing module is used to record operation logs.

2. The intelligent document batch processing system based on a large language model according to claim 1, characterized in that: The field prompt word automatic generation engine has pre-set basic templates for scientific research, PICOS framework templates for medical fields, and financial analysis templates for financial fields, and supports the import and saving of user-defined templates; The field mapping mechanism associates user-defined fields with template placeholders, automatically adds field description context, and uses string interpolation technology to inject fields into the template to generate complete prompts. Users can also manually modify or add constraints.

3. The intelligent document batch processing system based on a large language model according to claim 1, characterized in that: The multi-mode PDF text extraction module integrates four strategies: beginning-first, ending-first, beginning-and-end combined, and full-text extraction. The built-in priority rule is: beginning-and-end combined > full-text extraction > beginning-first > ending-first. The module parses the PDF text stream through a rule engine, identifies the position markers of "Abstract", "Introduction", "Conclusion" and "References", calculates the confidence level based on the document length and structural complexity, and automatically selects the extraction strategy.

4. The intelligent document batch processing system based on a large language model according to claim 1, characterized in that: The large language model API interaction and fault tolerance module integrates a multi-level JSON repair algorithm, adopting a progressive repair strategy at the syntax level, structure level, and semantic level. Syntax-level fixes include fixing bracket matching issues, quotation mark mismatches, and escape character issues. Structural-level fixes include using strict regular expressions to extract key-value pairs and using loose regular expressions to match key-value pairs. Semantic-level fixes involve calling a lightweight large language model or rule engine to perform semantic verification on the fixed JSON.

5. The intelligent document batch processing system based on a large language model according to claim 1, characterized in that: The multi-source large language model API proxy module uniformly encapsulates the API interfaces of AI service providers containing different large language models, and supports automatic switching of service nodes based on service load and cost; the real-time status monitoring and logging module records API call status, processing progress, error type and repair records.

6. A method for batch processing intelligent documents based on a large language model, executed in the intelligent document batch processing system based on a large language model as described in any one of claims 1-5, characterized in that, The steps are as follows: S1. System Initialization: Start the system and load the preset configuration and historically saved user configuration information; S2. Task Configuration and Queue Creation: Users configure API parameters, select the domain, PDF extraction mode, processing options, customize analysis requirements and output options, select required fields, and select the folder containing the PDF documents to be processed through a graphical interface. The system traverses the folder to search for all PDF documents and creates a batch processing task queue. S3, PDF Text Extraction: Extract text content from PDF documents based on the configured extraction mode or the extraction mode intelligently recommended by the system; S4. Cloud API Call: Prepare API call parameters, construct system prompt words, send the extracted text to the cloud-based large language model API for analysis, and monitor the API call status in real time. S5. Response Preprocessing and Fault Tolerance Repair: Organize the API response. The organization process includes validating JSON, removing HTML tags, cleaning up error message prefixes, standardizing the format, removing non-JSON content, and determining whether the preprocessed response is valid JSON. If it is invalid JSON, a multi-level progressive repair strategy is initiated. S6. Field Integrity Validation: Perform field name mapping conversion, required field checking, custom column processing, and default value setting on the repaired data; S7. Document organization and storage: Create new file names, create category folders, handle duplicate files, copy PDF documents to the target directory, and rename PDF files according to document category; S8. Output Results: Organize the analysis results into structured data, add processing time and record file path, construct data rows and add them to the data frame, generate a standardized Excel report, and support exporting JSON and CSV format files.

7. The intelligent document batch processing method based on a large language model according to claim 6, characterized in that: In step S3, the logic for the system's intelligent recommendation extraction mode includes: If the "Abstract" or "Introduction" is located in the first 10% of the text area of ​​the PDF document and accounts for 5%-15% of the length of the full text, the "First to Enter" mode is recommended. If the "Conclusion" or "Discussion" section is located in the last 10% of the text area of ​​the PDF document and is clearly separated from the preceding text, the "End First" mode is recommended. If the PDF document has a complex structure with multiple chapters and numerous appendices, adopt the "head and tail combined" mode; If the above features are not significant or the user requests complete content extraction, the "full text extraction" mode will be used.

8. The intelligent document batch processing method based on a large language model according to claim 6, characterized in that: In step S4, after the cloud API call is made, the call status is judged. If the call fails, tiered retries are performed according to the retry threshold and time interval configured by the user. After the retries are exhausted, the failure log is recorded and the processing of the current PDF document is terminated without affecting the processing flow of other PDF documents in the task queue. After all documents are processed, the system automatically summarizes and generates a batch processing report.

9. The intelligent document batch processing method based on a large language model according to claim 6, characterized in that: In step S6, field integrity verification includes: achieving intelligent matching of Chinese and English fields through a dynamic field mapping mechanism; required fields include document classification, author, author affiliation, publication year, journal, title, and introduction; mandatory verification is performed on required fields, and standardized default values ​​are automatically filled in for missing fields.

10. The intelligent document batch processing method based on a large language model according to claim 6, characterized in that: In step S7, the document metadata includes title, author, publication year, field classification, journal name, and impact factor; Category folders are created based on domain classification, processing result tags, or user-defined rules; duplicate files are distinguished by serial numbers.

Images